Managing Evaluation – from the Program Manager’s Point of View

 

Much interest exists in evaluating and improving programs that serve our communities. Staff in nonprofit organizations and government agencies of all types want to know which services work and which do not work, so that they can continually increase their effectiveness. Funders want to know how best to invest their resources. 

I’ve written a new book, The Manager’s Guide to Program Evaluation (2^nd edition), to help those of you who value the benefits of high quality program evaluation and have responsibility for making sure that evaluation occurs, but need help understanding how best to make it happen. This new edition provides several examples of evaluation research, based on work completed at Wilder Research. Those examples can help readers to better understand how evaluation might occur in their organizations. 

The Manager’s Guide takes a common sense approach to explaining program evaluation. All of us intuitively make decisions, based on the best information that we have, regarding activities in our daily lives. We discover the best way to travel to a destination; we assess products in order to decide which ones to purchase; and so on. Program evaluation formalizes that intuitive process. It offers overall discipline, specific rules, and reliable methods so that we can ensure the validity of whatever information we collect and the soundness of whatever judgements we make. It offers us confidence that we will move forward to increase our impacts. 

The Manager’s Guide walks readers through all the steps in the program evaluation process, providing the information needed for planning, contracting, and managing a helpful evaluation. The book takes the program manager’s point of view. It informs program managers about what they need to know as managers – not about the technical skills they would need as program evaluation specialists. We hope that readers find that slant valuable. 

Ground yourself in the evaluation essentials

One chapter covers the basics – offering definitions and key concepts important for understanding the evaluation process. That chapter describes the essential information that an evaluation must have. It also indicates other information which, though not absolutely necessary, can enable a program manager to answer more questions and get at more nuances in determining the effectiveness of their services. That chapter also describes the importance of having a program theory or logic model. 

Understand your role throughout the evaluation process

Another chapter guides readers step by step through the four phases of an evaluation: design, data collection, analysis, and reporting. For each step, the book indicates what major activities occur and the roles of the program manager and the evaluation researcher in those activities. For example, at one point in the design phase, a program manager will need to take a lead role in determining the major questions for the evaluation to answer. With knowledge of the questions to address, the researcher will take a lead role in identifying the most appropriate methods for obtaining information to respond to those questions. 

The Manager’s Guide also provides answers to practical questions. For example, how do you decide what to do on your own and what to hire an evaluation researcher to do? If hiring a program evaluator, how should someone select that person or organization? Many options exist, including working with a large firm, with an academic research center, with an independent solo practitioner, or with a smaller team or firm (for profit or nonprofit). Each of those options has its advantages and disadvantages, and the book points them out. 

Plan for the necessary resources

How much does an evaluation cost? That question often arises, and the book provides some guidance. However, spoiler alert: Asking, “How much does an evaluation cost?” resembles asking, “How much does a house cost?” The answer obviously is “it depends.” To estimate costs of program evaluation, you will need to consider what questions you must answer and develop an understanding of the methods required to respond to those questions. Knowing those things, you can reasonably predict the staff and consultant hours necessary and the associated costs. 

Make an impact!

Michael Patton has served as an inspiration, a guide, a teacher, a friend, and a mentor to many of us in the evaluation field. In reviewing the book, he offered the comment that “for managers committed to evidence-informed decision making, no better book exists.” I very much appreciate that assessment. In addition, I greatly appreciate the opportunity to have created this book to put tools into the hands of people who seek to make a difference in the world through their work – so that we can act in an equitable fashion to improve the lives of people everywhere, using sound information and making sound judgements for the betterment of humanity. 

The Manager’s Guide to Program Evaluation (2^nd edition) becomes available in January 2022. You can pre-order your copy from Turner Publishing.