Managing Evaluation – from the Program Manager’s Point of View

 

Much interest exists in evaluating and improving programs that serve our communities. Staff in nonprofit organizations and government agencies of all types want to know which services work and which do not work, so that they can continually increase their effectiveness. Funders want to know how best to invest their resources. 

I’ve written a new book, The Manager’s Guide to Program Evaluation (2^nd edition), to help those of you who value the benefits of high quality program evaluation and have responsibility for making sure that evaluation occurs, but need help understanding how best to make it happen. This new edition provides several examples of evaluation research, based on work completed at Wilder Research. Those examples can help readers to better understand how evaluation might occur in their organizations. 

The Manager’s Guide takes a common sense approach to explaining program evaluation. All of us intuitively make decisions, based on the best information that we have, regarding activities in our daily lives. We discover the best way to travel to a destination; we assess products in order to decide which ones to purchase; and so on. Program evaluation formalizes that intuitive process. It offers overall discipline, specific rules, and reliable methods so that we can ensure the validity of whatever information we collect and the soundness of whatever judgements we make. It offers us confidence that we will move forward to increase our impacts. 

The Manager’s Guide walks readers through all the steps in the program evaluation process, providing the information needed for planning, contracting, and managing a helpful evaluation. The book takes the program manager’s point of view. It informs program managers about what they need to know as managers – not about the technical skills they would need as program evaluation specialists. We hope that readers find that slant valuable. 

Ground yourself in the evaluation essentials

One chapter covers the basics – offering definitions and key concepts important for understanding the evaluation process. That chapter describes the essential information that an evaluation must have. It also indicates other information which, though not absolutely necessary, can enable a program manager to answer more questions and get at more nuances in determining the effectiveness of their services. That chapter also describes the importance of having a program theory or logic model. 

Understand your role throughout the evaluation process

Another chapter guides readers step by step through the four phases of an evaluation: design, data collection, analysis, and reporting. For each step, the book indicates what major activities occur and the roles of the program manager and the evaluation researcher in those activities. For example, at one point in the design phase, a program manager will need to take a lead role in determining the major questions for the evaluation to answer. With knowledge of the questions to address, the researcher will take a lead role in identifying the most appropriate methods for obtaining information to respond to those questions. 

The Manager’s Guide also provides answers to practical questions. For example, how do you decide what to do on your own and what to hire an evaluation researcher to do? If hiring a program evaluator, how should someone select that person or organization? Many options exist, including working with a large firm, with an academic research center, with an independent solo practitioner, or with a smaller team or firm (for profit or nonprofit). Each of those options has its advantages and disadvantages, and the book points them out. 

Plan for the necessary resources

How much does an evaluation cost? That question often arises, and the book provides some guidance. However, spoiler alert: Asking, “How much does an evaluation cost?” resembles asking, “How much does a house cost?” The answer obviously is “it depends.” To estimate costs of program evaluation, you will need to consider what questions you must answer and develop an understanding of the methods required to respond to those questions. Knowing those things, you can reasonably predict the staff and consultant hours necessary and the associated costs. 

Make an impact!

Michael Patton has served as an inspiration, a guide, a teacher, a friend, and a mentor to many of us in the evaluation field. In reviewing the book, he offered the comment that “for managers committed to evidence-informed decision making, no better book exists.” I very much appreciate that assessment. In addition, I greatly appreciate the opportunity to have created this book to put tools into the hands of people who seek to make a difference in the world through their work – so that we can act in an equitable fashion to improve the lives of people everywhere, using sound information and making sound judgements for the betterment of humanity. 

The Manager’s Guide to Program Evaluation (2^nd edition) becomes available in January 2022. You can pre-order your copy from Turner Publishing.

The Presidential Election, Dr. Fauci, and Our Commitment to Science

 

“A presidential election in the United States may be looked upon as a time of national crisis. As the election draws near, intrigues intensify, and agitation increases and spreads. The citizens divide into several camps, each behind its candidate. The entire nation falls into a feverish state. The election becomes the daily grist of the public papers, the subject of private conversations, the aim of all activity, the object of all thought, the sole interest of the moment.”*

So wrote Alexis de Tocqueville in the 1830s. Not much about the election dynamics he described almost two centuries ago has changed. But much seems to have intensified during 2020, especially the “division into several camps.”

Unfortunately, one camp more than the other in the current Presidential election has chosen to act contrary to science. In many cases, the members of that camp blatantly reject strong evidence of what can protect the health of the population.

However, we should not be shortsighted. During my 40+ years of research for the public good, I have witnessed people across the political continuum reject clear, well-established facts. Some proportion of liberals, centrists, and conservatives succumb to the temptation to ignore or reject truths that don’t neatly correspond with their pre-determined assumptions or that don’t conveniently support their goals.

The natural tendency to bend reality to fit our preconceptions and biases, along with the sometimes subconscious purging of information that does not align with our worldview constitute aspects of human frailty that underlie the rationale for independent news media, nonpartisan research organizations, and other institutions and practices that can ground us in accurate representations of reality.

Mask wearing and social distancing constitute “no brainers” – actions everyone should readily accept without hesitation in order to protect both health and economic vitality. In 2007, long before controversies regarding how best to handle COVID-19, the AmericanJournal of Medicine reported strong evidence that, during the pandemic of 1918-19, “nonpharmaceutical interventions,” such as school closings, cancelation of public gatherings, quarantines, and mask wearing, reduced deaths.

Yet, politicians have perverted these sensible, evidence-based practices into partisan symbols. That’s very sad and has already had deleterious effects.

Ironically, China, the country in which the virus initially launched itself, successfully controlled COVID and restored economic activity quickly – because it placed public health measures first. Good health seems to support economic growth. At Wilder Research, one of our studies sponsored by the Robert Wood Johnson Foundation and reported last year, showed that metropolitan regions in the U.S. with stronger health seem to have more vibrant economic capacity and greater resilience when they encounter shocks and challenges. In short, it's not either/or. We can have good health and economic prosperity.

The politicization of established health practices has killed and will kill. That’s why Dr. Fauci, this country’s North Star during the pandemic, steadfastly refuses to become partisan. His objective is to serve the public good, regardless of the political party in power, and despite whatever the prevailing attitudes, fads, and fashions of the times might be.

On Election Day and any other day, I encourage all of us to keep an open mind in searching for the best approach to today’s issues. In voting, give preference to candidates – of any party – who base their platforms on sound, reliable information. Everyone deserves their own opinion. In fact, it is good to have a mix of opinions so that we can learn with one another and evolve over time our understanding of what our planet needs to survive and thrive. However, everyone does not deserve their own separate set of facts.

If someone says that masks do not work to prevent infection, we have a hundred years of evidence to prove that statement incorrect. The intrigues, agitation, and feverishness that de Tocqueville mentioned should not cloud our judgement, shape our thinking, and influence our votes. Science, while imperfect and always improving, can nourish our decision-making about the people, programs, and policies that our communities need.

Facts matter, because

People matter.

*From Democracy in America, translated and paraphrased from several sources.

Celebrating All Who Labor

 “Labor Day, the first Monday in September, is a creation of the labor movement and is dedicated to the social and economic achievements of American workers. It constitutes a yearly national tribute to the contributions workers have made to the strength, prosperity, and well-being of our country.” (U.S. Department of Labor)

The day emerged during a transitional period for the United States economy that lasted well into the 20^th century. Rates of participation in the manufacturing sector increased, while rates of participation in the agricultural sector decreased. Safety laws, health ordinances, and other protections for workers largely did not exist at the turn of the century. Immigrants, such as my grandmother, faced harsh working conditions.

The U.S. celebrated the first federal Labor Day holiday in September of 1894, although some cities and states had established the holiday during the preceding 10 years. Pressure to improve conditions for working people motivated legislators to act to establish the holiday. Labor strikes, violence, and unrest in 1894 pushed them to enact legislation, subsequently signed by the President, with the hope that the symbolism of the holiday would prompt change.

A railroad strike in the spring of 1894 and a sympathy boycott of the railroads led by Eugene Victor Debs (I worked for radio station WEVD, named after him, but that’s another story.) precipitated a feud between federal and local officials regarding whose troops should restore order in Chicago:

“President Cleveland dispatched federal troops to the city to enforce the injunction. Illinois’ pro-labor governor, John Peter Altgeld, who had already called out state militia troops to prevent violence, was outraged, calling the government’s actions unconstitutional. With the arrival of federal troops, the Pullman strike turned bloody, with some rioters destroying hundreds of railroad cars in South Chicago on July 6, and National Guardsmen firing into a mob on July 7, killing as many as 30 people and wounding many others.” (History.com)

Does that sound eerily parallel to events we have witnessed in 2020?

History does not show that improvement in workers’ lives occurred at the hoped-for pace. Enactment of laws to protect the health and safety of workers, and everyone else, occurred slowly during the last decades of the nineteenth century and the first half of the twentieth century. Minnesota passed safety legislation as early as the 1880s. However, it tended to have only modest effects because it relied on voluntary compliance. The Fair Labor Standards Act of 1938 initiated the minimum wage. The Civil Rights Act of 1964 banned employment discrimination based on race, gender, and other personal characteristics.

Coincidentally, the first study published by Wilder Research, in 1917, prompted the development of municipal health and safety ordinances in cities around the U.S.

Despite legislation, advocacy, and good intentions, the benefits of employment remain unavailable to some in our communities. Racial and gender differences persist in the proportion of adults who work, wages, and corporate leadership, for example. Minnesota Compass shows how these disparities exist in Minnesota.

On this Labor Day, my wish is that we can find ways to distribute equitably the fruits of our economy to all people in the U.S.

In addition, on this day originally intended to honor people employed in the paid labor force, I suggest that we might broaden our definition of “labor” to honor all of us who contribute to our economy and our communities. We can treat the day as an opportunity to celebrate people who engage in paid employment, but also those who engage in raising children or in other forms of caregiving, in volunteer work to make our communities better, or in any other type of endeavor where they apply their energy and talents. Most of us engage in at least one, if not more, of those activities.

Let’s celebrate all who labor!

Awakening

“Awakening” – President Obama used that word several weeks ago. It seems appropriate. It also seems to have caught on among others. I certainly hope so.

Beyond a doubt, our society requires fundamental change. That change can only occur through diligent commitment and unflagging effort.

The sight of George Floyd pinned to the ground begging for air shocked and upset me. Watching the video also made me realize that reactions of shock and dismay are part of my privilege. I don’t live on a constant basis with the traumatic fear that the abuse which Floyd experienced might extend one day to one of my children or one of my brothers just because of the color of their skin.

We witness heightened attention to racism now because of the murder of George Floyd. Subsequent sentiments have come from many new places, including corporations and political leaders. But don’t confuse statements of solidarity, made in the heat of the moment, with commitments to persevere with long-term change. To paraphrase Reverend Al Sharpton at Floyd’s Houston memorial service, we need to keep up the work after the last TV truck leaves.

My resolve has intensified to strengthen the impacts of Wilder Research. As a group of about 80 people, we can’t change the world by ourselves, but we will persist and resist. We will persist in using our research talents to address noxious aspects of our society, such as systemic racism and the institutional structures that lead to disparities and inequities. We will resist -- through our sound, deliberate, nonpartisan research techniques -- those forces that have emerged to distort the truth and obfuscate the public in order to create division and perpetuate injustice.

To this point in the history of Wilder Research, we have attempted in various ways to shed light on racial disparities. Many community organizations, nonprofits, and media consult our Minnesota Compass, for example, for such information. Compass also includes an anti-racism resource guide. We have engaged in projects intended to address disparities in health, housing, and criminal justice. We have worked on initiatives that give voice to the new arrivals to our region. We try to address the long-term conditions that produce social, health, and economic inequities. 

Long-term, systemic alteration of attitudes, behavior, structure, and culture constitutes the only way that we will prevent occurrences such as what happened to George Floyd. Regardless of what we’ve done in the past, we at Wilder Research can do more. So, we will ask what we can do better, how we can align with those who have wisdom that we don’t have, and what activities of ours can produce the most meaningful anti-racist influence.

However, talk is cheap. Therefore, in closing I introduce another “A” word: Accountability. 

The Wilder Research website reflects almost everything that we do. Please return to it six months and a year from now, “after the last TV truck has left.”. Hold us accountable.

• Have we at Wilder Research intensified our efforts to address systemic racism through work that sheds light on disparities and social determinants of health, wealth and well-being?
• Have we collaborated with communities and organizations to test interventions to eradicate racial disparities?
• Have we incorporated a racial equity perspective as much as possible in our projects, even when the primary focus is not race?
• Does a substantial and meaningful proportion of our activities address racism-related issues?

If you do not feel that each of those questions merits a “yes” response, please bring that to my attention. Get in touch. 

As Martin Luther King noted, we are all “caught in an inescapable network of mutuality, tied in a single garment of destiny.” As we direct our attention to the future, we should continually strive, in our professional and personal lives, to design that garment to fit everyone equitably and optimally.

Eliminating What Sets Us Apart: Some Thoughts for Martin Luther King Day

“I am convinced that men hate each other because they fear each other. They fear each other because they don’t know each other, and they don’t know each other because they don’t communicate with each other, and they don’t communicate with each other because they are separated from each other.”

So stated the Reverend Dr. Martin Luther King, Jr., in 1962.

Class and race boundaries have a lot of rigidity. In combination with geographic boundaries, they separate people and foster misinformation, stereotypes, and fear.

The 1917 report from Wilder Research on health and housing in Saint Paul showed a concentration of unacceptable conditions in a central city area that we currently call the Thomas-Dale neighborhood. Seventy years later, data we analyzed for use by community groups showed that children living in roughly that same area more likely lived in poverty and had lower educational outcomes than children had in other neighborhoods. Use Minnesota Compass today to compare income levels and other characteristics across neighborhoods in Saint Paul, and what neighborhood district will you find at or near the bottom? (Hint: the same one that showed up in Wilder Research data a century earlier.)

In 1967, Dr. King noted the lack of progress for Black citizens.

“Negroes generally live in worse slums today than 20 or 25 years ago. In the North schools are more segregated today than they were in 1954 when the Supreme Court's decision on desegregation was rendered. Economically the Negro is worse off today than he was 15 and 20 years ago. And so the unemployment rate among Whites at one time was about the same as the unemployment rate among Negroes. But today the unemployment rate among Negroes is twice that of Whites. And the average income of the Negro is today 50% less than Whites.”

How much change has occurred since then? The Brookings Institution highlights research showing how race and economic disparities persist into the 21^st century and fuel educational disparities that threaten our society. Reflecting the importance of economics, which Dr. King identified, they note that poverty-based segregation and race-based segregation contribute to the educational disparities we witness today. In fact, segregation based on poverty appears statistically to account for differences that we sometimes attribute to the effects of race and racial discrimination.

What maintains these rigid boundaries that produce separation with deleterious consequences? We can easily point to outspoken racists and advocates of segregation, of course. Dr. King recognized and confronted the visible, tangible elements of racism. He also faced observable white backlash. However, overt attempts to maintain inequality only partly explain our longstanding social problem.

Another insidious, though less visible, element also explains the persistence of economic and social disparities: a form of “backlash” that occurs through complacency. People who want to do good, but seek to avoid too much disruption occurring too rapidly in the status quo, can impede progress as much as vocal opponents of change.

“I'm absolutely convinced that the forces of ill-will in our nation, the extreme rightists in our nation, have often used time much more effectively than the forces of good will. And it may well be that we will have to repent in this generation not merely for the vitriolic words of the bad people and the violent actions of the bad people, but for the appalling silence and indifference of the good people who sit around and say wait on time. Somewhere we must come to see that social progress never rolls in on the wheels of inevitability. It comes through the tireless efforts and the persistent work of dedicated individuals. And without this hard work time itself becomes an ally of the primitive forces of social stagnation. And so we must help time, and we must realize that the time is always ripe to do right.”

Let’s acknowledge that we need to win a Super Bowl of real life – much more important than the mere football game that takes place a couple of weeks after today’s important holiday. We cannot let the forces of evil play the clock to win, or we will experience a lose-lose for humanity.