The Census: An Eye on the Past

 On April 1st of this year (no fooling), the National Archives released the original census records from 1950. Interesting stuff if you seek to obtain information for historical, genealogical or other purposes.

With a name and the county / city of residence, you can search for a specific person. The more unique the name, the easier the search, especially in larger cities. You can also look for specific addresses.

I searched for information about parents and grandparents. The results helped me piece together some ancestral facts that I knew in part, but did not fully understand.

Do you want to know who lived in your house (if it existed) in 1950, or even earlier? I learned that the seven occupants in 1950 of my current house included three generations. The “head of household” reported the occupation of university professor. So, I explored the professor’s background and discovered that he had written several important books about local history.

I looked as well at other residents on my city block and nearby blocks. As I read names, ages, occupations of these people, it set me wondering about their stories – how they built their lives, how they arrived where they found themselves in 1950. What shaped their experiences, and how did their lives in these places shape our present experiences? I have looked at earlier census records as well. Comparing records over time provides details of changes at a microscopic level in a neighborhood. It tells us something about the individuals who lived in the houses, and also something about social processes and structures, as they changed or as they did not change, over the decades.

Understanding the past supports our efforts to move into the future. If you take a look at census records, who can predict what fascinating little historical tidbit you might discover – which will change your understanding of the world? 

Give it a try!

https://1950census.archives.gov/

Program evaluation: Gathering good data to increase positive impacts

 

 

A community group wants to improve its food distribution program and seeks to understand how best to increase the accessibility of food that has high nutritional value and meets varied cultural standards of the community’s residents.

 

A multi-service, nonprofit agency provides assistance to people across the life span. It asks: What can we do to increase the numbers of people who find our services helpful? Why do older people seem to use us more than younger people?

 

A museum seeks to enrich the lives of children in a neighborhood whose residents do not know about the museum’s services.

 

A foundation has a mission to provide funding for housing and wants to know how to best direct the funding toward the types of housing that fit the needs and preferences of people in the area the foundation serves.

 All of these organizations want to learn whether they accomplish their intended purpose. In the simplest form, they want to know: “If we do A, does it lead to B? 

Program evaluation is a systematic process for an organization to obtain information on its activities, its impacts, and the effectiveness of its work, so that it can improve its activities and describe its accomplishments.

Gathering solid evidence about effectiveness

A useful program evaluation typically builds upon four types of information:

• Client/participant characteristics - e.g., demographics
• Service data - type and amount of services, activities, treatments, etc.
• Documentation of results or outcomes – evidence of changes that occurred or needs met among the people served
• Perceptions about services – how people feel about their experience with the organization

·       The organization might have such information available, or it may need to develop the means to acquire that information. Ideally, the organization will have, or will create, a process for obtaining information in a complete and accurate form. Having information on clients and the services they receive requires a consistent method of tabulating service use, along with a database that stores that information for retrieval. Understanding outcomes and client satisfaction requires collection of authentic information from and about the people served by a program.

That’s where program evaluation enters the picture. Techniques for gathering information enable programs who want solid evidence about their effectiveness to gather it in reliable ways.

Logic models: Identifying the path to impact

Evaluation can, and should, stretch the limits of our thinking. Many years ago, after a couple of initial, and seemingly productive, meetings with a program that wanted to initiate an evaluation, they canceled an upcoming meeting. They did not return my phone calls (before the days of email!). After about a month, their director called and said, “Paul, we’re sorry we didn’t get back to you. We had an issue with ourselves, not with you. When you asked us to outline how we work and how we produce our impact, those questions raised existential concerns for us. We realized that we really did not have a solid process for working together; we realized that we really didn’t have a system to collaborate as a team to know what to do when in order to produce the best outcomes for our clients.”

Encouraging that organization to identify their program theory of change (e.g., a logic model) led to a pivotal moment – a juncture where they suddenly had new insight and, in a relatively short period of time, developed a new way to work together. Over the years, many organizations have told Wilder Research staff that the most useful part of the evaluation process is the development of a logic model. It clarifies what programs expect to accomplish. It offers a guide that enables staff to better plan and make decisions to improve the accessibility and effectiveness of their services. It assists in communicating about impact with many different kinds of stakeholders.

Showing progress and where we need to do better

Beyond helping us to improve our work, evaluation provides much other value. Staff and volunteers can feel a sense of pride and feel increased motivation by seeing the number of people served by their organization. They can feel empowered by knowing the number of people who benefited as expected from the services they received, and the number who did not. Organizations can share such information with other organizations in their field to compare notes and jointly discuss how they might improve effectiveness. Organizations can share the information with funders, to document what the funders’ resources supported and to point toward areas of additional need.

The single biggest mistake I’ve witnessed regarding evaluation is when people look at evaluation information as if it represents the score at the end of a sporting event – assuming that it shows whether we “won” or “lost”. Viewing evaluation this way makes people fearful, and it suppresses innovation.

One of the most successful fund-raising institutions in the United States – St. Jude Children’s Research Hospital – has often published statistics on its success rates for treatment of different types of cancer. Over several decades, some success rates began extremely low – even close to zero. Did that mean “give up”? Did that mean “hide the facts”? Absolutely not. In fact, the opposite. It served to rally people around a cause, and it offered a baseline against which to judge new approaches. Acknowledging the difficulty of achieving success increased determination to find cures; it strengthened appeals for funding and for other resources to improve the health of children.

So too with other types of issues that nonprofit, community-oriented organizations face. Some social issues might seem intractable. Adapting organizational activities to fit changing needs and new populations might seem bewildering. That does not mean “game over, we lost.” It means, whatever our level of effectiveness – high, medium, or low – let’s build on that to strengthen our work and increase our impact.

Evaluation comprises part of an ongoing cycle of using information to design and deliver services, gathering more information to see how well the services achieve intended outcomes, and then using that new information to make revisions and adjustments for improvement. Evaluation creates a platform for evidence-based decision-making: We can never have total certainty that what we do will be effective, but with good data, we increase the probability that we will create positive impacts.

The development of programs and policies that benefit people involves a constant search. We seriously err if we think we know it all, or if we remain rigid in our thinking and our approaches to helping people, solving problems, and addressing issues. Through program evaluation, we ask questions, seek better paths, and make progress with humility to create a better world.

 

The Manager's Guide to Program Evaluation: 2nd Edition: Planning, Contracting, & Managing for Useful Results is now available. This revised edition is an invaluable resource with brand new real-world examples taken from recent evaluation research projects conducted by Wilder Research staff. Get your copy here from: Turner Publishing.

Our New Year 2022

 

Today’s New York Times stated: “January 2022 arrives as our ways of keeping time seem broken. Calendar pages turn, yet time feels lost.” Yes. It can feel as if time froze. Recent events can seem distant; events of a year ago can seem like they recently occurred. Pandemic time does not have the feel of “real time”.

Of course, time did not freeze. Throughout 2021, the world continued to need research, and we at Wilder Research persevered, despite many obstacles, to do social research to improve the lives of individuals, families, and communities.

I never anticipated that the last of my 40 years as Executive Director of this 104 year old research institute would include navigating through a pandemic. The plague of Covid-19 forced us to balance work and personal responsibilities in unique ways, cope with sickness of ourselves and/or family members, learn how to accomplish a lot of tasks remotely, participate in meetings knowing that a child or grandchild might pop onto the screen, and overcome myriad other challenges – while nevertheless maintaining high standards of quality for our products and services.

Wilder Research staff acted in heroic ways to move our products and services out the door and into the hands of research users. Our mission maintained our motivation because community needs did not freeze in time; organizations striving to make the world a better place did not halt operations. Community leaders, decision-makers, public officials, philanthropic organizations, news media sought greater insight (which research can provide) regarding social issues, to cope with unprecedented experiences.

Whatever 2022 brings – it will not include yesteryear’s standard patterns of living and working. Amidst the “new abnormal”, we might not settle into any predictable arrangements for our lives, for at least some time. But at Wilder Research, the values we hold – such as dedication to community, faith in our colleagues, commitment to high quality research products and services – those will persist and transcend whatever the future drops upon us.

Happy New Year!

Managing Evaluation – from the Program Manager’s Point of View

 

Much interest exists in evaluating and improving programs that serve our communities. Staff in nonprofit organizations and government agencies of all types want to know which services work and which do not work, so that they can continually increase their effectiveness. Funders want to know how best to invest their resources. 

I’ve written a new book, The Manager’s Guide to Program Evaluation (2^nd edition), to help those of you who value the benefits of high quality program evaluation and have responsibility for making sure that evaluation occurs, but need help understanding how best to make it happen. This new edition provides several examples of evaluation research, based on work completed at Wilder Research. Those examples can help readers to better understand how evaluation might occur in their organizations. 

The Manager’s Guide takes a common sense approach to explaining program evaluation. All of us intuitively make decisions, based on the best information that we have, regarding activities in our daily lives. We discover the best way to travel to a destination; we assess products in order to decide which ones to purchase; and so on. Program evaluation formalizes that intuitive process. It offers overall discipline, specific rules, and reliable methods so that we can ensure the validity of whatever information we collect and the soundness of whatever judgements we make. It offers us confidence that we will move forward to increase our impacts. 

The Manager’s Guide walks readers through all the steps in the program evaluation process, providing the information needed for planning, contracting, and managing a helpful evaluation. The book takes the program manager’s point of view. It informs program managers about what they need to know as managers – not about the technical skills they would need as program evaluation specialists. We hope that readers find that slant valuable. 

Ground yourself in the evaluation essentials

One chapter covers the basics – offering definitions and key concepts important for understanding the evaluation process. That chapter describes the essential information that an evaluation must have. It also indicates other information which, though not absolutely necessary, can enable a program manager to answer more questions and get at more nuances in determining the effectiveness of their services. That chapter also describes the importance of having a program theory or logic model. 

Understand your role throughout the evaluation process

Another chapter guides readers step by step through the four phases of an evaluation: design, data collection, analysis, and reporting. For each step, the book indicates what major activities occur and the roles of the program manager and the evaluation researcher in those activities. For example, at one point in the design phase, a program manager will need to take a lead role in determining the major questions for the evaluation to answer. With knowledge of the questions to address, the researcher will take a lead role in identifying the most appropriate methods for obtaining information to respond to those questions. 

The Manager’s Guide also provides answers to practical questions. For example, how do you decide what to do on your own and what to hire an evaluation researcher to do? If hiring a program evaluator, how should someone select that person or organization? Many options exist, including working with a large firm, with an academic research center, with an independent solo practitioner, or with a smaller team or firm (for profit or nonprofit). Each of those options has its advantages and disadvantages, and the book points them out. 

Plan for the necessary resources

How much does an evaluation cost? That question often arises, and the book provides some guidance. However, spoiler alert: Asking, “How much does an evaluation cost?” resembles asking, “How much does a house cost?” The answer obviously is “it depends.” To estimate costs of program evaluation, you will need to consider what questions you must answer and develop an understanding of the methods required to respond to those questions. Knowing those things, you can reasonably predict the staff and consultant hours necessary and the associated costs. 

Make an impact!

Michael Patton has served as an inspiration, a guide, a teacher, a friend, and a mentor to many of us in the evaluation field. In reviewing the book, he offered the comment that “for managers committed to evidence-informed decision making, no better book exists.” I very much appreciate that assessment. In addition, I greatly appreciate the opportunity to have created this book to put tools into the hands of people who seek to make a difference in the world through their work – so that we can act in an equitable fashion to improve the lives of people everywhere, using sound information and making sound judgements for the betterment of humanity. 

The Manager’s Guide to Program Evaluation (2^nd edition) becomes available in January 2022. You can pre-order your copy from Turner Publishing.

The Presidential Election, Dr. Fauci, and Our Commitment to Science

 

“A presidential election in the United States may be looked upon as a time of national crisis. As the election draws near, intrigues intensify, and agitation increases and spreads. The citizens divide into several camps, each behind its candidate. The entire nation falls into a feverish state. The election becomes the daily grist of the public papers, the subject of private conversations, the aim of all activity, the object of all thought, the sole interest of the moment.”*

So wrote Alexis de Tocqueville in the 1830s. Not much about the election dynamics he described almost two centuries ago has changed. But much seems to have intensified during 2020, especially the “division into several camps.”

Unfortunately, one camp more than the other in the current Presidential election has chosen to act contrary to science. In many cases, the members of that camp blatantly reject strong evidence of what can protect the health of the population.

However, we should not be shortsighted. During my 40+ years of research for the public good, I have witnessed people across the political continuum reject clear, well-established facts. Some proportion of liberals, centrists, and conservatives succumb to the temptation to ignore or reject truths that don’t neatly correspond with their pre-determined assumptions or that don’t conveniently support their goals.

The natural tendency to bend reality to fit our preconceptions and biases, along with the sometimes subconscious purging of information that does not align with our worldview constitute aspects of human frailty that underlie the rationale for independent news media, nonpartisan research organizations, and other institutions and practices that can ground us in accurate representations of reality.

Mask wearing and social distancing constitute “no brainers” – actions everyone should readily accept without hesitation in order to protect both health and economic vitality. In 2007, long before controversies regarding how best to handle COVID-19, the AmericanJournal of Medicine reported strong evidence that, during the pandemic of 1918-19, “nonpharmaceutical interventions,” such as school closings, cancelation of public gatherings, quarantines, and mask wearing, reduced deaths.

Yet, politicians have perverted these sensible, evidence-based practices into partisan symbols. That’s very sad and has already had deleterious effects.

Ironically, China, the country in which the virus initially launched itself, successfully controlled COVID and restored economic activity quickly – because it placed public health measures first. Good health seems to support economic growth. At Wilder Research, one of our studies sponsored by the Robert Wood Johnson Foundation and reported last year, showed that metropolitan regions in the U.S. with stronger health seem to have more vibrant economic capacity and greater resilience when they encounter shocks and challenges. In short, it's not either/or. We can have good health and economic prosperity.

The politicization of established health practices has killed and will kill. That’s why Dr. Fauci, this country’s North Star during the pandemic, steadfastly refuses to become partisan. His objective is to serve the public good, regardless of the political party in power, and despite whatever the prevailing attitudes, fads, and fashions of the times might be.

On Election Day and any other day, I encourage all of us to keep an open mind in searching for the best approach to today’s issues. In voting, give preference to candidates – of any party – who base their platforms on sound, reliable information. Everyone deserves their own opinion. In fact, it is good to have a mix of opinions so that we can learn with one another and evolve over time our understanding of what our planet needs to survive and thrive. However, everyone does not deserve their own separate set of facts.

If someone says that masks do not work to prevent infection, we have a hundred years of evidence to prove that statement incorrect. The intrigues, agitation, and feverishness that de Tocqueville mentioned should not cloud our judgement, shape our thinking, and influence our votes. Science, while imperfect and always improving, can nourish our decision-making about the people, programs, and policies that our communities need.

Facts matter, because

People matter.

*From Democracy in America, translated and paraphrased from several sources.