Позитивные изменения, Том 3 №1, 2023. Positive changes. Volume 3, Issue 1 (2023) - Редакция журнала «Позитивные изменения»

In a rapidly changing world, we need fast-turnaround techniques that produce real data in real time and provide them just as quickly to people.

If you have to work on topics like pandemics, climate change, or civil unrest when things are changing rapidly, randomized controlled trials are not appropriate – they are too rigid and too time-consuming.

The speed of change means that the evaluation must produce results quickly, in real time, and provide them just as quickly to people who need to make decisions in a time-constrained environment.

The evaluator’s profession is changing in response to what is going on, but evaluators themselves are also changing the world. What do you think this profession has already given to the world?

I think we are living in a time of a worldwide battle between approaches based on evidence, facts, and science, on the one hand, and a part of the world that ignores them and only seeks to promote ideology. Evaluation is part of this worldwide data. Evaluation proves that if you use data, you make better decisions and help people more.

We’ve seen it work with people who refused COVID vaccination, who don’t believe the pandemic and vaccination data. In the so-called post-truth or anti-science world, where people can say whatever they believe and assume it to be true, evaluators help people appreciate the importance of evidence, looking at reality and working with it rather than with subjective perceptions and beliefs. This is our contribution.

In the so-called post-truth or anti-science world evaluators help people appreciate the importance of evidence, looking at reality and working with it.

You are the author of many program evaluation concepts used by evaluators around the world. We would like to discuss some of them. The first question concerns the concept of utilization-focused evaluation. In it, you emphasize the client’s role and interest, their desire to participate in the use of the evaluation results. Nowadays, investors in social projects and even the state (as in the social impact bond model, SIB), often become the evaluators’ clients. They are not involved in the project itself, but are its customers. Is it possible to use the evaluation principle with an emphasis on its use in this specific situation? Are there any peculiarities of its application in this case?

This is an important question. Utilization-focused evaluation requires considering different levels and different types of stakeholders – we call them intended users. Who is this evaluation for? If the intended users are social investors, then I would want to engage them in defining the criteria they are seeking. What do they want to know to make a better investment?

When I work with social investors, part of what I do is help them understand as clients that there are different ways to do evaluation, because there are many different types of programs. For example, when I work with boards of directors investing in social projects or charitable foundations, I do an exercise with them. I make a list of different types of financial instruments: blue chip stocks, high growth stocks, undervalued stocks, that is, stocks of companies in transition or struggling, long-term bonds, short-term bonds, mutual funds. These people are well aware of the different types of financial instruments and the differences in their profitability. Then I take a set of projects and draw parallels between different kinds of projects and different kinds of financial instruments.

Blue chips are well known programs like the World Food Program or UNICEF; they are world-famous, stable, and have a good reputation. A program that seeks to scale, to enter other territories within or outside the country is the equivalent of high-growing stock in the financial market.

Undervalued stock is a program that is in transition and needs help to get through and adapt to change. Long-term bonds are the equivalent of operating support for a program without specifying what specific project the program will implement.

This exercise helps social investors understand that they value different financial instruments differently. They look at different types of stocks, investment funds or bonds using different criteria, and we also evaluate different programs using different criteria depending on the nature of the program and the nature of the investment. Using the logic of evaluating financial instruments helps them understand the logic of evaluating programs.

The theory of transformation, trying to change systems, must consist of many different theories of change. The Blue Marble evaluation is related to the theory of transformation: once we start dealing with large systems, we are dealing with global impacts.

Investors in social projects also have internal networks and have developed a number of methodological approaches. For example, they are very fond of the social return on investment (SROI) method. Sometimes it is even taken as the only possible, no-alternative method for evaluating the results of any project. This seems to go at odds with the evaluation specialists’ views – again, based on the utilization-focused evaluation concept.

The social effectiveness of investments still needs to be interpreted and adapted, depending on whether we are talking about long-term or short-term social effectiveness of investments, about innovative high-risk programs, high-risk programs with high performance potential, or low-risk, low-return programs. When I talk about parallels between different kinds of financial instruments and different kinds of programs, I’m talking about how much risk social project investors are willing to take, how innovative they want to be, what the time span is for investment and return, whether they are looking for several different indicators or just one measure of success.

There is a lot to negotiate and a lot to adapt, even with the general criteria of investing in social projects. It is also necessary to determine what it means to invest in social projects and the effectiveness of such investments in a particular situation. And this is where utilization-focused evaluation is needed, because it provides a dialogue with clients