How to Evaluate the Effectiveness of State Programmes?

On June 29^th the second plenary session of the APPAM (Association for Public Policy Analysis and Management) international conference ‘Improving the Quality of Public Services’ took place at the HSE.

The second day of the conference started with a presentation by Douglas Besharov, Professor at the University of Maryland School of Public Policy, dedicated to a comparison of two different approaches to implementing state programmes: management by results and effectiveness evaluation. These approaches, if they are used to analyze one programme, can sometimes give different answers to the question of whether it is worth continuing the programme. Moreover, some positive results can be deceptive or can change over time.

Evaluating the effectiveness of a specific state programme can take time, and over this period project leaders or even the principles of operational administration can change. That’s why evaluation in ‘real-time’ is very important, allowing prompt corrections to the programme. In addition to this, it is necessary to distinguish the ‘products’ produced during the programme and its final results. As an example, Douglas Besharov described several programmes fighting child poverty started in the U.S. in 1960s. One of the ‘products’ of those programmes could be considered that children were taught to read and write, but this did not mean that any child who learned this skill was guaranteed to have a wealthy life.

While evaluating a state programme it makes sense to use contrast, believes Maureen Pirog, Professor at the Indiana University School of Public and Environmental Affairs. In other words, the existing (or the missing) effect can be compared with what would be if the programme had not been implemented at all. This approach also allows comparison between new and old programmes.

Douglas Besharov

Maureen Pirog also spoke about the topic of different cultural approaches to the evaluation of state programmes which exist in Europe and the USA. In her view, Europeans tend to consider average figures which smooth over social differences, while in the U.S. the focus is on individual indicators. While in the U.S. a social programme would be considered successful if it leads to an increase in the individual revenue of a citizen, in Europe they would try to improve the system of revenues distribution and decrease the material deprivation and social isolation of the poorest strata of the population.

Another question is – What can a state afford? What to choose – a low-effective, but cheap programme or an effective but costly one? The response is often given not by researchers but by bureaucracy.

Jacob Klerman, Principal Associate at the Abt Associates research and consulting center (USA) suggested another view of the problem: is it possible to create an interface of state programmes which would combine management by results and also effectiveness evaluation? But then, ‘instead of focusing on methodology’, J. Klerman revealed ‘a little secret’: ‘A considerable part of what is being done with state programmes lies in the sphere of ‘pleasures’. It is pleasant to be involved in them, whether or not they bring a real effect or even worsen the situation. The experience proves that the better the evaluation of a programme, the less obvious its results. And, in fact, quite the opposite, programmes which have an obvious result can look bad when regressive and other types of analysis are used’.

 Alexey Barabashev, Dean of the HSE Faculty of Public Administration gave a speech dedicated to the history of public service reforms in contemporary Russia and the development of a new system of evaluationg the effectiveness of public officers. In particular, he talked about new job descriptions and classifications for officers and shared the results of an experiment on the implementation of a new effectiveness evaluation system for 150 members of the State Duma staff.

These results, according to A. Barabashev, were shocking even for the lawmakers themselves. It revealed that the public officers who were responsible for decision-making were busy only ‘thoughtlessly’ approving analytical documents which were prepared for them by ordinary members of staff. ‘Almost all decisions which came from below were approved, A. Barabashev stated. Then there is a question: who is our authority? It turns outs they are humble secretaries. And it is logical to suppose that such system of administration penetrates all bodies of the Russian government’.

Alexey Barabashev

Alexey Barabashev believes that in Russia it is necessary to develop ‘a new theory of effective government which would take into account the power of civil society, and have clear distribution of responsibilities for professionals in the public administration service. Their work (which means the process of administration itself) and its results should be evaluated as two separate procedures’. It is also necessary to measure the contribution of governmental bodies in the growth of the nation’s economic, cultural and social progress – and this will be the reflection of the effectiveness of the government’s work.

Steve Kelman, Professor at the Harvard University spoke about how the system of recruiting officials can be improved, how their responsibilities can be distributed and results evaluated. While the differences are obvious, an effective public administration, in his view, can be considered as a business administration, where the result is both the goal and a guarantee for the existence of the organization itself.

Alexey Germanovich, Public Projects’ Director at the Moscow School of Management Skolkovo, told the conference participants about the history and the goals of the Skolkovo project. ‘For us it is important to teach students not only to analyze available materials, but to apply them in practice’ he emphasized.

The plenary session closed with a presentation by Jason Turner from the Practical Government Solutions research center (Milwaukee, USA). Mr. Turner was head of several governmental programmes in social welfare, the results of which he unveiled at the conference. One such programme was carried out by J. Turner in Wisconsin. As a result of the change in the work scheme and financial support for employment offices, the number of the paid unemployment allowances in the state decreased by more than 60 percent: employment offices started looking for new workplaces for their clients far more effectively.

A similar project was implemented in New York in the time of Rudolph Giuliani. Some measures implemented in Wisconsin and New York were borrowed by the Labour government of the Great Britain and continue under the new Conservative government, proving their effectiveness.

Oleg Seregin, HSE News Service

Photos by Nikita Benzoruk