How to Evaluate the Effectiveness of State Programmes?
On June 29th the second plenary session of the APPAM (Association for Public Policy Analysis and Management) international conference ‘Improving the Quality of Public Services’ took place at the HSE.
The second day of the conference started with a presentation by Douglas Besharov, Professor at the University of Maryland School of Public Policy, dedicated to a comparison of two different approaches to implementing state programmes: management by results and effectiveness evaluation. These approaches, if they are used to analyze one programme, can sometimes give different answers to the question of whether it is worth continuing the programme. Moreover, some positive results can be deceptive or can change over time.
Evaluating the effectiveness of a specific state programme can take time, and over this period project leaders or even the principles of operational administration can change. That’s why evaluation in ‘real-time’ is very important, allowing prompt corrections to the programme. In addition to this, it is necessary to distinguish the ‘products’ produced during the programme and its final results. As an example, Douglas Besharov described several programmes fighting child poverty started in the U.S. in 1960s. One of the ‘products’ of those programmes could be considered that children were taught to read and write, but this did not mean that any child who learned this skill was guaranteed to have a wealthy life.
While evaluating a state programme it makes sense to use contrast, believes Maureen Pirog, Professor at the Indiana University School of Public and Environmental Affairs. In other words, the existing (or the missing) effect can be compared with what would be if the programme had not been implemented at all. This approach also allows comparison between new and old programmes.
Another question is – What can a state afford? What to choose – a low-effective, but cheap programme or an effective but costly one? The response is often given not by researchers but by bureaucracy.
Jacob Klerman, Principal Associate at the Abt Associates research and consulting center (USA) suggested another view of the problem: is it possible to create an interface of state programmes which would combine management by results and also effectiveness evaluation? But then, ‘instead of focusing on methodology’, J. Klerman revealed ‘a little secret’: ‘A considerable part of what is being done with state programmes lies in the sphere of ‘pleasures’. It is pleasant to be involved in them, whether or not they bring a real effect or even worsen the situation. The experience proves that the better the evaluation of a programme, the less obvious its results. And, in fact, quite the opposite, programmes which have an obvious result can look bad when regressive and other types of analysis are used’.
Alexey Barabashev, Dean of the HSE Faculty of Public Administration gave a speech dedicated to the history of public service reforms in contemporary Russia and the development of a new system of evaluationg the effectiveness of public officers. In particular, he talked about new job descriptions and classifications for officers and shared the results of an experiment on the implementation of a new effectiveness evaluation system for 150 members of the State Duma staff.
These results, according to A. Barabashev, were shocking even for the lawmakers themselves. It revealed that the public officers who were responsible for decision-making were busy only ‘thoughtlessly’ approving analytical documents which were prepared for them by ordinary members of staff. ‘Almost all decisions which came from below were approved, A. Barabashev stated. Then there is a question: who is our authority? It turns outs they are humble secretaries. And it is logical to suppose that such system of administration penetrates all bodies of the Russian government’.
Steve Kelman, Professor at the Harvard University spoke about how the system of recruiting officials can be improved, how their responsibilities can be distributed and results evaluated. While the differences are obvious, an effective public administration, in his view, can be considered as a business administration, where the result is both the goal and a guarantee for the existence of the organization itself.
Alexey Germanovich, Public Projects’ Director at the Moscow School of Management Skolkovo, told the conference participants about the history and the goals of the Skolkovo project. ‘For us it is important to teach students not only to analyze available materials, but to apply them in practice’ he emphasized.
The plenary session closed with a presentation by Jason Turner from the Practical Government Solutions research center (Milwaukee, USA). Mr. Turner was head of several governmental programmes in social welfare, the results of which he unveiled at the conference. One such programme was carried out by J. Turner in Wisconsin. As a result of the change in the work scheme and financial support for employment offices, the number of the paid unemployment allowances in the state decreased by more than 60 percent: employment offices started looking for new workplaces for their clients far more effectively.
A similar project was implemented in New York in the time of Rudolph Giuliani. Some measures implemented in Wisconsin and New York were borrowed by the Labour government of the Great Britain and continue under the new Conservative government, proving their effectiveness.
Oleg Seregin, HSE News Service
Photos by Nikita Benzoruk