The Evaluation Gap An International Initiative to Build Knowledge Presentation by

The Evaluation Gap An International Initiative to Build Knowledge

• Presentation by

• William D. Savedoff

• Sr. Partner, Social Insight

• Project Director, Center for Global Development

• Meeting of the DAC Evaluation Network

• Paris

• November 10, 2004

The Evaluation Gap Overview

• Who are we?

• What do we mean by an evaluation gap?

• Why does this evaluation gap occur?

• What other initiatives are being taken?

• What are we doing?

• Where are we going?

Who are we?

Working Group Members

• Members do not represent their institutions but participate in their individual capacity, bringing a wide range of experience from:

• Evaluation Offices

• NGOs

• Development Agencies

• Universities

• Philanthropic Foundations

Working Group Members

• Nancy Birdsall

• Francois Bourguignon

• Esther Duflo

• Paul Gertler

• Judith Gueron

• Indrani Gupta

• Jean-Pierre Habicht

• Dean Jamison

• Patience Kuruneri

What is “the evaluation gap”? Range of Analytical Work

• Project preparation studies

• Monitoring implementation

• Process & Operational evaluation

• Outcome evaluation

• Impact evaluation

Impact Evaluation

• Studies that measure changes in a target population that can be attributed to a particular program or policy

Importance of Attribution

• Different studies answer different questions

• Attribution should not be ignored

• Imagine a hypothetical Project for combatting HIV/AIDS in a large country with limited funds …

Importance of Attribution

• Different studies answer different questions

• Attribution should not be ignored

• Imagine a hypothetical Project for combatting HIV/AIDS in a large country with limited funds …

Results from Millions Saved Cases of public health interventions

Results from Millions Saved Cases of public health interventions

• “The gap in evaluation inhibits the documentation of successes, and prevents policymakers from being able to tell the difference between a well told story and a hard fact as they make decisions about which programs to support.”

Community Health Insurance

• Community Health Insurance was proposed as early as 1978 in a WHO Technical Expert Report

• What have we learned about community health insurance in the 36 years that have followed?

Community Health Insurance ILO/Universitas Review

Community Health Insurance Ekman, Health Policy & Planning (2004)

Why does this gap occur?

• Knowledge from impact evaluations is a public good

• Costs are visible, benefits seem far-off & resources are limited

• Demand is diffuse (institutions & time)

• “It Pays to be Ignorant”

• Low-quality studies crowd out the good

• Methodological challenges

Guarded optimism

• Changing profile of agency staff

• Worldwide capacity for good studies

• Recognition of impact evaluation value

• Pressure from skeptical donors

• Methodological & practical advances in research design (also highly visible & successful)

• PROGRESA/Oportunidades, IMCI, Guinea Worm, etc.

Other Initiatives

• Increasing access to existing information

• Developing aggregate indicators

• Improving capacity

• Promoting evaluation w/funds & data

• Conducting research & demonstrating good evaluation practices

Some Examples

• OECD/DAC Evaluation Network

• World Bank Research Department

• Health Metrics Network

• UN M.E.R.G.

• USAID/MACRO Surveys

• MIT Poverty Action Lab

We need an initiative that:

• Focuses specifically on the Public Good aspect of impact evaluation

• Develops a collective response to the problem

• Mobilizes & appropriately channels new funds

• Acts selectively where the most can be learned

The Evaluation Gap Working Group Process

• Preliminary interviews & research

• Convene working group

• Meetings, teleconference, and e-list debates

• Consultation group

• Draft paper & action plan

• Dissemination and broader debate

Likely characteristics of recommendations

• Identifying a reliable source of funding

• Establishing collective mechanism for selecting “enduring questions”, guidance, involvement & commitment

• Developing institutional mechanisms for channeling funds into appropriate studies & projects

What do you think?

• What is the fundamental problem from your perspective?

• Do you have other examples (reviews?)

• What are your current initiatives?

• What kinds of solutions should be considered?

• Who else should we consult?

Contact Us

• Center for Global Development www.cgdev.org

• Nancy Birdsall, President [e-mail]

• Ruth Levine, Senior Fellow [e-mail]

• William D. Savedoff, Project Director [e-mail]

References

• Christensen, Jon. Asking the Do-Gooders to Prove They Do Good. The New York Times . 1-3-2004.

• Development Assistance Committee. Principles for Evaluation of Development Assistance. Organisation for Economic Co-operation and Development. 1991. Paris, OECD. 2004.

• Development Assistance Committee. Review of the DAC Principles for Evaluation of Development Assistance. Organisation for Economic Co-operation and Development. 1-120. 1998. Paris, OECD. 2004.

• Development Assistance Committee. Glossary of Key Terms in Evaluation and Results Based Management. 6, 1-37. 2002. Paris, OECD. Evaluation and Aid Effectiveness.

References

• Dugger, Celia. “World Bank Challenged: Are the Poor Really Helped?” The New York Times . 7-28-2004. New York.

• Ekman, Björn. "Community-based health insurance in low-income countries: a systematic review of the evidence." Health Policy and Planning, 2004, 19 (5), 249-270.

• France, Ministère de l'Economie des Finances et de l'Industrie. Partners in Development Evaluation: Learning and Accountability. Partners in Development Evaluation: Learning and Accountability. 3-25-2003. Paris.

References

• International Labour Office. 2002. “Extending Social Protection in Health Through Community Based Health Organizations: Evidence and Challenges”, Dicussion Paper, Universitas Programme, ILO, Geneva.

• Kremer, Michael. "Randomized Evaluations of Educational Programs in Developing Countries: Some Lessons." American Economic Review Papers and Proceedings, 2003, 93 (2), 102-115.

• Pritchett, Lant. "It Pays to Be Ignorant: A Simple Political Economy of Rigorous Program Evaluation." The Journal of Policy Reform, December 2002, 5 (4), 251-269.

References

• Victora, Cesar G, Habicht, Jean-Pierre, Bryce, Jennifer. "Evidence-Based Public Health: Moving Beyond Randomized Trials." American Journal of Public Health, March 2004, 94 (3), 400-405.

• World Bank. Influential Evaluations: Evaluations that Improved Performance and Impacts of Development Programs. 1-24. 2004. Washington, DC, World Bank.

• World Health Organization. Financing of Health Services. 625. 1978. Geneva. Technical Report Series.

EG Working Group Members

• Nancy Birdsall, President, Center for Global Development

• Francois Bourguignon, Chief Economist & Sr. Vice President, World Bank

• Esther Duflo, Associate Professor of Economics, MIT

• Paul Gertler, Professor of Economics, Haas School of Business

• Judith Gueron, President, MDRC

• Indrani Gupta, Reader, Institute of Economic Growth

• Jean-Pierre Habicht, Professor, Cornell University

• Dean Jamison, Senior Fellow, National Institutes of Health

• Patience Kuruneri, Senior Policy Analyst, World Health Organization

• Ruth Levine, Senior Fellow, Center for Global Development

• Richard Manning, Chair, Development Assistance Committee

• Stephen Quick, Director, Inter-American Development Bank

• William D. Savedoff, Senior Partner, Social Insight

• Raj Shah, Senior Policy Officer & Senior Economist, Bill & Melinda Gates Foundation

• Smita Singh, Special Advisor for Global Affairs, William & Flora Hewlett Foundation

• Miguel Szekely, Ministry of Social Development, Mexico

• Cesar Victora, Professor, Universidade Federal de Pelotas, Brazil