Upload
margery-wade
View
221
Download
2
Embed Size (px)
Citation preview
Ce document est conçu comme un soutien à la présentation orale et n'est pas destiné à être utilisé séparément
Major methodological lessons learnt
Ex post evaluationof the impact of 4 projects of
the governmental cooperation
Brussels, Seminar on Impact Evaluation, 27th March 2014
2
Goals and definitions
Case studies: projects Senegal and DRC
Formative lessons learnt
Agenda
Goals
3
Drawing lessons– Which impact? – How to evaluate impact?– Feasibility / Utility of impact evaluation?
4 projects ≠ evaluated EX POST– Implementing phase of the projects : 2003-2008– Mixed methods (quali & quanti)– Mixed team (ADE/CRED)
Double goal: sommative and formative
3 definitions of impact
4
A major source of mis-
understanding!
A logic of action that seems shared…
5
OUTCOMES
OUTPUTS
INPUT / ACTIVITIES IMPACT
… but with which definition for each of these elements?
3 definitions of impact
① DAC : « Positive and negative long-term effects
produced by a development intervention, directly or
indirectly, intended or unintended »
② Logical framework: « Ultimate goal » =
« contribution of the intervention to changes at a
more global level »
③ Quanti methods: « effects on the beneficiaries that
can be attributed to the project » (« outcomes » of
the logical framework)
6
3 definitions of impact
① CAD : « long-term effects… »
Effects are time bound
② Logical framework: « Ultimate goal »
Effects at a global level
③ Quanti methods: «effects that can be attributed…» = « outcomes » of logical framework
Effects on beneficiaries + attribution
7
QUALI
QUANTI
3 definitions of impact
8
A major source of mis-
understanding!
Development interest: impact (global level)Scientific interest: outcomes (tunnel approach)
Mixed methods
9
OUTCOMES
OUTPUTS
INPUT / ACTIVITIES
QUALITATIVE« Impact evaluation »
IMPACT
Process
Relevant causal chain? Conditions / hypothesis verified?
Mixed methods
10
OUTCOMES
OUTPUTS
INPUT / ACTIVITIES
QUALITATIVE« Impact evaluation »
QUANTITATIVE« Impact
evaluation »
IMPACT
Measure & Attribution
Process
Mixed methods
Qualitative methods Reasoned judgement on impact
– Logical framework analysis– Document analysis, interviews and observations
Quantitative methods
Measure of attributable outcomes– Statistical and econometrical analysis– Sufficient reliable data ; beneficiaries and non-
beneficiaries ; before/after the project. 11
Theory-based evaluation
12
OUTCOMES
OUTPUTS
INPUT / ACTIVITIES
QUALITATIVE« Impact evaluation »
QUANTITATIVE« Impact
evaluation »
IMPACT
In depth analysis of causal chain and
conditions for success
Six principles (© H. White 2009)
13
(1) Map out the causal chain(programme theory)
(2) Understand context (3) Anticipate heterogeneity (4) Use a credible counterfactual (5) Rigorous factual analysis(6) Use mixed methods
Limits of this study
14
EX POST evaluations
No optimal use of quantitative methods
No representative sample of projects
15
Goals and definitions
Case studies:
–Project in Senegal
–Project in DRC
Formative lessons learnt
Agenda
16
Project: Water supply (rural),
2003-2008 Methods of analysis:
Mixed methods incl. quasi-experimental design with counterfactual
• Reconstruction of the intervention logic• Document study ; literature review• 217 household surveys (4 boreholes), with counterfactual
(no baseline available)• Difference of means test (t-test) and Independence test on first hand
data (household and individual level)• Double difference tests• 34 Focus groups (users, village authorities)• Interviews with resource persons in Belgium and in Senegal• Site visits
PARPEBA in Senegal
17
Project: Water supply (rural)
OUTPUTS: overall achieved OUTCOMES
– Time and energy gain (mostly for private taps)• Gain in wellbeing and more social cohesion• Quantity of water / distance : no evolution
– Water quality is a problem– Effect on education, health?– Strong willingness to pay
IMPACT– Project better water access for 15.5% of the
population (! water quality)
PARPEBA in Senegal
18
Goals and definitions
Case studies:
–Project in Senegal
–Project in DRC
Formative lessons learnt
Agenda
19
AETP2 in RDC
Project: Support to technical/vocational training,
2005-2008
Methods of analysis:
Case studies with counterfactual• Reconstruction of intervention logic• Document study; literature review• Interviews with resource persons in Belgium + DRC • Statistical analysis of second hand quantitative data
pre/post project (annual school performance)• Qualitative comparison of target and reference groups • 2 target schools + 2 reference schools (no baseline)• « Focus groups » with direct beneficiaries
(schools – teachers – students - graduates)
20
AETP2 in DRC
Project: Support to technical/vocational training
OUTPUTS: globally achieved OUTCOMES
- No tangible effects- Causes:
• Very unfavourable context• Partial use of outputs
IMPACT?- No outcomes no impact- In any case, limited coverage at national level: AETP1&2
= 5% of schools- Positive evolution at Ministry level
21
Goals and definitions
Case studies: projects Senegal and DRC
Formative lessons learnt
1. Mixed methods 2. Monitoring - evaluation3. Objective4. Types
Agenda
Lesson 1
1. Measure and demonstrate the attribution of certain outcomes
2. Appreciate and argue the attribution of all the outcomes
3. Argued judgment on the obtained impact
4. Understand the ‘why’ and the ‘how’ of effects or lack of effects
22
Mixed methods = adequate solution to appreciate and explain the outcomes and the
impact
Mixed methods = ideal
a)Quanti enriches quali
Via its objectives• Measure and attribution extrapolation
possible
Via its mechanism of information gathering• Number and reliability of quali data• Focus on beneficiaries (and non beneficiaries)• At random
23
Mixed methods = ideal
b)Quali allows rigorous quanti
4 challenges of quanti: ①A credible counterfactual
②Reliable data in sufficient quantity
③Technical rigour
④Realistic interpretation of results
Quali brings in depth understanding of programme theory + context
24
Mixed methods = ideal
c) Quali to understand the effects/non-effect Analysis of logical framework
• Coherence : means and fixed goals• Risks in the chain of results
Analysis of the context • Relevance• Social, economic, political, cultural, diplomatic
elements, local leadership, historic perspective…
25
Mixed methods = ideal
d) Consultation of all the actors
Quali: ‘a bit of all the actors’
Quanti : (non-) beneficiaries ; at random; numerous
Triangulation and + objectivity26
Mixed methods, Feasibility?
4 factors ① Numerous reliable data (baseline + ex post)
② Credible counterfactual
③ Agreement of national authorities
④ Value for money • Probability to have an effect • Use of results
27
28
Goals and definitions
Case studies: projects Senegal and DRC
Formative lessons learnt
1. Mixed methods 2. Monitoring - evaluation3. Objective4. Types
Agenda
Lesson 2
29
Importance of adequate design, implementation and M&E systems
Intervention logic (programme theory) must be ‘well thought out’
Precise and realistic objectives Focus on outcomes and impact M&E systems
30
Goals and definitions
Case studies: projects Senegal and DRC
Formative lessons learnt
1. Mixed methods 2. Monitoring - evaluation3. Objectives4. Types
Agenda
Lesson 3
31
Clarify the objective of the impact evaluation
Define the concepts univocally– « outcomes », « impact », etc.
Define the objective of every impact evaluation– Accountability, supporting decision making, or
capitalisation & sharing lessons learnt?– What role in the overall evaluation policy?– Of what use for the organisation?
32
Goals and definitions
Case studies: projects Senegal and DRC
Formative lessons learnt
1. Mixed methods 2. Monitoring - evaluation3. Objectives4. Types
Agenda
Lesson 4
33
Impact evaluation types
Rigorous evaluations of outcomes/impact; mixed methods based on the intervention logic (theory based evaluation)
Important elements: – Objectives – Timing – Budget– Data– Field
Methodological, financial and operational feasibility
Quality impact evaluation
34
EX ANTE evaluation= ideal case Logical framework = true management tool
+ quality in design• Explicit formulation of causal links• Context and issues at stake• Better defined, more realistic objectives
+ rigour in the evaluation• M&E (precise definitions of indicators; collection
modes) Baseline, monitoring, ex post data• Credible counterfactual (‘pipeline’)
Quality impact evaluation
EX POST evaluation= possible and rich
Interesting alternatives
Restitution = important step– To make the evaluation public– Reactions to feed the debates actions
35
36
Questions or information requests?
Office of the Special Evaluator - OSE:Dominique.deCrombrugghedeLooringhe@[email protected]
ADE:[email protected]@ade.eu
37