SlideShare ist ein Scribd-Unternehmen logo
1 von 20
Lessons Learned In Using the Most
Significant Change Technique in Evaluation
Mary Freyder, MPH, LMSW
Bridgit Adamou, MPH
January 8, 2020
• Data for Impact (D4I) overview
• Sharing lessons from the field – Most Significant
Change (MSC) in five countries under MEASURE
Evaluation Project
• Deep dive - evaluating the Local Capacity Initiative
(LCI) in Uganda
• Questions and answers
Presentation Overview
• Generate evidence
• Strengthen capacity
• Ensure data quality
• Integrate gender
• Promote data use
• Learn
D4I works to:
• Participatory approach
• Participants give personal stories of significant
change, directly or indirectly related to intervention
• Stories typically analyzed and filtered through
various organization levels
Introduction to MSC
MSC process in three steps
1. Decide the types of stories to be collected
2. Gather stories from participants
3. Share stories with all stakeholders to learn
• A program adapts to different or changing
contexts, leading to differences in implementation
and outcomes
• Cause and effect are poorly
understood
MSC: Useful when…
• Multiple intervention
components exist
• Evaluation is focused on
learning, not just
accountability
• Flexible, captures broad range of results
• Addresses limitations of other, more
traditional methods
• Simple process
MSC: Strengths
• Goal-free, no
predetermined outcome
• Bias toward success stories
• Subjectivity in story selection
• Bias toward popular views
• Bias toward views of those who
are good storytellers
MSC: Limitations
MEASURE Evaluation’s applications of MSC
• Feed the Future’s Integrating Nutrition in Value
Chains (INVC) program in Malawi
• M&E systems strengthening in Côte d’Ivoire and
Nigeria
• USAID’s Gender, Policy, and Measurement
Program, in partnership with the Suaahara Project
and the Government of Nepal
• USAID’s Sustainable Comprehensive Responses
(SCORE) Project in Uganda
• PEPFAR’s Local Capacity Initiative (LCI) in Uganda
and Ghana
• Part of the evaluation of Feed the Future’s INVC
program
• Nutrition promoters trained on MSC in two districts
• 26 volunteer nutrition promoters collected 277 stories
from project beneficiary households
RESULTS:
• Stories were short, lacked details, and were repetitive
• Many of the stories covered multiple domains
• MSC was useful for understanding participants’
perceptions of the key benefits of INVC
Application of MSC in Malawi
• In both countries, researchers sought to determine
the most significant changes experienced by
stakeholders in the improvement of each country’s
national HIV M&E system.
• Using a facilitator well-trained in MSC, a
stakeholder workshop was held in each of the
countries to identify stories.
• Groups used an adapted self-assessment tool to
guide discussion, agree on a response, and then
provide sources of evidence for the responses.
• Researchers validated the stories and conducted
key informant interviews to obtain additional details.
Application of MSC in Côte d’Ivoire and
Nigeria
RESULTS:
• Engaging stakeholders in the MSC method helped
them identify important changes resulting from
interventions to strengthen M&E systems.
Application of MSC in Côte d’Ivoire and
Nigeria
• Researchers clearly heard
stakeholders’ voices and could
comprehensively and systematically
quantify improvements in each of
the 12 components of M&E
systems strengthening.
• Impact evaluation was conducted
to understand the contributions
of two approaches to strengthening
the capacity of Health Facility
Operation & Management
Committees (HFOMCs).
• MSC method was used for focus group discussions
with HFOMC members and community members to
see if significant changes had occurred in their lives.
• A local research organization collected stories from
community members and presented them to the
HFOMC members for group discussion.
Application of MSC in Nepal
Application of MSC in Nepal
RESULTS:
• Using the MSC approach with community members
was challenging because the intervention was
implemented by the same organization that
implemented another program and community
members confused the two.
• HFOMC members shared useful stories about how
the project helped them obtain information.
• The SCORE project aimed to benefit orphans and
vulnerable children and their caregivers.
• The program’s aims were to build
economic resilience, enhance food
security, improve child protection,
and increase access to education and
critical services.
• Forty randomly selected beneficiaries
and project staff were asked about
positive and negative changes resulting
from the program.
Application of MSC in Uganda
RESULTS:
• Because of time and resource constraints, the MSC
framework was used to develop open-ended
questions for the participants.
• The MSC method generated
rich and interesting responses.
Application of MSC in Uganda
• Analyzing the answers revealed
that the layered, multifaceted
components of the intervention
led to positive change,
particularly at the family level.
• The MSC approach is time-consuming and requires
careful facilitation.
• People collecting stories should
be well trained and supervised.
• The MSC approach is easy for
participants to understand.
• It is worthwhile to allow ample
time for the interviews—even if
this means gathering fewer stories.
Lessons learned across MSC studies
• Follow-up interviews with
beneficiaries, program
staff, or donors can further
strengthen the learning
aspect of this approach
and complement MSC
data.
Lessons learned across MSC studies
• The MSC technique is useful when evaluators
must narrow down the components of participant
observations of what changed.
• This method tends to have a bias for positive
responses.
• Research brief: Experiences and Lessons Learned:
Implementing the Most Significant Change Method
• Report: Integrating Nutrition in Value Chains in Malawi:
Using Most Significant Change stories to understand
community experiences
• Case study: A case study to measure national HIV M&E
systems strengthening: Nigeria
• Case study: A case study to measure national HIV M&E
systems strengthening: Côte d’Ivoire
• Report: Uganda’s SCORE program for vulnerable children
and their families: Mixed-methods performance evaluation
D4I resources on the MSC technique
This presentation was produced with the support of the United States
Agency for International Development (USAID) under the terms of the
Data for Impact (D4I) associate award 7200AA18LA00008, which is
implemented by the Carolina Population Center at the University of
North Carolina at Chapel Hill, in partnership with Palladium
International, LLC; ICF Macro, Inc.; John Snow, Inc.; and Tulane
University. The views expressed in this publication do not necessarily
reflect the views of USAID or the United States government.
www.data4impactproject.org

Weitere ähnliche Inhalte

Was ist angesagt?

Monitoring And Evaluation Presentation
Monitoring And Evaluation PresentationMonitoring And Evaluation Presentation
Monitoring And Evaluation PresentationEquiGov Institute
 
Rapid Assessment Process (1st Edition)
Rapid Assessment Process (1st Edition)Rapid Assessment Process (1st Edition)
Rapid Assessment Process (1st Edition)James Beebe
 
6 M&E - Monitoring and Evaluation of Aid Projects
6 M&E - Monitoring and Evaluation of Aid Projects6 M&E - Monitoring and Evaluation of Aid Projects
6 M&E - Monitoring and Evaluation of Aid ProjectsTony
 
Monitoring and Evaluation of Health Services
Monitoring and Evaluation of Health ServicesMonitoring and Evaluation of Health Services
Monitoring and Evaluation of Health ServicesNayyar Kazmi
 
Health Information System Measurement: Assessing, Monitoring, and Evaluating
Health Information System Measurement: Assessing, Monitoring, and EvaluatingHealth Information System Measurement: Assessing, Monitoring, and Evaluating
Health Information System Measurement: Assessing, Monitoring, and EvaluatingMEASURE Evaluation
 
Basics of Monitoring & Evaluation
Basics of Monitoring & EvaluationBasics of Monitoring & Evaluation
Basics of Monitoring & EvaluationFida Karim 🇵🇰
 
Integrating Gender in the M&E of Health Programs: A Toolkit
Integrating Gender in the M&E of Health Programs: A ToolkitIntegrating Gender in the M&E of Health Programs: A Toolkit
Integrating Gender in the M&E of Health Programs: A ToolkitMEASURE Evaluation
 
Introduction to monitoring and evaluation
Introduction to monitoring and evaluationIntroduction to monitoring and evaluation
Introduction to monitoring and evaluationMeshack Lomoywara
 
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...MEASURE Evaluation
 
Capacity Development For Monitoring And Evaluation
Capacity Development For Monitoring And EvaluationCapacity Development For Monitoring And Evaluation
Capacity Development For Monitoring And EvaluationKnowledge Management Center
 
Impact evaluation
Impact evaluationImpact evaluation
Impact evaluationCarlo Magno
 
M& e slide share
M& e   slide shareM& e   slide share
M& e slide shareSelf
 
Monitoring and evaluation (2)
Monitoring and evaluation (2)Monitoring and evaluation (2)
Monitoring and evaluation (2)Dr.RAJEEV KASHYAP
 
Advances in Outcome Monitoring
Advances in Outcome MonitoringAdvances in Outcome Monitoring
Advances in Outcome MonitoringMEASURE Evaluation
 
Monotoring and evaluation principles and theories
Monotoring and evaluation  principles and theoriesMonotoring and evaluation  principles and theories
Monotoring and evaluation principles and theoriescommochally
 

Was ist angesagt? (20)

Monitoring And Evaluation Presentation
Monitoring And Evaluation PresentationMonitoring And Evaluation Presentation
Monitoring And Evaluation Presentation
 
Rapid Assessment Process (1st Edition)
Rapid Assessment Process (1st Edition)Rapid Assessment Process (1st Edition)
Rapid Assessment Process (1st Edition)
 
6 M&E - Monitoring and Evaluation of Aid Projects
6 M&E - Monitoring and Evaluation of Aid Projects6 M&E - Monitoring and Evaluation of Aid Projects
6 M&E - Monitoring and Evaluation of Aid Projects
 
M&E Plan
M&E PlanM&E Plan
M&E Plan
 
Monitoring and Evaluation of Health Services
Monitoring and Evaluation of Health ServicesMonitoring and Evaluation of Health Services
Monitoring and Evaluation of Health Services
 
Health Information System Measurement: Assessing, Monitoring, and Evaluating
Health Information System Measurement: Assessing, Monitoring, and EvaluatingHealth Information System Measurement: Assessing, Monitoring, and Evaluating
Health Information System Measurement: Assessing, Monitoring, and Evaluating
 
Basics of Monitoring & Evaluation
Basics of Monitoring & EvaluationBasics of Monitoring & Evaluation
Basics of Monitoring & Evaluation
 
Integrating Gender in the M&E of Health Programs: A Toolkit
Integrating Gender in the M&E of Health Programs: A ToolkitIntegrating Gender in the M&E of Health Programs: A Toolkit
Integrating Gender in the M&E of Health Programs: A Toolkit
 
Data Demand and Use Workshop
Data Demand and Use WorkshopData Demand and Use Workshop
Data Demand and Use Workshop
 
Introduction to monitoring and evaluation
Introduction to monitoring and evaluationIntroduction to monitoring and evaluation
Introduction to monitoring and evaluation
 
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...
 
Capacity Development For Monitoring And Evaluation
Capacity Development For Monitoring And EvaluationCapacity Development For Monitoring And Evaluation
Capacity Development For Monitoring And Evaluation
 
Impact evaluation
Impact evaluationImpact evaluation
Impact evaluation
 
M& e slide share
M& e   slide shareM& e   slide share
M& e slide share
 
Monitoring and evaluation (2)
Monitoring and evaluation (2)Monitoring and evaluation (2)
Monitoring and evaluation (2)
 
Advances in Outcome Monitoring
Advances in Outcome MonitoringAdvances in Outcome Monitoring
Advances in Outcome Monitoring
 
Monotoring and evaluation principles and theories
Monotoring and evaluation  principles and theoriesMonotoring and evaluation  principles and theories
Monotoring and evaluation principles and theories
 
Monitoring & Evaluation Framework - Fiinovation
Monitoring & Evaluation Framework - FiinovationMonitoring & Evaluation Framework - Fiinovation
Monitoring & Evaluation Framework - Fiinovation
 
Methods of data collection
Methods of data collectionMethods of data collection
Methods of data collection
 
M&E.ppt
M&E.pptM&E.ppt
M&E.ppt
 

Ähnlich wie Lessons Learned In Using the Most Significant Change Technique in Evaluation

Evaluating a community media approach to promote nutrition in Niger
Evaluating a community media approach to promote nutrition in NigerEvaluating a community media approach to promote nutrition in Niger
Evaluating a community media approach to promote nutrition in NigerJSI
 
Sustaining quality approaches for locally embedded community health services ...
Sustaining quality approaches for locally embedded community health services ...Sustaining quality approaches for locally embedded community health services ...
Sustaining quality approaches for locally embedded community health services ...REACHOUTCONSORTIUMSLIDES
 
Quality improvement and Community Health Worker performance: A mixed method r...
Quality improvement and Community Health Worker performance: A mixed method r...Quality improvement and Community Health Worker performance: A mixed method r...
Quality improvement and Community Health Worker performance: A mixed method r...REACHOUTCONSORTIUMSLIDES
 
Improving Agricultural Productivity through Training; a Case of Heifer Projec...
Improving Agricultural Productivity through Training; a Case of Heifer Projec...Improving Agricultural Productivity through Training; a Case of Heifer Projec...
Improving Agricultural Productivity through Training; a Case of Heifer Projec...Waswa Balunywa
 
Day 2 panel 4 quality improvement for mnh tz 108020
Day 2 panel 4 quality improvement for mnh tz 108020Day 2 panel 4 quality improvement for mnh tz 108020
Day 2 panel 4 quality improvement for mnh tz 108020ea-imcha
 
ICCM_Sara Riese_10.16.13
ICCM_Sara Riese_10.16.13ICCM_Sara Riese_10.16.13
ICCM_Sara Riese_10.16.13CORE Group
 
Transitioning from reach every district to reach every community
Transitioning from reach every district to reach every communityTransitioning from reach every district to reach every community
Transitioning from reach every district to reach every communityJSI
 
Two Examples of Program Planning, Monitoring and Evaluation
Two Examples of Program Planning, Monitoring and EvaluationTwo Examples of Program Planning, Monitoring and Evaluation
Two Examples of Program Planning, Monitoring and EvaluationMEASURE Evaluation
 
Implementation of the British framework and tools for assessing the needs of ...
Implementation of the British framework and tools for assessing the needs of ...Implementation of the British framework and tools for assessing the needs of ...
Implementation of the British framework and tools for assessing the needs of ...BASPCAN
 
Cies 2015 global literacy & going to scale sc presentation 11march2015
Cies 2015 global literacy & going to scale sc presentation 11march2015Cies 2015 global literacy & going to scale sc presentation 11march2015
Cies 2015 global literacy & going to scale sc presentation 11march2015Cecilia Ochoa
 
Big lottery fund evidence briefing
Big lottery fund evidence briefing Big lottery fund evidence briefing
Big lottery fund evidence briefing Ecorys
 
Day 2 panel 4 replicating mama toto tz 108024
Day 2 panel 4 replicating mama toto tz 108024Day 2 panel 4 replicating mama toto tz 108024
Day 2 panel 4 replicating mama toto tz 108024ea-imcha
 
Using case-based methods to assess scalability and sustainability: Lessons fr...
Using case-based methods to assess scalability and sustainability: Lessons fr...Using case-based methods to assess scalability and sustainability: Lessons fr...
Using case-based methods to assess scalability and sustainability: Lessons fr...Barb Knittel
 
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...Measuring National M&E System Strengthening in Nigeria: Application of the Mo...
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...MEASURE Evaluation
 
Pathway to Practice: Incorporating Evidence into Military Family Services
Pathway to Practice: Incorporating Evidence into Military Family ServicesPathway to Practice: Incorporating Evidence into Military Family Services
Pathway to Practice: Incorporating Evidence into Military Family ServicesMFLNFamilyDevelopmnt
 
Day 2 panel 4 improving standards based management mw 108031
Day 2 panel 4 improving standards based management mw 108031Day 2 panel 4 improving standards based management mw 108031
Day 2 panel 4 improving standards based management mw 108031ea-imcha
 
The Current State of Play of Community Health Workers Training Programs in Su...
The Current State of Play of Community Health Workers Training Programs in Su...The Current State of Play of Community Health Workers Training Programs in Su...
The Current State of Play of Community Health Workers Training Programs in Su...germainsky
 
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...Office of Adolescent Health
 
Developing the India smallholder dairy value chain impact pathway(s)
Developing the India smallholder dairy value chain impact pathway(s) Developing the India smallholder dairy value chain impact pathway(s)
Developing the India smallholder dairy value chain impact pathway(s) ILRI
 

Ähnlich wie Lessons Learned In Using the Most Significant Change Technique in Evaluation (20)

Evaluating a community media approach to promote nutrition in Niger
Evaluating a community media approach to promote nutrition in NigerEvaluating a community media approach to promote nutrition in Niger
Evaluating a community media approach to promote nutrition in Niger
 
Sustaining quality approaches for locally embedded community health services ...
Sustaining quality approaches for locally embedded community health services ...Sustaining quality approaches for locally embedded community health services ...
Sustaining quality approaches for locally embedded community health services ...
 
Quality improvement and Community Health Worker performance: A mixed method r...
Quality improvement and Community Health Worker performance: A mixed method r...Quality improvement and Community Health Worker performance: A mixed method r...
Quality improvement and Community Health Worker performance: A mixed method r...
 
Improving Agricultural Productivity through Training; a Case of Heifer Projec...
Improving Agricultural Productivity through Training; a Case of Heifer Projec...Improving Agricultural Productivity through Training; a Case of Heifer Projec...
Improving Agricultural Productivity through Training; a Case of Heifer Projec...
 
Day 2 panel 4 quality improvement for mnh tz 108020
Day 2 panel 4 quality improvement for mnh tz 108020Day 2 panel 4 quality improvement for mnh tz 108020
Day 2 panel 4 quality improvement for mnh tz 108020
 
ICCM_Sara Riese_10.16.13
ICCM_Sara Riese_10.16.13ICCM_Sara Riese_10.16.13
ICCM_Sara Riese_10.16.13
 
Transitioning from reach every district to reach every community
Transitioning from reach every district to reach every communityTransitioning from reach every district to reach every community
Transitioning from reach every district to reach every community
 
Two Examples of Program Planning, Monitoring and Evaluation
Two Examples of Program Planning, Monitoring and EvaluationTwo Examples of Program Planning, Monitoring and Evaluation
Two Examples of Program Planning, Monitoring and Evaluation
 
Implementation of the British framework and tools for assessing the needs of ...
Implementation of the British framework and tools for assessing the needs of ...Implementation of the British framework and tools for assessing the needs of ...
Implementation of the British framework and tools for assessing the needs of ...
 
Cies 2015 global literacy & going to scale sc presentation 11march2015
Cies 2015 global literacy & going to scale sc presentation 11march2015Cies 2015 global literacy & going to scale sc presentation 11march2015
Cies 2015 global literacy & going to scale sc presentation 11march2015
 
Big lottery fund evidence briefing
Big lottery fund evidence briefing Big lottery fund evidence briefing
Big lottery fund evidence briefing
 
Day 2 panel 4 replicating mama toto tz 108024
Day 2 panel 4 replicating mama toto tz 108024Day 2 panel 4 replicating mama toto tz 108024
Day 2 panel 4 replicating mama toto tz 108024
 
Using case-based methods to assess scalability and sustainability: Lessons fr...
Using case-based methods to assess scalability and sustainability: Lessons fr...Using case-based methods to assess scalability and sustainability: Lessons fr...
Using case-based methods to assess scalability and sustainability: Lessons fr...
 
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...Measuring National M&E System Strengthening in Nigeria: Application of the Mo...
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...
 
Pathway to Practice: Incorporating Evidence into Military Family Services
Pathway to Practice: Incorporating Evidence into Military Family ServicesPathway to Practice: Incorporating Evidence into Military Family Services
Pathway to Practice: Incorporating Evidence into Military Family Services
 
Day 2 panel 4 improving standards based management mw 108031
Day 2 panel 4 improving standards based management mw 108031Day 2 panel 4 improving standards based management mw 108031
Day 2 panel 4 improving standards based management mw 108031
 
The Current State of Play of Community Health Workers Training Programs in Su...
The Current State of Play of Community Health Workers Training Programs in Su...The Current State of Play of Community Health Workers Training Programs in Su...
The Current State of Play of Community Health Workers Training Programs in Su...
 
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...
 
Developing the India smallholder dairy value chain impact pathway(s)
Developing the India smallholder dairy value chain impact pathway(s) Developing the India smallholder dairy value chain impact pathway(s)
Developing the India smallholder dairy value chain impact pathway(s)
 
Role of Evaluation in Decision Making
Role of Evaluation in Decision MakingRole of Evaluation in Decision Making
Role of Evaluation in Decision Making
 

Mehr von MEASURE Evaluation

Tuberculosis/HIV Mobility Study: Objectives and Background
Tuberculosis/HIV Mobility Study: Objectives and BackgroundTuberculosis/HIV Mobility Study: Objectives and Background
Tuberculosis/HIV Mobility Study: Objectives and BackgroundMEASURE Evaluation
 
How to improve the capabilities of health information systems to address emer...
How to improve the capabilities of health information systems to address emer...How to improve the capabilities of health information systems to address emer...
How to improve the capabilities of health information systems to address emer...MEASURE Evaluation
 
LCI Evaluation Uganda Organizational Network Analysis
LCI Evaluation Uganda Organizational Network AnalysisLCI Evaluation Uganda Organizational Network Analysis
LCI Evaluation Uganda Organizational Network AnalysisMEASURE Evaluation
 
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...MEASURE Evaluation
 
Understanding Referral Networks for Adolescent Girls and Young Women
Understanding Referral Networks for Adolescent Girls and Young WomenUnderstanding Referral Networks for Adolescent Girls and Young Women
Understanding Referral Networks for Adolescent Girls and Young WomenMEASURE Evaluation
 
Local Capacity Initiative (LCI) Evaluation
Local Capacity Initiative (LCI) EvaluationLocal Capacity Initiative (LCI) Evaluation
Local Capacity Initiative (LCI) EvaluationMEASURE Evaluation
 
Development and Validation of a Reproductive Empowerment Scale
Development and Validation of a Reproductive Empowerment ScaleDevelopment and Validation of a Reproductive Empowerment Scale
Development and Validation of a Reproductive Empowerment ScaleMEASURE Evaluation
 
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...MEASURE Evaluation
 
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...MEASURE Evaluation
 
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...MEASURE Evaluation
 
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...MEASURE Evaluation
 
Sustaining the Impact: MEASURE Evaluation Conversation on Health Informatics
Sustaining the Impact: MEASURE Evaluation Conversation on Health InformaticsSustaining the Impact: MEASURE Evaluation Conversation on Health Informatics
Sustaining the Impact: MEASURE Evaluation Conversation on Health InformaticsMEASURE Evaluation
 
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...MEASURE Evaluation
 
Sexual Orientation and Gender Identity Measures for Global Survey Research
Sexual Orientation and Gender Identity Measures for Global Survey ResearchSexual Orientation and Gender Identity Measures for Global Survey Research
Sexual Orientation and Gender Identity Measures for Global Survey ResearchMEASURE Evaluation
 
What’s Next? Practical Implementation Lessons from the Partnership for HIV-Fr...
What’s Next?Practical Implementation Lessons from the Partnership for HIV-Fr...What’s Next?Practical Implementation Lessons from the Partnership for HIV-Fr...
What’s Next? Practical Implementation Lessons from the Partnership for HIV-Fr...MEASURE Evaluation
 
Measuring Outcomes for Vulnerable Children: A Global Snapshot
Measuring Outcomes for Vulnerable Children: A Global SnapshotMeasuring Outcomes for Vulnerable Children: A Global Snapshot
Measuring Outcomes for Vulnerable Children: A Global SnapshotMEASURE Evaluation
 
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...MEASURE Evaluation
 
Les dialogues communautaires pour diffuser des résultats de recherche Example...
Les dialogues communautaires pour diffuser des résultats de recherche Example...Les dialogues communautaires pour diffuser des résultats de recherche Example...
Les dialogues communautaires pour diffuser des résultats de recherche Example...MEASURE Evaluation
 
Seven Steps to EnGendering Evaluations of Public Health Programs
 Seven Steps to EnGendering Evaluations of Public Health Programs Seven Steps to EnGendering Evaluations of Public Health Programs
Seven Steps to EnGendering Evaluations of Public Health ProgramsMEASURE Evaluation
 
HIV Risk and Service Use: Results of a Survey of Men in Port-au-Prince and St...
HIV Risk and Service Use: Results of a Survey of Men in Port-au-Prince and St...HIV Risk and Service Use: Results of a Survey of Men in Port-au-Prince and St...
HIV Risk and Service Use: Results of a Survey of Men in Port-au-Prince and St...MEASURE Evaluation
 

Mehr von MEASURE Evaluation (20)

Tuberculosis/HIV Mobility Study: Objectives and Background
Tuberculosis/HIV Mobility Study: Objectives and BackgroundTuberculosis/HIV Mobility Study: Objectives and Background
Tuberculosis/HIV Mobility Study: Objectives and Background
 
How to improve the capabilities of health information systems to address emer...
How to improve the capabilities of health information systems to address emer...How to improve the capabilities of health information systems to address emer...
How to improve the capabilities of health information systems to address emer...
 
LCI Evaluation Uganda Organizational Network Analysis
LCI Evaluation Uganda Organizational Network AnalysisLCI Evaluation Uganda Organizational Network Analysis
LCI Evaluation Uganda Organizational Network Analysis
 
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...
 
Understanding Referral Networks for Adolescent Girls and Young Women
Understanding Referral Networks for Adolescent Girls and Young WomenUnderstanding Referral Networks for Adolescent Girls and Young Women
Understanding Referral Networks for Adolescent Girls and Young Women
 
Local Capacity Initiative (LCI) Evaluation
Local Capacity Initiative (LCI) EvaluationLocal Capacity Initiative (LCI) Evaluation
Local Capacity Initiative (LCI) Evaluation
 
Development and Validation of a Reproductive Empowerment Scale
Development and Validation of a Reproductive Empowerment ScaleDevelopment and Validation of a Reproductive Empowerment Scale
Development and Validation of a Reproductive Empowerment Scale
 
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...
 
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...
 
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...
 
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...
 
Sustaining the Impact: MEASURE Evaluation Conversation on Health Informatics
Sustaining the Impact: MEASURE Evaluation Conversation on Health InformaticsSustaining the Impact: MEASURE Evaluation Conversation on Health Informatics
Sustaining the Impact: MEASURE Evaluation Conversation on Health Informatics
 
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...
 
Sexual Orientation and Gender Identity Measures for Global Survey Research
Sexual Orientation and Gender Identity Measures for Global Survey ResearchSexual Orientation and Gender Identity Measures for Global Survey Research
Sexual Orientation and Gender Identity Measures for Global Survey Research
 
What’s Next? Practical Implementation Lessons from the Partnership for HIV-Fr...
What’s Next?Practical Implementation Lessons from the Partnership for HIV-Fr...What’s Next?Practical Implementation Lessons from the Partnership for HIV-Fr...
What’s Next? Practical Implementation Lessons from the Partnership for HIV-Fr...
 
Measuring Outcomes for Vulnerable Children: A Global Snapshot
Measuring Outcomes for Vulnerable Children: A Global SnapshotMeasuring Outcomes for Vulnerable Children: A Global Snapshot
Measuring Outcomes for Vulnerable Children: A Global Snapshot
 
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...
 
Les dialogues communautaires pour diffuser des résultats de recherche Example...
Les dialogues communautaires pour diffuser des résultats de recherche Example...Les dialogues communautaires pour diffuser des résultats de recherche Example...
Les dialogues communautaires pour diffuser des résultats de recherche Example...
 
Seven Steps to EnGendering Evaluations of Public Health Programs
 Seven Steps to EnGendering Evaluations of Public Health Programs Seven Steps to EnGendering Evaluations of Public Health Programs
Seven Steps to EnGendering Evaluations of Public Health Programs
 
HIV Risk and Service Use: Results of a Survey of Men in Port-au-Prince and St...
HIV Risk and Service Use: Results of a Survey of Men in Port-au-Prince and St...HIV Risk and Service Use: Results of a Survey of Men in Port-au-Prince and St...
HIV Risk and Service Use: Results of a Survey of Men in Port-au-Prince and St...
 

Kürzlich hochgeladen

SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA .pdf
SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA    .pdfSGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA    .pdf
SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA .pdfHongBiThi1
 
Different drug regularity bodies in different countries.
Different drug regularity bodies in different countries.Different drug regularity bodies in different countries.
Different drug regularity bodies in different countries.kishan singh tomar
 
Basic structure of hair and hair growth cycle.pptx
Basic structure of hair and hair growth cycle.pptxBasic structure of hair and hair growth cycle.pptx
Basic structure of hair and hair growth cycle.pptxkomalt2001
 
Female Reproductive Physiology Before Pregnancy
Female Reproductive Physiology Before PregnancyFemale Reproductive Physiology Before Pregnancy
Female Reproductive Physiology Before PregnancyMedicoseAcademics
 
ayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologypptayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologypptPradnya Wadekar
 
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdf
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdfPAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdf
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdfDolisha Warbi
 
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdfSGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdfHongBiThi1
 
Physiology of Smooth Muscles -Mechanics of contraction and relaxation
Physiology of Smooth Muscles -Mechanics of contraction and relaxationPhysiology of Smooth Muscles -Mechanics of contraction and relaxation
Physiology of Smooth Muscles -Mechanics of contraction and relaxationMedicoseAcademics
 
Role of Soap based and synthetic or syndets bar
Role of  Soap based and synthetic or syndets barRole of  Soap based and synthetic or syndets bar
Role of Soap based and synthetic or syndets barmohitRahangdale
 
Mental health Team. Dr Senthil Thirusangu
Mental health Team. Dr Senthil ThirusanguMental health Team. Dr Senthil Thirusangu
Mental health Team. Dr Senthil Thirusangu Medical University
 
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptx
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptxDNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptx
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptxMAsifAhmad
 
power point presentation of Clinical evaluation of strabismus
power point presentation of Clinical evaluation  of strabismuspower point presentation of Clinical evaluation  of strabismus
power point presentation of Clinical evaluation of strabismusChandrasekar Reddy
 
High-Performance Thin-Layer Chromatography (HPTLC)
High-Performance Thin-Layer Chromatography (HPTLC)High-Performance Thin-Layer Chromatography (HPTLC)
High-Performance Thin-Layer Chromatography (HPTLC)kishan singh tomar
 
Neurological history taking (2024) .
Neurological  history  taking  (2024)  .Neurological  history  taking  (2024)  .
Neurological history taking (2024) .Mohamed Rizk Khodair
 
Adenomyosis or Fibroid- making right diagnosis
Adenomyosis or Fibroid- making right diagnosisAdenomyosis or Fibroid- making right diagnosis
Adenomyosis or Fibroid- making right diagnosisSujoy Dasgupta
 
SGK ĐIỆN GIẬT ĐHYHN RẤT LÀ HAY TUYỆT VỜI.pdf
SGK ĐIỆN GIẬT ĐHYHN        RẤT LÀ HAY TUYỆT VỜI.pdfSGK ĐIỆN GIẬT ĐHYHN        RẤT LÀ HAY TUYỆT VỜI.pdf
SGK ĐIỆN GIẬT ĐHYHN RẤT LÀ HAY TUYỆT VỜI.pdfHongBiThi1
 
"Radical excision of DIE in subferile women with deep infiltrating endometrio...
"Radical excision of DIE in subferile women with deep infiltrating endometrio..."Radical excision of DIE in subferile women with deep infiltrating endometrio...
"Radical excision of DIE in subferile women with deep infiltrating endometrio...Sujoy Dasgupta
 
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.Bulimia nervosa ( Eating Disorders) Mental Health Nursing.
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.aarjukhadka22
 

Kürzlich hochgeladen (20)

American College of physicians ACP high value care recommendations in rheumat...
American College of physicians ACP high value care recommendations in rheumat...American College of physicians ACP high value care recommendations in rheumat...
American College of physicians ACP high value care recommendations in rheumat...
 
SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA .pdf
SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA    .pdfSGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA    .pdf
SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA .pdf
 
Different drug regularity bodies in different countries.
Different drug regularity bodies in different countries.Different drug regularity bodies in different countries.
Different drug regularity bodies in different countries.
 
Basic structure of hair and hair growth cycle.pptx
Basic structure of hair and hair growth cycle.pptxBasic structure of hair and hair growth cycle.pptx
Basic structure of hair and hair growth cycle.pptx
 
Female Reproductive Physiology Before Pregnancy
Female Reproductive Physiology Before PregnancyFemale Reproductive Physiology Before Pregnancy
Female Reproductive Physiology Before Pregnancy
 
ayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologypptayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologyppt
 
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdf
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdfPAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdf
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdf
 
Rheumatoid arthritis Part 1, case based approach with application of the late...
Rheumatoid arthritis Part 1, case based approach with application of the late...Rheumatoid arthritis Part 1, case based approach with application of the late...
Rheumatoid arthritis Part 1, case based approach with application of the late...
 
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdfSGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
 
Physiology of Smooth Muscles -Mechanics of contraction and relaxation
Physiology of Smooth Muscles -Mechanics of contraction and relaxationPhysiology of Smooth Muscles -Mechanics of contraction and relaxation
Physiology of Smooth Muscles -Mechanics of contraction and relaxation
 
Role of Soap based and synthetic or syndets bar
Role of  Soap based and synthetic or syndets barRole of  Soap based and synthetic or syndets bar
Role of Soap based and synthetic or syndets bar
 
Mental health Team. Dr Senthil Thirusangu
Mental health Team. Dr Senthil ThirusanguMental health Team. Dr Senthil Thirusangu
Mental health Team. Dr Senthil Thirusangu
 
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptx
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptxDNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptx
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptx
 
power point presentation of Clinical evaluation of strabismus
power point presentation of Clinical evaluation  of strabismuspower point presentation of Clinical evaluation  of strabismus
power point presentation of Clinical evaluation of strabismus
 
High-Performance Thin-Layer Chromatography (HPTLC)
High-Performance Thin-Layer Chromatography (HPTLC)High-Performance Thin-Layer Chromatography (HPTLC)
High-Performance Thin-Layer Chromatography (HPTLC)
 
Neurological history taking (2024) .
Neurological  history  taking  (2024)  .Neurological  history  taking  (2024)  .
Neurological history taking (2024) .
 
Adenomyosis or Fibroid- making right diagnosis
Adenomyosis or Fibroid- making right diagnosisAdenomyosis or Fibroid- making right diagnosis
Adenomyosis or Fibroid- making right diagnosis
 
SGK ĐIỆN GIẬT ĐHYHN RẤT LÀ HAY TUYỆT VỜI.pdf
SGK ĐIỆN GIẬT ĐHYHN        RẤT LÀ HAY TUYỆT VỜI.pdfSGK ĐIỆN GIẬT ĐHYHN        RẤT LÀ HAY TUYỆT VỜI.pdf
SGK ĐIỆN GIẬT ĐHYHN RẤT LÀ HAY TUYỆT VỜI.pdf
 
"Radical excision of DIE in subferile women with deep infiltrating endometrio...
"Radical excision of DIE in subferile women with deep infiltrating endometrio..."Radical excision of DIE in subferile women with deep infiltrating endometrio...
"Radical excision of DIE in subferile women with deep infiltrating endometrio...
 
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.Bulimia nervosa ( Eating Disorders) Mental Health Nursing.
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.
 

Lessons Learned In Using the Most Significant Change Technique in Evaluation

  • 1. Lessons Learned In Using the Most Significant Change Technique in Evaluation Mary Freyder, MPH, LMSW Bridgit Adamou, MPH January 8, 2020
  • 2. • Data for Impact (D4I) overview • Sharing lessons from the field – Most Significant Change (MSC) in five countries under MEASURE Evaluation Project • Deep dive - evaluating the Local Capacity Initiative (LCI) in Uganda • Questions and answers Presentation Overview
  • 3. • Generate evidence • Strengthen capacity • Ensure data quality • Integrate gender • Promote data use • Learn D4I works to:
  • 4. • Participatory approach • Participants give personal stories of significant change, directly or indirectly related to intervention • Stories typically analyzed and filtered through various organization levels Introduction to MSC
  • 5. MSC process in three steps 1. Decide the types of stories to be collected 2. Gather stories from participants 3. Share stories with all stakeholders to learn
  • 6. • A program adapts to different or changing contexts, leading to differences in implementation and outcomes • Cause and effect are poorly understood MSC: Useful when… • Multiple intervention components exist • Evaluation is focused on learning, not just accountability
  • 7. • Flexible, captures broad range of results • Addresses limitations of other, more traditional methods • Simple process MSC: Strengths • Goal-free, no predetermined outcome
  • 8. • Bias toward success stories • Subjectivity in story selection • Bias toward popular views • Bias toward views of those who are good storytellers MSC: Limitations
  • 9. MEASURE Evaluation’s applications of MSC • Feed the Future’s Integrating Nutrition in Value Chains (INVC) program in Malawi • M&E systems strengthening in Côte d’Ivoire and Nigeria • USAID’s Gender, Policy, and Measurement Program, in partnership with the Suaahara Project and the Government of Nepal • USAID’s Sustainable Comprehensive Responses (SCORE) Project in Uganda • PEPFAR’s Local Capacity Initiative (LCI) in Uganda and Ghana
  • 10. • Part of the evaluation of Feed the Future’s INVC program • Nutrition promoters trained on MSC in two districts • 26 volunteer nutrition promoters collected 277 stories from project beneficiary households RESULTS: • Stories were short, lacked details, and were repetitive • Many of the stories covered multiple domains • MSC was useful for understanding participants’ perceptions of the key benefits of INVC Application of MSC in Malawi
  • 11. • In both countries, researchers sought to determine the most significant changes experienced by stakeholders in the improvement of each country’s national HIV M&E system. • Using a facilitator well-trained in MSC, a stakeholder workshop was held in each of the countries to identify stories. • Groups used an adapted self-assessment tool to guide discussion, agree on a response, and then provide sources of evidence for the responses. • Researchers validated the stories and conducted key informant interviews to obtain additional details. Application of MSC in Côte d’Ivoire and Nigeria
  • 12. RESULTS: • Engaging stakeholders in the MSC method helped them identify important changes resulting from interventions to strengthen M&E systems. Application of MSC in Côte d’Ivoire and Nigeria • Researchers clearly heard stakeholders’ voices and could comprehensively and systematically quantify improvements in each of the 12 components of M&E systems strengthening.
  • 13. • Impact evaluation was conducted to understand the contributions of two approaches to strengthening the capacity of Health Facility Operation & Management Committees (HFOMCs). • MSC method was used for focus group discussions with HFOMC members and community members to see if significant changes had occurred in their lives. • A local research organization collected stories from community members and presented them to the HFOMC members for group discussion. Application of MSC in Nepal
  • 14. Application of MSC in Nepal RESULTS: • Using the MSC approach with community members was challenging because the intervention was implemented by the same organization that implemented another program and community members confused the two. • HFOMC members shared useful stories about how the project helped them obtain information.
  • 15. • The SCORE project aimed to benefit orphans and vulnerable children and their caregivers. • The program’s aims were to build economic resilience, enhance food security, improve child protection, and increase access to education and critical services. • Forty randomly selected beneficiaries and project staff were asked about positive and negative changes resulting from the program. Application of MSC in Uganda
  • 16. RESULTS: • Because of time and resource constraints, the MSC framework was used to develop open-ended questions for the participants. • The MSC method generated rich and interesting responses. Application of MSC in Uganda • Analyzing the answers revealed that the layered, multifaceted components of the intervention led to positive change, particularly at the family level.
  • 17. • The MSC approach is time-consuming and requires careful facilitation. • People collecting stories should be well trained and supervised. • The MSC approach is easy for participants to understand. • It is worthwhile to allow ample time for the interviews—even if this means gathering fewer stories. Lessons learned across MSC studies
  • 18. • Follow-up interviews with beneficiaries, program staff, or donors can further strengthen the learning aspect of this approach and complement MSC data. Lessons learned across MSC studies • The MSC technique is useful when evaluators must narrow down the components of participant observations of what changed. • This method tends to have a bias for positive responses.
  • 19. • Research brief: Experiences and Lessons Learned: Implementing the Most Significant Change Method • Report: Integrating Nutrition in Value Chains in Malawi: Using Most Significant Change stories to understand community experiences • Case study: A case study to measure national HIV M&E systems strengthening: Nigeria • Case study: A case study to measure national HIV M&E systems strengthening: Côte d’Ivoire • Report: Uganda’s SCORE program for vulnerable children and their families: Mixed-methods performance evaluation D4I resources on the MSC technique
  • 20. This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of the Data for Impact (D4I) associate award 7200AA18LA00008, which is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Palladium International, LLC; ICF Macro, Inc.; John Snow, Inc.; and Tulane University. The views expressed in this publication do not necessarily reflect the views of USAID or the United States government. www.data4impactproject.org

Hinweis der Redaktion

  1. D4I, which is a MEASURE Evaluation associate award and is funded by United States Agency for International Development, works with countries to: Generate strong evidence for program and policy decision making Build individual and organizational capacity Enhance the use of data for global health programs and policies
  2. MEASURE Evaluation has implemented the MSC technique in evaluations of health programs. This presentation summarizes our experience and shares lessons learned in the application of MSC. But first, what is MSC? It is a participatory qualitative approach to monitoring and evaluation that involves assessing the changes and impacts as the result of a program from the perspective of program participants or beneficiaries. It is participatory because program participants themselves and stakeholders are usually involved in deciding types of change to explore, and in analyzing the data, or stories, collected. Project staff collect stories from beneficiaries, partners, or participants. Story collectors ask questions such as the following: “During the last period, in your opinion, what was the most significant change that took place for participants in the project?” Respondents describe both the change and the reasons they consider it significant. The stories are then analyzed and filtered through different levels. “Typically” is in italics here because MSC has been implemented in many different ways, ranging from the original version of it, with multiple levels of story review/analysis and voting, to just asking individuals for their stories and researchers analysing those. You will see this illustrated through the MEASURE Evaluation examples coming up.
  3. The process entails three basic steps: 1. Deciding the types of stories that should be collected from beneficiaries, partners, or participants 2. Gathering stories by asking respondents to describe both the change they experienced and the reasons they consider it significant, then collectively determining which stories are the most significant 3. Sharing the stories with stakeholders and contributors to learn about what is valued
  4. MSC is useful when you are evaluating a program that adapts to different or evolving contexts, leading to differences in implementation and outcomes. When cause and effect is not understood well. When there are multiple program components And when your evaluation is learning focused rather than solely focused on accountability.
  5. MSC is flexible: open to broader range of results, including those that may not have originally been intended by the program. Complementary to more traditional methods: can help address limitations of other methods. For example, if you are using a structured survey only, you may miss unanticipated changes, negative or positive. Simple, straightforward process that anybody can understand and use. Compared to other monitoring approaches, it is easy to communicate across cultures. There aren’t any indicators, so there is no need to explain what an indicator is. Everyone can tell stories about events they think were important. Goal-free: no assumption of what success should look like. Because it is not a linear model, there are no predetermined outcomes.
  6. This method is biased towards successes – particularly to make the storyteller look good. Evaluators can overcome this by having a domain of change for negative stories. There is subjectivity in story selection. Recording the reasons for selecting stories helps strengthen the process There are biases toward popular views. And there are also biases toward views of those who are good storytellers. This is why don’t use as stand-alone tool.
  7. MEASURE Evaluation applied most significant change in five different settings to monitor and evaluate various community based or complex programs. One was a nutrition program in Malawi. Another looked at how national government HIV M&E systems in Cote d’Ivoire and Nigeria were strengthened. In Nepal, MSC was used as part of an impact evaluation involving health facility operation and management committees. The Sustainable Comprehensive Responses, or SCORE, program in Uganda aimed to benefit orphans and vulnerable children and their caregivers. And then lastly there was the PEPFAR-supported global initiative to support local partners in HIV policy advocacy for highly vulnerable populations such as men who have sex with men, transgender women and sex workers. My colleague, Mary, will discuss this application more in-depth.
  8. MSC was intended to be one aspect of qualitative data collection for a large impact evaluation of Feed the Future’s Integrating Nutrition in Value Chains program. Volunteer nutrition promoters were trained on MSC in Malawi’s Mchinji and Lilongwe districts to obtain information from participants about their experiences with the INVC program. Over the course of a year, 26 nutrition promoters, who were also program implementers, collected 277 MSC stories from project beneficiary households. Outcomes: Because the nutrition promoters wrote the stories in their notebooks, summarizing them rather than writing them verbatim, the stories tended to be very short, had few details, and were repetitive. The similarities among stories made it difficult for the promoters to discuss, rank, and select the most significant stories. Many of the stories covered more than one domain of change, which also made it challenging for promoters to assign the stories to the domains. The MSC method was useful for understanding participants’ perceptions of the key benefits of and challenges to the INVC program. (Due to the cancellation of the impact evaluation, no other qualitative data was collected to supplement the MSC stories.)
  9. In two case studies, researchers sought to produce evidence of how Côte d’Ivoire and Nigeria’s national HIV M&E systems were strengthened. The MSC technique was applied in both countries to determine the MSCs experienced by stakeholders in the improvement of each country’s national HIV M&E system. With the support of a facilitator well-trained in MSC, a stakeholder workshop was held in each of the countries to identify stories. Participants started by self-selecting into groups. Next, each group used an adapted self-assessment tool to guide discussion, agree on a response, and then provide sources of evidence for the responses. Groups reported to the plenary to verify each other’s findings. Afterwards the researchers validated the stories and conducted key informant interviews to obtain additional details and perspectives on the MSCs that were identified in the workshop.
  10. Researchers found that engaging stakeholders in the MSC method helped them identify the important changes resulting from interventions to strengthen M&E systems, which also helped to raise interest in these interventions and the defined domains of change. Using the MSC method also meant that researchers clearly heard stakeholders’ voices and could comprehensively and systematically quantify improvements or successes in each of the 12 components of M&E systems strengthening.
  11. MEASURE Evaluation conducted an impact evaluation to understand the contributions of two approaches to strengthening the capacity of the Health Facility Operation and Management Committees (HFOMCs) in Nepal. At the end line of the evaluation, we used the MSC method to conduct eight focus group discussions with HFOMC members and community members to see if significant changes had occurred in their lives as a direct result of their participation in the program or as a result of the program’s interventions overall. The plan was to apply the MSC method in two ways: (1) HFOMC members were to engage in focus group discussions about changes they experienced through program participation and (2) HFOMC members were to collect stories of change from the community for the HFOMC to discuss and vote on. The latter story collection did not happen as planned, however, because of HFOMC members’ limited time and limited skills in facilitating discussions, among other reasons. Instead, a local research organization collected the stories from community members and presented them to the HFOMC members for group discussion.
  12. Using the MSC approach with community members proved challenging because the intervention was implemented by the same organization that implemented another program and community members confused the two. However, HFOMC members shared useful stories about how the project helped them obtain information.
  13. USAID’s Sustainable, Comprehensive Responses (SCORE) project operated in Uganda to benefit orphans and vulnerable children and their caregivers. The program’s aims were to build economic resilience, enhance food security, improve child protection, and increase access to education and critical services. MEASURE Evaluation used the MSC method to assess the project’s effects on beneficiaries and the strengths and challenges associated with implementation. Using the MSC techniquie, 40 randomly selected beneficiaries and project staff about positive and negative changes resulting from the program.
  14. Time and resource constraints prevented respondents from gathering to discuss and vote on the changes that had the most impact. Instead, we used the MSC framework to develop open-ended questions for the participants, which generated rich and interesting responses. Analysis of these answers revealed that the layered, multifaceted components of the intervention led to positive change, particularly at the family level.
  15. Before we dive deep into the LCI Uganda evaluation, I’ll share a few lessons learned across all four studies. Mary will highlight a few of these in her presentation. The MSC approach is time-consuming and requires careful facilitation. If people capturing stories lack sufficient time and skills, it is better to select others better suited to use it correctly, or plan to spend significant time training data collectors to record stories—or avoid the method altogether. People collecting stories should be well trained and supervised so the stories are thoroughly captured with detailed notes. The MSC approach is easy for participants to understand. It is worthwhile to allow ample time for the interviews—even if this means gathering fewer stories. It’s important that data collectors can fully explain the questions and that participants have enough time to think about their experiences and articulate their responses.
  16. Follow-up interviews with beneficiaries, program staff, or donors can further strengthen the learning aspect of this approach and complement the MSC data. The MSC technique is useful when evaluators must systematically narrow down the components of participant observations of what changed. As mentioned earlier, this method tends to have a bias for positive responses. Specifically asking a question about most significant negative change can address this potential drawback.