Design, Monitoring and Evaluation

Making Evaluations Matter - A Practical Guide for Evaluators

2011-10-27T17:19:00.002+04:00

Recently, the Centre for Development Innovation, Wageningen University & Research Centre, Wageningen, The Netherlands, published Making Evaluations Matter: A Practical Guide for Evaluators written by Cecile Kusters with Simone van Vugt, Seerp Wigboldus, Bob Williams and Jim Woodhill.

This guide emphasizes participatory evaluation and draws heavily upon the work of Michael Quinn Patton, especially from Utilization-Focused Evaluation.

I think this is a very handy guide NOT ONLY for evaluators, but also a handy guide for country directors, project managers, and project directors to read PRIOR to implementing a project as well as toward the end of a project when planning an evaluation.

The contents are the following:

1. Core Principles for guiding evaluations that matter.

2. Suggested steps for designing and facilitating evaluations that matter.

3. Getting stakeholders to contribute successfully.

4. Turning evaluation into a learning process.

5. Thinking through the possible influences and consequences of evaluation on change processes.
6. Conclusion

Annex A: Examples of (Learning) Purposes, Assessment Questions, Users, and Uses of an Evaluation for a Food Security Initiative.

Annex B: Contrasts between traditional evaluation and complexity-sensitive developmental evaluation.

Self-administered questionnaires

2011-10-19T18:19:00.004+04:00

In survey research, especially when questions related to sensitive topics are being asked, there are debates between which form of questionnaire administration is best: a) interviewer-administration or b) self-administration. More often that not, questionnaires are administered by a trained interviewer; however, there are times that some people feel its best that the respondent completes the questionnaire without the assistance of an interviewer (self-administered).

Currently, I'm dealing with survey data from a youth study that used a self-administered questionnaire and the data contain many "missing" cases, nonsensical responses, and numerous cases of Errors of Commission and Errors of Omission. An Error of Commission is one where the person responds where they should not and an Error of Omission is one where the person fails to respond when they should.

A questionnaire designed for an interviewer administered survey cannot be used for a self-administered survey! Interviewers are trained in understanding the questions and how to navigate through the questionnaire; however, a questionnaire that is designed for someone who has never seen it before and for them to understand the questions as well as navigate through the questionnaire requires special attention to many factors. Using pg.6 from the 2008 National Survey of College Graduates, conducted by the US Census Bureau to illustrate, some critical factors to consider for a self-administered questionnaire are:

Language - the instructions and questions need to be written in a vocabulary that is slightly lower than lowest education level of any respondent.
Section Heading - every section/topic needs a heading that is short, in bold font, slightly different color than the rest of the questionnaire, such as Part B - Past Employment.
Question Numbering - question numbers should carry the section lettering/numbering as well as the question number and should be in a slightly larger font than the question text and in bold font, such as B1.
Verbal navigation - instructions next to certain responses that tell the respondent where to clearly go next. In the example above, if the respondent answers "No" in question B1 there is a verbal instruction, in bold font, telling them both 1) the page and 2) the question # to go to.
Symbol navigation - these are generally arrows showing a respondent where to go next if they answer a certain response. Above, if a respondent answers "Yes" in question B1 the arrow shows them to go to question B2.
Adequate spacing - all to often to save printing costs, a questionnaire is too cluttered but generally this is ok for a trained interviewer but not for self-administration. A self-administered questionnaire should have adequate spacing between questions to reduce eye fatigue and confusion.
Coloring - if posible, use slightly different grays or colors to highlight different sections and responses, such as in the example above.

Youth Conflict & Tolerance Survey

2011-10-15T22:31:00.001+04:00

Conflict and tolerance are relevant issues for western (Republic of) Georgia, which has experienced civil war in the 1990s and the recent 2008 war with Russia. Save the Children, with funding from the EU, conducted formative research among youth in regions on both sides of the de facto administrative border between Samegrelo and Gali regions.

In cooperation with several colleagues, we used formative research (Focus group discussion, in-depth interviews, and key informant interviews), to develop a Youth Conflict and Tolerance Survey (YCTS) tool for western Georgia. The YCTS tool focuses on conflict and tolerance issues youth confront in three context settings: the home, at school and in the community. It is a 185 item survey instrument divided into 6 sections: 1) Respondent characteristics, 2) Knowledge of conflict resolution skills and attitudes, 3) Types of conflict in the home, 4) Types of conflict at school/education facility, 5) Types of conflict in the community, and 6) General attitudes. The YCTS underwent a time-stability (reliability) test which showed that the majority of the questions in each section are reasonably stable.

The YCTS tool is designed to be completed either via self-administration (where the youth reads and answers the questions on their own) or via oral administration (where a youth worker or teacher reads each question and the youth answers each question on their own). The YCTS tool is designed for program development (general assessment) and program evaluation (change resulting from program interventions). Change over time is tracked through the administration of the YCTS tool on at least two separate occasions (Time One and Time Two) -- using the same survey methodology, the same instrument with the same learners.

In addition, a YCTS handbook was developed to provide a simple, step-by-step guide to effectivelyadminister the Youth Conflict & Tolerance Survey (YCTS) Tool. Download the TCTS Handbook and please send me your comments, suggestions, and critiques.

YCTS Handbook

NGO Network Analysis Handbook

2011-10-10T13:39:00.002+04:00

Today, there is the UN NGO Network, NGO Global Network, International NGO Network, Voluntary Action Network and many other type of sectoral networks such as the Child Rights Network, Human Rights Network, and Environmental Protection Network.

However, more often than not, the term "network" is used as a metaphor and rarely is there much effort to actually measure and demonstrate if there is truly a network; that is, an interconnected group or system.

Since January 2011, I worked with Lilly Saganelidze and Tamuna Dagargulia to study, measure and map the network of youth-focused NGOs in western Georgia. The outcome of this study is: NGO Network Analysis Handbook: how to measure and map linkages between NGOs.

Please feel free to use download and use this handbook and please send us an email if you have suggestions in how to improve it.

GPS and Google Earth/Maps

2011-06-29T15:38:00.003+04:00

Two new handbooks are available for download, one in English and one in Russian, titled:

Community Based Disaster Risk Reduction Mapping Using Google Earth & GPS: A Handbook

This handbook, produced by Save the Children, to aid humanitarian organizations, local governments and central governments to use mapping methodologies with Google Earth virtual globe, map and geographic

information programs in disaster risk reduction. Using affordable GPS (global positioning system) units, and free Google Earth/Maps, local communities can produce highly accurate maps that can be easily understood by wider communities, and provide a useful picture for targeting disaster preparedness, prevention, and mitigation resources.

Not only can GPS units and Google Earth/Maps facilitate community disaster risk reduction, but in Jordan these are being used to map child labor locations as well as child labor services.

These handbooks are available in the DME Documents section to the left.

Catalog of Survey Quesitions

2011-06-15T21:02:00.000+04:00

How often have you had to develop a questionnaire for a needs assessment, baseline or end-line survey, and not only for one project, but several that address different issues? At times like these it would be nice to have a catalog of questions to review to see which ones may be appropriate and relevant or which ones with some adaptation could be appropriate and relevant.

The International Household Survey Network has now developed an easily accessible online Catalog of Survey Questions. Currently, it has now about 2500 questions that have been used in about 1300 different surveys. You can filter types of surveys and questions by country, year (1950-2011) and type. There are four types of surveys: 1) demographic and health survey, 2) income and expenditure, 3) multiple indicator cluster survey, and 4) population and housing census.

I think this is a good start, however, several of the links do not work. For example, since I'm involved in a child labor project in Amman, Jordan, I thought I would download the Child Labor Study questionnaire used in West Bank/Gaza. But, the link does not work. I have sent an email to IHSN and I'll let you know if they respond. Nonetheless, many of the links to the questionnaires do work.

Online Data and Statistics

2011-06-12T08:37:00.000+04:00

Data Market is a website that provides easy access to 13k data sets, 100 million time series, and 600 million facts! DataMarket is a type of "one stop shopping" to obtain data for countries ranging from Afghanistan to Zimbabwe and issue ranging from agriculture to youth.

Since I work in the Middle East, and with the recent "Arab Spring" I decided to graph internet users in the countries of Egypt, Jordan, Lebanon, Morocco, Tunisia, Palestine and Yemen. The graph below was produced and DataMarket provides various formats to export the data and graphs.

As the graph illustrates, since 2001 there was a dramatic increase in the number of internet users in Egypt, followed by Morocco and Tunisia with much lower increases in internet users were in Lebanon, Jordan and no real increase in Libya or Yemen.

So, give DataMarket a try and see if it can help you quickly get the data you need and illustrate it.

Monitoring & Evaluating a Project Related Website

2011-06-09T20:53:00.002+04:00

Increasingly, websites are being developed as one part of the projects/programs' outreach to their beneficiaries, especially urban-based youth projects. These project websites provide information about the project, allow beneficiaries such as youth to download information, such as information on reproductive health, as well as post comments and questions. However, I rarely projects monitoring these project websites and regularly reporting any website metrics, which means that these website are not evaluated at the end of the project.

Google Analytics has made monitoring and evaluating a project website very easy. Google Analytics provides very detailed information to help you monitor who is viewing the project website, how much they are using it, which pages of the website are viewed most, how long they stay at the website, as well as location of the viewers.

All that is required is that you have a Google Gmail account and Google Analytics is free. The process involves copying/pasting a script provided by Google into the code of your website that has a "tracking code." That's it. In usually 24hrs, you can start viewing a Google Analytic Report about your website.

To illustrate how to monitor and evaluate a project website, I will report my own Design, Monitoring & Evaluation website usage.

I began writing this blog regularly starting in March 2010. Google Analytics lets me put in any dates I want to analyze, so I will report usage since this time. So, since March 2010:

Visitors:

there have been 13,538 visits to this blog
there have been 10,740 unique visitors
there have been 21,954 page views
on average, visitors view one and one-half (1.6) pages
on average, visitors stay for 2.42 minutes

Location:

22.6% of visitors are from the US
the majority of US visitors are from District of Columbia, California and New York state
the majority of visitors from California are from Los Angels and San Francisco
the majority of visitors from New York state are from the city of New York and Brooklyn
the next largest percentage of visitors are from the UK, of which most are from London and Manchester
since I live in Georgia (Republic of) I know that 98% of visitors are from Tbilisi (the capital) and the remaining 2% are from Batumi (on the Black Sea)
all the 8 countries I cover in my work, and for which I started this website to assist, represent only about 1% of all the visitors

Content:

other than the opening page, the most viewed pages are Using Excel to Create a Ghant Chart, Essential Program Evaluation, and Quotes Related to Evaluation

Traffic:

the vast majority (71%) of visitor find this blog using a Search Engine (60% of these visitors use Google), 15% using a Referring Site, and 14% coming directly (they have bookmarked this blog).

Mobile Devices:

the two mobile devices most used to read this blog are IPhone and IPad.

In summary, these data help me monitor my website (I know who is visiting, how often, what are the favorite pages, where visitors are from).

Also, I can evaluate this blog as not accomplishing its intend goal, which was to serve Save the Children offices in the 8 countries I cover since very few visitors are from these countries and from the locations where Save the Children offices are located. However, I can assess that this blog is serving a general interest of a small group of people around the world.

So, if your project has a website, or is considering developing one, consider using Google Analytics to monitor and evaluate the website.

Training Videos on Program Evaluation

2011-06-05T09:15:00.000+04:00

For those people interested in random assignment program evaluations, the Abdul Latif Jameel Poverty Action Lab (located at the Massachusetts Institute of Technology) has videos of its training sessions on evaluating social programs online.

The videos are titled:

What is Evaluation
Why Randomize
How to Randomize 1
How to Randomize 2
Measurement and Outcome
Sample Size and Power Calculations
Managing Threats to Evaluation and Data Analysis
Analyzing Data

These videos are located here.

My M&E

2011-06-03T20:47:00.001+04:00

My M&E is an interactive WEB 2.0 platform to share knowledge on country-led M&E systems worldwide. In addition to being a learning resource, My M&E facilitates the strengthening of a global community, while identifying good practices and lessons learned about program monitoring and evaluation in general, and on country-led M&E systems in particular.

http://www.mymande.org/?q=wikimehome

While My M&E was founded by IOCE, UNICEF and DevInfo, it is managed by a consortium of partner organizations including IDEAS, IPDET, WHO/PAHO, UNIFEM, ReLAC, Preval, Agencia brasileira de Avaliacao, SLEvA and IPEN. If your organization wishes to join the consortium as a partner, please send an email to Marco Segone, UNICEF Evaluation Office, at msegone@unicef.org.

My M&E is a collaborative website whose content can be modified continuously by users. To develop and strengthen a global community on country-led M&E systems, registered users have the facility to complete their own social profile and exchange experiences and knowledge through blogs, discussion forums, documents, webinars and videos.

This is a great resource! Enjoy.

Systemic Approaches in Evaluation

2011-06-03T20:44:00.000+04:00

On January 25 and 26, 2011 the Evaluation and Audit Division of the Federal Ministry of Economic Cooperation and Development (BMZ) and the Evaluation Unit of GIZ offered a forum to discuss systemic approaches to evaluation at an international conference.

More than 200 participants from academia, consulting firms and NGOs discussed, amongst others, the following questions:

What are systemic approaches in evaluation?
For which kind of evaluations are systemic approaches (not) useful? Can they be used to enhance accountability, for example?
Are rigorous impact studies and systemic evaluations antipodes or can we combine elements of both approaches?
Which concrete methods and tools can be used in systemic evaluation?

http://www.evaluation-conference.de/en/index.html

You can find videos and download presentations from this website.

Online Training in Monitoring and Evaluation

2011-02-11T12:58:00.000+04:00

The Global Health eLearning Center is provide online training in various topics, some of which include monitoring and evaluation (M&E). Some of the M&E course include:

Data Quality
Data Use for Program Managers
Economic Evaluation
M&E Frameworks for HIV/AIDS Programs
M&E Fundamentals

In order to take these online courses you must register, but registration is free. Also, certificates can be earned for these courses.

So, go to Global Health eLearning Center and take an online M&E course!

Distance Surveying With Google Forms

2010-11-26T13:56:00.002+04:00

Distance learning refers to students who learn from a teacher but who are not physically in a classroom, but rather some distance away but connecting with the teacher and lesson via technology, such as an internet browser or other software. Distance surveying is basically the same principle in that it refers to surveying individuals from a distance via technology, such as telephone, email or the internet.

I'm currently involved with a child protection research project in Iraq. One component of the research is to survey key informants on child protection mechanisms, regulations and policies. These key informants include people from Iraqi ministries and agencies, lawyers, United Nations officials, local and international NGOs, and child protection professionals inside Iraq, both in the Arabic and Kurkish speaking governorates. The distance to travel as well as security issues makes conducting face-2-face interviews costly and risky.

A wonderful tool for distance surveying is Google Forms. It is a flexible form for survey development than includes a built-in data collection system and it does not require any coding and is free of charge! The only "catch" is that you must have an internet connection and a Google Gmail account. Once have get, or if you have a Gmail account, go to Documents and select Forms. To develop a survey tool is quite intuitive but for those who would like some help go here: Google Forms Tutorial

Also, you may have noticed that to use this type of distance survey the respondents will need to have access to an internet connection and an internet browser. Most of the key informants I mentioned above have access to an internet connection and an internet browser.

The survey process involves finding a "survey champion" in each of the groups, for example one person in the UN who will promote the survey among colleagues. The survey champion would call or send an email telling their colleague about the survey and the importance of their participation. A follow-up email would be sent with the link to the Google Form survey. Each key informant would complete the questionnaire. For those key informants who did not complete the online survey in the requested time would receive a follow-up from the "survey champion" in their group.

Once completed all responses are automatically compiled into a Google Spreadsheet which can then be analyzed online, using Google analysis, or downloaded into other data analysis software such as SPSS.

Software for Network Visualization and Analysis

2010-11-24T20:32:00.001+04:00

On 27 March 2010 I had blog called, Network Analysis and Visualization of Qualitative Data. In response to this blog I received several requests to provide software programs that can be used in network visualization and analysis.

Increasingly each year there are more software developed for network visualization, but all network visualization tools require some degree of knowledge about how to structure network data and databases PRIOR to visualization. The only network visualization tool I know of (at this time) that requires a minimum knowledge of network data structure (or theory), is free-of-charge, and has some degree of document support is NodeXL.

NodeXL is a MSExcel add-on template for visually analyzing networks but also provides basic network statistics (in-degree, density, etc.). Most network visualization software require network data to be entered in one format (e.g., txt, DL, or other) and then imported into a software (NetDraw, Pajek) for visualization. NodeXL allows you to enter data directly, via in a familiar spreadsheet format, then to visualize it, and modify various characteristics of the network easily.

NodeXL is up-dated frequently and can provide great visual illustrations of ties, links, clusters, communities, and networks for a better understanding of your data.

Measuring Empowerment

2010-11-23T19:37:00.000+04:00

A new handbook was brought to my attention recently, which I have not had a chance to review yet. It is called, Measuring Empowerment? Ask Them: Quantifying qualitative outcomes from people's own analysis (2010), published by the Swedish International Development Cooperation Agency (Sida). The authors are Dee Jupp and Shoel Ibn Ali. The Preface is written by Robert Chambers.

I have taken parts from the Premble to provide a quick summary of this handbook:

"Quantitative analyses of qualitative assessments of outcomes and impacts can be undertaken with relative ease and at low cost. It is possible to measure what many regard as unmeasurable. This publication suggests that steps in the process of attainment of rights and the process of empowerment are easy to identify and measure for those active in the struggle to achieve them.....This paper presents the experience of one social movement in Bangladesh, which managed to find a way to measure empowerment by letting the members themselves explain what benefits they acquired from the Movement and by developing a means to measure change over time. These measures, which are primarily of use to the members, have then been subjected to numerical analysis outside of the village environment to provide convincing quantitative data, which satisfies the demands of results-based management."

Participatory Impact Assessment

2010-11-23T19:05:00.000+04:00

The Feinstein International Center, a Tufts University, has a rather comprehensive guide on participatory assessment titled, Participatory Impact Assessment: A Guide for Practitioners (2007).The authors are Andrew Carley, John Burns, Dawit Abebe and Omeno Suji. This guide focuses on measuring the impact of livelihood projects.

An eight stage approach is outlined that includes:

Provide a framework for assessing the impact of livelihoods interventions.
Clarify the differences between measuring process and real impact .
Demonstrate how Participatory Impact Assessment (PIA) can be used to measure the impact of different projects in different contexts using community identified impact indicators .
Demonstrate how participatory methods can be used to measure impact where no baseline data exists .
Demonstrate how participatory methods can be used to attribute impact to a project .
Demonstrate how qualitative data from participatory tools can be systematically collected and numerically presented to give representative results of project impact .

The three fundamental questions the PIA attempts to answer are: a) what changes have there been in the community since the start of the project?, b) which of these changes are attributable to the project?, and c) what differences have these changes made to people's lives?

Not all donors are open to participatory methods, especially if it entails the development of indicators by community members during or even after a project. Most donors prefer a list of result and impact indicators prior to project implementation. But, for those projects with greater donor flexibility to use participatory assessment techniques this guide presents some basic steps and methods.

Visualization Methods

2010-11-21T21:09:00.002+04:00

Two guys (Ralph Lengler & Martin J. Eppler) form the Institute of Corporate Communication have developed a "Periodic table" of visualization methods that shows examples of about 100 visualization methods in six categories, which are:

Data Visualization - representation of quantitative data in schematic form (either with or without axes).
Information Visualization - the use of interactive visual representations of data to amplify knowledge...that is, data is transformed into an image.
Concept Visualization - methods to elaborate (mostly) quantitative concepts, ideas, plans, and analyses.
Strategy Visualization - the systematic use of complementary visual representations in the analysis, development, formulation, communication, and implementation of strategies in organizations.
Metaphor Visualization - displays information graphically to organize and structure information.
Compound Visualization - complementary use of different graphic representation formats in one single schema or frame.

Examples:

SM (Stakeholder Map)

FF (Force Field)

The online version is here: A Periodic Table of Visualization Methods

This is a great source for thinking of how to illustrate your ideas and data.

Participatory Photo Mapping

2010-11-20T20:06:00.008+04:00

Participatory Photo Mapping (PPM) is a tool for exploring the "experience of place" and for communicating this experience to community stakeholders and decision-makers. Using Participatory Photo Mapping helps uncover supports and barriers to well-being, especially related to the built environment. The PPM approach photography, narrative stories, and mapping.

The PPM process has four steps:
Step 1: Provide participants with digital cameras and GPS units and have them take pictures of their neighborhood, documenting routine use of community and recreation environments.

Step 2: These photos become the objects of focus group sessions in which open dialogue creates emerging themes that are attached to particular images. Conduct focus group and narrative sessions where the photographs are projected onto a wall and community people talk about the images and are engaged in exploring perceptions of their neighborhood environment.

Step 3: The images are then geocoded as part of a neighborhood-level geographic information system that includes other demographic and spatial data, such as population, household characteristics and crime statistics, to create a qualitative GIS focused on the experience of community and recreation environments.

Step 4: Use learned knowledge to communicate the information to local decision-makers, such as health professionals, business owners, community organizations, and policy makers.
Below are some links to videos by Dr. , originally designed, to develop and design collaborative projects and networks to improve health and well-being of communities by strengthening health information systems and sharing that information with community stakeholders and public health decision-makers.

PPM allows you to:

assess the community and environmental contributions to health, safety and well-being,
address peoples’ perceptions of their neighborhood environments,
identify environmental factors that impact health and well-being,
identify community supports and barriers to health and well-being, present this information to stakeholders and decision-makers.

Watch the videos (links below) to learn more about this tool.

With the cost of digital cameras declining each day, and the ability to instantly print these photos, this technique of community participation (from youth to adults) in identifying community problems and or issues and allowing for multiple interpretations of what the photo is about and why it is important to start the discussion on how it can be resolved.

Livelihood Strategy Videos from Gaza, Palestine

2010-11-02T17:11:00.002+04:00

The third video focuses on normal Gazan life (Abd, 28 Years old) from Beit Lahya in the Northern Gaza Strip. He is a young farmer confronting challenges with earning an income from his farm to take care of his children and helping friends overcome many personal and economic challenges.

Livelihood Strategy Videos from Gaza, Palestine

2010-11-02T14:07:00.001+04:00

The second livelihoods video from Gaza, Palestine, is about Mohammed, who is 30 years of age and works in Save the Children's Livelihood Department. This video presents a young man trying to use his education and commitment to help youth organizations and communities in Gaza.

Livelihood Strategy Videos from Gaza, Palestine

2010-11-02T11:01:00.001+04:00

Save the Children's office in Gaza, Palestine, has produced three short videos on livelihood strategies. The first video is about Torfa, a 56 year old woman from Khan Younis, Southern Gaza Strip, who is trying to support her family. I will be featuring the other two videos shortly.

Networks of Self-Categorised Stories

2010-10-30T14:02:00.000+04:00

Rick Davies, developer and manager of MandE website, discusses an interesting technique to analyze stories generated from his Most Significant Change method for program/project evaluation. He calls the technique, Networks of Self-Catorgoris(z)ed Stories.

This technique illustrates several of the future trends in evaluation highlighted by Michael Patton in a webinar that I mentioned in an earlier post: transdiscipline (graph theory, content analysis), systems thinking (interdependence), and complex concepts (emergent categories). An illustration of networks of self-categorisation by Rick is below.

For most people in the field, the most challenging aspect of this technique will be obtaining the UCINET (by Analytic Technologies) software, which must be purchased and takes some time to comfortably use, but can quickly and easily generate two-mode data. However, the software, Netdraw, which graphs the data can be download, installed and used for free.

Michael Quinn Patton: Future Trends in Evaluation

2010-10-30T13:10:00.000+04:00

On the website, My M&E, in the Webniars page, under the title, Developing Capacities for Country M&E Systems, you can find an informative webinar by Michael Quinn Patton who discusses future trends in program and/or project evaluation.

The webinar was recorded using Elluminate and therefore when you click on the link, if you computer does not already have the most up-to-date Java script, it will download the Java script automatically, and the Elluminate interface will open.

If you are not interested in the opening remarks, you can move the playback slider forward to the 6:30 minute point, which is about where Michaels presentation begins. Also, at the end of the presentation Michael takes questions from an online, international audience.

In his presentation, Michael highlights 6 new trends in evaluation:

Globalization of the evaluation profession.
Evaluation as a transdiscipline.
Increased political intent in accountability, performance, indicators and transparency.
Growing interest on evaluation capacity-building and skill development.
Debates about methods.
Using system thinking and complex concepts.

The Road to Results: Designing and Conducting Effective Development Evaluations

2010-09-19T10:47:00.005+04:00

A very practical handbook, which also has some practical exercises, is the 2009 World Bank publication by Linda G. Morra Imas and Ray C. Rist, titled: The Road to Results: Designing and Conduction Effective Development Evaluations. It is a 585 page book that covers a wide variety of topic on program evaluation. The nice thing is that you can read it online for FREE. The detailed table of contents is below. I have also added the link to the Documents section of this blog.

The Road to Results: Designing and Conduction Effective Development Evaluations

FOUNDATIONS

Chapter 1. Introducing Development Evaluation

Evaluation: What is it?

The Origins and History of the Evaluation Discipline

The Development Evaluation Context

Principles and Standards for Development Evaluation

Examples of Development Evaluations

Chapter 2. Understanding the Issues Driving Development Evaluation

Overview of Evaluation in Developed and Developing Countries

Implications of Emerging Development Issues

PREPARING AND CONDUCTING EFFECTIVE DEVELOPMENT EVALUATIONS

Chapter 3. Building a Results-Based Monitoring and Evaluation System

Importance of Results-Based Monitoring and Evaluation

What is Results-Based Monitoring and Evaluation?

Traditional Versus Results-Based Monitoring and Evaluation

Ten Steps to Building a Results-Based Monitoring and Evaluation System

Chapter 4. Understanding the Evaluation Context and the Program Theory of Change

Front-End Analysis

Identifying the Main Client and Key stakeholders

Understanding the Context

Tapping Existing Knowledge

Constructing, Using, and Assessing a Theory of Change

Chapter 5. Considering the Evaluation Approach

General Approaches to Evaluation

DESIGNING AND CONDUCTING

Chapter 6. Developing Evaluation Questions and Starting the Design Matrix

Sources of Questions

Types of Questions

Identifying and Selecting Questions

Developing Good Questions

Designing the Evaluation

Chapter 7. Selecting Designs for Cause-and-Effect, Descriptive, and Normative Evaluation Questions

Connecting Questions to Design

Designs for Cause-and-Effect Questions

Designs for Descriptive Questions

Designs for Normative Questions

The Need for More Rigorous Evaluation Designs

Chapter 8. Selecting and Constructing Data Collection Instruments

Data Collection Strategies

Characteristics of Good Measures

Quantitative and Qualitative Data

Tools for Collecting Data

Chapter 9. Choosing the Sampling Strategy

Introduction to Sampling

Types of Samples: Random and Nonrandom

Determining the Sample Size

Chapter 10. Planning for and Conducting Data Analysis

Data Analysis Strategy

Analyzing Qualitative Data

Analyzing Quantitative Data

Linking Qualitative Data and Quantitative Data

MEETING CHALLENGES

Chapter 11. Evaluating Complex Interventions

Big-Picture Views of Development Evaluation

Joint Evaluations

Country Program Evaluations

Sector Program Evaluations

Thematic Evaluations

Evaluation of Global and Regional Partnership Programs

LEADING

Chapter 12. Managing an Evaluation

Managing the Design Matrix

Contracting the Evaluation

Roles and Responsibilities of Different Players

Managing People, Tasks, and Budgets

Chapter 13. Presenting Results

Crafting a Communication Strategy

Writing an Evaluation Report

Displaying information Visually

Making an Oral Presentation

Chapter 14. Guiding the Evaluator: Evaluation Ethics, Politics, Standards, and Guiding Principles

Ethical Behavior

Politics and Evaluation

Evaluation standards and Guiding Principles

Chapter 15. Looking to the Future

Past to Present

The Future

Pareto's 80/20 Principle in Development Projects

2010-09-13T14:37:00.000+04:00

In the early 1900s an Italian economist, Vilfredo Pareto, observed that in most countries 20% of the people owned 80% of the wealth. It was soon described as the Pareto's Principle. In the 1930s and 1940s, Dr. Juran who studied Quality Management began noticing a similar pattern in organizations, in that 20% of something in an organization accounted for 80% of the results. Dr. Juran began referring to the "vital few and the trivial many."

Other examples of the 80:20 ratio are:

80% of all deaths on account of sickness happen from 20% percent of diseases.
80% of the nutrition you acquire comes from 20% of the foodstuff you eat.
80% and above marks are scored by only 20% of children in examinations.
80% of the work in an office is done by 20% of the staff.
80% market share of a product is owned by 20% of business houses.
80% of what an presenter presents is understood only by 20% of the audience.
80% of the people browsing the Internet go to 20% of the web-sites.
80% of the most listened music will be from 20% of the albums produced.

What are some examples from projects that might follow Pareto's Principle? For example:

80% of project results come from 20% of the project activities.
80% of project results come from the efforts of 20% of the project staff.
80% of project results come from 20% of project funds.
80% of project of beneficiaries benefit from 20% of the project activities.

Can you think of other aspects of projects that do or may follow Pareto's Principle?

Using Excel to Create a Gantt Chart

2010-09-12T11:54:00.004+04:00

Every project or program must present with the proposal a Gantt Chart illustrating its list of activities and completing dates. After being awarded, the Gantt Chart must be revised and updated.

Not only does every project need a Gantt Chart but virtually all project staff use MS Excel spreadsheet application. What is very nice, several people have published helpful instructions on how to use Excel to quickly create useful and easy to understand Gantt Charts.

Michelle McDonough has published her set of instructions here: Gantt Charts from MS Excel

The site, Techblissonline, presents a downloadable Excel Gantt Chart template. An example is shown below.

Guide to Statistical Charts

2010-09-06T15:00:00.000+04:00

For those who produce reports that include various types of statistical data, you know the importance of graphically illustrating data. Most people readily understand data graphs than data tables.

The UK government has published a nice little guide on that sets out some principles and conventions for making statistical charts. The first part sets out some general principles to follow for any chart and then covers the default formatting used by the Social & General Statistics Section of the Library in the UK.

Guide to Statistical Charts

A chart works on visual and different analytical levels is open to greater interpretation. There are many more options to make a good graph or chart when blending chart type, colour, size, dimensions, labelling, scales etc. There are few rules, but the general principles highlighted in this paper help improve any graph or chart presentation and thus understanding.

Google Maps for Project Description

2010-09-04T20:34:00.000+04:00

Google offers many great services. One great service is My Maps in Google Maps. Once you have an account in Google Maps you can geographically locate any aspect of your project for others to view. For example, Save the Children is implementing the Youth Empowerment Program (YEP) in the country of Yemen in schools primarily in the south.

Using Google Maps, via the Satellite View, the project staff located all the schools the YEP project. Once located, the schools were tagged with an icon (GPS units can be used to help locate project activities). Not only did the staff locate the schools and place icons were they were located in Yemen, but they also created pop-up windows so that when various stakeholders (donors, Ministry of Education officials, SC HQ staff) clicked on the icon, a "ballon window" appears providing specific details about the school. It is also possible to include pictures and video besides text.

The YEP project schools can be viewed here: Schools Save the Children is working with in Yemen. Click on any of the icons to the left the school will be immediately identified via a balloon-window that will show various details about the school (not all schools have details entered yet). If you are interested, keep zooming in and you will actually see the school building.

Also, an organization can add its Logo to Google Maps and can be integrated in to your organization's website or your project/program website. So, when appropriate, use Google Maps to show others where your projects or programs are being implemented.

Understanding the Use of Focus Group Discussions (FGDs)

2010-09-02T17:09:00.003+04:00

Recently, I was asked to assist with a research project. The main aim of the research is to obtain prevalence rates for violence against children as well as the context in which they occur. The methodology proposed was the use of focus groups discussions (FGDs).

The original design estimated the number of interviews (sample size) that would be needed to be a representative of the larger population of children and parents. These samples sizes were then divided by 10, the average number of people in a FGDs, and the research project determined the number of FGDs that would be needed to obtain prevalence rates would be a total of 980 FGDs. Yes, 980 FGDs among children and parents on violence against children.

In addition, the research protocols stated that the FGDs would:

recruit participants randomly
cover a wide range of issues
include confidentiality
not attempt to get individual accounts of violence
would solicit consensus on type and contexts of violence
data analysis would be completed in a short period of time

When I read the aims of the research project and that FGDs would be used as the main methodology and the protocols, I was quite shocked. Why? Here are the reasons.

The sampling was based on individuals not groups. The sampling parameters (confidence interval, margin-of-error, etc.) would not be applicable once you took individuals and formed them into groups.
FGDs findings can not provide prevalence rates nor be generalized of a larger population.
Random sampling does not apply to FGDs; instead FGDs use either purposive or convenience samples. FGD participants are selected because of some common characteristic(s) not randomly.
Generally, FGDs should focus on a few issues with sufficient time to "dig deeper" into these few issues rather than discussing a wide range of issues lightly.
FGDs are not good for discussing sensitive issues such as child violence or exploitation, which is better handled in private interviews or in-depth interviews.
FGDs cannot ensure confidentiality of what is discussed. FGDs organizers and the moderator cannot control what participants may tell others what was discussed in the FGDs afterwards.
FGDs should solicit as many diverse opinions and views and NOT attempt to impose consensus.
Data generated by FGDs are not cheap and easy to enter, analyze and interpret. Despite what may project staff think and/or believe, qualitative data entry and analysis must be systematic and rigorous and is as challenging to analyze as quantitative data.
A good FGD requires an experience moderator and consistency across FGDs, which would be almost next to impossible with 980 FGDs.

Anyone interested in the best practices of using FGDs should read, International Focus Group Research: a hanbook for the health and social sciences, by Monique Hennink.

New Monitoring & Evaluation Site by UNICEF's Evaluation Office

2010-08-31T08:45:00.000+04:00

UNICEF's Evaluation Office has developed a website called, MyMandE. As stated on the websites homepage, MyMandE "is an interactive WEB 2.0 platform to share knowledge on country-led M&E systems worldwide. In addition to being a learning resource, My M&E facilitates the strengthening of a global community, while identifying good practices and lessons learned about program monitoring and evaluation in general, and on country-led M&E systems in particular."

"While My M&E was founded by IOCE, UNICEF and DevInfo, it is managed by a consortium of partner organizations including IDEAS, IPDET, WHO/PAHO, UNIFEM, ReLAC, Preval, Agencia brasileira de Avaliacao, SLEvA and IPEN. If your organization wishes to join the consortium as a partner, please send an email to Marco Segone, UNICEF Evaluation Office, at msegone@unicef.org."

"My M&E is a collaborative website whose content can be modified continuously by users. To develop and strengthen a global community on country-led M&E systems, registered users have the facility to complete their own social profile and exchange experiences and knowledge through blogs, discussion forums, documents, webinars and videos."

Marco Segone discusses MyMandE in an presentation available online. MyMandE has the following sections:

Wiki
Community
Webinars
Videos
How to
Trainings
Virtual Library
Jobs
Roster

Under Videos, my 3-part series on impact evaluation is listed! So, for those interested in international program/project evaluation this site provides many resources (and hopefully more will be added) and allows the international community of M&E people to contribute.

Standardized Tools for Measuring Child Abuse and Violence Against Children

2010-08-26T21:19:00.001+04:00

In 2006, the World Report on Violence Against Children was published by the United Nations, authored by Paulo Sergio Pinheiro. This report was the first comprehensive global attempt to describe the scale of
all forms of violence against children and highlighted the large scale rate of violence against children at home, in schools, in their communities, and by the state despite most countries having signed the Convention of the Rights of the Child.

From a monitoring & evaluation perspective, why should I discuss child abuse and violence against children? One of the major findings of this study was that too many complicated and different tools were being used across countries which made it difficult to assess and compare child abuse rates and trends.

In response to this finding, recommendations were made to develop a approaches and standardized ways to measure voilence agaisnt children that would allow for cross-national comparisons. A set of tools were
developed by 122 experts and are discussed in an article published in 2009 in the journal, Child Abuse & Neglect. These tools are called the International Child Abuse Screening Tools (or ICAST). The ICAST tools were piloted in convenience samples in 7 countries representing all regions of the world. As stated in the journal article mentioned above, these tools were specifically designed to be used in cross-cultural, multi-national, multi-cultural studies so that comparisons could be made across time and countries.

There are 4 ICAST tools to measure child abuse: a) the parent report for young children, b) the child report, c) the institutionalized child report, and d) the young adult retrospective report. These 4 ICAST tools can be found on the International Child Abuse and Neglect website. They have been translated into the following languages: Spanish, Arabic, Icelandic, Hindi and Russian.

If your project or program is considering a baseline assessment of child abuse or violence, these tools may be useful.

Personal Digital Assistants (PDAs) and Child Protection

2010-08-25T19:45:00.004+04:00

Recently, Save the Children (SC) received a grant from Google to test some innovative ideas. One of the innovative projects that SC is testing is the use of personal digital assistants (PDAs) in child protection. PDAs are hand-held computer that is a mobile information manager.

The use of PDAs for child protection is being field tested in the country of Azerbaijan. Once part of the Soviet Union, Azerbaijan-- like many other Soviet states-- institutionalized children for even minor mental or physical disabilities or even if the parents could not afford to support them. Since independence, international organizations have encourage the Azeri government to pass de-institutionalization reforms; that is, as much as possible returning children to their families, or foster homes, and supporting both the children and the families. One approach to meeting de-institutionalization of children in Azerbaijan is the community case management approach. Case management involves assessing, monitoring and evaluating each child and family. Community case management involves ensuring local referral and support services are available, accessible and used by the children and their families.

Currently, child case management involves the use of paper-based in-take assessment forms, child and family monitoring forms, and referral service follow-up. Collecting all this data requires time and money. In addition, the time to process all these data (data entry, data cleaning, data analysis, case reports produced) means that the children and families may not have immediate care or support they need.

With a current project in Azerbaijan, SC is conducting a comparative study of paper-based vs. PDA child case management. In this study, 10 of 60 child case management social workers have been randomly selected to participate in two groups. The first group will be 5 randomly selected social workers selected to use paper-based child case management forms and the second group will be 5 randomly selected social workers to use PDAs that have digital versions of the child case management forms.

After social workers in each group has completed case management forms for approximately 10 children and families, SC will examine the following:

The time and cost differences between paper-based vs. PDAs for each child and family, which will determine the cost effectiveness of PDAs. That is, do PDAs reduce costs and if so, how much;
Data quality, which will determine if the error rates are different. That is, does direct data entry and transmission to the central database reduce errors and if so which errors and by how much; and
User satisfaction, which will determine if social workers are most satisfied with paper-based on PDAs as a case management data tool.

The pilot testing of PDAs for child protection is occurring in Azerbaijan as I write (end of Aug 2010) but should be completed in late September. Once the results are in I'll share them on this blog.

Admitting Project Failure

2010-08-25T12:08:00.001+04:00

Buckminster Fuller said, "If I ran a school, I'd give the average grade to the ones who gave me all the right answers, for being good parrots. I'd give the top grades to those who made a lot of mistakes and told me about them, and then told me what they learned from them."

This quote characterizes the new website, FAILFaire, which provides an online forum to report and discuss project failures. As I have mentioned in a previous blog, all too often most project evaluations do not mention any project shortcomings or failures. The international development community often highlights success and files away failures. But this is a mistake!!! Only by "talking openly and seeing where we have failed may help us learn, make better decisions, and avoid making the same mistakes again." I believe it is no secret that many project just don't work...for various reasons.

Hopefully, though, most projects should be based on sound evidence that they will work prior to implementation. But, for those occasions when projects do fail they should be talk about for lessons learned.

Besides the website, FAILFaire holds conferences. The 1st was held in New York by MobileActive and focused on technology and the 2nd was held in Washington DC in July by the World Bank.

One example reported at a FAILFaire conference was by UNICEF. It was the 5 Million Stories by
2010 Project. UNICEF Innovations’ Chris Fabian and Erica Kochi co-presented what they jokingly referred to as a “zombie project”, because despite the fact that the project couldn’t get off the ground, it kept being half-heartedly restarted over the years. “Our Stories” was designed to give children around the world the chance to tell their stories to be published online as part of a look at the global experience of childhood, with the ultimate goal of having 5 million stories posted by 2010.

Launched in 2007, Kochi and Fabian estimate the project had a .008 success rate, since it only gathered 400 stories. They say that this project was a failure of real world application, in that although the idea was good, there was no real desire for it among the community it targeted. As Kochi explained, “No one asked for this.” Other problems included using proprietary, non-open source code so that they couldn’t adjust when there were problems, a lack of ownership and commitment to the project by key stakeholders, and a long timeline that mean that resources never aligned with needs – in 2007 there was money for PR, in 2008 pro-bonoe design resources, in 2009 the software development. In 2010, they finally shelved the project.

So, if you have a failed project, please proudly post it on FAILFaire.

On Home Leave (to the US) Until July 2010

2010-05-21T09:16:00.001+04:00

I will be taking my annual Home Leave to the US during the month of June, 2010. When I return home to Tbilisi, Georgia, in July 2010 I will resume my program/project design, monitoring and evaluation blog.

Competencies Needed by Evaluators

2010-05-14T15:05:00.002+04:00

In 2001, the American Journal of Evaluation published an article by Jean A. King, Laurie Stevahn, Gail Ghere and Jane Minnema titled, "Toward a Taxonomy of Essential Evaluator Competencies," (22; 229). These authors conducted an exploratory study to determine the extent to which 31 evaluation professionals from diverse backgrounds and approaches could reach agreement on a proposed taxonomy of essential evaluator competencies.

Using the weighted scores, I entered the competencies into Wordle to get a graphic representation of these competencies. Those competencies in large font size were weighted as more important than those in small font, though all were considered important.

The most important competencies were Framing a Research Question, Research Methods and Research Design. Of comparatively less important were competencies of Training Others, Supervising, and Responding to RFPs (Request for Proposals).

Quotes Related to Evaluation

2010-05-14T10:20:00.001+04:00

True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information. Winston Churchill
The only man who behaves sensibly is my tailor; he takes my measurements anew every time he sees me, while all the rest go on with their old measurements and expect me to fit them. George Bernard Shaw
Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted. Albert Einstein
We cannot discover what ought to be the case by examining what is the case. We must decide what ought to be the case. Paul Taylor
The most serious mistakes are not being made as a result of wrong answers. The truly dangerous this is asking the wrong question. Peter Drucker
One of the great mistakes is to judge policies and programs by their intentions rather than their results. Milton Friedman
The pure and simple truth is rarely pure and never simple! Oscar Wilde
First get your facts; then you can distort them at your leisure. Mark Twain
My mind is made up, don't confuse me with the facts! Unknown
A funeral eulogy is a belated plea for the defense delivered after the evidence is all in. Irvin S. Cobb
There's a world of difference between truth and facts. Facts can obscure the truth. Maya Angelou
It is easier to believe a lie that one has heard a thousand times than to believe a fact that no one has heard before. Unknown
We want the facts to fit the preconceptions. When they don't it is easier to ignore the facts than to change the preconceptions. Jessamyn West
If at first you don't succeed, destroy all evidence that you tried. Unknown
Absence of evidence is not evidence of absence. Carl Sagan
Evaluate what you want -- because what gets measured, gets produced. James Belasco
Price is what you pay. Value is what you get. Warren Buffett
For changes to be of any true value, they've got to be lasting and consistent. Tony Robbins
Extraordinary claims require extraordinary evidence.Carl Sagan

Country-Level Monitoring & Evaluation Dashboard

2010-05-07T11:24:00.003+04:00

This past week (2-7 May 2010) I worked with a great team in SC's Cairo office to sketch-out a system that would provide the Senior Management Team (SMT) a consolidated "dashboard" report of key project
indicators on a quarterly basis.

The goal of the dashboard report is to provide the SMT an at-a-glance view of the progress of all projects to meet 1) quarterly benchmarks on key indicators, 2) successes and challenges for that quarter, 3) the burn-rate of project funds, 4) as well as number of beneficiaries reached, thus allowing better informed decision-making by the SMT on project improvement BEFORE the end of the reporting year or the project.

Like the dashboard of a car, this Dashboard Report will carry only those indicators that the SMT consider "key" to measuring the accomplisment of the project NOT ALL indicators.

Example of a Dashboard Report

A Dashboard Report is primarily based on comparing quarterly achievements with quarterly benchmarks for activity-level indicators. The Dashboard Report will present result-level indicators on an annual basis. Therefore, if projects do not have end-of-project targets and established quarterly (or semi-annual) benchmarks then developing a meaningfuly Dashboard Report to show process is more difficult.

In addition, identifying over or under achievment of quarterly benchmarks, or annual targets, will only be the beginning of the process; if, for example, there is an under achievement of a quarterly benchmark, then the SMT can "drill-down" deeper with the Project Director as to possible reasons which can be discussed and hopefully resolved before the next quarter.

A Dashboard Report should be simple and easy to read; however, this takes a lot of work! Behind a good Dashboard Report is consensus of key indicators, systems to ensure data quality, adequate staff resources, long-term SMT committment, userfriendly IT tools, clearly defined roles and flow of data, and project with M&E plans that clearly state end-of-project targets to be achieved and quarterly or semi-annual bechmarks to track progress toward the targets.

The Egypt Country Office will be working on developing these systems and has planned the 1st draft of a Dashboard Report this summer. Once it is developed I will present it here.

Barriers to Learning within NGOs...as well as Regional & Country Offices

2010-04-21T21:10:00.001+04:00

Just as individuals must learn in order to survive and grow in the new complexities of a global environment, so must organizations. There are different motives for learning: profit (maximize monetary gain), value (emphasis on ethics and reconciling interests), and altruism (doing good for others).

I would say that most people working in NGOs are motived by altruism. Organizations employing staff who are primarliy movtivated by altruism is positive but can negatively affect organizational learning in specific ways. The organization, Networking for International Development, conducted a survey of NGOs regarding organizational learning and published a report called, "Working with Barriers to Organisational Learning."

The focus of the report is on barriers to that seem to limit learning within NGOs. Ten barriers are discussed:

Bias for Action - often altruistic motives can lead to an activist tendency in which staff feel that there is not time to slow down, clarify issues, and reflect on what is happening.
The Undiscussables - altruism to do good can lead to the tendency to avoid issues because of fear of upsetting others or to avoid conflict.
Commitment to 'the Cause'- similar to the Bias for Action above, this is a sense that the altruistic 'cause' has to be achieved and that taking time to reflect may lead to questionning of what is being done may not achieve the ultimate goal.
A Cultural Bias - many people working in international NGOs come from the US or western cultures. The dominant culture of the organization may ignore other means and methods of interaction, discussion, reflection and learning.
Advocacy at the Expense of Inquiry - the altruistic urge can lead to emphasis on advocating and defending a position at the expense of learning about other views.
The Role of Leadership - combining most or many of the points above, organizational leaders often set the tone for what is acceptable forms of questioning, inquiry, interaction all of which affects overall learning.
Learning to Unlearn - possibly surrounded by others who have similar altruistic motives, the trap of doing what is easy because of habits and assumptions that have been relied on for years with few challenges of learning new habits.
Practicing What We Preach - as part of the altruistic zeal there can be a tendency to promote processes, methods and practices that the organization itself does not do.
The Funding Environment - altruism often relies on outside funding and all too often funding (ie., donor) limits the extent of innovating and testing as well as enacting change based on learning.
Thinking Strategically About Learning - even though many NGO staff are motivated by altruism, there can still be a tendency to be competetive, which results in a core practice of learning being an "internal" activity with little priority placed on learning from peer organizations.

Responding to Evaluation Findings

2010-04-08T08:23:00.007+04:00

John Scott Bayley, an Evaluation Specialist at the Independent Evaluation Department in the Asian Development Bank published an article in the Evaluation Journal of Australasia about Handy Hints for Program Managers.

Though meant to be both lighthearted as well as serious, one of his handy hints for Program Managers is if they feel threatened by the results of an evaluation study they should consider responding with one of the following strategies:

Attack the evaluation’s methodology;
Attack the data's interpretation and resulting conclusions;
Attack the evaluation’s assumptions;
Attack the recommendations;
Substitute previously unstated goals for the official program goals;
Attack the evaluators personally, claim they are biased or unfamiliar with the program;
Attack the evaluation’s key issues and research questions;
Do not participate in the evaluation, but argue that the findings lack an adequate contextual background;
Rally together those who are threatened by the findings;
Indicate the findings are reasonable, but unable to be implemented due to a lack of resources, political opposition, the staff need training etc;
Complain about a lack of consultation;
Argue that the evaluators did not appreciate the subtleties of the program;
Simply pretend that the evaluation never occurred, ignore it;
State that the program’s environment has changed, and the findings are no longer relevant;
Stall for time until the evaluation is forgotten about;
Argue that the union and staff will not accept the recommendations;
Argue that while the program has not achieve it's goals, it does achieve other important things that are too subtle to be easily measured;
Say that the evaluation leaves important questions unanswered, its' significance is questionable;
Argue that the data is open to alternative interpretations, the evaluation’s conclusions have been questions by others;
Attack the steering committee;
Claim that the results contradict commonsense experience, and testimonials from clients;
Claim that the findings are contradicted by other research conducted by various experts in the field;
Agree with the findings and indicate that you have known about this for some time, and you started making changes months ago;
Argue that the findings contradict the spirit and philosophy of the dept/program;
Make up quotations that support your case and attribute them to knowledgeable sources; and
Argue about definitions and interpretations.

Some others that I have heard that are not listed above are:

Argue that the project did not have a sufficient budget to monitoring the results and thus cannot be held responsible for not achieving them.
Argue that the results the project was trying to achieve are so unique that they are not measurable.
Argue that the "real" results of the project will occur years after the evaluation.

Development Assets of Yemeni Youth

2010-04-01T15:06:00.015+04:00

ave the Children (SC) has been working in Yemen since 1963 with programming in education, health, child protection and civil society. In 2008, SC was awarded a grant from the United States Agency for International Development (USAID) for a Youth Empowerment Program (YEP), which operates in the governorates of Sa'ana, Ibb, Aden and Abyan.

In late 2008 to early 2009, I was involved in an assessment to better understand how youth in Yemen are fairing personally and socially, which would help inform the programming in the YEP program. One of the measures we used to assess how youth were doing was the well regarded Search Institute's Developmental Assets Profile (DAP).

The DAP consists of 58 questions that comprise two domains, Internal and External Assets. The Internal Assets domain is comprise of four sub-scales: Support, Empowerment, Boundaries & Expectations, and Constructive Use of Time. The External Assets domain is also comprised of four sub-scales: Commitment to Learning, Positive Values, Social Competencies, and Positive Identity.

It took about 3 months to adapt the tool to the Yemen context. Adapting the DAP involved initial translation from a Egyptian Arabic version (from a another project) by Yemen project staff, pilot-testing this version with youth via individual interviews and focus group discussion which highlighted issues and wording that required further refinement. Once we had an Arabic version that was adapted for youth in Yemen, a pilot-test was conducted with youth who completed the entire DAP. The results were analyzed for internal reliability (based on Alpha reliability) and temporal stability over a 1-week period of time (Pearson correlation). The internal and temporal reliability test were satifactory, a random sample of 600 youth in the four governorates were interviewed in their homes away from parents and siblings as much as possible. Each question is answered by youth using a scale ranging from "not at all" (=0) to a high of "almost always" (=3) regarding the presence of various situations and conditions in their life in the previous 3 months. The scores are totaled and the developmental assets are categorized as low, fair, good, or excellent.

The graph below present the results, average scores, for each of the four sub-scales that comprise External and Internal Assets. The average scores for each sub-scale has been connected with a line to provide a profile.

Several findings are quickly apparent about how Yemeni youth are fairing personally and socially. First, oveall, Yemeni youth have few constructive opportunities as indicated by the sub-scale Constructive Use of Time having the lowest scores regardless of location. Second, where a youth lives influences their level of developmental assts as show by the substantial differences on these sub-scales depending on which governorate the youth lives (each of the lines represents a governorate). Third, that the Yemeni youth are fairing well on Internal Assets but not so well on External Assets.

The implications of these findings for programming are that youth need more constructive opportunities and outlets. Currently, there are few opportunites in schools, neighborhoods or communities for youth to be involved in structured activities such as sports, music, mentoring, drop-in centers, social groups, or camps. This is particulary the case for youth living in the interior (Sana'a and Ibb), whereas youth living along the coast are more likely to be involved in activities such as boating or fishing. Being involved in constructive activities has been shown to improve empowerment through increasing self-esteem, providing a sense of belonging, developing cognitive, physical and social skills, enhancing a sense of self-worth, and developing relationships.

Although not shown in the graph, further analysis of the DAP shows that Yemeni youth reported low scores for schooling and neighborhood safety. For schooling, these youth did not feel that the schools they attend enforce rules fairly, cares about them, or encourages them to do their best. Thus, empowering youth requires improving the quality of schools. For neigborhood safety, basically, youth felt that neighbors do not help watch out for them, which means working with neighbors to also empower youth in Yemen.

Unhealthy Evaluation Practices!?

2010-03-30T20:51:00.003+04:00

Winston Churchill said, "Criticism may not be agreeable, but it is necessary. It fulfils the same function as pain in the human body. It calls attention to an unhealthy state of things." (No I did not quote this from hearing him.) The following article calls to attention some unhealth things in the (non)use of evlauation within International NGOs (INGOs) especially in trying to convinence the public they are accomplishing their mission statements through effective strategies and interventions. The article is titled, "Measuring Performance versus Impact: Evaluation Practices and their Implications on Governance and Accountability of Humanitarian NGOs," by Claude Bruderlein and MaryAnn Dakkak (June 30, 2009, SSRN).

The authors say that their study "confirms also a growing frustration among humanitarian professionals themselves that, while much is measured and evaluated, it is rarely the actual impact of their work. Instead it is apparent that evaluation as it mostly takes place today reflects primarily the needs of donors; is irrelevant for serious organizational learning and programming efforts; adds considerably to the burden of local staff and partners; and does little to shed light on the roles, influence and impact of INGOs as central actors in humanitarian action and protection."

One quote in the article, from a high ranking person in an INGO, "Evaluation as it is used today is the worst way to learn:It is done post-program (often after the new program has started),it is unhelpful, doesn’t address what produces good programming,focuses on attribution and doesn’t delve into the ambiguities of relationships;They are largely unused and a waste of resources and time."

The main critisms of evaluations in INGOs (the "pains" Churchill mentioned) are:

While organizations want evaluations for moral reasons, they only do what is actually required by donors.
Evaluations are often not useful.
Evaluations are often not used.
New evaluation materials will help little as existing ones are not enforced.
Evaluation criteria are often inappropriate.
Impact evaluation as the one really meaningful approach is almost never done, and is just at the beginning of its development.

In order to treat these unhealth evaluation practices, the authors recommend:

Ensure that evaluations have leverage on programming, including through the direct involvement of evaluators, e.g. by scoring INGOs based on their resolution of identified problems and their integration of evaluator recommendations. Incidentally, these measures are also likely to have implications on the overall quality of evaluations.
Clarify and separate competing organizational accountabilities, by effectively dividing INGO operations into for-profit and non-profit activities, or by partnering with outside for-profit entities. As they exist, most INGOs examined do neither adequately fulfill their internal governance accountability, nor their external business accountability.
Develop and invest in dedicated evaluation research capacity, in-house or through partnerships with academic institutions that provide a rigorous basis and feedback mechanism to INGOs, their donors and the general public.
Increase collaboration among INGOs and donors, based on existing efforts to consolidate, integrate and simplify evaluation methodologies in the interest of less time-consuming yet more meaningful and outcome-focused approaches.
Develop a common approach towards donors and the public on what good humanitarian practice requires, in terms of minimum organizational overheads for rigorous and professional standards of evaluation, programming and organizational learning.
Create a consortium of advocacy organizations, similar as they exist in other areas as an effective way of creating space for dialogue and inter-agency collaboration towards the definition of shared standards in advocacy.
Share evaluations and learn collaboratively, in particular from failures and problems presently not included (or well hidden) in evaluation reports – primarily by fostering collective approaches for open evaluation dialogue.
Experiment with a system of peer-reviewed evaluations, initially internal and confidential to each organization allowing for rigorous and open reviews of evaluation methods – similar to methods applied by ALNAP as an effective collaborative of evaluators but with more effective ways to actually enforce and ensure good practice.
Agree on standardized quantitative and qualitative metrics of impact that would allow for a sufficiently practical and pertinent measurement of impact – as part and priority focus of an improved dialogue, even if it involved superceding existing collaboration successes in consolidating agency methods and indicators.
Ensure that timelines and resources for evaluations are flexible and sufficient, including to undertake meaningful qualitative research of impact over the long-term and to ensure that evaluations on advocacy and policy can be adjusted to affect relevant processes.
Preserve flexibility and check for unintended consequences, especially in advocacy and policy programming to take into account the dynamics of relevant political contexts.
Agree on a simple but shared evaluation language, integrated into all stages of evaluation and programming that allows for the effective involvement of professionals and beneficiaries at and across all levels of humanitarian assistance.

Of all the criticisms, from my experience I agree that organizational learning from evaluation findings is quite rare. All too often, we (myself included) are too busy in search of the next funding to apply evaluation finding to current or future programs and projects; most evaluations focus on achieving results but rarely assess the "operational" aspects on how those results were (not)achieved; and that the unintended consequences are rarely investigated.

Demonstrating Project "Impact"

2010-03-28T20:30:00.026+04:00

When I conduct workshops in monitoring and evaluation, one of the topics discussed is "impact." When impact is defined in a workshop as, "the net change directly attributed to the project interventions," then it requires using and explaining its related terminology, such as "randomization," "selection bias," "attribution," "counter-factual," "double-difference," and "net-change." Attempting to define each of these terms and have them understood by workshop participants who may be unfamiliar with experimental design is challenging.

To help illustrate these concepts and terms I use an game on the first and last days of the workshop. On the first day of the workshop, as just an ice-breaker, a sheet of paper with a number is placed on the notebook of each workshop participant. Using a randon number generator on my computer, I choose two numbers and the two workshop particpants who have these numbers form one team. Then I randomly generate two more numbers and these two particpants form the second team.

On a table in the workshop, I have the game, Perfection, by Milton Bradley (see picture below). (For those unfamiliar with Perfection it is a plastic box with holes of 16 different shapes in a 4x4 arrangement. The goal is to take 16 plastic shapes and place them in their matching hole in the least amount of time.) The rest of the workshop participants are on the other side of the table either cheering or jeering. One person is chosen to be the timekeeper. All 16 plastic pieces are placed in a pile on the table in front of Team 1 and when the timekeeper says "go" Team 1 starts putting the pieces in their matching holes in the Perfection box. Once all pieces are in, the timekeeper shouts how much time it took them. For example, "1 minute, 45 seconds!"

Perfection, a game by Milton Bradley.

Then Team 2 gets their chance to place all 16 pieces in their matching holes, with the timekeeper shouting out the time it took them. (Of course, there are the usual arguments if the timekeeper is correct.)

On the first day that is all that I do....just use the game as an energizer. HOWEVER, at the end of the first day of the workshop I randomly select one of the teams (in this case Team 2), and I gave them the Perfection game and asked them to SECRETLY practice the game until the last day of the workshop.

On the last day of the workshop, again as an energizer, I asked both teams to come to the table and redo the Perfection game, with the timekeeper to record their time, to see which team was faster. After having both teams redo the Perfection game, I along with the secretly chosen team (Team 2), told the other workshop particpants that they had been practing the Perfection game since the 1st day of the workshop.

After Team 1 settles down from being upset since they were not allowed to practice too, we all gathered at a flip chart with the timekeeper and a list of the impact evaluation terminology I mentioned above. We discussed why I randomized the team members, how this was meant to reduce selection bias (most coordinated participants were not necessarily selected nor people who had played games together before), and how Team 2 formed the factual (the effect of practicing) and Team 1 formed the counter-factual (not practicing).

Next, I had the timekeeper calculate the single differences and the double-difference of the change in time for each team to complete the Perfection game. So, on the flip chart paper, the timekeeper calculated:

Single Differences (absolute change):
Team 1: 90 secs (Time 2) - 120 secs (Time 1) = -30 secs
Team 2: 125 secs (Time 2) - 180 secs (Time 1) = -55 secs

Double-Difference:
55 secs (Team 2: factual) - 30 secs (Team 1: counter-factual) = 25 secs

Net Change: 25 secs

Attribution: That without practicing, having played the Perfection game at least once can decrease the amount of time it takes to complete it a second time. However, practicing for about an hour each day of the 3-day workshop results in even a greater decrease in the amount of time to complete the Perfection game. In this case, of the 55 second decrease in time for Team 2, 25 seconds can be attributed to practicing (the intervention).

Thus, if this were a project that had a training activity that conducted a baseline and end-line of training participants, without the counter-factual a project would report that its training reduced the amount of time to complete the Perfection game by 30.6% (55 secs/180 secs); however, the counter-factual shows that the training had only a 13.9% effect (25 secs/180 secs) on reducing time.

And, as you may have already thought, after this blog I will have to change my "impact" exercise for future workshops!

Visualization of Focus Group Discussion Results

2010-03-28T12:10:00.004+04:00

A study in youth livelihood pathways was conducted in rural Azerbaijan from April to July in 2008. The primary objective of this study was to understand the perceptions, practices and opportunities for livelihood strategies among youth by youth and their parents.

Both quantitative (survey) and qualitative (focus group discussions - FGDs) were used. The main focus of this blog is on the FGDs findings. A total of 345 youth and 108 parents were involved in 35 FGDs to disucss youth livelihood pathways. There were four FGD groups: girls, boys, mothers, fathers. In these FGDs, one of the topics discussed was, "what is needed for a successful start-up of a livelihood in your community?"

Youth and adults discussed and listed what was necessary for a successful start-up. Then at the end of each FGDs, participants were asked to score (vote) on the listed items. The results of this process was a matrix comprised of rows of issues, column of four groups (boys, girls, mothers and fathers), with the score values in the cells.

Using the free network drawing tool, Netdraw (Borgatti), the diagram below presents the connections and consensus among girls, boys, mothers and fathers on what is needed for a successful start-up of a livelihood in their Azer community.

The darker lines represent more votes and the seven issues at the top left corner were issues mentioned in the FGDs but did not receive a score by any of the participants at the end of the FGDs.

The diagram quickly shows the issues but also the degree of consensus among the four groups. In the yellow circle, the three issues of 1) tools and equipment, 2) location, and 3) financial support were mentioned and scored by all groups. Also, though not completely, but in general boys place priority on hard (material) issues compared to girls who place more priority on soft (interpersonal) issues as needed for a successful start-up.

Also, interesting, is that in this post-soviet state parents still see the need for a successful start-up business to have a "backer/supporter" which still highlights relality.

From these results, the program was able to quickly learn what were some of the livelihood issues and concerns of the youth and their parents to design more appropriate livelihood programming.

Network Visualizations of Qualitative Data

2010-03-27T22:06:00.000+04:00

I would like to highlight a great website that discusses the use of network analysis to analyze and visualize data. The website founder and maintained by Rick Davies and you can find a special page on this site devoted to network visualization of qualitative data.

Rick's discussion of using network analysis and visualization of pile sorts is especially useful. He provides interesting diagrams for the connections between sorted items, categories, and participants, which is helpful for project planning or evaluation.

Network Analysis & Visualization of Qualitative Data

2010-03-27T21:30:00.003+04:00

An approach to measuring and mapping connections between concepts, opinions, formal and informal relationships and/or exchange of resources between individuals, groups or organizations is network analysis. Network analysis is a technique that allows you to both quantitatively (statistically) and qualitiatively (graphically) analyze connections or linkages between and within various units, whatever those are.

In 2003, I used network analysis for measuring and mapping inter-organizational coalition building for a Prevention Task Force (PTF) in an HIV/AIDS prevention project funded by USAID. One objective of the project was to promote advocacy and policy reform in the area of surveillance, services and prevention of STIs and HIV/AIDS. The project invited a broad, cross-sectional group of stakeholders who had previously been working on issues of STI/HIV/AIDS in the country to join the PTF, which included the government ministries, UN agencies, local NGOs, international PVO/NGOs, and various donors. Initially, the PTF was comprised 32 member organizations and agencies.

The objective of using network analysis was to measure and map the intial interactions of the young coalition of PTF members on 1) exchaning information, 2) sharing data, and 3) sharing technical assistance.

Using a questionnaire, PTF members were asked to report separately how often over the last year their organization had exchanged HIV/AIDs information, data or technical assistance with other PTF members.

Using "on a monthly basis" as the cut-off, the follwing graph shows the PTF coalition network on only "exchanging technical assistance," at the baseline in 2004 and at the end-line in 2007. I will present both the quantitative and qualitive findings. First, the quantitative findings.

Membership: reduction from 32 to 22 members
Isolates: reduction from 12 to 0 members
% of members receiving technical assistance: increase from 53% to 96%
% of members giving technical assistance: increase from 41% to 68%
Main inter-organizational brokers: National AIDS Center and Save the Children in 2003 to National AIDS Center and a local NGO in 2007.

Second, the qualitative findings are the two network graphs of exchanging technical assistance within the PTF. The dots and colors represent the different organizations or government ministries who were members of the PTF.

Prevention Task Force (PTF) 2004

Prevention Task Force (PTF) 2007

In addition, the baseline mapping was used as a type of action research with the PTF...to show who was or was not exchanging technical information and why. As you can readily see, many of the "isolate" organization in 2003 were not members in 2007.

So, network analysis is a powerful tool for measuring and illustrating all types of connections or relationships between various types of people, households, groups, organizations, districts, and nations as well as concepts!

Post-Test of Only Project Group, Design # 8

2010-03-27T10:10:00.002+04:00

Of all the evaluation designs I have presented, this is the weakest and any findings from using ONLY this design would need to be viewed with a lot of caution!

Some of the reasons this design is so weak at evaluating a project are: a) without a pre-test or baseline it is difficult to show that a change has occurred, b) if change did occur how much, and c) without a comparative group it is difficult to argue how much the project interventions were responsible for any of this change.

This type of evaluation design is best used only with very small projects, in rather isolated contexts, using and adhering to proven strategies and interventions, and including mixed methods (secondary data, key informant interviews, focus groups, etc.) to bolster the findings.

Pre- and Post-Test Project Group (no comparison), Design #7

2010-03-26T15:04:00.006+04:00

This design begins at the start of the project; however, baseline data is collected on for the project group and not for a comparative group. The reasons for not including a comparative group could be lack of awareness of evaluation design, budget, time, or other constraints.

This design works best IF (yes, a big if) the project is based on and adheres to already reasonably proven theory of change, strategies, and interventions related to the outcomes being measured in a similar context. This design is more useful to understand how the process of project implementation accomplished expected outcomes.

The basic flaws of this design are that it cannot provide very precise estimates of project's impact on the outcomes and that due to lack of a comparative group it is difficult to determine the potential for scaling-up the project.

Again, to improve on this design, other methods need to be used such as key informant interviews, focus group discussions, in-depth interviews, and personal history recall.

Post-test Comparison of Project and Comparison Groups, Design #6

2010-03-26T14:36:00.002+04:00

In this design, no baseline data was collected for the project group or a comparative (non-project) group and relies completely on end-of-project comparisons between the two groups. This design can be used when do baseline data study was conducted due to time, funding, staffing or situational (e.g., conflict) constraints, or if a baseline cannot be reconstructed (Michael Bamberger has written extensively on reconstructing baselines). It is more effective when used in isolated communities or settings that have little or minimal outside influences (which is getting hard to find these days).

The basic flaws of this design are that it does not account for possible influencial historical events, substantial pre-existing differences, or possible differential trajectories over time between the two groups.

To improve on this design, other methods need to be used such as key informant interviews, focus group discussions, in-depth interviews, and personal history recall.

Pre-test of Project Group with Post-test of Project and Comparison Groups, Design #5

2009-12-03T16:16:00.003+04:00

The next set of project evaluation designs that I will presenting are considered weaker than the previous designs; weaker meaning that they can not provide good evidence that a project's interventions are directly attributable to the measured outcomes or to what degree the project contributed to the measured outcomes (net impact).

The pre-test with a project group combined with a post-test of both project and comparison group design is shown below. As the name implies, the pre-test (baseline study) is conducted only with the people involved in the project. There could be a number of reasons why this might occur: save money, project team didn't like the idea of a comparison group initially, or some technical reasons. However, later there may be the budget, interest or feasibility to include a comparsion group.

The advantages of this design are that it can assess reasonably well how a project is being implemented and whether intended outputs from activities were produced. IF the comparison group studied at the end of the project is quite similar to the project group in characteristics, and adequate mixed methods can demonstrate the comparison group were similar at the baseline as the project group on the outcomes that will be measured, then this design MIGHT demonstrate project effects.

The disadvantages are that is that it is difficult to conclusively determine if the differences between the project and comparison group at the end-line study are due to the project or other factors. Another weakness of this design is that local context events can effect outcomes in the comparison group. For example, an agricultural project compares agricultural output of its farmers to a comparison group of farmers without knowing that the the comparison group of farmers received more irrigated water than previous years thus increasing their output. To monitor local context events, retrospective information can be obtained from the comparison group.

Truncated Pre-test and Post-test with Comparison Group, Design # 4

2009-12-03T14:53:00.003+04:00

How do you evaluate a project if no baseline was conducted and the project is being implemented? One possible approach, if a project has an adequate budget and has a reasonable amount of time left before it is completed, is the use of a truncated pre-test and post-test with a comparison group.

The truncated, or shortened, quasi-experimental project evaluation design uses a mid-term study as a proxy measure for the baseline even while recognizing that the project has been underway for some time. Again, as in all the other more rigorous designs, a comparison group is studied so as to estimate the net impact of the program interventions; however, with no baseline this design is weaker than the previous ones.

The post-test or final study, is conducted at the end of the project. One advantage of this design is that smaller sample sizes can be used since the time frame is shorter which reduces the possibility of respondent loss (attrition). One of the drawbacks of this design is that without having a baseline measure it is not possible to know the total amount of change over the life of the project, rather inferences have to be made based on contextual analysis and incorporating mixed methods approaches.

Project Evaluation: General Purpose Quantiative Evaluation Design #3

2009-11-27T14:12:00.004+04:00

The next RealWorld (Bamberger, Rugh, Mabry: 2006) project evaluation design (#3) is a more simplified version of the previous two designs, but still considered one of the more robust project designs. It is simplified since it has only two studies: baseline and end-line, which is basically a pre- and post-test. But, similar to the two previous rigorous designs, it includes both participants and a match comparative group.

The cost, time, and data constraints are reduced with this design compared to the previous two designs, but still are relatively substantial, yet nonetheless provide more convincing evidence of project success.

With all three robust quantitative project designs discussed so far it is essential to combine them mixed method qualitative methods that especially focus on overall project implementation (quality), context (social, cultural, economic and political) in which the project occurs, and cases (case studies).

So, the three most rigorous project designs have been presented. Let me know what you think of them or if you have used them, what were the benefits, challenges or drawbacks. Just use the Comments section below.

Second Most Robust Quantitative Evaluation-Design #2

2009-11-27T10:15:00.002+04:00

In this series on various types of quantitative project evaluation designs, let's look at another design that is considered very rigorous using quasi-experimental methods. This design is quite similar to Design #1 presented in an earlier blog, in that two groups (participants and a comparative non-participant group) are studied over the life of the project; HOWEVER, this desgin does not include the the 4th study that was in Design #1, the post-project follow-up, but rather has three: baseline, mid-point, and end-line.

As in the most rigorous project evaluation design, what makes this design rigorous are a) the use of a matched comparative group who help establishe the counterfactual [i.e., what would have happened if the project had not occurred], and b) measurements taken at three-points in time. What makes this design slightly less rigorous is without the post-project study the sustainabiliy or trajectory of the results is not known. In other words, after a certain period of time with no interventions, were the results among the participant at the end of the project able to be sustained, increased or did they eventually decline?

Many of the limitations that applied to the first design also apply to this design, which partly explain why this design is not often used among NGOs. First, it requies more time and costs to collect data at 3 points in time and among two groups. Second, the sample size must be relatively large to account for loss or attrition of members in both groups over this period of time. Third, data management and analysis can be a challenge.

In an effort to clearly demonstrate "What Works" this is a project evaluation design that should be considered more often than is currently being used, especially for longer-term projects that span 3 or more years. BUT, such designs must be included at the propsal development phase, otherwise trying to fund, arrange and organize such a project evaluation becomes difficult.

Where Every Project Succeeds and Every Intervention is Above Average!

2009-11-25T16:14:00.000+04:00

On 20 November 2009, Nicholas Kristof published an article in the New York Times called "How Can We Help the World's Poor?" He discussed three views on this question, those who think: 1) that aid is crucial to help the poor, 2) aid, and aid organizations, don't help but actually hurt the poor, and 3) there are both shortcomings and successes with aid but that this should be demonstrated on a case by case basis.

The first view holds that more money is needed from the developed countries to assist the underdeveloped or developing countries. And, one of the main reasons that there are still so many poor in the world is becasuse of so little development aid. Kristof cites Jeffery Sachs and his book "The End of Poverty" as one of the main proponent of this view.

The second view holds that from the years of foriegn aid, there is no correlation between aid given and develoment. In fact, they say that aid systematically fails, undermines self-reliance, entrepreneurship, and can even harm people. The proponents of this view are William Easterly and his book "The White Man's Burden: Why the West's Efforts to Aid the Rest Have Done So Much Ill and So Little Good," and Dambisa Moyo and her book "Dead Aid: Why Aid is Not Working & How There is a Better Way for Africa."

The third view acknowledges that aid has had and continues to have shortcomings, but that some aid programs and projects do make a difference in the lives of people. Proponents in this camp rely on quality evaluations to empirically, not theoretically, show if a project is successful and the interventions make a difference. These proponents are against evaluations showing that every project is successful, where project failures are buried, and every intervention is "above average," and as Kristof says, "these evaluations are often done by the organizations themselves."

Recently, I had a director of a project tell me, "my projet is too unique to be evaluated!" It is this view that quickly provides fuel to the critics of aid.

I think all too often those who work in foreign aid take it for granted that whatever they do is good and appreciated, by both the world community and beneficiaries, but this is simply not the case. There are many who believe that those funds should be used in other ways to improve the lives of people.

Aid projects and interventions can be good or bad. However, to determine this, systematic methods of evaluating the merits of a project, as objectively as possible are needed with the findings---both good and bad ---being made public. In other words, the aid community needs to willingly face "what works" but also "what does not work" and not view project evaluations as a way to report "All our projects succeed and all our interventions are above average."

Project Evaluation: Most Robust Quantitative Evaluation Design #1

2009-11-19T14:15:00.004+04:00

I will be starting a series on 7 different types of quantitative project evaluations, from the strongest (or most rigorous) to the weakest (or least rigorous) designs, that use are based on a quasi-experimental design (i.e., randomization is not used but rather the use of a matched comparison group). These seven quantitative project designs are discussed in more detials in the book, RealWorld Evaluation: working under budget, time, data and political constraints, by Michael Bamberger, Jim Rugh, and Linda Mabry (2006).

The most robust or strongest quantitative project design has also the longest name: Comprehensive longitudinal design with pre-, mid-term, post- and ex-post observations on the project and comparison groups. This design is one of the strongest quantitative project evaluation designs but also the most time consuming and expensive.

As shown in the diagram above, there are several characteristics of this design that make it one of most rigorous, but also expesive and time consuming. First, data is collected at 4 points of time (1-Baseline, 2- Mid-Term, 3-End-line and 4-After Project). In addition, it involves data collection among two groups: those people/households that are involved in the project as well as a match compartive group who are as similiar as project participant BUT who are NOT involved or effected by the project.

The reason for the matched, comparative group is to establish what is called the "counter factual", which attempts to answer the question: "What would have happened to these individuals/households IF the project had not occurred?" Thus, any differences between the project participants and the matched group at the end of the project is estimated to be the impact of the project. The reason the 4th data collection point (After Project Study) is included is to understand the "trajectory" or sustainabilty of any results or impact(s); that is, do the results tend to increase, level off or decrease over time.

There are limitations to being able use this type of design, which is why it is not very often used. First, as mentioned earlier, it requies more time and costs to collect data at 4 points in time and among two groups. Second, the sample size must be relatively large to account for loss or attrition of group members over this period of time. Third, data management and analysis can be a challenge. Fourth, due to the longer period of time, there is potential for larger macro-level influences, such as policy changes, that can affect results.

Despite these limitations and challenges, this design should be considered in new projects, or projects that want to scale-up regionally or nationally, to clearly demonstrate project interventions produce the expected outcomes and results, as well as how sustainable the results are.

Using WORDLE to Illustrate Reports

2009-11-15T13:36:00.001+04:00

When submitting reports or papers, or even giving a Power Point presentation, it is nice to have an illustration of what your presenting. There is an online tool which allows you to generate a "word cloud" which you can place at the beginning of any report, paper, or presentation. This tool is called, Wordle. Wordle is a tool that generates word clouds from any text that you provide. These word clouds give greater prominence to words that appear more frequently in the source text; that is, words used most often are large and words used less often are smaller. You can tweak your clouds with different fonts, layouts, color schemes and backgrounds. The images you create with Wordle can be copy and pasted into documents or presentations. To illustrate, I have made two word clouds.

This "word cloud" represents text I copied and pasted into Wordle from a semi-annual report submitted by a project for street-children in Georgia.

This "word cloud" represents text from a proposal written for supporting children in the marsh lands of Iraq.

So, this is a great tool to embellish a report or presentation and gives the reader a "snap shot" of what will be presented. In both word clouds, the most prominent theme is "children" with other relevant themes and issues. It is a great way to check if children are the most common theme in your report, proposal or presentation. The link to Wordle is: http://www.wordle.net/

Measuring Advocacy and Policy Outcomes

2009-11-13T19:14:00.005+04:00

One of the more challenging aspects of projects in Save the Children is measuring IR-4 performance (Enhanced Enabling Environment), especially if advocacy and policy outcomes are envisioned. This challenge was addressed by various authors and organizaitons in The Evaluation Exchange (Vol. XIII, No.1, Spring 2007), sponsored by Harvard Research Project.

In this issue of The Evaluation Exchange the authors attempted to define advocacy and policy change and how to evaluate this change. Excerpts from one article, which presents a illustrative menu of outcomes and strategies for various types of advocacy and policy objectives, are below.

Objective: Shifts in social norms. Social norms are the knowledge, attitudes, values, and behaviors that comprise the normative structure of culture and society. Advocacy and policy work and intervention increasingly has focused on this area because of the importance of aligning advocacy and policy goals with core and enduring social values and behaviors.

Examples of outcomes
+ Changes in awareness
+  Increased agreement about the definition of a problem
+ Changes in beliefs
+  Changes in attitudes
+  Changes in values
+  Changes in the salience of an issue
+  Increased alignment of campaign goal with core societal values
+  Changes in public behavior

Examples of strategies to achieve these outcomes
+  Framing issues
+  Media campaign
+  Message development (e.g., defining the problem, framing,
   naming)
+  Development of trusted messengers and champions

Objective: Strengthened organizational capacity. Organizational capacity is another name for the skill set, staffing and leadership, organizational structure and management systems, finances, and strategic planning of nonprofits and formal coalitions that do advocacy and policy work. Development of these core capacities is critical to advocacy and policy change efforts.

Examples of outcomes
+ Improved management of organizational capacity of organizations
   involved with advocacy and policy work

+  Improved strategic abilities of organizations involved with
   advocacy and policy work
+  Improved capacity to communicate and promote advocacy
   messages of organizations involved with advocacy and policy work
+  Improved stability of organizations involved with advocacy and
   policy work

Examples of strategies to achieve these outcomes
+ Leadership development
+ Organizational capacity building
+ Communication skill building
+ Strategic planning

Objective: Strengthened alliances. Alliances among advocacy partners vary in levels of coordination, collaboration, and mission alignment and can include nontraditional alliances such as bipartisan alliances or relationships between unlikely allies. Alliances bring about structural changes in community and institutional relationships and are essential to presenting common messages, pursuing common goals, enforcing policy changes, and protecting policy “wins.”

Examples of outcomes
+ Increased number of partners supporting an issue
+ Increased level of collaboration (e.g., coordination)
+ Improved alignment of partnership efforts (e.g., shared priorities,
   shared goals, common accountability system)
+ Strategic alliances with important partners (e.g., stronger or more
   powerful relationships and alliances)
+ Increased ability of coalitions working toward policy change to
   identify policy change process (e.g., venue of policy change, steps
   of policy change based on strong understanding of the issue and
   barriers, jurisdiction of policy change)

Examples of strategies to achieve these outcomes
+ Partnership development
+ Coalition development
+ Cross-sector campaigns
+ Joint campaigns
+ Building alliances among unlikely allies

Objective: Strengthened base of support. Nonprofits draw on grassroots, leadership, and institutional support in working for policy changes. The breadth, depth, and influence of support among the general public, interest groups, and opinion leaders for particular issues are a major structural condition for supporting
policy changes. This outcome category spans many layers of culture and societal engagement including increases in civic participation and activism, “allied voices” among informal and formal groups, the coalescence of dissimilar interest groups, actions of opinion leader champions, and positive media attention.

Examples of outcomes
+ Increased public involvement in an issue
+ Increased level of actions taken by champi ons of an issue
+ Increased voter registration
+ Changes in voting behavior
+ Increased breadth of partners supporting an issue (e.g., number
   of “unlikely allies” supporting an issue)
+ Increased media coverage (e.g., quantity, prioritization, extent
   of coverage, variety of media "beats,” message echoing)
+ Increased awareness of campaign principles and messages
   among selected groups (e.g., policymakers, general public,
   opinion leaders)
+ Increased visibility of the campaign message (e.g., engagement
   in debate, presence of campaign message in the media)
+ Changes in public will

Examples of strategies to achieve these outcomes
+ Community organizing
+ Media campaigns
+ Outreach
+ Public/grassroots engagement campaign
+ Voter registration campaign
+ Coalition development
+ Development of trusted messengers and champions
+ Policy analysis and debate
+ Policy impact statements

Objective: Improved policies. Change in the public policy arena occurs in stages—including policy development, policy proposals, demonstration of support (e.g., co-sponsorship), adoption, funding, and implementation. Advocacy and policy evaluation frequently focuses on this area as a measure of success. While and important focus, improved policies are rarely achieved without changes in the preconditions to policy change identified in other outcome categories.

Examples of outcomes
+ Policy development
+ Policy adoption (e.g., ordinance, ballot measure, legislation,
   legally binding agreements)
+ Policy implementation (e.g., equity, adequate funding, other
   resources for implementing policy)
+ Policy enforcement (e.g., holding the line on bedrock legislation)

Examples of strategies to acheive these outcomes
+ Scientific research
+ Development of “white papers”
+ Development of policy proposals
+ Pilots/demonstration programs
+ Educational briefings of legislators
+ Watchdog function

Objective: Changes in impact. Changes in impact are the ultimate and long-term changes in social and physical lives and conditions (i.e., individuals, populations, and physical environments) that motivate policy change efforts. These changes are important to monitor and evaluate when grantmakers and advocacy organizations are partners in social change. Changes in impact are influenced by policy change but typically involve far more strategies, including direct interventions, community support, and personal and family behaviors.

Examples of outcomes
+ Improved social and physical conditions (e.g., poverty,
   habitat diversity, health, equality, democracy).

Outcome Mapping

2009-11-06T13:16:00.000+04:00

Outcome Mapping: building learning and reflection into development programs (2001), is a book by Sarah Earl, Fred Carden and Terry Smutylo, with a forward by Michael Quinn Patton. Outcome Mapping focuses on intermediate results (outcomes) of change in behavior, relationships, activities, or actions of people or groups; thus the focus is on people rather than things such as cleaner water or improved economy.

Outcome Mapping is most effective when used at the planning stage of a project or program. The parts of the Outcome Mapping exercise can then be adapted into a Results Framework or Logical Framework. And, importantly, successful Outcome Mapping requires commitments in knowing the strategic direction of the project, type of monitoring and evaluation data needed, reporting, participatory learning, team consenus, and resource commitments.

Outcome Mapping has 3 Stages and 12 Steps:

The outline of the book is:
1. Outcome Mapping: The Theory
2. Outcome Mapping: The Workshop Approach
3. Stage 1: Intentional Design
4. Stage 2: Outcome & Performance Monitoring
5. Stage 3: Evaluation Planning
Appendix A: Sample Intentional Design Framework
Appendix B: Overview of Evaluation Methods
Appendix C: Glossary
Appendix D: Terms in French, English, Spanish

If this book sounds useful, you can download a PDF version of it under the DME Documents section to the right.

Matching Results Framework and Logical Framwork Terminology

2009-11-05T16:04:00.000+04:00

Long before the need to monitoring and evaluate a project/program is the fundamental need to design a project/program. The acronym, DME, as in DME Advisor refers to design, monitoring and evaluation. Yes, I like to get involved in the initial design of a project, which is important in how it will monitored and evaluated.

There are two basic project/program designs being used in SC at this time. There is the Results Framework (RF), recommended by SC and generally used by US and Canadian-based donors. In addition, as more funding comes from non-US sources, the Logical Framework Approach (LFA), which is generally used by European donors.

Since SC primarily uses the RF most project/program directors or manager are familiar with it; however, increasingly they are being asked to use the LFA.

There are basic differences between the two design approaches, of which the two main are format and terminology. The basic format of a LFA is that of a matrix, whereas the basic format of a RF is a graphic illustration.

The largest challenge though for staff is the terminology differences. Below I have tried to match the RF and LFA terminology as closely as possible. Of course, there are slightly different versions of the RF and LFA, so this table is for the generic versions of both approaches.

Results Framework Logical Framework

Goal Long-term Objective/Goal

Strategic Objective (SO) Purpose/Short-term Objective

Intermediate Results (IRs) Outputs

Strategies ----

Activities Activities

---- Inputs

---- Risks/Assumptions

Benchmarks Milestones

Targets Targets

Let me know if there are some terminology I'm missing or your think I've mismatched.

Narrative Methods of Project/Program Evaluation

2009-11-05T14:40:00.000+04:00

Have you been involved in a project/program in which a rigorous project/program evaluation was not possible or simply not wanted? For such times there are other methods to conduct evaluations, especially those with few pre-determined indicators, difficulties in implementing rigorous studies, or with the possibiliity of many unforseen or unitended results/outcomes.

One broad approach to evaluating projects/programs is called the "narrative" method. The narrative method is described by Charles McClintock (Dean of the Fielding Graduate Institute’s School of Human and Organization Development, in his article "Using narrative methods to link program evaluation and organization development" published in The Evaluation Exchange Volume 9, Number 4, Winter 2003/ 2004 by the Harvard Family Research Project).

The narrative method is fundamentally storytelling and is related to participatory change processes because it relies on people themselves to make sense of their own experiences as it relates to the project/program. And, the participant and beneficiary's stories can be systematically gathered and claims verified from independent sources or methods.

The narrative method can be divided into three basic types, depending on the purpose of the evaluation.

Success stories
Positive and negative outcomes
Emerging themes

1. Success Stories: One of the most prominant narrative methods for the purpose of success stories related to intermediate outcomes and impact is Most Significant Change, or MSC (Davies and Dart 2003, see MSC document list on the right tab). This method is highly structured and designed to engage stakedholders at all levels. Davies and Dart recommend MSC when a project or program is:

complex and produce diverse and emergent outcomes
large with numerous organisational layers
focused on social change
emphasizes participation
designed with repeated contact between field staff and participants
struggling with conventional monitoring systems
highly customised services to a small number of beneficiaries (such as family counselling).

2. Positive and negative outcomes: this narrative method is called the Success Case method (Brinkerhoff, 2003). This Success Case method has two phases: a) a short questionnaire sent to all project/program participants to identify those for whom the project/program has made a difference and those for whom it did not make a difference; b) next, a number of extreme cases are selected from those two ends of the success continuum (i.e., did and did not make a difference) and respondents are asked to tell stories about both the features of the project/program that were or were not helpful as well as other factors that facilitated or impeded success. Based on the logic of journalism and legal inquiry, independent evidence is sought during these storytelling interviews that would corroborate the success claims.

These stories serve both to document outcomes, but also to guide management about needed change in project/program interventions that will accomplish higher level outcomes and impacts.

3. Emerging Themes: This narrative methods is basically qualitative case studies (Costantino & Greene, 2003). Here, stories are used to understand context, culture, and participants’ experiences in relation to program activities and outcomes. As with most case studies, this method can require site visits, review of documents, participant observation, and personal and telephone interviews. Stories can include verbatim transcripts, some of which contained interwoven mini stories. Selection of a few cases, studied in-depth, are able to develop many more "themes" that are involved in a project/program and of relationships among participants and staff.

Why NGOs Are Hesitant to Share Lessons Learned

2009-10-28T16:05:00.004+04:00

Ricardo Wilson-Grau Consulting (17 April 2007 from CSO survey) asked a civil society organizations (CSOs) if they had lessons learned and, if so, how many were not from just "better practices" but just as importantly from "bad practices." Surprisingly, virtually none of the CSOs reported sharing "bad practices" as one aspect of lessons learned.
A follow-up question was asked: Why are so few CSOs willing to share their "bad practices" so that others can learn from them. The question had 5 closed-ended responses and 1 open-ended response. The result were:

Organizations are reluctant to think about negative experiences - 36.6%
They are uncomfortable sharing weaknesses with a donor - 64.1%
They are uncomfortable sharing weaknesses with other organizations - 57.3%
Organizations have little information and knowledge available to explain failures - 36.6%
Organizations are interested in what does work and not in spending time on what does not - 35.9%
Other: 36.6% were by and large nuances of the five multiple choice options above.

In a blog by Daniel O'Neil (The Change Agent), he cites the paper, "Lessons Not Learned: Why don't NGO workers collaborate more?" by Wade Channell. In this paper, Wade highlights why development workers are lousy at learning from each other. Although we might be friends and socialize together, the world of international development workers does not foster learning. He cited four problems of learning:

Incentives for Knowing, But Not for Learning
High Incentives for Repetition, Low Incentives for Innovation
High Incentives for Guarding Information
Disconnection between Performance and Awards

All too often staff are hired because of their proficiency in an area, and due to the pace of project implementation, there is little time or incentive to read, reflect and interact with others in a learning process. Access to conferences, workshops, and journals for those workers in the field are both a physical and financial challenge. Thus, bad practices can "creep" into projects because staff are not able to keep up-to-date.

As repetition, Daniel states, "Donors only want to fund proven successes and NGOs write their proposals to satisfy what the donor wants to hear. This is especially critical when entering a competitive bid. The NGOs seek to divine what the donor wants to hear, rather than to come up with the best approach. The Gates Foundation has made significant waves because they are willing to fund projects that take risky approaches."

When it comes to guarding information, occasionally, project staff and NGOs are not willing to share project evaluations due to potentially unfavorable findings. Also, there are not many forums to share project evaluations even with the same organization so "bad practices" can be avoided.

Finally, the fear is that to expose "bad practices" and the lessons learned from that will not be rewarded. Certainly, good planning should reduce bad practices, but no project can be completely flawless.

How Many In-depth Interviews Are Enough?

2009-10-22T15:21:00.002+04:00

Have you ever had a limited amount of time and budget and wondered what are the fewest number in-depth interviews you could get by with but still get an adequate amount of data and information on a specific topic?

In the article, "How Many Interviews Are Enough? An Experiment with Data Saturation," (Field Methods, Vol. 18, No. 1, February 2006) the authors Greg Guest, Arwen Bunce and Laura Johnson investigate this question.

Specifically, these authors were interested in the minimum number of in-depth interviews does it take to get a reliable sense of themes and issues and variability. That is, does it take 6 interviews, 18 interviews, 100 interviews to render a useful understanding of most of the issues; or another way to ask the question is, when does adding more interviews not make a difference in rendering substantial more information? When is enough?

To answer these questions, they conducted a study among a group of women (sex workers) in two African countries. The in-depth interview guide consisted of six structured demographically oriented questions, sixteen open-ended main questions, and fourteen open-ended sub-questions. To determine the degree of data saturation (useful understanding of most themes/issues), the authors used the point in data collection and analysis when new information produced little or no change to the codebook.

After collecting and analyzing their data, data saturation occurred at when they had analyzed 12 interviews.That is, 92% of the total number of codes they developed for the entire study were developed by the 12th interview.

Benefits:

1. Maximum data saturation obtained with minimum number of interviews.

2. Time and cost-effective

Limitations:

1. Must be done with a purposive sample: people specifically interviewed because of their knowledge or experience related to the specific topic.

2. The individuals should be relatively similar (homogeneous), for example female sex workers, street children, etc.

Participatory Video: A Qualitative Method of Monitoring & Evaluation

2009-10-20T15:47:00.004+04:00

The following have been taken from Insight Into Participatory Video: A handbook for the field, written by Nick and Chris Lunch, 2006. Published by Insight. (Find this publication under DME Documents.)

Participatory Video (PV) is a set of techniques to involve a group or community in shaping and creating their own film. The idea behind this is that making a video is easy and accessible, and is a great way of bringing people together to explore issues, voice concerns or simply to be creative and tell stories. This process can be very empowering, enabling a group or community to take action to solve their own problems and also to communicate their needs and ideas to decision-makers and/or other groups and communities. As such, PV can be a highly effective tool to engage and mobilize marginalized people and to help them implement their own forms of sustainable development based on local needs.

Insight has its own YouTube channel that allows you to view PV from around the world. Insight's YouTube channel is located at: http://www.youtube.com/Insightshare

Nick and Chris Lunch were recently interviewed as part of OneWorldTV's series focusing on pioneering individuals and organizations using video as a tool for social change.

How Does It Work

Participants (men, women and youth) rapidly learn how to use video equipment through games and exercises.
Facilitators help groups to identify and analyze important issues in their community by adapting a range of Participatory Rural Appraisal (PRA)- type tools with PV techniques (for example, social mapping, action search, prioritizing, etc. See ‘Chambers’ in Appendix 7, References).
Short videos and messages are directed and filmed by the participants.
Footage is shown to the wider community at daily screenings.
A dynamic process of community-led learning, sharing and exchange is set in motion.
Completed films can be used to promote awareness and exchange between various different target groups. Insight has worked with pastoralists, farmers, marginalized communities and youth in rural and urban settings, street children, refugees and asylum seekers,people with mental health problems, learning difficulties and physical disabilities (see Part Five, Case Studies). PV films or video messages can be used to strengthen both horizontal communication (e.g. communicating with other communities) and vertical communication (e.g. communicating with decision-makers).

Tichezerane AIDS Support Group Participatory Video

What Does PV Offer

PV engages: Video is an attractive technological tool, which gives immediate results.

PV empowers: A rigorous but fun participatory process gives participants control over a project.

PV clarifies: Participants find their voices and focus on local issues of concern.

PV amplifies: Participants share their voices with other communities, including decision-makers.

PV catalyzes: Participants become a community, which takes further action. PV is inclusive and flexible: Insight have worked with a wide range of groups in the UK and internationally.

PV is accessible: Findings, concerns and living stories are captured by communities themselves on video; projects

can be documented and evaluated; policy information and decisions can also be transferred back to the community level through PV.

PV equips people with skills and positive attitudes: Skills developed include good group-working skills, listening skills, self-esteem building and motivation techniques; PV projects encourage better awareness of community, identity and place; PV develops an active role for participants in improving their quality of life.

PV disseminates good practice: A range of impressive initiatives and suggestions can be documented by those directly involved, cheaply and effectively, and shared across the country and even further abroad;policymakers can be deeply affected by powerful stories and images captured in this way at, and by, the grassroots.

Waiting for Water

Applications of PV

Marginalised social group to wider community: showing a PV film made by one group and using as a tool to stimulate discussion and participation among other groups in society. Participants may want to conduct filmed interviews to gauge reactions among the audience and record feedback. Facilitators can use such screenings to identify and congregate new groups to work with using the same PV methods.
Community to community: produced films shown to other communities and used as a tool to inspire and initiate same process of analysis and local action in the second community. Spreading impacts of the work and "PV strikes me as especially well suited to enabling rural people, after only a little training and at moderate cost, to create vivid accounts of their own experience. Very suitable for sharing with their counterparts elsewhere in the country or even abroad." Claire Milne, ICT Telecoms Consultant awareness raising, but also a chance to bring in new groups, highlight differences as well as similarities.
Community to community PV exchange visits: introducing PV into this process as a tool for wider sharing, equitable exchange and team building (i.e. focusing on a shared task and having fun together!). Exchange visits can be costly and usually only benefit a handful of community members, with PV the learning and exchange can be documented enabling the wider community and other communities to benefit from the exchange.
Policy to community PV visits: as with the community to community PV exchange visits above, but getting policymakers to the field. This can be difficult to arrange and maybe only one or two individuals can be prized out of their offices! A policymaker sharing a PV documentation task with the community members can be a good way to equalize relationships. They will have fun together and create something which the policymaker can show to his/her network of colleagues and superiors.
Facilitating multi-stakeholder workshops using PV: A means of getting different groups together on a more equal footing, empowering populations who feel uncomfortable in a workshop setting, or are illiterate. Community members present their films and these become the starting point for discussion and group work which is all documented using PV tools rather than written notes. This also allows the workshop outcomes to be shared widely among communities, personal and professional networks of the workshop participants and the general public (if relevant).
Campaigns: PV has tremendous potential to bring out personal stories to support campaigns and build understanding and consensus in potentially fraught situations. Decision-makers may respond better to the voices of people on the ground than to organizations, academics or activists campaigning on their behalf. Participatory videos are raw, direct and show a fuller picture of what is at stake.
Participatory Research: Generate knowledge, initiate local action, raise awareness, monitor and spread widely.
Community-led Research: Assist groups in the target communities to carry out their own research using the video as a tool for them to document local knowledge and ideas, as well as generate new knowledge and fresh solutions. Local people’s findings can be included in multimedia reports and publications, bringing their authorship into the process and developing a synthesis of local and scientific knowledge.
Participatory Monitoring & Evaluation: Using video rather than an attitudes survey to look at progress during the research can put the community in control. It is visual and accessible to all. It allows the community to highlight issues and areas of interest that we could not necessarily conceive of as outsiders. Things emerge from the films they produce that open up new lines of enquiry and can also help shape the kinds of quantifiable questions partners focus on.
Sharing Best Practices: The groups involved can document and communicate their achievements in their own words. The use of PV to collect and share the best practices and lessons learned. Often while collecting the lessons learned staff and experts obtain the information from the project implementing parties and having analyzed such information they may then prepare the manuals and adjust the vision expressed by the local communities while looking at such data from their own professional perspective. When receiving the project outcomes and developments, NGOs and local communities may have difficulty to fully understand the essence of the project outcomes. The use of PV can enable people to have a virtual interaction with their colleagues from other villages. While watching video material they obtain the information directly without the "university" filter of the professionals."

Participatory Video made by semi-nomadic shepherds of Kazakhstan.

Psycho-Social Programming for Children in Crisis

2009-10-20T09:21:00.002+04:00

In 2004, Save the Children produced a handbook on psycho-social programming for children (Children in Crisis: Good Practices in Evaluating Psycho-social Programming, by Joan Duncan and Laura Arntson). You can download this handbook under the DME Documents section to the right. Those country offices doing, or considering psycho-social programming, should read this handbook if you have not already. It provides a good overview of psycho-social programming, from theory, definitions, issues, types of interventions, and measuring program/project outputs and outcomes. And, there is a very helpful chapter that discusses the differences between outcome and impact measurement.

Some short summaries from the handbook are:

What does psycho-social refer to? “The term “psychosocial” implies a very close relationship between psychological and social factors. When applied to child development, the term underlines the close, ongoing connections between a child’s feelings, thoughts, perceptions and understanding, and the development of that child as a social being in interaction with his or her social environment.”

What are the levels of severity children face in crisis? 1) Severely Affected Group- these are children in which their psychological and social functioning abilities may be severely compromised. While generally a small percentage of the overall population, this group requires intensive psychological attention because they are unable to manage on their own. Children forced to view and/or commit violent acts, such as child soldiers, are likely to fall into this group. More time-intensive, individualized approaches are likely to be the most appropriate responses, where social and cultural resources permit. This group is in need of one-on-one attention in order to address the more severe traumatic and/or depression disorders, for example. 2) At-Risk Group- A second segment of the community consists of those who have experienced severe losses and disruption, are significantly distressed, and may be experiencing despair and hopelessness, but whose social and psychological capacity to function has not yet been overwhelmed. Children in this category may be suffering from acute stress disorder (the most extreme, or exaggerated normal reaction to violence and trauma). They may have lost family members in the violence, they may have witnessed deaths, or they may be victims of violence. This group is at particular risk for psychological and social deterioration if their psychological, social, cognitive, and development needs are not addressed through timely community and social support mechanisms. 3) Generally Affected Group- The third and broadest segment of the population consists of individuals who may not have been directly affected by crisis events and whose families may be largely intact. Children in this group may be suffering from physical and mental exhaustion, for example, but are not experiencing the level of distress felt by those in the severely affected or at-risk groups. Community-based interventions that include not only normalization activities but also theme- and body-based activities can preserve and augment positive coping strategies among this population in a shorter time-frame and contribute effectively and more immediately to children’s and youths’ social, cognitive, and emotional development.

What are psycho-social programs/projects? Child-focused psychosocial projects are those that promote the psychological and social well-being and development of children. The orientation here is that child development is promoted most effectively in the context of the family, community, and culture. At its most fundamental level, psychosocial programming consists of activities designed to advance children's psychological and social development, to strengthen protective and preventive factors that can limit the negative consequences of complex emergencies, and to promote peace-building processes and reduce tensions between groups.

What are the primary issues psycho-social program attempt to address?

+ Secure attachments with caregivers - Child feels safe and cared for by supportive adult caregivers.

+ Meaningful peer relations or social competence - Child has the capacity to create and maintain relationships with peers and adults. Feels he/she is able to effectively navigate his or her social world.

+ Sense of Belonging - Child is socially connected to a community and feels he/she is part of a larger social whole. Child adopts the values, norms and traditions of his/her community.

+ Sense of self-worth and value, self-esteem, well-being - Child thinks of him/herself as worthy and capable of achieving desired goals. Child has a sense of empowerment and a sense of being valued. Child participates in larger community and feels in harmony with norms of his/her society. Child has the capacity and/or possibility to participate in decisions affecting his/her own life and to form independent opinions.

+ Trust in others – Child has a belief that he/she can rely on others for nurturance, help, and advice. Child feels that he/she will not be hurt by others.

+ Access to opportunities – Child has a sense of being in a supportive environment. Child has access to opportunities for cognitive, emotional, and spiritual development and economic security.

+ Physical and economic security – Child’s physical health, livelihood/economic security and environment are supportive and do not pose threats to the child’s emotional or physical wellbeing.

+ Hopefulness or optimism about the future – Childs feels confident that the world offers positive outcomes and a hopeful future.

Should psycho-social programs/projects be similar to each other? Some elements of psychosocial development are specific to a particular culture, meaning that there is not a “one size fits all” approach to psychosocial programming. A key challenge facing project designers is how cultural factors minimize or increase risk, and promote or impede resiliency. However, child development theory and research does point to a set of concepts that are useful building blocks for psychosocial projects regardless of where they are established. These include understanding what makes children resilient and the role that protective factors play throughout development. Identifying the ways these concepts are expressed within a particular culture should guide psychosocial project development and implementation. Through the study of children who have grown up under difficult circumstances, we have learned that some have certain characteristics and social supports that have enabled them to overcome adversity. Similarly, features of the social world have been identified that buffer the consequences of negative experiences on children. These features are often referred to as protective factors.

What are the content areas for interventions? Since children and adults experience and react to complex emergencies in unique ways, the types of projects designed to address their needs will also differ. Projects range and include those that are curative, preventive, and those that promote psychosocial well-being. Curative projects address the diagnosed psychological effects of complex emergencies on children and families, such as treatment of trauma. Preventive projects seek to prevent further psychosocial deterioration and may focus on a particular group or social environment. Lastly, projects may seek to promote healthy psychosocial development through, for example, opportunities to engage in educational, social, and spiritual activities that support the development of children.

What are the basic intervention approaches? There are different approaches to psychosocial programming, depending on the population being targeted and the project to be implemented. It is possible to identify three major groupings:

1. Psychological: Some projects focus more on psychological factors than on social factors. For example, some projects may provide individual counseling to children who have had traumatic experiences or provide training to key community members to identify, refer, or counsel children. These projects will most likely target children and caregivers who have been most severely impacted by crisis events and require a higher level of individualized attention than community-based interventions can provide.

2. Predominately Psychosocial: Some psychosocial projects are predominately or exclusively psychosocial in focus. The project is self-contained and not integrated into other projects with different foci health, food security, shelter) that may co-exist and are co-located. Examples include stand-alone recreation projects, art therapy, or various community-based interventions that promote positive cognitive, emotional, and educational development and functioning. Staff working in these psychosocial projects may have only minimal contact with staff working on other projects. Predominately psychosocial projects are likely to target their activities toward generally affected and at-risk populations, and provide screening and referral (to individualized mental health services or counseling programs) for those more severely affected by conflict or violence.

3. Integrated/Holistic: In some cases psychosocial interventions are integrated into a holistic and total response to the needs of a community. In this case, the “psychosocial” elements may not be as visible. For example, income generation or vocational training projects are not typically thought to be psychosocial. Yet, addressing the economic livelihood of families is fundamental to psychosocial health both in terms of reducing the daily stress of how a family will feed itself, and in terms of providing a pathway to stability and hope for the future. Similarily, such an intervention may have an educational component that supports cognitive development and at the same time fosters good peer relationships and social skills. An income generation project or vocational training project may be a conduit for improved self-esteem and self-worth and the establishment of peer friendships. The position here is that projects that are based on such a holistic approach are to be preferred since they maximize a mutually reinforcing effect when responding to different aspects of child development simultaneously. These projects are most likely to focus on those in the at-risk or generally affected group.

It is useful to organize projects into six broad areas that encompass the diverse social and psychological needs of children during and after a crisis: The Primacy of Family, Education, Engaging Activities, Economic Security, Community Connections, and Reconciliation and Restoration of Justice.

20 Essential Program/Project Evaluation Books

2009-10-20T08:52:00.009+04:00

If you have an interest in having several books on program/project evaluation as a reference, you may want to include those book that are most often purchased together.

Using the online book store, Amazon.com, I began my search to determine which books might be best to have in an evaluation library by starting with the most known evaluation book, Evaluation: A Systematic Approach, by Peter Rossi. Amazon.com also provides a list of other books purchased by those people who purchased a certain book. Starting with Rossi's book, which is generally recognized as an evaluation primer, I made a list of all evaluation books people purchased with Rossi's Evaluation: A Systematic Approach. Then I looked-up each one of those books (as of October 2009) and made the same list of books purchased with each on of them until I had reached almost 550 citations for 110 books on evaluation. (Of course, there is a degree of selection bias in this method since this list will show primarily American readership and only for those who purchase books online.)

The map below shows the purchasing patterns for the 110 books on evaluation purchased in the US. The cluster in the middle of the map is those evaluation books most often purchased together.Taking those books that received the most citations of being purchased together I created the top 10 list, and the next 10 list, which are then the 20 most essential books on program/project evaluation.

Taking those books that received the most citations of being purchased together I created the top 10 list, and the next 10 list, which are then the 20 most essential books on program/project evaluation.

Top 10 Program/Project Evaluation Books Purchased Together

Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, by John W. Creswell.
Qualitative Research & Evaluation Methods, by Michael Quinn Patton (Editor).
Qualitative Inquiry and Research Design: Choosing among Five Approaches, by John W. Creswell.
Evaluation: A Systematic Approach, Peter H. Rossi, Mark W. Lipsey, and Howard E. Freeman.
Handbook of Practical Program Evaluation, Joseph S. Wholey, Harry P. Hatry, Kathryn E. Newcomer.
Program Evaluation: Alternative Approaches and Practical Guidelines, Jody L Fitzpatrick, James R Sanders, Blaine R Worthen.
Utilization-Focused Evaluation, Michael Quinn Patton.
Logic Modeling Methods in Program Evaluation, Joy A. Frechtling.
Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation, E. Jane Davidson.
Evaluation Theory, Models, and Applications, Daniel L. Stufflebeam, Anthony J. Shinkfield.

Next 10

11. Program Evaluation and Performance Measurement: An Introduction to Practice, James C. McDavid and Laura R. L. Hawthorn.
12. RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints, Michael J. Bamberger, Jim Rugh, and Linda Mabry.
13. The Program Evaluation Standards: How to Assess Evaluations of Educational Programs, James R. Sanders.
14. Evaluation, Carol H. Weiss.
15. Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness, Huey Tsyh Chen
16. Case Study Research: Design and Methods, Robert K. Yin
17. Designing and Conducting Mixed Methods Research, John W. Creswell and Dr. Vicki L. Plano Clark.
18. Experimental and Quasi-Experimental Designs for Generalized Causal Inference, William R. Shadish, Thomas D. Cook, and Donald T. Campbell.
19. The Research Methods Knowledge Base, William Trochim and James P Donnelly.
20. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, Juliet Corbin and Anselm C. Strauss.

Other books I found of interest:

What Counts as Credible Evidence in Applied Research and Evaluation Practice?, Stewart I. Donaldson, Christina A. Christie, and Dr. Melvin (Mel) M. Mark.
Program Theory-Driven Evaluation Science: Strategies and Applications, Stewart I. Donaldson
Counterfactuals and Causal Inference: Methods and Principles for Social Research, Stephen L. Morgan and Christopher Winship.
How to Measure Anything: Finding the Value of "Intangibles" in Business, Douglas W. Hubbard
Quasi-Experimentation: Design and Analysis Issues for Field Settings, Thomas D. Cook and Donald T. Campbell.
Damned Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists, Joel Best.

Use of Mobile Phone SMS in Program Delivery and Monitoring

2009-10-17T09:01:00.003+04:00

Increasingly, mobile phones are becoming cheaper and thus more widespread around the world and owned and used by the poor. SMS is most likely the most widely used electronic means of communication in the world, with an estimated 2.5 billion active users.

Percentage of people covered by mobile services, by country in the MEE Region:

% population covered Mobile phone service

Country by mobile service per 100 people

Armenia: 88% 63

Azerbaijan 94% 53

Egypt 98% 40

Georgia 96% 59

Iraq 72% n/a

Jordan 99% 23

Kazakhstan 94% 80

Kyrgyzstan 90% 41

Tajikistan 4% 1

West Band/Gaza 95% n/a

Yemen 68% 3

Why use mobile phone SMS in development?

1. Very cost effective for communication (data and information).

2. Fast because SMS is sent in real time and immediately.

3. High exposure in that, generally, 90% of all SMS messages are opened.

4. Potential for large reach depending on how many use mobile phones.

5. Personal in the SMS can be personalized.

6. Interactive in that receives can respond.

If you would like to read more, below are links to sites that provide more information about mobile phones in development work.

Livelihoods and employment (Souktel http://www.souktel.org/ and http://mobileactive.org/improving-livelihoods-and-incomes-mobile-phones)

Farming advice (http://mobileactive.org/farming-advice-cell-phone)

Health (RapidSMS http://www.rapidsms.org/about/about/)

Nutrition surveillance (http://mobileactive.org/case-studies/nutritional-surveillance-rapidsms-malawi)

Pre-natal child care (http://mobileactive.org/prenatal-care-through-sms)

Emergency Preparedness & Emergency Warning (FloodSMS http://www.netsquared.org/projects/floodsms-–-early-detection-and-warning-catastrophic-flooding-sms)

Community mobilization (FrontlineSMS http://www.frontlinesms.com/)

Education (http://mobileactive.org/bridgeit-mobiles-classroom)

Literacy & community empowerment (http://mobileactive.org/how-tostan-using-mobiles-literacy-and-community-empowerment)

Cash transfer (http://mobileactive.org/concerns-cash-aid-mobile-kenya-evaluation

Why does SC use a Results Framework?

2009-10-16T15:40:00.010+04:00

Often, country offices ask me why Save the Children recommends a Results Framework rather than some other type of program/project design tool, such as Logical Framework. Several employees of Save the Children (see attached article on the right titled, “A Results Framework Services Both Program Design & Delivery Science” under the Documents section).

Some of the reasons these authors cite include:

1. The entire program/project logic and “theory of change” can be visually grasped without extensive reading.

2. Different disciples or technical specialists (health, food security, livelihoods, education) can use the same
basic model.

3. The ability to clarify assumptions as well as state hypotheses.

4. Facilitates in the design programs and projects.

5. Helps in the evaluation designs.

6. Informs action research

The Results Framework has the following components:
Goal- a) States the long-term end status that is to be achieved, b) Usually expensive to measure since it requires large population-based surveys.
Strategic Objective (SO) – a) Is the most ambitious result that programs can reasonably effect and for which implementing agencies are willing to be held accountable.
Intermediate Results (IRs) – These are essential steps toward achieving the SO. Save the Children recommends the use of the following 4 IRs, since SC’s programming is based on behavior change:

IR-1: Availability & Access (as service must be available as well as spatially and economically accessible)

IR-2: Quality (services meet technical as well as client perceived standards)

IR-3: Demand (knowledge, skills, attitudes, or beliefs that hinder or promote service usage)

IR-4: Enabling Environment (facilitates both the supply and demand side of services)
IR Strategies – specific steps to achieve the Intermediate Results
IR Activities – specific program/project activities related to each IR strategy.

Results Framework (Health Example)

What are some of the limitations of the Results Framework?

1. IR2, Quality, has many dimensions, such as technical (i.e., meeting national or international standards) and perceived (client’s perception of quality services). However, this model combines both types into one box even though these are separate dimensions.

2. The Enabling Environment (IR4) is often highly related to achieving IR3, a change in demand via more informed clientele.

3. The Results Framework, unlike the Logical Framework, omits external environmental factors (apart from those in IR4) that can ease or constrain achieving the results and that are beyond the reach of programmers.

4. Finally, the framework lacks the operational details (such as those found in Logical Frameworks) that managers and some donors need; however, standard detailed implementation and monitoring plans that are based on the framework provide these.

Overall, the Results Framework is a simplistic way to illustrate the relationships between higher level Goals all the way down to activities for small as well as large, complex programs/projects regardless of the sector. But, as the authors conclude, that simplicity has its disadvantages.

US White House announcement: Program Evaluations

2009-10-08T15:14:00.002+04:00

In an 7 October 2009 memorandum, issued by the Executive Office of the US President, the issue of program evaluation is emphasized (http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-01.pdf). In short, an inter-agency working group of evaluation experts under the Performance Improvement Council established will be revived. The purpose of the working group will be: (a) to help build agency evaluation capacity and create effective evaluation networks that draw on the best expertise inside and outside the Federal government; (b) to share best practices from agencies with strong, independent evaluation offices; (c) to make research expertise available to agencies that need assistance in selecting appropriate research designs in different contexts; (d) to devise strategies for using data and evaluation to drive continuous improvement in program policy and practice; and (e) to develop government-wide guidance on program evaluation practices across the Federal government while allowing agencies flexibility to adopt practices suited to their specific needs. A key goal of the working group will be to help agencies determine the most rigorous study designs appropriate for different programs given their size, stage of development, and other factors.

The question to me is: Will this also apply to programs receiving US foreign aid via USAID during the Obama administration? Is so, when?

Reconstructing a baseline via retrospective pretest

2009-10-07T17:35:00.005+04:00

A common approach to project evaluation is using a pre- and post-test among beneficiaries. Although this type of design is not very rigorous (I will write about why in a later post), occasionally there are training projects that get started so quickly that only at the end of the training do staff realize that they did not conduct a pretest and subsequently do not have baseline to compare a change in knowledge from the training course.

In an article by Debra Moore and Cynthia Tananis (American Journal of Evaluation, 2009) these authors discuss the issue of not only how to reconstruct a baseline but also the validity and reliability of data when doing so. This method is called a retrospective pretest design.

Now, the authors clearly state that this is a method best used with short-term, intensive training programs and may not be as reliable in other types of activities and interventions.

The authors mention that in a both a pretest and posttest design, or a retrospective pretest design, one of the primary concerns is something called response-shift bias. Response-shift bias occurs when a participant understands the concept being measured at the pretest differently than at the posttest. For example, youth asked on pretest to answer questions about empowerment (the concept) before the training begins may answer differently when asked the same questions about empowerment at the posttest when the training ends because after taking a training course they understand the concept of empowerment differently. Thus, the authors wanted to test if the degree of response-shift bias when a pretest and posttest was conducted for a training course and when a pretest was NOT done and a retrospective pretest was used.

The basic research question was: Do participant’s responses more accurately represent their level of knowledge/awareness at the beginning or after the training?

For example, before taking a training course on DME I many think I know a lot and would willing respond on pretest questionnaire high levels of knowledge and abilities in doing DME. Then after taking a DME course, and being exposed to more detailed and complex issues that I was not previously aware of, I may reassess that my level of knowledge and abilities were not as great as I thought. But, sadly it’s too late to change my pretest responses.

The authors conclude that pretest scores tend to overestimate a particular level of knowledge or ability(larger response-shift bias) than with a retrospective pretest. They also report that other studies have found that self-report retrospective pretest scores are more highly correlated with scores on objective pretest measures of skill development or knowledge than the self-report pretest scores.

The basic message: In projects that include short, intensive training courses, reconstructing a baseline through the use of a retrospective pretest conducted at the end of the training may provide more accurate results than a pretest at the beginning of the training course.

Youth Livelihood Developmental Index (YLDI): Measuring Youth Assets and Competencies

2009-10-05T19:33:00.004+04:00

For my first blog, I thought I would review several tools that I have been involved with the Egypt Country Office in measuring outcomes for their youth livelihoods projects, and has been used in Yemen, as well as in the Africa and Asia regions.

The YLDI is a set of three tools: 1) the Developmental Assets Profile referred to as the DAP, 2) the Livelihoods Competencies Profile referred to as the LCP , and 3) the Tangible Assets Profile referred to as the TAP. Each of these tools is a standardized index to measure the level of assets, competencies and resources that youth have at a given point in time.

The DAP was developed by the Search Institute and contains 58 questions that can be completed either by the youth themselves or in groups. The scores are totaled and the levels of the 40 developmental assets are categorized as low, fair, good, or excellent. Profiled results portray the type and degree of developmental assets among the youth. The DAP produces quantitative scores for two domains (External and Internal Assets) and on eight asset categories (Support, Empowerment, Boundaries & Expectations, Constructive Use of Time, Commitment to Learning, Positive Values, Social Competencies, and Positive Identity) as well as four Context Area domains (Personal, Family, School, Community, Social).

The LCP was developed by Global Youth Livelihoods and contains 69 questions that can be completed individually by the youth or in a group. The LCP is designed to measure a youth’s self-assessment of the level to which s/he possess one or more of 17 basic competencies needed to generate or maintain an income and livelihood. These competencies are grouped into four domains: Human Capital, Social Capital, Financial Capital, and Physical Capital. The scores are totaled and the levels the youth possess of the four types of capital are categorized as low, fair, good, or excellent. Profiled results portray the type and degree of livelihood competencies among the youth.

The TAP, again developed by Global Youth Livelihoods, contains 32 questions that can be completed individually by the youth or in a group. The TAP is designed to measure a youth’s self-assessment of the level to which s/he possess or has access to 8 tangible assets that can generate or maintain an income and livelihood. The 8 tangible assets are grouped into two domains: Financial Capital and Physical Capital.

The YLDI is accompanied by a database template that allows for easy data entry and some basic data analysis and reports.

Due to the total number of questions for all three tools they are administered at separate times. These three tools that comprise the YLDI can be used for three general objectives:

To assess the status (prevalence) of developmental and livelihood assets of youth in a given area to assist with new program/project design;
To evaluate a program or project via a baseline and end-line survey to measure developmental asset and livelihood competency outcomes and or results;
To establish youth profiles based on various characteristics so as to better tailor program/project activities and interventions.

To date, the YLDI has been used in Upper Egypt for a project evaluation (with Mona Moneer) and in Yemen (with Lucienne Mass) for a general assessment to develop youth livelihood programming.

I have attached the English versions of the DAP, LCP and TAP (in the Documents list to the right). HOWEVER, please do not translate or use because they are copyrighted materials. If you would like to use them in a project contact either me or Sita Conklin (MEE Livelihoods Advisor).

If you have a livelihoods project or are considering a livelihoods component in a future project, and are interested in learning more about the results of the use of the YLDI in Upper Egypt or Yemen, feel free to contact me.

In my next post I will discuss how to reconstruct a baseline for a training program at the end of a project!

Design, Monitoring and Evaluation

Making Evaluations Matter - A Practical Guide for Evaluators

Self-administered questionnaires

Youth Conflict & Tolerance Survey

NGO Network Analysis Handbook

GPS and Google Earth/Maps

Catalog of Survey Quesitions

Online Data and Statistics

Monitoring & Evaluating a Project Related Website

Training Videos on Program Evaluation

My M&E

Systemic Approaches in Evaluation

Online Training in Monitoring and Evaluation

Distance Surveying With Google Forms

Software for Network Visualization and Analysis

Measuring Empowerment

Participatory Impact Assessment

Visualization Methods

Participatory Photo Mapping

Livelihood Strategy Videos from Gaza, Palestine

Livelihood Strategy Videos from Gaza, Palestine

Livelihood Strategy Videos from Gaza, Palestine

Networks of Self-Categorised Stories

Michael Quinn Patton: Future Trends in Evaluation

The Road to Results: Designing and Conducting Effective Development Evaluations

Pareto's 80/20 Principle in Development Projects

Using Excel to Create a Gantt Chart

Guide to Statistical Charts

Related articles by Zemanta

Google Maps for Project Description

Understanding the Use of Focus Group Discussions (FGDs)

New Monitoring & Evaluation Site by UNICEF's Evaluation Office

Standardized Tools for Measuring Child Abuse and Violence Against Children

Personal Digital Assistants (PDAs) and Child Protection

Admitting Project Failure

On Home Leave (to the US) Until July 2010

Competencies Needed by Evaluators

Quotes Related to Evaluation

Country-Level Monitoring & Evaluation Dashboard

Barriers to Learning within NGOs...as well as Regional & Country Offices

Responding to Evaluation Findings

Development Assets of Yemeni Youth

Unhealthy Evaluation Practices!?

Demonstrating Project "Impact"

Visualization of Focus Group Discussion Results

Network Visualizations of Qualitative Data

Network Analysis & Visualization of Qualitative Data

Post-Test of Only Project Group, Design # 8

Pre- and Post-Test Project Group (no comparison), Design #7

Post-test Comparison of Project and Comparison Groups, Design #6

Pre-test of Project Group with Post-test of Project and Comparison Groups, Design #5

Truncated Pre-test and Post-test with Comparison Group, Design # 4

Project Evaluation: General Purpose Quantiative Evaluation Design #3

Second Most Robust Quantitative Evaluation-Design #2

Where Every Project Succeeds and Every Intervention is Above Average!

Project Evaluation: Most Robust Quantitative Evaluation Design #1

Using WORDLE to Illustrate Reports

Measuring Advocacy and Policy Outcomes

Outcome Mapping

Matching Results Framework and Logical Framwork Terminology

Narrative Methods of Project/Program Evaluation

Why NGOs Are Hesitant to Share Lessons Learned

How Many In-depth Interviews Are Enough?

Participatory Video: A Qualitative Method of Monitoring & Evaluation

Psycho-Social Programming for Children in Crisis

20 Essential Program/Project Evaluation Books

Use of Mobile Phone SMS in Program Delivery and Monitoring

Why does SC use a Results Framework?

US White House announcement: Program Evaluations

Reconstructing a baseline via retrospective pretest

Youth Livelihood Developmental Index (YLDI): Measuring Youth Assets and Competencies