The Road to Results: Designing and Conducting Effective Development Evaluations

A very practical handbook, which also has some practical exercises, is the 2009 World Bank publication by Linda G. Morra Imas and Ray C. Rist, titled: The Road to Results: Designing and Conduction Effective Development Evaluations. It is a 585 page book that covers a wide variety of topic on program evaluation. The nice thing is that you can read it online for FREE. The detailed table of contents is below. I have also added the link to the Documents section of this blog.

The Road to Results: Designing and Conduction Effective Development Evaluations

FOUNDATIONS

Chapter 1. Introducing Development Evaluation

Evaluation: What is it?

The Origins and History of the Evaluation Discipline

The Development Evaluation Context

Principles and Standards for Development Evaluation

Examples of Development Evaluations

Chapter 2. Understanding the Issues Driving Development Evaluation

Overview of Evaluation in Developed and Developing Countries 

Implications of Emerging Development Issues

PREPARING AND CONDUCTING EFFECTIVE DEVELOPMENT EVALUATIONS

Chapter 3. Building a Results-Based Monitoring and Evaluation System

Importance of Results-Based Monitoring and Evaluation

What is Results-Based Monitoring and Evaluation?

Traditional Versus Results-Based Monitoring and Evaluation

Ten Steps to Building a Results-Based Monitoring and Evaluation System

Chapter 4. Understanding the Evaluation Context and the Program Theory of Change

Front-End Analysis

Identifying the Main Client and Key stakeholders

Understanding the Context

Tapping Existing Knowledge

Constructing, Using, and Assessing a Theory of Change

Chapter 5. Considering the Evaluation Approach

General Approaches to Evaluation

DESIGNING AND CONDUCTING

Chapter 6. Developing Evaluation Questions and Starting the Design Matrix

Sources of Questions

Types of Questions

Identifying and Selecting Questions

Developing Good Questions

Designing the Evaluation

Chapter 7. Selecting Designs for Cause-and-Effect, Descriptive, and Normative Evaluation Questions

Connecting Questions to Design

Designs for Cause-and-Effect Questions

Designs for Descriptive Questions

Designs for Normative Questions

The Need for More Rigorous Evaluation Designs

Chapter 8. Selecting and Constructing Data Collection Instruments

Data Collection Strategies

Characteristics of Good Measures

Quantitative and Qualitative Data

Tools for Collecting Data

Chapter 9. Choosing the Sampling Strategy

Introduction to Sampling

Types of Samples: Random and Nonrandom

Determining the Sample Size

Chapter 10. Planning for and Conducting Data Analysis

Data Analysis Strategy

Analyzing Qualitative Data

Analyzing Quantitative Data

Linking Qualitative Data and Quantitative Data

MEETING CHALLENGES

Chapter 11. Evaluating Complex Interventions

Big-Picture Views of Development Evaluation

Joint Evaluations

Country Program Evaluations

Sector Program Evaluations

Thematic Evaluations

Evaluation of Global and Regional Partnership Programs

LEADING

Chapter 12. Managing an Evaluation

Managing the Design Matrix

Contracting the Evaluation

Roles and Responsibilities of Different Players

Managing People, Tasks, and Budgets

Chapter 13. Presenting Results

Crafting a Communication Strategy

Writing an Evaluation Report

Displaying information Visually

Making an Oral Presentation 

Chapter 14. Guiding the Evaluator: Evaluation Ethics, Politics, Standards, and Guiding Principles

Ethical Behavior

Politics and Evaluation

Evaluation standards and Guiding Principles

Chapter 15. Looking to the Future

Past to Present

The Future

Pareto's 80/20 Principle in Development Projects

In the early 1900s an Italian economist, Vilfredo Pareto, observed that in most countries 20% of the people owned 80% of the wealth. It was soon described as the Pareto's Principle. In the 1930s and 1940s, Dr. Juran who studied Quality Management began noticing a similar pattern in organizations, in that 20% of something in an organization accounted for 80% of the results. Dr. Juran began referring to the "vital few and the trivial many."

Other examples of the 80:20 ratio are:

• 80% of all deaths on account of sickness happen from 20% percent of diseases.
• 80% of the nutrition you acquire comes from 20% of the foodstuff you eat.
• 80% and above marks are scored by only 20% of children in examinations.
• 80% of the work in an office is done by 20% of the staff.
• 80% market share of a product is owned by 20% of business houses.
• 80% of what an presenter presents is understood only by 20% of the audience.
• 80% of the people browsing the Internet go to 20% of the web-sites.
• 80% of the most listened music will be from 20% of the albums produced.

What are some examples from projects that might follow Pareto's Principle? For example:

• 80% of project results come from 20% of the project activities.
• 80% of project results come from the efforts of 20% of the project staff.
• 80% of project results come from 20% of project funds.
• 80% of project of beneficiaries benefit from 20% of the project activities.

Can you think of other aspects of projects that do or may follow Pareto's Principle?

Using Excel to Create a Gantt Chart

Every project or program must present with the proposal a Gantt Chart illustrating its list of activities and completing dates. After being awarded, the Gantt Chart must be revised and updated.

Not only does every project need a Gantt Chart but virtually all project staff use MS Excel spreadsheet application. What is very nice, several people have published helpful instructions on how to use Excel to quickly create useful and easy to understand Gantt Charts.

Michelle McDonough has published her set of instructions here: Gantt Charts from MS Excel

The site, Techblissonline, presents a downloadable Excel Gantt Chart template. An example is shown below.

Guide to Statistical Charts

For those who produce reports that include various types of statistical data, you know the importance of graphically illustrating data. Most people readily understand data graphs than data tables.

The UK government has published a nice little guide on that sets out some principles and conventions for making statistical charts. The first part sets out some general principles to follow for any chart and then covers the default formatting used by the Social & General Statistics Section of the Library in the UK.

Guide to Statistical Charts

A chart works on visual and different analytical levels is open to greater interpretation. There are many more options to make a good graph or chart when blending chart type, colour, size, dimensions, labelling, scales etc. There are few rules, but the general principles highlighted in this paper help improve any graph or chart presentation and thus understanding.

Related articles by Zemanta

• 22 Awesome Visualization Libraries: Charts and Diagrams | Onextrapixel - Showcasing Web Treats Without A Hitch (onextrapixel.com)

Google Maps for Project Description

Google offers many great services. One great service is My Maps in Google Maps. Once you have an account in Google Maps you can geographically locate any aspect of your project for others to view. For example, Save the Children is implementing the Youth Empowerment Program (YEP) in the country of Yemen in schools primarily in the south. 

Using Google Maps, via the Satellite View, the project staff located all the schools the  YEP project. Once located, the schools were tagged with an icon (GPS units can be used to help locate project activities). Not only did the staff locate the schools and place icons were they were located in Yemen, but they also created pop-up windows so that when various stakeholders (donors, Ministry of Education officials, SC HQ staff) clicked on the icon, a "ballon window" appears providing specific details about the school. It is also possible to include pictures and video besides text.


The YEP project schools can be viewed here: Schools Save the Children is working with in Yemen. Click on any of the icons to the left the school will be immediately identified via a balloon-window that will show various details about the school (not all schools have details entered yet). If you are interested, keep zooming in and you will actually see the school building.

Also, an organization can add its Logo to Google Maps and can be integrated in to your organization's website or your project/program website. So, when appropriate, use Google Maps to show others where your projects or programs are being implemented.

Understanding the Use of Focus Group Discussions (FGDs)

Recently, I was asked to assist with a research project. The main aim of the research is to obtain prevalence rates for violence against children as well as the context in which they occur. The methodology proposed was the use of focus groups discussions (FGDs).

The original design estimated the number of interviews (sample size) that would be needed to be a representative of the larger population of children and parents. These samples sizes were then divided by 10, the average number of people in a FGDs, and the research project determined the number of FGDs that would be needed to obtain prevalence rates would be a total of 980 FGDs. Yes, 980 FGDs among children and parents on violence against children.

In addition, the research protocols stated that  the FGDs would:

• recruit participants randomly
• cover a wide range of issues
• include confidentiality
• not attempt to get individual accounts of violence
• would solicit consensus on type and contexts of violence
• data analysis would be completed in a short period of time

When I read the aims of the research project and that FGDs would be used as the main methodology and the protocols, I was quite shocked. Why? Here are the reasons.

1. The sampling was based on individuals not groups. The sampling parameters (confidence interval, margin-of-error, etc.) would not be applicable once you took individuals and formed them into groups.
2. FGDs findings can not provide prevalence rates nor be generalized of a larger population.
3. Random sampling does not apply to FGDs; instead FGDs use either purposive or convenience samples. FGD participants are selected because of some common characteristic(s) not randomly.
4. Generally, FGDs should focus on a few issues with sufficient time to "dig deeper" into these few issues rather than discussing a wide range of issues lightly.
5. FGDs are not good for discussing sensitive issues such as child violence or exploitation, which is better handled in private interviews or in-depth interviews.
6. FGDs cannot ensure confidentiality of what is discussed. FGDs organizers and the moderator cannot control what participants may tell others what was discussed in the FGDs afterwards.
7. FGDs should solicit as many diverse opinions and views and NOT attempt to impose consensus.
8. Data generated by FGDs are not cheap and easy to enter, analyze and interpret. Despite what may project staff think and/or believe, qualitative data entry and analysis must be systematic and rigorous and is as challenging to analyze as quantitative data.
9. A good FGD requires an experience moderator and consistency across FGDs, which would be almost next to impossible with 980 FGDs. 

Anyone interested in the best practices of using FGDs should read, International Focus Group Research: a hanbook for the health and social sciences, by Monique Hennink.