Sunday, September 19, 2010

The Road to Results: Designing and Conducting Effective Development Evaluations

A very practical handbook, which also has some practical exercises, is the 2009 World Bank publication by Linda G. Morra Imas and Ray C. Rist, titled: The Road to Results: Designing and Conduction Effective Development Evaluations. It is a 585 page book that covers a wide variety of topic on program evaluation. The nice thing is that you can read it online for FREE. The detailed table of contents is below. I have also added the link to the Documents section of this blog.

FOUNDATIONS
Chapter 1. Introducing Development Evaluation
Evaluation: What is it?
The Origins and History of the Evaluation Discipline
The Development Evaluation Context
Principles and Standards for Development Evaluation
Examples of Development Evaluations

Chapter 2
. Understanding the Issues Driving Development Evaluation
Overview of Evaluation in Developed and Developing Countries 
Implications of Emerging Development Issues

PREPARING AND CONDUCTING EFFECTIVE DEVELOPMENT EVALUATIONS
Chapter 3. Building a Results-Based Monitoring and Evaluation System
Importance of Results-Based Monitoring and Evaluation
What is Results-Based Monitoring and Evaluation?
Traditional Versus Results-Based Monitoring and Evaluation
Ten Steps to Building a Results-Based Monitoring and Evaluation System

Chapter 4. Understanding the Evaluation Context and the Program Theory of Change
Front-End Analysis
Identifying the Main Client and Key stakeholders
Understanding the Context
Tapping Existing Knowledge
Constructing, Using, and Assessing a Theory of Change

Chapter 5. Considering the Evaluation Approach
General Approaches to Evaluation

DESIGNING AND CONDUCTING
Chapter 6. Developing Evaluation Questions and Starting the Design Matrix
Sources of Questions
Types of Questions
Identifying and Selecting Questions
Developing Good Questions
Designing the Evaluation

Chapter 7. Selecting Designs for Cause-and-Effect, Descriptive, and Normative Evaluation Questions
Connecting Questions to Design
Designs for Cause-and-Effect Questions
Designs for Descriptive Questions
Designs for Normative Questions
The Need for More Rigorous Evaluation Designs

Chapter 8. Selecting and Constructing Data Collection Instruments
Data Collection Strategies
Characteristics of Good Measures
Quantitative and Qualitative Data
Tools for Collecting Data

Chapter 9. Choosing the Sampling Strategy
Introduction to Sampling
Types of Samples: Random and Nonrandom
Determining the Sample Size

Chapter 10. Planning for and Conducting Data Analysis
Data Analysis Strategy
Analyzing Qualitative Data
Analyzing Quantitative Data
Linking Qualitative Data and Quantitative Data

MEETING CHALLENGES
Chapter 11. Evaluating Complex Interventions
Big-Picture Views of Development Evaluation
Joint Evaluations
Country Program Evaluations
Sector Program Evaluations
Thematic Evaluations
Evaluation of Global and Regional Partnership Programs

LEADING
Chapter 12. Managing an Evaluation
Managing the Design Matrix
Contracting the Evaluation
Roles and Responsibilities of Different Players
Managing People, Tasks, and Budgets

Chapter 13. Presenting Results
Crafting a Communication Strategy
Writing an Evaluation Report
Displaying information Visually
Making an Oral Presentation 

Chapter 14. Guiding the Evaluator: Evaluation Ethics, Politics, Standards, and Guiding Principles
Ethical Behavior
Politics and Evaluation
Evaluation standards and Guiding Principles

Chapter 15. Looking to the Future
Past to Present
The Future
Enhanced by Zemanta

Monday, September 13, 2010

Pareto's 80/20 Principle in Development Projects

In the early 1900s an Italian economist, Vilfredo Pareto, observed that in most countries 20% of the people owned 80% of the wealth. It was soon described as the Pareto's Principle. In the 1930s and 1940s, Dr. Juran who studied Quality Management began noticing a similar pattern in organizations, in that 20% of something in an organization accounted for 80% of the results. Dr. Juran began referring to the "vital few and the trivial many."

Other examples of the 80:20 ratio are:
  • 80% of all deaths on account of sickness happen from 20% percent of diseases.
  • 80% of the nutrition you acquire comes from 20% of the foodstuff you eat.
  • 80% and above marks are scored by only 20% of children in examinations.
  • 80% of the work in an office is done by 20% of the staff.
  • 80% market share of a product is owned by 20% of business houses.
  • 80% of what an presenter presents is understood only by 20% of the audience.
  • 80% of the people browsing the Internet go to 20% of the web-sites.
  • 80% of the most listened music will be from 20% of the albums produced.
What are some examples from projects that might follow Pareto's Principle? For example:
  • 80% of project results come from 20% of the project activities.
  • 80% of project results come from the efforts of 20% of the project staff.
  • 80% of project results come from 20% of project funds.
  • 80% of project of beneficiaries benefit from 20% of the project activities.
Can you think of other aspects of projects that do or may follow Pareto's Principle?

Sunday, September 12, 2010

Using Excel to Create a Gantt Chart

Every project or program must present with the proposal a Gantt Chart illustrating its list of activities and completing dates. After being awarded, the Gantt Chart must be revised and updated.
Not only does every project need a Gantt Chart but virtually all project staff use MS Excel spreadsheet application. What is very nice, several people have published helpful instructions on how to use Excel to quickly create useful and easy to understand Gantt Charts.

Michelle McDonough has published her set of instructions here: Gantt Charts from MS Excel
The site, Techblissonline, presents a downloadable Excel Gantt Chart template. An example is shown below.

Monday, September 6, 2010

Guide to Statistical Charts

For those who produce reports that include various types of statistical data, you know the importance of graphically illustrating data. Most people readily understand data graphs than data tables.

The UK government has published a nice little guide on that sets out some principles and conventions for making statistical charts. The first part sets out some general principles to follow for any chart and then covers the default formatting used by the Social & General Statistics Section of the Library in the UK.


A chart works on visual and different analytical levels is open to greater interpretation. There are many more options to make a good graph or chart when blending chart type, colour, size, dimensions, labelling, scales etc. There are few rules, but the general principles highlighted in this paper help improve any graph or chart presentation and thus understanding.

Saturday, September 4, 2010

Google Maps for Project Description



Google offers many great services. One great service is My Maps in Google Maps. Once you have an account in Google Maps you can geographically locate any aspect of your project for others to view. For example, Save the Children is implementing the Youth Empowerment Program (YEP) in the country of Yemen in schools primarily in the south. 

Using Google Maps, via the Satellite View, the project staff located all the schools the  YEP project. Once located, the schools were tagged with an icon (GPS units can be used to help locate project activities). Not only did the staff locate the schools and place icons were they were located in Yemen, but they also created pop-up windows so that when various stakeholders (donors, Ministry of Education officials, SC HQ staff) clicked on the icon, a "ballon window" appears providing specific details about the school. It is also possible to include pictures and video besides text.


The YEP project schools can be viewed here: Schools Save the Children is working with in Yemen. Click on any of the icons to the left the school will be immediately identified via a balloon-window that will show various details about the school (not all schools have details entered yet). If you are interested, keep zooming in and you will actually see the school building.

Also, an organization can add its Logo to Google Maps and can be integrated in to your organization's website or your project/program website. So, when appropriate, use Google Maps to show others where your projects or programs are being implemented.

Thursday, September 2, 2010

Understanding the Use of Focus Group Discussions (FGDs)

Recently, I was asked to assist with a research project. The main aim of the research is to obtain prevalence rates for violence against children as well as the context in which they occur. The methodology proposed was the use of focus groups discussions (FGDs).
The original design estimated the number of interviews (sample size) that would be needed to be a representative of the larger population of children and parents. These samples sizes were then divided by 10, the average number of people in a FGDs, and the research project determined the number of FGDs that would be needed to obtain prevalence rates would be a total of 980 FGDs. Yes, 980 FGDs among children and parents on violence against children.
In addition, the research protocols stated that  the FGDs would:
  • recruit participants randomly
  • cover a wide range of issues
  • include confidentiality
  • not attempt to get individual accounts of violence
  • would solicit consensus on type and contexts of violence
  • data analysis would be completed in a short period of time
When I read the aims of the research project and that FGDs would be used as the main methodology and the protocols, I was quite shocked. Why? Here are the reasons.
  1. The sampling was based on individuals not groups. The sampling parameters (confidence interval, margin-of-error, etc.) would not be applicable once you took individuals and formed them into groups.
  2. FGDs findings can not provide prevalence rates nor be generalized of a larger population.
  3. Random sampling does not apply to FGDs; instead FGDs use either purposive or convenience samples. FGD participants are selected because of some common characteristic(s) not randomly.
  4. Generally, FGDs should focus on a few issues with sufficient time to "dig deeper" into these few issues rather than discussing a wide range of issues lightly.
  5. FGDs are not good for discussing sensitive issues such as child violence or exploitation, which is better handled in private interviews or in-depth interviews.
  6. FGDs cannot ensure confidentiality of what is discussed. FGDs organizers and the moderator cannot control what participants may tell others what was discussed in the FGDs afterwards.
  7. FGDs should solicit as many diverse opinions and views and NOT attempt to impose consensus.
  8. Data generated by FGDs are not cheap and easy to enter, analyze and interpret. Despite what may project staff think and/or believe, qualitative data entry and analysis must be systematic and rigorous and is as challenging to analyze as quantitative data.
  9. A good FGD requires an experience moderator and consistency across FGDs, which would be almost next to impossible with 980 FGDs. 
Anyone interested in the best practices of using FGDs should read, International Focus Group Research: a hanbook for the health and social sciences, by Monique Hennink.