Making Evaluations Matter - A Practical Guide for Evaluators

Recently, the Centre for Development Innovation, Wageningen University & Research Centre, Wageningen, The Netherlands, published Making Evaluations Matter: A Practical Guide for Evaluators written by Cecile Kusters with Simone van Vugt, Seerp Wigboldus, Bob Williams and Jim Woodhill.

This guide emphasizes participatory evaluation and draws heavily upon the work of Michael Quinn Patton, especially from Utilization-Focused Evaluation.

I think this is a very handy guide NOT ONLY for evaluators, but also a handy guide for country directors, project managers, and project directors to read PRIOR to implementing a project as well as toward the end of a project when planning an evaluation.

The contents are the following:

1. Core Principles for guiding evaluations that matter.

2. Suggested steps for designing and facilitating evaluations that matter.

3. Getting stakeholders to contribute successfully.

4. Turning evaluation into a learning process.

5. Thinking through the possible influences and consequences of evaluation on change processes.
6. Conclusion

Annex A: Examples of (Learning) Purposes, Assessment Questions, Users, and Uses of an Evaluation for a Food Security Initiative.

Annex B: Contrasts between traditional evaluation and complexity-sensitive developmental evaluation.

Self-administered questionnaires

In survey research, especially when questions related to sensitive topics are being asked, there are debates between which form of questionnaire administration is best: a) interviewer-administration or b) self-administration. More often that not, questionnaires are administered by a trained interviewer; however, there are times that some people feel its best that the respondent completes the questionnaire without the assistance of an interviewer (self-administered).

Currently, I'm dealing with survey data from a youth study that used a self-administered questionnaire and the data contain many "missing" cases, nonsensical responses, and numerous cases of Errors of Commission and Errors of Omission. An Error of Commission is one where the person responds where they should not and an Error of Omission is one where the person fails to respond when they should.

A questionnaire designed for an interviewer administered survey cannot be used for a self-administered survey! Interviewers are trained in understanding the questions and how to navigate through the questionnaire; however, a questionnaire that is designed for someone who has never seen it before and for them to understand the questions as well as navigate through the questionnaire requires special attention to many factors. Using pg.6 from the 2008 National Survey of College Graduates, conducted by the US Census Bureau to illustrate, some critical factors to consider for a self-administered questionnaire are:

• Language - the instructions and questions need to be written in a vocabulary that is slightly lower than lowest education level of any respondent.
• Section Heading - every section/topic needs a heading that is short, in bold font, slightly different color than the rest of the questionnaire, such as Part B - Past Employment.
• Question Numbering - question numbers should carry the section lettering/numbering as well as the question number and should be in a slightly larger font than the question text and in bold font, such as B1.
• Verbal navigation - instructions next to certain responses that tell the respondent where to clearly go next. In the example above, if the respondent answers "No" in question B1 there is a verbal instruction, in bold font, telling them both 1) the page and 2) the question # to go to.
• Symbol navigation - these are generally arrows showing a respondent where to go next if they answer a certain response. Above, if a respondent answers "Yes" in question B1 the arrow shows them to go to question B2.
• Adequate spacing - all to often to save printing costs, a questionnaire is too cluttered but generally this is ok for a trained interviewer but not for self-administration. A self-administered questionnaire should have adequate spacing between questions to reduce eye fatigue and confusion.
• Coloring - if posible, use slightly different grays or colors to highlight different sections and responses, such as in the example above.

Youth Conflict & Tolerance Survey

Conflict and tolerance are relevant issues for western (Republic of) Georgia, which has experienced civil war in the 1990s and the recent 2008 war with Russia. Save the Children, with funding from the EU, conducted formative research among youth in regions on both sides of the de facto  administrative border between Samegrelo and Gali regions.

In cooperation with several colleagues, we used formative research (Focus group discussion, in-depth interviews, and key informant interviews), to develop a Youth Conflict and Tolerance Survey (YCTS) tool for western Georgia. The YCTS tool focuses on conflict and tolerance issues youth confront in three context settings: the home, at school and in the community. It is a 185 item survey instrument divided into 6 sections: 1) Respondent characteristics, 2) Knowledge of conflict resolution skills and attitudes, 3) Types of conflict in the home, 4) Types of conflict at school/education facility, 5) Types of conflict in the community, and 6) General attitudes. The YCTS underwent a time-stability (reliability) test which showed that the majority of the questions in each section are reasonably stable.

The YCTS tool is designed to be completed either via self-administration (where the youth reads and answers the questions on their own) or via oral administration (where a youth worker or teacher reads each question and the youth answers each question on their own). The YCTS tool is designed for program development (general assessment) and program evaluation (change resulting from program interventions). Change over time is tracked through the administration of the YCTS tool on at least two separate occasions (Time One and Time Two) -- using the same survey methodology, the same instrument with the same learners.

In addition, a YCTS handbook was developed to provide a simple, step-by-step guide to effectively administer the Youth Conflict & Tolerance Survey (YCTS) Tool. Download the TCTS Handbook and please send me your comments, suggestions, and critiques.

YCTS Handbook

NGO Network Analysis Handbook

Today, there is the UN NGO Network, NGO Global Network, International NGO Network, Voluntary Action Network and many other type of sectoral networks such as the Child Rights Network, Human Rights Network, and Environmental Protection Network.

However, more often than not, the term "network" is used as a metaphor and rarely is there much effort to actually measure and demonstrate if there is truly a network; that is, an interconnected group or system.

Since January 2011, I worked with Lilly Saganelidze and Tamuna Dagargulia to study, measure and map the network of youth-focused NGOs in western Georgia. The outcome of this study is: NGO Network Analysis Handbook: how to measure and map linkages between NGOs.

Please feel free to use download and use this handbook and please send us an email if you have suggestions in how to improve it.