Project Evaluation: General Purpose Quantiative Evaluation Design #3

The next RealWorld (Bamberger, Rugh, Mabry: 2006) project evaluation design (#3) is a more simplified version of the previous two designs, but still considered one of the more robust project designs. It is simplified since it has only two studies: baseline and end-line, which is basically a pre- and post-test. But, similar to the two previous rigorous designs, it includes both participants and a match comparative group.

The cost, time, and data constraints are reduced with this design compared to the previous two designs, but still are relatively substantial, yet nonetheless provide more convincing evidence of project success.

With all three robust quantitative project designs discussed so far it is essential to combine them mixed method qualitative methods that especially focus on overall project implementation (quality), context (social, cultural, economic and political) in which the project occurs, and cases (case studies).

So, the three most rigorous project designs have been presented. Let me know what you think of them or if you have used them, what were the benefits, challenges or drawbacks. Just use the Comments section below.

Second Most Robust Quantitative Evaluation-Design #2

In this series on various types of quantitative project evaluation designs, let's look at another design that is considered very rigorous using quasi-experimental methods. This design is quite similar to Design #1 presented in an earlier blog, in that two groups (participants and a comparative non-participant group) are studied over the life of the project; HOWEVER, this desgin does not include the the 4th study that was in Design #1, the post-project follow-up, but rather has three: baseline, mid-point, and end-line.

As in the most rigorous project evaluation design, what makes this design rigorous are a) the use of a matched comparative group who help establishe the counterfactual [i.e., what would have happened if the project had not occurred], and b) measurements taken at three-points in time. What makes this design slightly less rigorous is without the post-project study the sustainabiliy or trajectory of the results is not known. In other words, after a certain period of time with no interventions, were the results among the participant at the end of the project able to be sustained, increased or did they eventually decline?

Many of the limitations that applied to the first design also apply to this design, which partly explain why this design is not often used among NGOs. First, it requies more time and costs to collect data at 3 points in time and among two groups. Second, the sample size must be relatively large to account for loss or attrition of members in both groups over this period of time. Third, data management and analysis can be a challenge.

In an effort to clearly demonstrate "What Works" this is a project evaluation design that should be considered more often than is currently being used, especially for longer-term projects that span 3 or more years. BUT, such designs must be included at the propsal development phase, otherwise trying to fund, arrange and organize such a project evaluation becomes difficult.

Where Every Project Succeeds and Every Intervention is Above Average!

On 20 November 2009, Nicholas Kristof published an article in the New York Times called "How Can We Help the World's Poor?" He discussed three views on this question, those who think: 1) that aid is crucial to help the poor, 2) aid, and aid organizations, don't help but actually hurt the poor, and 3) there are both shortcomings and successes with aid but that this should be demonstrated on a case by case basis.

The first view holds that more money is needed from the developed countries to assist the underdeveloped or developing countries. And, one of the main reasons that there are still so many poor in the world is becasuse of so little development aid. Kristof cites Jeffery Sachs and his book "The End of Poverty" as one of the main proponent of this view.

The second view holds that from the years of foriegn aid, there is no correlation between aid given and develoment. In fact, they say that aid systematically fails, undermines self-reliance, entrepreneurship, and can even harm people. The proponents of this view are William Easterly and his book "The White Man's Burden: Why the West's Efforts to Aid the Rest Have Done So Much Ill and So Little Good," and Dambisa Moyo and her book "Dead Aid: Why Aid is Not Working & How There is a Better Way for Africa."

The third view acknowledges that aid has had and continues to have shortcomings, but that some aid programs and projects do make a difference in the lives of people. Proponents in this camp rely on quality evaluations to empirically, not theoretically, show if a project is successful and the interventions make a difference. These proponents are against evaluations showing that every project is successful, where project failures are buried, and every intervention is "above average," and as Kristof says, "these evaluations are often done by the organizations themselves."

Recently, I had a director of a project tell me, "my projet is too unique to be evaluated!" It is this view that quickly provides fuel to the critics of aid.

I think all too often those who work in foreign aid take it for granted that whatever they do is good and appreciated, by both the world community and beneficiaries, but this is simply not the case. There are many who believe that those funds should be used in other ways to improve the lives of people.

Aid projects and interventions can be good or bad.  However, to determine this, systematic methods of evaluating the merits of a project, as objectively as possible are needed with the findings---both good and bad ---being made public. In other words, the aid community needs to willingly face "what works" but also "what does not work" and not view project evaluations as a way to report "All our projects succeed and all our interventions are above average."

Project Evaluation: Most Robust Quantitative Evaluation Design #1

I will be starting a series on 7 different types of quantitative project evaluations, from the strongest (or most rigorous) to the weakest (or least rigorous) designs, that use are based on a quasi-experimental design (i.e., randomization is not used but rather the use of a matched comparison group). These seven quantitative project designs are discussed in more detials in the book, RealWorld Evaluation: working under budget, time, data and political constraints, by Michael Bamberger, Jim Rugh, and Linda Mabry (2006).

The most robust or strongest quantitative project design has also the longest name: Comprehensive longitudinal design with pre-, mid-term, post- and ex-post observations on the project and comparison groups. This design is one of the strongest quantitative project evaluation designs but also the most time consuming and expensive.



As shown in the diagram above, there are several characteristics of this design that make it one of most rigorous, but also expesive and time consuming. First, data is collected at 4 points of time (1-Baseline, 2- Mid-Term, 3-End-line and 4-After Project). In addition, it involves data collection among two groups: those people/households that are involved in the project as well as a match compartive group who are as similiar as project participant BUT who are NOT involved or effected by the project.

The reason for the matched, comparative group is to establish what is called the "counter factual", which attempts to answer the question: "What would have happened to these individuals/households IF the project had not occurred?" Thus, any differences between the project participants and the matched group at the end of the project is estimated to be the impact of the project. The reason the 4th data collection point (After Project Study) is included is to understand the "trajectory" or sustainabilty of any results or impact(s); that is, do the results tend to increase, level off or decrease over time.

There are limitations to being able use this type of design, which is why it is not very often used. First, as mentioned earlier, it requies more time and costs to collect data at 4 points in time and among two groups. Second, the sample size must be relatively large to account for loss or attrition of group members over this period of time. Third, data management and analysis can be a challenge. Fourth, due to the longer period of time, there is potential for larger macro-level influences, such as policy changes, that can affect results.

Despite these limitations and challenges, this design should be considered in new projects, or projects that want to scale-up regionally or nationally, to clearly demonstrate project interventions produce the expected outcomes and results, as well as how sustainable the results are.

Using WORDLE to Illustrate Reports

When submitting reports or papers, or even giving a Power Point presentation, it is nice to have an illustration of what your presenting. There is an online tool which allows you to generate a "word cloud" which you can place at the beginning of any report, paper, or presentation. This tool is called, Wordle. Wordle is a tool that generates word clouds from any text that you provide. These word clouds give greater prominence to words that appear more frequently in the source text; that is, words used most often are large and words used less often are smaller. You can tweak your clouds with different fonts, layouts, color schemes and backgrounds. The images you create with Wordle can be copy and pasted into documents or presentations. To illustrate, I have made two word clouds.

This "word cloud" represents text I copied and pasted into Wordle from a semi-annual report submitted by a project for street-children in Georgia.

This "word cloud" represents text from a proposal written for supporting children in the marsh lands of Iraq.

So, this is a great tool to embellish a report or presentation and gives the reader a "snap shot" of what will be presented. In both word clouds, the most prominent theme is "children" with other relevant themes and issues. It is a great way to check if children are the most common theme in your report, proposal or presentation. The link to Wordle is: http://www.wordle.net/

Measuring Advocacy and Policy Outcomes

One of the more challenging aspects of projects in Save the Children is measuring IR-4 performance (Enhanced Enabling Environment), especially if advocacy and policy outcomes are envisioned. This challenge was addressed by various authors and organizaitons in The Evaluation Exchange (Vol. XIII, No.1, Spring 2007), sponsored by Harvard Research Project.

In this issue of The Evaluation Exchange the authors attempted to define advocacy and policy change and how to evaluate this change. Excerpts from one article, which presents a illustrative menu of outcomes and strategies for various types of advocacy and policy objectives, are below.

Objective: Shifts in social norms. Social norms are the knowledge, attitudes, values, and behaviors that comprise the normative structure of culture and society. Advocacy and policy work and intervention increasingly has focused on this area because of the importance of aligning advocacy and policy goals with core and enduring social values and behaviors.

Examples of outcomes
+ Changes in awareness
+  Increased agreement about the definition of a problem
+  Changes in beliefs
+  Changes in attitudes
+  Changes in values
+  Changes in the salience of an issue
+  Increased alignment of campaign goal with core societal values
+  Changes in public behavior

Examples of strategies to achieve these outcomes
+  Framing issues
+  Media campaign
+  Message development (e.g., defining the problem, framing,
    naming)
+  Development of trusted messengers and champions 

Objective: Strengthened organizational capacity. Organizational capacity is another name for the skill set, staffing and leadership, organizational structure and management systems, finances, and strategic planning of nonprofits and formal coalitions that do advocacy and policy work. Development of these core capacities is critical to advocacy and policy change efforts.

Examples of outcomes
+ Improved management of organizational capacity of organizations
   involved with advocacy and policy work

+  Improved strategic abilities of organizations involved with
    advocacy and policy work
+  Improved capacity to communicate and promote advocacy
    messages of organizations involved with advocacy and policy work
+  Improved stability of organizations involved with advocacy and
    policy work

Examples of strategies to achieve these outcomes
+ Leadership development
+ Organizational capacity building
+ Communication skill building
+ Strategic planning

Objective: Strengthened alliances. Alliances among advocacy partners vary in levels of coordination, collaboration, and mission alignment and can include nontraditional alliances such as bipartisan alliances or relationships between unlikely allies. Alliances bring about structural changes in community and institutional relationships and are essential to presenting common messages, pursuing common goals, enforcing policy changes, and protecting policy “wins.”

Examples of outcomes
+ Increased number of partners supporting an issue
+ Increased level of collaboration (e.g., coordination)
+ Improved alignment of partnership efforts (e.g., shared priorities,
   shared goals, common accountability system)
+ Strategic alliances with important partners (e.g., stronger or more
    powerful relationships and alliances)
+ Increased ability of coalitions working toward policy change to
   identify policy change process (e.g., venue of policy change, steps
   of policy change based on strong understanding of the issue and
   barriers, jurisdiction of policy change)

Examples of strategies to achieve these outcomes
+ Partnership development
+ Coalition development
+ Cross-sector campaigns
+ Joint campaigns
+ Building alliances among unlikely allies

Objective: Strengthened base of support. Nonprofits draw on grassroots, leadership, and institutional support in working for policy changes. The breadth, depth, and influence of support among the general public, interest groups, and opinion leaders for particular issues are a major structural condition for supporting
policy changes. This outcome category spans many layers of culture and societal engagement including increases in civic participation and activism, “allied voices” among informal and formal groups, the coalescence of dissimilar interest groups, actions of opinion leader champions, and positive media attention.

Examples of outcomes
+ Increased public involvement in an issue
+ Increased level of actions taken by champi ons of an issue
+ Increased voter registration
+ Changes in voting behavior
+ Increased breadth of partners supporting an issue (e.g., number
   of “unlikely allies” supporting an issue)
+ Increased media coverage (e.g., quantity, prioritization, extent
   of coverage, variety of media "beats,” message echoing)
+ Increased awareness of campaign principles and messages
   among selected groups (e.g., policymakers, general public,
   opinion leaders)
+ Increased visibility of the campaign message (e.g., engagement
   in debate, presence of campaign message in the media)
+ Changes in public will

Examples of strategies to achieve these outcomes
+ Community organizing
+ Media campaigns
+ Outreach
+ Public/grassroots engagement campaign
+ Voter registration campaign
+ Coalition development
+ Development of trusted messengers and champions
+ Policy analysis and debate
+ Policy impact statements

Objective: Improved policies. Change in the public policy arena occurs in stages—including policy development, policy proposals, demonstration of support (e.g., co-sponsorship), adoption, funding, and implementation. Advocacy and policy evaluation frequently focuses on this area as a measure of success. While and important focus, improved policies are rarely achieved without changes in the preconditions to policy change identified in other outcome categories.

Examples of outcomes
+ Policy development
+ Policy adoption (e.g., ordinance, ballot measure, legislation,
   legally binding agreements)
+ Policy implementation (e.g., equity, adequate funding, other
   resources for implementing policy)
+ Policy enforcement (e.g., holding the line on bedrock legislation)

Examples of strategies to acheive these outcomes
+ Scientific research
+ Development of “white papers”
+ Development of policy proposals
+ Pilots/demonstration programs
 + Educational briefings of legislators
+ Watchdog function

Objective: Changes in impact. Changes in impact are the ultimate and long-term changes in social and physical lives and conditions (i.e., individuals, populations, and physical environments) that motivate policy change efforts. These changes are important to monitor and evaluate when grantmakers and advocacy organizations are partners in social change. Changes in impact are influenced by policy change but typically involve far more strategies, including direct interventions, community support, and personal and family behaviors.

Examples of outcomes
+ Improved social and physical conditions (e.g., poverty,
   habitat diversity, health, equality, democracy).

Outcome Mapping

Outcome Mapping: building learning and reflection into development programs (2001), is a book by Sarah Earl, Fred Carden and Terry Smutylo, with a forward by Michael Quinn Patton. Outcome Mapping focuses on intermediate results (outcomes) of change in behavior, relationships, activities, or actions of people or groups; thus the focus is on people rather than things such as cleaner water or improved economy.

Outcome Mapping is most effective when used at the planning stage of a project or program. The parts of the Outcome Mapping exercise can then be adapted into a Results Framework or Logical Framework. And, importantly, successful Outcome Mapping requires commitments in knowing the strategic direction of the project, type of monitoring and evaluation data needed, reporting, participatory learning, team consenus, and resource commitments.

Outcome Mapping has 3 Stages and 12 Steps:

The outline of the book is:
1. Outcome Mapping: The Theory
2. Outcome Mapping: The Workshop Approach
3. Stage 1: Intentional Design
4. Stage 2: Outcome & Performance Monitoring
5. Stage 3: Evaluation Planning
Appendix A: Sample Intentional Design Framework
Appendix B: Overview of Evaluation Methods
Appendix C: Glossary
Appendix D: Terms in French, English, Spanish

If this book sounds useful, you can download a PDF version of it under the DME Documents section to the right.

Matching Results Framework and Logical Framwork Terminology

Long before the need to monitoring and evaluate a project/program is the fundamental need to design a project/program. The acronym, DME, as in DME Advisor refers to design, monitoring and evaluation. Yes, I like to get involved in the initial design of a project, which is important in how it will monitored and evaluated.

There are two basic project/program designs being used in SC at this time. There is the Results Framework (RF), recommended by SC and generally used by US and Canadian-based donors. In addition, as more funding comes from non-US sources, the Logical Framework Approach (LFA), which is generally used by European donors.

Since SC primarily uses the RF most project/program directors or manager are familiar with it; however, increasingly they are being asked to use the LFA.

There are basic differences between the two design approaches, of which the two main are format and terminology. The basic format of a LFA is that of a matrix, whereas the basic format of a RF is a graphic illustration.

The largest challenge though for staff is the terminology differences. Below I have tried to match the RF and LFA terminology as closely as possible. Of course, there are slightly different versions of the RF and LFA, so this table is for the generic versions of both approaches.

Results Framework                     Logical Framework

Goal                                              Long-term Objective/Goal

Strategic Objective (SO)                Purpose/Short-term Objective

Intermediate Results (IRs)              Outputs

Strategies                                       ----

Activities                                        Activities

----                                                Inputs

----                                                Risks/Assumptions

Benchmarks                                   Milestones

Targets                                          Targets

Let me know if there are some terminology I'm missing or your think I've mismatched.

Narrative Methods of Project/Program Evaluation

Have you been involved in a project/program in which a rigorous project/program evaluation was not possible or simply not wanted? For such times there are other methods to conduct evaluations, especially those with few pre-determined indicators, difficulties in implementing rigorous studies, or with the possibiliity of many unforseen or unitended results/outcomes.

One broad approach to evaluating projects/programs is called the "narrative" method. The narrative method is described by Charles McClintock (Dean of the Fielding Graduate Institute’s School of Human and Organization Development, in his article "Using narrative methods to link program evaluation and organization development" published in The Evaluation Exchange Volume 9, Number 4, Winter 2003/ 2004 by the Harvard Family Research Project).

The narrative method is fundamentally storytelling and is related to participatory change processes because it relies on people themselves to make sense of their own experiences as it relates to the project/program. And, the participant and beneficiary's stories can be systematically gathered and claims verified from independent sources or methods.

The narrative method can be divided into three basic types, depending on the purpose of the evaluation.

1. Success stories
2. Positive and negative outcomes
3. Emerging themes

1. Success Stories: One of the most prominant narrative methods for the purpose of success stories related to intermediate outcomes and impact is Most Significant Change, or MSC (Davies and Dart 2003, see MSC document list on the right tab). This method is highly structured and designed to engage stakedholders at all levels. Davies and Dart recommend MSC when a project or program is:

• complex and produce diverse and emergent outcomes 
• large with numerous organisational layers 
• focused on social change 
• emphasizes participation 
• designed with repeated contact between field staff and participants 
• struggling with conventional monitoring systems 
• highly customised services to a small number of beneficiaries (such as family counselling). 

2. Positive and negative outcomes: this narrative method is called the Success Case method (Brinkerhoff, 2003). This Success Case method has two phases: a) a short questionnaire sent to all project/program participants to identify those for whom the project/program has made a difference and those for whom it did not make a difference; b) next, a number of extreme cases are selected from those two ends of the success continuum (i.e., did and did not make a difference) and respondents are asked to tell stories about both the features of the project/program that were or were not helpful as well as other factors that facilitated or impeded success. Based on the logic of journalism and legal inquiry, independent evidence is sought during these storytelling interviews that would corroborate the success claims.

These stories serve both to document outcomes, but also to guide management about needed change in project/program interventions that will accomplish higher level outcomes and impacts.

3. Emerging Themes: This narrative methods is basically qualitative case studies (Costantino & Greene, 2003). Here, stories are used to understand context, culture, and participants’ experiences in relation to program activities and outcomes. As with most case studies, this method can require site visits, review of documents, participant observation, and personal and telephone interviews. Stories can include verbatim transcripts, some of which contained interwoven mini stories. Selection of a few cases, studied in-depth, are able to develop many more "themes" that are involved in a project/program and of relationships among participants and staff.