Why does SC use a Results Framework?

Often, country offices ask me why Save the Children recommends a Results Framework rather than some other type of program/project design tool, such as Logical Framework. Several employees of Save the Children (see attached article on the right titled, “A Results Framework Services Both Program Design & Delivery Science” under the Documents section).

Some of the reasons these authors cite include:

1. The entire program/project logic and “theory of change” can be visually grasped without extensive reading.

2. Different disciples or technical specialists (health, food security, livelihoods, education) can use the same
    basic model.

3. The ability to clarify assumptions as well as state hypotheses.

4. Facilitates in the design programs and projects.

5. Helps in the evaluation designs.

6. Informs action research

The Results Framework has the following components:
Goal- a) States the long-term end status that is to be achieved, b) Usually expensive to measure since it requires large population-based surveys.
Strategic Objective (SO) – a) Is the most ambitious result that programs can reasonably effect and for which implementing agencies are willing to be held accountable.
Intermediate Results (IRs) – These are essential steps toward achieving the SO. Save the Children recommends the use of the following 4 IRs, since SC’s programming is based on behavior change:

   IR-1: Availability & Access (as service must be available as well as spatially and economically accessible)

   IR-2: Quality (services meet technical as well as client perceived standards)

   IR-3: Demand (knowledge, skills, attitudes, or beliefs that hinder or promote service usage)

   IR-4: Enabling Environment (facilitates both the supply and demand side of services)
IR Strategies – specific steps to achieve the Intermediate Results
IR Activities – specific program/project activities related to each IR strategy.

Results Framework (Health Example)

What are some of the limitations of the Results Framework?

1. IR2, Quality, has many dimensions, such as technical (i.e., meeting national or international standards) and perceived (client’s perception of quality services). However, this model combines both types into one box even though these are separate dimensions.

2. The Enabling Environment (IR4) is often highly related to achieving IR3, a change in demand via more informed clientele.

3. The Results Framework, unlike the Logical Framework, omits external environmental factors (apart from those in IR4) that can ease or constrain achieving the results and that are beyond the reach of programmers.

4. Finally, the framework lacks the operational details (such as those found in Logical Frameworks) that managers and some donors need; however, standard detailed implementation and monitoring plans that are based on the framework provide these.

Overall, the Results Framework is a simplistic way to illustrate the relationships between higher level Goals all the way down to activities for small as well as large, complex programs/projects regardless of the sector. But, as the authors conclude, that simplicity has its disadvantages.

US White House announcement: Program Evaluations

In an 7 October 2009 memorandum, issued by the Executive Office of the US President, the issue of program evaluation is emphasized (http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-01.pdf). In short, an inter-agency working group of evaluation experts under the Performance Improvement Council established will be revived. The purpose of the working group will be: (a) to help build agency evaluation capacity and create effective evaluation networks that draw on the best expertise inside and outside the Federal government; (b) to share best practices from agencies with strong, independent evaluation offices; (c) to make research expertise available to agencies that need assistance in selecting appropriate research designs in different contexts; (d) to devise strategies for using data and evaluation to drive continuous improvement in program policy and practice; and (e) to develop government-wide guidance on program evaluation practices across the Federal government while allowing agencies flexibility to adopt practices suited to their specific needs. A key goal of the working group will be to help agencies determine the most rigorous study designs appropriate for different programs given their size, stage of development, and other factors.

The question to me is: Will this also apply to programs receiving US foreign aid via USAID during the Obama administration? Is so, when?

Reconstructing a baseline via retrospective pretest

A common approach to project evaluation is using a pre- and post-test among beneficiaries. Although this type of design is not very rigorous (I will write about why in a later post), occasionally there are training projects that get started so quickly that only at the end of the training do staff realize that they did not conduct a pretest and subsequently do not have baseline to compare a change in knowledge from the training course.

In an article by Debra Moore and Cynthia Tananis (American Journal of Evaluation, 2009) these authors discuss the issue of not only how to reconstruct a baseline but also the validity and reliability of data when doing so. This method is called a retrospective pretest design.

Now, the authors clearly state that this is a method best used with short-term, intensive training programs and may not be as reliable in other types of activities and interventions.

The authors mention that in a both a pretest and posttest design, or a retrospective pretest design, one of the primary concerns is something called response-shift bias. Response-shift bias occurs when a participant understands the concept being measured at the pretest differently than at the posttest. For example, youth asked on pretest to answer questions about empowerment (the concept) before the training begins may answer differently when asked the same questions about empowerment at the posttest when the training ends because after taking a training course they understand the concept of empowerment differently. Thus, the authors wanted to test if the degree of response-shift bias when a pretest and posttest was conducted for a training course and when a pretest was NOT done and a retrospective pretest was used.

The basic research question was: Do participant’s responses more accurately represent their level of knowledge/awareness at the beginning or after the training?

For example, before taking a training course on DME I many think I know a lot and would willing respond on pretest questionnaire high levels of knowledge and abilities in doing DME. Then after taking a DME course, and being exposed to more detailed and complex issues that I was not previously aware of, I may reassess that my level of knowledge and abilities were not as great as I thought. But, sadly it’s too late to change my pretest responses.

The authors conclude that pretest scores tend to overestimate a particular level of knowledge or ability(larger response-shift bias) than with a retrospective pretest. They also report that other studies have found that self-report retrospective pretest scores are more highly correlated with scores on objective pretest measures of skill development or knowledge than the self-report pretest scores.

The basic message: In projects that include short, intensive training courses, reconstructing a baseline through the use of a retrospective pretest conducted at the end of the training may provide more accurate results than a pretest at the beginning of the training course.

Youth Livelihood Developmental Index (YLDI): Measuring Youth Assets and Competencies

For my first blog, I thought I would review several tools that I have been involved with the Egypt Country Office in measuring outcomes for their youth livelihoods projects, and has been used in Yemen, as well as in the Africa and Asia regions.

The YLDI is a set of three tools: 1) the Developmental Assets Profile referred to as the DAP, 2) the Livelihoods Competencies Profile referred to as the LCP , and 3) the Tangible Assets Profile referred to as the TAP. Each of these tools is a standardized index to measure the level of assets, competencies and resources that youth have at a given point in time.

The DAP was developed by the Search Institute and contains 58 questions that can be completed either by the youth themselves or in groups. The scores are totaled and the levels of the 40 developmental assets are categorized as low, fair, good, or excellent. Profiled results portray the type and degree of developmental assets among the youth. The DAP produces quantitative scores for two domains (External and Internal Assets) and on eight asset categories (Support, Empowerment, Boundaries & Expectations, Constructive Use of Time, Commitment to Learning, Positive Values, Social Competencies, and Positive Identity) as well as four Context Area domains (Personal, Family, School, Community, Social).

The LCP was developed by Global Youth Livelihoods and contains 69 questions that can be completed individually by the youth or in a group. The LCP is designed to measure a youth’s self-assessment of the level to which s/he possess one or more of 17 basic competencies needed to generate or maintain an income and livelihood. These competencies are grouped into four domains: Human Capital, Social Capital, Financial Capital, and Physical Capital. The scores are totaled and the levels the youth possess of the four types of capital are categorized as low, fair, good, or excellent. Profiled results portray the type and degree of livelihood competencies among the youth.

The TAP, again developed by Global Youth Livelihoods, contains 32 questions that can be completed individually by the youth or in a group. The TAP is designed to measure a youth’s self-assessment of the level to which s/he possess or has access to 8 tangible assets that can generate or maintain an income and livelihood. The 8 tangible assets are grouped into two domains: Financial Capital and Physical Capital.

The YLDI is accompanied by a database template that allows for easy data entry and some basic data analysis and reports.

Due to the total number of questions for all three tools they are administered at separate times. These three tools that comprise the YLDI can be used for three general objectives:

• To assess the status (prevalence) of developmental and livelihood assets of youth in a given area to assist with new program/project design;
• To evaluate a program or project via a baseline and end-line survey to measure developmental asset and livelihood competency outcomes and or results;
• To establish youth profiles based on various characteristics so as to better tailor program/project activities and interventions.

To date, the YLDI has been used in Upper Egypt for a project evaluation (with Mona Moneer) and in Yemen (with Lucienne Mass) for a general assessment to develop youth livelihood programming.

I have attached the English versions of the DAP, LCP and TAP (in the Documents list to the right). HOWEVER, please do not translate or use because they are copyrighted materials. If you would like to use them in a project contact either me or Sita Conklin (MEE Livelihoods Advisor).

If you have a livelihoods project or are considering a livelihoods component in a future project, and are interested in learning more about the results of the use of the YLDI in Upper Egypt or Yemen, feel free to contact me.

In my next post I will discuss how to reconstruct a baseline for a training program at the end of a project!