On Home Leave (to the US) Until July 2010

I will be taking my annual Home Leave to the US during the month of June, 2010. When I return home to Tbilisi, Georgia, in July 2010 I will resume my program/project design, monitoring and evaluation blog.

Competencies Needed by Evaluators

In 2001, the American Journal of Evaluation published an article by Jean A. King, Laurie Stevahn, Gail Ghere and Jane Minnema titled, "Toward a Taxonomy of Essential Evaluator Competencies," (22; 229). These authors conducted an exploratory study to determine the extent to which 31 evaluation professionals from diverse backgrounds and approaches could reach agreement on a proposed taxonomy of essential evaluator competencies.

Using the weighted scores, I entered the competencies into Wordle to get a graphic representation of these competencies. Those competencies in large font size were weighted as more important than those in small font, though all were considered important.

The most important competencies were Framing a Research Question, Research Methods and Research Design. Of comparatively less important were competencies of Training Others, Supervising, and Responding to RFPs (Request for Proposals).

Quotes Related to Evaluation

• True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information. Winston Churchill
  
• The only man who behaves sensibly is my tailor; he takes my measurements anew every time he sees me, while all the rest go on with their old measurements and expect me to fit them. George Bernard Shaw
  
• Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted. Albert Einstein
  
• We cannot discover what ought to be the case by examining what is the case. We must decide what ought to be the case. Paul Taylor
  
• The most serious mistakes are not being made as a result of wrong answers. The truly dangerous this is asking the wrong question. Peter Drucker
  
• One of the great mistakes is to judge policies and programs by their intentions rather than their results. Milton Friedman
  
• The pure and simple truth is rarely pure and never simple! Oscar Wilde
  
• First get your facts; then you can distort them at your leisure. Mark Twain
  
• My mind is made up, don't confuse me with the facts! Unknown
  
• A funeral eulogy is a belated plea for the defense delivered after the evidence is all in. Irvin S. Cobb
  
• There's a world of difference between truth and facts. Facts can obscure the truth. Maya Angelou
  
• It is easier to believe a lie that one has heard a thousand times than to believe a fact that no one has heard before. Unknown
  
• We want the facts to fit the preconceptions. When they don't it is easier to ignore the facts than to change the preconceptions. Jessamyn West
  
• If at first you don't succeed, destroy all evidence that you tried. Unknown
  
• Absence of evidence is not evidence of absence. Carl Sagan
  
• Evaluate what you want -- because what gets measured, gets produced. James Belasco
  
• Price is what you pay. Value is what you get. Warren Buffett
  
• For changes to be of any true value, they've got to be lasting and consistent. Tony Robbins
  
• Extraordinary claims require extraordinary evidence.Carl Sagan

Country-Level Monitoring & Evaluation Dashboard

This past week (2-7 May 2010) I worked with a great team in SC's Cairo office to sketch-out a system that would provide the Senior Management Team (SMT) a consolidated "dashboard" report of key project
indicators on a quarterly basis.

The goal of the dashboard report is to provide the SMT an at-a-glance view of the progress of all projects to meet 1) quarterly benchmarks on key indicators, 2) successes and challenges for that quarter, 3) the burn-rate of project funds, 4) as well as number of beneficiaries reached, thus allowing better informed decision-making by the SMT on project improvement BEFORE the end of the reporting year or the project.

Like the dashboard of a car, this Dashboard Report will carry only those indicators that the SMT consider "key" to measuring the accomplisment of the project NOT ALL indicators.

Example of a Dashboard Report

A Dashboard Report is primarily based on comparing quarterly achievements with quarterly benchmarks for activity-level indicators. The Dashboard Report will present result-level indicators on an annual basis. Therefore, if projects do not have end-of-project targets and established quarterly (or semi-annual) benchmarks then developing a meaningfuly Dashboard Report to show process is more difficult.

In addition, identifying over or under achievment of quarterly benchmarks, or annual targets, will only be the beginning of the process; if, for example, there is an under achievement of a quarterly benchmark, then the SMT can "drill-down" deeper with the Project Director as to possible reasons which can be discussed and hopefully resolved before the next quarter.

A Dashboard Report should be simple and easy to read; however, this takes a lot of work! Behind a good Dashboard Report is consensus of key indicators, systems to ensure data quality, adequate staff resources, long-term SMT committment, userfriendly IT tools, clearly defined roles and flow of data, and project with M&E plans that clearly state end-of-project targets to be achieved and quarterly or semi-annual bechmarks to track progress toward the targets.

The Egypt Country Office will be working on developing these systems and has planned the 1st draft of a Dashboard Report this summer. Once it is developed I will present it here.

Barriers to Learning within NGOs...as well as Regional & Country Offices

Just as individuals must learn in order to survive and grow in the new complexities of a global environment, so must organizations. There are different motives for learning: profit (maximize monetary gain), value (emphasis on ethics and reconciling interests), and altruism (doing good for others).

I would say that most people working in NGOs are motived by altruism. Organizations employing staff who are primarliy movtivated by altruism is positive but can negatively affect organizational learning in specific ways. The organization, Networking for International Development, conducted a survey of NGOs regarding organizational learning and published a report called, "Working with Barriers to Organisational Learning."

The focus of the report is on barriers to that seem to limit learning within NGOs. Ten barriers are discussed:

1. Bias for Action - often altruistic motives can lead to an activist tendency in which staff feel that there is not time to slow down, clarify issues, and reflect on what is happening.
2. The Undiscussables - altruism to do good can lead to the tendency to avoid issues because of fear of upsetting others or to avoid conflict.
3. Commitment to 'the Cause'- similar to the Bias for Action above, this is a sense that the altruistic 'cause' has to be achieved and that taking time to reflect may lead to questionning of what is being done may not achieve the ultimate goal.
4. A Cultural Bias - many people working in international NGOs come from the US or western cultures. The dominant culture of the organization may ignore other means and methods of interaction, discussion, reflection and learning.
5. Advocacy at the Expense of Inquiry - the altruistic urge can lead to emphasis on advocating and defending a position at the expense of learning about other views.
6. The Role of Leadership - combining most or many of the points above, organizational leaders often set the tone for what is acceptable forms of questioning, inquiry, interaction all of which affects overall learning.
7. Learning to Unlearn - possibly surrounded by others who have similar altruistic motives, the trap of doing what is easy because of habits and assumptions that have been relied on for years with few challenges of learning new habits.
8. Practicing What We Preach - as part of the altruistic zeal there can be a tendency to promote processes, methods and practices that the organization itself does not do.
9. The Funding Environment - altruism often relies on outside funding and all too often funding (ie., donor) limits the extent of innovating and testing as well as enacting change based on learning.
10. Thinking Strategically About Learning - even though many NGO staff are motivated by altruism, there can still be a tendency to be competetive, which results in a core practice of learning being an "internal" activity with little priority placed on learning from peer organizations.

Responding to Evaluation Findings

John Scott Bayley, an Evaluation Specialist at the Independent Evaluation Department in the Asian Development Bank published an article in the Evaluation Journal of Australasia about Handy Hints for Program Managers.

Though meant to be both lighthearted as well as serious, one of his handy hints for Program Managers is if they feel threatened by the results of an evaluation study they should consider responding with one of the following strategies:

1. Attack the evaluation’s methodology; 
2. Attack the data's interpretation and resulting conclusions;
3. Attack the evaluation’s assumptions; 
4. Attack the recommendations;
5. Substitute previously unstated goals for the official program goals;
6. Attack the evaluators personally, claim they are biased or unfamiliar with the program;
7. Attack the evaluation’s key issues and research questions; 
8. Do not participate in the evaluation, but argue that the findings lack an adequate contextual background;
9. Rally together those who are threatened by the findings; 
10. Indicate the findings are reasonable, but unable to be implemented due to a lack of resources, political opposition, the staff need training etc;
11. Complain about a lack of consultation;
12. Argue that the evaluators did not appreciate the subtleties of the program; 
13. Simply pretend that the evaluation never occurred, ignore it; 
14. State that the program’s environment has changed, and the findings are no longer relevant; 
15. Stall for time until the evaluation is forgotten about; 
16. Argue that the union and staff will not accept the recommendations; 
17. Argue that while the program has not achieve it's goals, it does achieve other important things that are too subtle to be easily measured;
18. Say that the evaluation leaves important questions unanswered, its' significance is questionable; 
19. Argue that the data is open to alternative interpretations, the evaluation’s conclusions have been questions by others;
20. Attack the steering committee;
21. Claim that the results contradict commonsense experience, and testimonials from clients;
22. Claim that the findings are contradicted by other research conducted by various experts in the field; 
23. Agree with the findings and indicate that you have known about this for some time, and you started making changes months ago;
24. Argue that the findings contradict the spirit and philosophy of the dept/program;
25. Make up quotations that support your case and attribute them to knowledgeable sources; and
26. Argue about definitions and interpretations.

Some others that I have heard that are not listed above are:

1. Argue that the project did not have a sufficient budget to monitoring the results and thus cannot be held responsible for not achieving them.
2. Argue that the results the project was trying to achieve are so unique that they are not measurable.
3. Argue that the "real" results of the project will occur years after the evaluation.

Development Assets of Yemeni Youth

ave the Children (SC) has been working in Yemen since 1963 with programming in education, health, child protection and civil society. In 2008, SC was awarded a grant from the United States Agency for International Development (USAID) for a Youth Empowerment Program (YEP), which operates in the governorates of Sa'ana, Ibb, Aden and Abyan.

In late 2008 to early 2009, I was involved in an assessment to better understand how youth in Yemen are fairing personally and socially, which would help inform the programming in the YEP program. One of the measures we used to assess how youth were doing was the well regarded Search Institute's Developmental Assets Profile (DAP).

The DAP consists of 58 questions that comprise two domains, Internal and External Assets. The Internal Assets domain is comprise of four sub-scales: Support, Empowerment, Boundaries & Expectations, and Constructive Use of Time. The External Assets domain is also comprised of four sub-scales: Commitment to Learning, Positive Values, Social Competencies, and Positive Identity.

It took about 3 months to adapt the tool to the Yemen context. Adapting the DAP involved initial translation from a Egyptian Arabic version (from a another project) by Yemen project staff, pilot-testing this version with youth via individual interviews and focus group discussion which highlighted issues and wording that required further refinement. Once we had an Arabic version that was adapted for youth in Yemen, a pilot-test was conducted with youth who completed the entire DAP. The results were analyzed for internal reliability (based on Alpha reliability) and temporal stability over a 1-week period of time (Pearson correlation). The internal and temporal reliability test were satifactory, a random sample of 600 youth in the four governorates were interviewed in their homes away from parents and siblings as much as possible. Each question is answered by youth using a scale ranging from "not at all" (=0) to a high of "almost always" (=3) regarding the presence of various situations and conditions in their life in the previous 3 months. The scores are totaled and the developmental assets are categorized as low, fair, good, or excellent.

The graph below present the results, average scores, for each of the four sub-scales that comprise External and Internal Assets. The average scores for each sub-scale has been connected with a line to provide a profile.

Several findings are quickly apparent about how Yemeni youth are fairing personally and socially. First, oveall, Yemeni youth have few constructive opportunities as indicated by the sub-scale Constructive Use of Time having the lowest scores regardless of location. Second, where a youth lives influences their level of developmental assts as show by the substantial differences on these sub-scales depending on which governorate the youth lives (each of the lines represents a governorate). Third, that the Yemeni youth are fairing well on Internal Assets but not so well on External Assets.

The implications of these findings for programming are that youth need more constructive opportunities and outlets. Currently, there are few opportunites in schools, neighborhoods or communities for youth to be involved in structured activities such as sports, music, mentoring, drop-in centers, social groups, or camps. This is particulary the case for youth living in the interior (Sana'a and Ibb), whereas youth living along the coast are more likely to be involved in activities such as boating or fishing. Being involved in constructive activities has been shown to improve empowerment through increasing self-esteem, providing a sense of belonging, developing cognitive, physical and social skills, enhancing a sense of self-worth, and developing relationships.

Although not shown in the graph, further analysis of the DAP shows that Yemeni youth reported low scores for schooling and neighborhood safety. For schooling, these youth did not feel that the schools they attend enforce rules fairly, cares about them, or encourages them to do their best. Thus, empowering youth requires improving the quality of schools. For neigborhood safety, basically, youth felt that neighbors do not help watch out for them, which means working with neighbors to also empower youth in Yemen.