Project Evaluation: Most Robust Quantitative Evaluation Design #1

I will be starting a series on 7 different types of quantitative project evaluations, from the strongest (or most rigorous) to the weakest (or least rigorous) designs, that use are based on a quasi-experimental design (i.e., randomization is not used but rather the use of a matched comparison group). These seven quantitative project designs are discussed in more detials in the book, RealWorld Evaluation: working under budget, time, data and political constraints, by Michael Bamberger, Jim Rugh, and Linda Mabry (2006).

The most robust or strongest quantitative project design has also the longest name: Comprehensive longitudinal design with pre-, mid-term, post- and ex-post observations on the project and comparison groups. This design is one of the strongest quantitative project evaluation designs but also the most time consuming and expensive.



As shown in the diagram above, there are several characteristics of this design that make it one of most rigorous, but also expesive and time consuming. First, data is collected at 4 points of time (1-Baseline, 2- Mid-Term, 3-End-line and 4-After Project). In addition, it involves data collection among two groups: those people/households that are involved in the project as well as a match compartive group who are as similiar as project participant BUT who are NOT involved or effected by the project.

The reason for the matched, comparative group is to establish what is called the "counter factual", which attempts to answer the question: "What would have happened to these individuals/households IF the project had not occurred?" Thus, any differences between the project participants and the matched group at the end of the project is estimated to be the impact of the project. The reason the 4th data collection point (After Project Study) is included is to understand the "trajectory" or sustainabilty of any results or impact(s); that is, do the results tend to increase, level off or decrease over time.

There are limitations to being able use this type of design, which is why it is not very often used. First, as mentioned earlier, it requies more time and costs to collect data at 4 points in time and among two groups. Second, the sample size must be relatively large to account for loss or attrition of group members over this period of time. Third, data management and analysis can be a challenge. Fourth, due to the longer period of time, there is potential for larger macro-level influences, such as policy changes, that can affect results.

Despite these limitations and challenges, this design should be considered in new projects, or projects that want to scale-up regionally or nationally, to clearly demonstrate project interventions produce the expected outcomes and results, as well as how sustainable the results are.