The next set of project evaluation designs that I will presenting are considered weaker than the previous designs; weaker meaning that they can not provide good evidence that a project's interventions are directly attributable to the measured outcomes or to what degree the project contributed to the measured outcomes (net impact).
The pre-test with a project group combined with a post-test of both project and comparison group design is shown below. As the name implies, the pre-test (baseline study) is conducted only with the people involved in the project. There could be a number of reasons why this might occur: save money, project team didn't like the idea of a comparison group initially, or some technical reasons. However, later there may be the budget, interest or feasibility to include a comparsion group.
The advantages of this design are that it can assess reasonably well how a project is being implemented and whether intended outputs from activities were produced. IF the comparison group studied at the end of the project is quite similar to the project group in characteristics, and adequate mixed methods can demonstrate the comparison group were similar at the baseline as the project group on the outcomes that will be measured, then this design MIGHT demonstrate project effects.
The disadvantages are that is that it is difficult to conclusively determine if the differences between the project and comparison group at the end-line study are due to the project or other factors. Another weakness of this design is that local context events can effect outcomes in the comparison group. For example, an agricultural project compares agricultural output of its farmers to a comparison group of farmers without knowing that the the comparison group of farmers received more irrigated water than previous years thus increasing their output. To monitor local context events, retrospective information can be obtained from the comparison group.
Hi Larry,
ReplyDeleteCompliments on a nice blog! I have one on similar topics in Dutch, but no more very active.
I work with Woord en Daad in exactly the same field: design of M&E systems together with colleagues and partners in several countries. We recently widened M&E to include Planning an Learning to PMEL.
ditto. excellent blog. concise and clear writing. please keep it up!
ReplyDeleteThanks Wouter for you compliments. Save the Children...for whom I work and this blog is generally targeted toward...has considered Planning and Learning added to M&E, but Planning is included in the Design of "DME" (I made this blog only M&E to shorten the name) and Learning is included as "Lessons Learned "in both monitoring and evaluation, not as a separate component. That said, in a DME capacity assessment tool I'm designing Learning is a separate component! :-)
ReplyDelete