Evaluating Aid Policies and Specific Initiatives. Applications at macro-, meso- and micro-levels of analysis

By Laura Fantini, Sarah Hassan

The Working Group on Evaluation and Development held two sessions titled Evaluating Aid Policies and Specific Initiatives. Applications at macro-, meso- and micro-levels of analysis”. Both sessions have a twofold aim: to stimulate a debate around innovative approaches to strategic evaluation and to facilitate a rethinking about what such an evaluation should be.

The debate took into account the changing international context and the evolving development policies and mechanisms of development assistance, focusing the attention on the evaluation of complex programs, policies, and specific initiatives, and on cross-cutting issues (such as environmental sustainability, climate change, and good governance).

The first session held on 24th of June hosted five contributions was chaired by Sara Hassan from Cespi, Rome. The first one “Evaluating budget support to fighting Climate Change: is the standard evaluation framework relevant?“ was presented by Marc Raffinot from Université Paris Dauphine. It focused on the three steps approach by OECD DAC for evaluating Budget Support (BS) on poverty reduction and growth. It tested the relevance of such approach in evaluating BS on climate change mitigation and adaptation in middle-income countries and proposed an adapted methodological framework.

Another case of evaluating BS was presented by Johannes Schmitt from the German Institute for Development Evaluation. His paper “Tracing mechanisms as a means to evaluate the governance-effectiveness of Budget Support”, starting from the assumption that BS follows a non-linear program logic in which multiple donors provide multiple inputs in order to achieve multiple outcomes, aims to answer to the following question: how can theory-based approaches better incorporate the analysis of causal mechanisms and increase their capacity for causal inference?

The paper suggested that an in-depth Process-Tracing of mechanisms linking the intervention with outcomes is a particularly well-suited case study tool for evaluating the governance-effectiveness of BS as well as other forms of complex policy interventions.

The third paper “Evaluating Aid for Good Governance” was presented by Karin ter Horst from the Departmentof Policy and Operations Evaluation of the Ministry of Foreign Affairs, Netherlands. The paper investigated how the governance interventions at the micro level supported by international donors in developing context are evaluated and what do we know on the results of these interventions. The paper focuses in particular, on projects aiming to promote the rule of law.

Laura Fantini, from Sapienza University of Rome, presented the fourth contribution titled “Positive thinking approaches and learning in evaluative research. The case of a climate change adaptation initiative in Dar es Salaam”. The paper aims to examine which kind of learning could be provided by using the Positive Thinking approach in evaluating the specific local context to inform the (re)design of an action-research project in the city of Dar es Salaam (Tanzania) focused on supporting local government authorities capacities on climate change adaptation. The study shows how detecting “positive cases” offer a new understanding of the situation, strengthening or undermining the assumptions of the intervention and helping to (re)design the project conceptual framework

The last presentation from Andrea Ferranini (ARCO Lab, Florene – Italy) was titled “Assessing aid effectiveness within a multilevel governance perspective: the case of a value chains project in rural Ethiopia”. The purpose of the paper is twofold: to place the discussion on aid effectiveness and Paris principles within a multilevel governance perspective, and to evaluate ownership, alignment, harmonization, managing for results and mutual accountability in terms of dynamic complex processes rather than simply outcomes, thus entailing the need for participatory evaluations in a mixed-methods perspective.

The second session held on 25th of June hosted six presentations and was chaired by Laura Fantini from Sapienza University of Rome. The first presented paper, “How to choose methods for evaluation research: a test of the relevance of systematic review” was written by Aogán Delaney, an independent consultant. It aimed at assessing the potentialities of a specific methodological tool in evaluation, reporting on a test of the suitability of systematic review for the identification of research methods appropriate for evaluation research on the effects of primary health care delivery on peace and conflict dynamics in conflict-affected regions. It was argued from the paper’s discussants that such an approach could determine a bias due to the sample’s selection, as it excludes non – English language sources, that in any case represent around 10% of the total production on the topic.

Dawit Tadesse Tiruneh, from the Ethiopian Revenues and Customs Authority, presented an evaluative study “An Analysis of Chinese FDI in Sub-Saharan Africa, With a Particular Focus on Ethiopia”, in order to assess the effect of such an increasing phenomenon in Africa, and particularly on Ethiopian labor market, on safety standards, national market -oriented manufacturing production and consumption, as a tool to put in place appropriate policies and strategies, including strong institutional arrangements to deal with such a complex and controversial issue. The presentation generated an interesting debate focused on whether Chinese FDIs could be seen as a benefit for African economies, and in this sense the author was asked to explain to what extent his conclusions, derived from an assessment of the secondary literature on the theme, are confirmed by interviews with key actors and stakeholder with reference to the Ethiopian context.

The third paper, “The Development Project as an Institution for Agency and Capability Expansion: The Case of the Njombe Milk Project “ was presented by Francesco Burchi from the German Development Institute, and it dealt with an evaluation of a multi-sectoral project in Tanzania, centred on the production, selling and distribution of milk in schools. Going beyond the adoption of the Logical Framework approach to evaluate such an intervention, the paper proposes a combination of quantitative and qualitative methods and the reference to the capability approach as the conceptual framework, in order to analyse the project’s effects on achieving functioning and agency by the different stakeholders. The following debate focused on the possible institutional explanations for the poor relevance of collective organizations of farmers in the framework of the activities, and on the implications of a baseline lack in considering the evaluation’s methodology constraints.

The fourth paper, “Evaluating a long-term decentralised cooperation policy: the Trentino case study”, presented by Stefano Rossi (Training Centre for International Development) dealt with the challenging attempt to evaluate a complex intervention, whose protagonist is an Italian Region, Trentino, involved in a long-term decentralized cooperation process since the late Nineties in three municipalities in the Balkans and in a rural district in Mozambique. The research represents an ex-ante reflective evaluation which mainly investigates the reasons behind programme and projects assumptions and strategies. The following debate focused on the cognitive strategies used to evaluate such a complex scenario, highlighting: i) the importance to focus on processes and strategies rather than on results, ii) the self-relevance and the actors’ ownership and iii) learning as key issues to be explored in carrying out such a kind of evaluation.

The paper presented by Gianni Vaggi and Marta Manson (University of Pavia), entitled “The empowerment approach to evaluation: the case of Project Malawi”, deals with combining a traditional evaluative approach, based on the log frame assessment, with the empowerment approach with its focus on the interaction between subjective and objective elements for capacitation to be achieved, in evaluating a program launched in 2005 by Intesa Sanpaolo and Cariplo Foundation to create a barrier against AIDS, starting from the prevention of mother-to-child transmission. The empowerment approach moved from the log frame-based evaluation and was carried out through a desk analysis conducted on results and indicators provided by the log frame itself in order to examine how far the two pillar notions of this approach (i.e. agency and opportunity structure) can help to assess the development/empowerment outcomes, also taking into account of the separation into three main domains: State, Market and Society. The participants’ comments following the presentation focused on the need to individuate and develop a more coherent theory of change underlying the evaluative work.

Finally, Stefano Moncada from the University of Malta provided a presentation “Assessing Local Adaptive Capacity – A Framework for Impact Evaluation Studies” that, starting from an ongoing impact evaluation study on a biogas and sanitation project in an informal urban settlement in Ethiopia, tries to assess the effects of small-scale development interventions on local adaptive capacity.The findings of this research point towards a greater integration of mixed methodologies, particularly in the realm of small-scale interventions in the field of climate change and development, stressing the importance to get in-depth information from PRAs to add context-specific value, and the use of a Propensity-Score Matching (PMS) technique with Difference-in-Difference in order to investigate to what extent the observed  changes are attributable to the intervention.

Report by Laura Fantini (Sapienza University of Rome) and Sara Hassan (Cespi, Rome), conveners and organizers of the two sessions titled “Evaluating Aid Policies and Specific Initiatives. Applications at macro-, meso- and micro-levels of analysis”.