Development Effectiveness Review of International Fund For Agricultural Development (IFAD) 2007 - 2013

Synthesis Report

August 2014

Acknowledgments

Foreign Affairs, Trade and Development Canada (DFATD)'s Development Evaluation Division wishes to thank all who have contributed to this review exercise for their valued input.

Our thanks go to the independent team of the firm DevPar Financial Consulting Ltd. in Canada that carried out the review. The team was made up of team leader and senior evaluator Ms. Promil Paul, analysts Henry Knight, Shishir Shahnawaz and James Chester.

The Development Evaluation Division would also like to thank the management team of DFATD's Global Issues and Development Branch at Headquarters in Gatineau for its valuable support.

Our thanks also go to the representatives of the International Fund for Agricultural Development for their helpfulness and their useful, practical advice to the reviewers.

From the DFATD's Development Evaluation Division, we wish to thank Shaylyn Gaffney, Junior Evaluation Officer, and Vivek Prakash, Senior Evaluation Officer, for their assistance with the review, and Michelle Guertin, Evaluation Team Leader, for guiding this review to completion.

James Melanson
Head of Development Evaluation
Foreign Affairs, Trade and Development Canada

List of Abbreviations

Executive Summary

Background

This report presents the results of a review of the development effectiveness of the International Fund for Agricultural Development (IFAD). Established in 1977, IFAD is a United Nations Specialized Agency and an International Financial Institution dedicated to mobilizing resources for agriculture and rural development in developing countries. Its mandate, as stated in its Strategic Framework, is to improve food security and nutrition and enable rural women and men to overcome poverty.^{Footnote 1} This mandate is underpinned by five strategic objectives:

• A natural resource and economic asset base for poor rural women and men that is more resilient to climate change, environmental degradation and market transformation;
• Access for poor rural women and men to services that reduce poverty, improve nutrition, raise incomes and build resilience in a changing environment;
• Poor rural women and men, and their organizations, being able to manage profitable, sustainable and resilient farm and non-farm enterprises or take advantage of decent work opportunities;
• Poor rural women and men, and their organizations, being able to influence policies and institutions that affect their livelihoods; and
• Enabling institutional and policy environments to support agricultural production and the full range of related non-farm activities.

IFAD's multilateral orientation provides a global platform for discussing rural policy issues and increasing awareness of why investment in agriculture and rural development is critical to reducing poverty and improving global food security. IFAD works in Sub-Saharan Africa, Asia and Pacific, Latin America and the Caribbean, Near East, North Africa, Central Asia and Eastern Europe region.

Canada is a founding member of IFAD^{Footnote 2} and has been a strong supporter since 1977. It has had a permanent seat on the Executive Board since the establishment of the organization and was the fifth-largest contributor for the 8th funding replenishment (2010 to 2012). The Government of Canada's 2013 Economic Action Plan affirmed Canada's commitment to international development investments, which included increasing food security. Foreign Affairs, Trade and Development Canada (DFATD) is expected to continue to maintain its focus and mandate of poverty alleviation^{Footnote 3} through support to multilateral organizations such as IFAD.^{Footnote 4}

Purpose

This review is intended to provide an independent, evidence-based assessment of the development effectiveness of IFAD programs to satisfy the requirements established by the Government of Canada's Policy on Evaluation and to provide DFATD and other stakeholders with evidence on the effectiveness of IFAD.

Methodology

The IFAD review applies the methodological approach developed under the guidance of the Organization for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC)'s Network on Development Evaluation (EVALNET). This approach was originally established to be complementary to the assessments conducted by the Multilateral Organization Performance Assessment Network (MOPAN).^{Footnote 5} The methodology involves a systematic and structured meta-synthesis of the findings from a sample of IFAD evaluations, according to six main criteria and nineteen sub-criteria that are considered essential elements of effective development (see Chapter 2.3). The six criteria are:

• Achievement of development objectives and expected results;
• Crosscutting themes (gender equality and environmental sustainability):
• Sustainability of results/benefits;
• Relevance of intervention;
• Efficiency; and
• Using evaluation and monitoring to improve development effectiveness.

This review undertook a meta-synthesis of 32 evaluations (listed in Appendix 1), conducted and published by the Independent Office of Evaluation (IOE) at IFAD between 2007 and 2013. One evaluation was removed from the universe of 33 evaluations, during the screening process, for scoring too low on the three screening criteria for use of multiple lines of evidence, good evaluation design, and relevant/evidence based conclusions.^{Footnote 6} In the final sample, 23 country program evaluations and 9 corporate-level evaluations have been included.

After being screened for quality (see Appendix 3 for the Evaluation Quality Scoring Guide and results), each evaluation was reviewed to identify findings related to six main criteria, divided into nineteen sub-criteria. The review team classified the evaluation findings using a four-point scale: "highly satisfactory", "satisfactory", "unsatisfactory" and "highly unsatisfactory". A grid guided the classification of findings with specific instructions for each rating across all sub-criteria (see Appendix 4 for the classification guide). The review team also identified factors contributing to or detracting from the achievement of results.

To reflect recent reforms of IFAD, which are not necessarily reflected in the findings of the review, the meta-synthesis of evaluation reports was supplemented by a review of IFAD corporate documents related to evaluation (Appendix 5) and reporting on effectiveness, as well as interviews with a small number of DFATD and IFAD personnel responsible for managing the relationship between DFATD and IFAD. Additional interviews were conducted with donor partners that work closely with Canada and IFAD.

Key Findings

IFAD is Achieving Most of its Development Objectives and Expected Results

IFAD programs were judged to have achieved their stated development objectives and resulted in clear positive benefits for the target groups, in fifty-six percent and sixty-nine percent of the evaluations reviewed, respectively. The evaluations noted, however, lower scores regarding the programs' ability to make substantial contribution to national development goals (44% of evaluations rated "satisfactory" or better) and to significant changes in national development policies (34% of evaluations rated "satisfactory or better).

An analysis of factors that contributed to positive results highlighted the importance of partnership arrangements and the quality of upfront program design. Setting clear strategic objectives and embedding them in well thought-out logic models was also correlated with effectiveness.

IFAD Faces Challenges in Contributing to Gender Equality and Environmental Sustainability

Gender equality and environmental sustainability were inadequately addressed. IFAD's programming was determined as "unsatisfactory" or worse in more than almost two-thirds of the evaluations reviewed with respect to its effectiveness in addressing gender equality, while 53% were rated as weak for their contribution to environmental sustainability. It was recognized that IFAD faces challenges in programming and reporting on both gender equality and environmental sustainability.

However, recent reports, such as IFAD'S 2013 Annual Report on Results and Impact (ARRI)^{Footnote 7}, the Evaluation Cooperation Group's 2012 synthesis report on gender equality and development evaluation units^{Footnote 8} and the 2013 UN System Wide Action Plan Reporting have noted some improvements in the area of gender equality, for example, the establishment of a new Policy on Gender Equality in 2012.

The 2013 ARRI also noted improvements in the areas of natural resources and the environment and climate change^{Footnote 9} and IFAD has since launched an Adaptation for Smallholder Agriculture Program, which provides financing to scale-up and integrate climate change adaptation across the organization's new investments.

The review of contributing factors suggests that more attention is needed at the program and project design stage. Clear understanding of target groups' needs and focused responsive strategies have been identified as a prerequisite for success in this area.

Sustainability of Benefits/Results is Generally Satisfactory

The evaluations reviewed reported broadly positive results for the sustainability of benefits/results for IFAD programs. Benefits were judged likely to continue after program completion in 56% of evaluations reviewed. Contribution to institutional and community capacity, and to strengthening the enabling environment, were found in 66% and 47% of cases, respectively.

Sustainability was best assured when planned and addressed early in the implementation of projects and programs. The support and endorsement of government was considered important to contribute to the sustainability of IFAD programs' results, along with adequate and timely availability of funds and staff to ensure continuity of the benefits of the program.

IFAD Programs are Aligned and Relevant to Stakeholder Needs

Relevance was one of the stronger elements highlighted in the review. Over eighty percent of evaluations reviewed noted that programs were aligned with national development goals. The extent to which programs were suited to the needs and priorities of the target group was found to be "satisfactory" or better in over half of evaluations. Effective partnerships were also noted in over half of the reports.

Direct effort to ensure that country assistance is aligned and integrated into national development programs and priorities during the design stage strengthened the relevance of IFAD programs.

Evaluations Report Less Positive Results in Efficiency

All three sub-criteria for efficiency received low scores. Programs were evaluated as cost/resource efficient in 47% of the evaluations reviewed, and the achievement on time of program implementation and objectives was noted in only 22% of cases. Systems and procedures for program implementation were considered unsatisfactory or worse in 78% of the evaluations reviewed.

IFAD is taking measures to address these challenges, and has prepared an action plan to follow-up on efficiency commitments made as part of its ninth replenishment and on recommendations of a 2013 Corporate-level evaluation of IFAD's institutional efficiency and efficiency of IFAD-funded operations.

Factors that contributed positively to efficiency included adequate planning, predictable funding, and staff capacity.

Evaluation is Effective and Useful, but Weaknesses Exist in Monitoring, Reporting and Results-Based Management.

Although evaluation is a tool used by IFAD to improve development effectiveness (53% of evaluations were rated satisfactory or better), the evaluations indicate that the systems and processes for monitoring and tracking progress on the achievement of results need to be improved (only 34% of the evaluations received positive scores). Results-based management systems were rated similarly with 31% of the evaluations scoring satisfactory or better. The effectiveness of systems and processes for evaluation did, however, score higher with 59% of evaluations rating satisfactory or better.

An important element that contributed to poor results in this area was the insufficient knowledge and capacity on the part of partnering governments for evaluation and monitoring.

Evaluations carried out by the IOE between 2007 and 2013 indicate that IFAD programming is achieving most of its development objectives and expected results and is relevant to the needs of target groups and developing country governments. The evaluations also reflect sustainability of results. IFAD has not been as effective in addressing crosscutting themes of gender equality and environmental sustainability. The cost efficiency and timeliness of its programming also represent areas for improvement. While IFAD made effective use of evaluations to improve program effectiveness, monitoring, reporting and results-based systems need to be improved.

IFAD programming contributes directly to Canada's thematic priorities of increasing food security and stimulating sustainable economic growth. Indirect contribution to a third thematic priority, securing the future of children and youth, was noted.

DFATD's institutional strategy for IFAD focuses on three strategic objectives that contribute to addressing the challenges identified in the review. These are:

• increasing investments in smallholder farmers with emphasis on benefiting women and strengthening environmental sustainability;
• ensuring development programming effectively fits within food security systems and engages other organizations involved in agricultural development; and
• Building on recent reforms aimed at improving organizational effectiveness and ability to measure, report and achieve results in the field.

Canada's strategic objectives are being addressed through IFAD programming for increased productivity of smallholder farmers and linkages to markets. IFAD programming is committed to promoting food security systems and expanded opportunities for the rural poor.

Recommendations for DFATD

The following recommendations to DFATD are based on the findings and conclusions outlined above. Implementation will of course require engagement with other shareholders.

DFATD should:

• Encourage IFAD management to ensure that issues related to sustainability, early in project cycles with clearly stated objectives, expected results, exit strategies, and attention to follow on resourcing.
• Emphasize the need to effectively address and promote gender equality throughout IFAD's programming. Gender-based analyses, results frameworks, and monitoring of gender equality results would contribute to strengthening the mainstreaming of this crosscutting theme.
• Continue to engage with IFAD to ensure that the environmental sustainability of its operations receives sufficient attention during the design phase and throughout the implementation process, and that results in this area are improved over time. Encourage IFAD to make significant efforts to improve the efficiency of its programming in line with IOE's 2013 evaluation on IFAD's institutional efficiency.
• Emphasize the need to strengthen systems for results-based management, including results monitoring and reporting, and the use of appropriate and focused indicators and associated targets.

1.0 Introduction

1.1   Background

This report presents the results of a review of the development effectiveness of the International Fund for Agricultural Development (IFAD). The methodology was developed under the guidance of the Organization for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC)'s Network on Development Evaluation (EVALNET). The review relies on the content of published evaluation reports produced by IFAD and is supplemented with a review of IFAD and DFATD corporate documents, and interviews with IFAD and DFATD personnel and other donor partners that work closely with Canada and IFAD.

The method uses a common set of assessment criteria derived from the DAC's evaluation criteria (see Chapter 2.3). It was pilot tested in 2010 using the Asian Development Bank and the World Health Organization. The methodology was endorsed by the members of DAC-EVALNET as an acceptable approach for assessing the development effectiveness of multilateral organizations in June 2011.

From its beginnings, the process of developing and implementing the reviews of development effectiveness has been coordinated with the work of MOPAN. By focusing on development effectiveness and carefully selecting assessment criteria, the reviews seek to avoid duplication or overlap with the MOPAN process. There has been an effort to conduct the development effectiveness reviews in the same year as a MOPAN survey, in order to achieve a complementary perspective on the development and organizational effectiveness of any given multilateral organization. In the case of IFAD, a MOPAN assessment was completed in December 2013.

1.2   Why conduct this review

The purpose of the review is to provide an independent, evidence-based assessment of the development effectiveness of IFAD programs for use by the Government of Canada and other interested shareholders.

The objectives of the review are:

• To provide DFATD with evidence of the development effectiveness of IFAD which can be used to plan the next period of long term institutional support;^{Footnote 10} and,
• To provide evidence on the development effectiveness of IFAD that can be used in the ongoing relationship between DFATD and IFAD to ensure that Canada's international development priorities are served by investments of all types.^{Footnote 11}

Although this report is intended, in part, to support DFATD's accountability requirements within the Government of Canada, the results are expected to be useful to other bilateral shareholders as well.

1.3   IFAD: Enabling the Rural Poor to Overcome Poverty

1.3.1   Background and Mandate

IFAD is a United Nations Specialized Agency and an International Financial Institution established in 1977 as a major outcome of the World Food Conference in 1974. It is dedicated to mobilizing resources for agriculture and rural development in developing countries. Its mandate is to improve food security and nutrition and enable rural women and men to overcome poverty.^{Footnote 12}

1.3.2   Strategic Plan

IFAD's mandate is underpinned by five strategic objectives, namely:^{Footnote 13}

• A natural resource and economic asset base for poor rural women and men that is more resilient to climate change, environmental degradation and market transformation;
• Access for poor rural women and men to services for reduce poverty, improve nutrition, raise incomes and build resilience in a changing environment;
• Poor rural women and men and their organizations able to manage profitable, sustainable and resilient farm and non-farm enterprises or take advantage of decent work opportunities;
• Poor rural women and men and their organizations able to influence policies and institutions that affect their livelihoods; and
• Enabling institutional and policy environments to support agricultural production and the full range of related non-farm activities.

Gender equality and social inclusion will be addressed as crosscutting themes under each of the objectives. In order to build resilience during transition from non-market-oriented agriculture to sustainable agricultural and non-farm activities, household livelihood and nutrition strategies will continue to be addressed.

In delivering on strategic framework objectives, IFAD aims to improve quality and efficiency, strengthen its ability to work effectively with the private sector, step up advocacy, and amplify the voices of poor rural women and men in decisions that affect their lives.^{Footnote 14}

1.3.3   Scale and Geographic Coverage

According to IFAD's 2012 Program of Work, the organization was financing 255 active programs and projects with investments of US$5.3 billion in 97 countries and Gaza and the West Bank. External co-financing and funds from domestic sources for the ongoing portfolio amounted to US$6.6 billion, bringing the total value of ongoing programs and projects in 2012 to US$11.9 billion. Levels of disbursement rose during 2012, keeping pace with the growing portfolio. Total disbursements of loans and debt sustainability framework grants during 2012 were US$652.9 million (see Table 1). For 2013, IFAD proposed to commit approximately US$1.066 billion in loans and grants from IFAD's own resources.^{Footnote 15}

Source: Project and Portfolio Management System, IFAD financial statements for 1978-2012, IFAD's Accounting System.

1.3.4   Evaluation and Results Reporting

Evaluation

IFAD's Independent Office of Evaluation (IOE) is headed by a Director and reports directly to the Executive Board. The IOE is responsible for conducting independent evaluations of IFAD's financed policies, strategies and operations to promote accountability and learning.^{Footnote 17} To fulfil this function, IOE conducts different types of evaluations (Figure1). In addition, the IOE prepares evaluation syntheses on selected topics of importance to both the Executive Board and IFAD management.

IFAD's Evaluation Policy was revised in 2011 following a 2008 Peer Review of the IOE and evaluation function by the Evaluation Co-operation Group of the Multilateral Development Banks.^{Footnote 18} IFAD's Evaluation Manual^{Footnote 19} guides the methodology and outlines the processes used for carrying out evaluations. An evaluation work program, which identifies the number and type of the evaluations to be conducted annually by the IOE, is prepared and approved by the Executive Board. A Report on the Implementation Status of Evaluation Recommendations and Management Actions is also prepared annually and presented to the Executive Board for discussion. All evaluations are subject to internal peer review. Senior Independent Advisors are also engaged to provide input at key stages of corporate-level evaluations and country program evaluations.^{Footnote 20}

Figure 1: Types of IFAD Evaluations^{Footnote 21}

The 2010 Peer Review concluded that IFAD had created a credible, independent evaluation function and prepared a comprehensive Evaluation Manual. It also noted that independent evaluation is supported and valued in IFAD and that there has been some notable use of evaluations, with some affecting IFAD corporate policies and country strategies. However, the review panel recommended that IOE:

• harmonize its approach to evaluation with that of the Evaluation Cooperation Group (ECG) good practice by basing its portfolio and project assessment more heavily on evidence drawn from validated project completion reports;
• further strengthen the use of evaluation findings, learning and the feedback loop; and,
• identify ways to further improve quality through use of a broader range of evaluation approaches and methodologies.
• The IOE implemented all recommendations immediately following the completion of the Peer Review.

The review team conducted its own quality review of the evaluations included in this review. The results were positive, with 87.5% of the reviewed evaluations scoring 31 points or more out of a possible total of 40 (see Appendix 3: Evaluation Quality Scoring Guide and Results).

Results Reporting

IFAD produces an annual Report on IFAD's Development Effectiveness, which presents the main features of IFAD's performance in three broad areas, namely the:

• Relevance of IFAD's mandate and operations in the context of the changing framework of international development assistance;
• Development effectiveness of IFAD-financed operations in generating development results "on the ground"; that support national and global efforts to reduce rural poverty and contribute to achieving the Millennium Development Goals, and the first MDG in particular; and
• Organizational effectiveness and efficiency of the Fund in delivering results based on indicators in its Results Measurement Framework and the implementation of IFAD'S replenishment commitments.^{Footnote 22}

IFAD has also been producing annual reports on results and impact of IFAD operations (ARRI) since 2003. These reports summarize findings from evaluation, identify results and impact of IFAD activities, examine lessons learned, and discuss systemic issues with a view to further enhancing the organization's development effectiveness. These are presented to the Board, which provides guidance to IFAD management.

2.0  Methodology

2.1   Rationale

Although multilateral organizations produce annual reports for their management and/or boards, bilateral shareholders felt they were not receiving a comprehensive overview of performance on the ground. The "Development Effectiveness Review" methodology was designed to fill this information gap, and to be implemented in a rapid and cost effective way. It was intended to be complementary to the assessments conducted by MOPAN.

The methodology used in this review is a systematic and structured meta-synthesis of the findings of 32 quality assured evaluations published by the IOE between 2007 and 2013, as they relate to six main criteria and 19 sub-criteria that are considered to be essential elements of effective development (see Chapter 2.3). A more detailed description is presented in Appendix 2.

2.2   Scope

The IOE published thirty-three (33) evaluations at the programming and corporate levels between 2007 and 2013 (described in more detail in Appendix 1). Thirty two (32)^{Footnote 23} of these evaluations were included for the conduct of this review, comprising of nine (9) corporate-level evaluations and twenty three (23) country program evaluations.

Geographically, the distribution of the 23 country program evaluations was as follows: 6 evaluations from East and Southern Africa; 5 evaluations from Asia and the Pacific; 5 evaluations from West and Central Africa; 4 evaluations from Near East, North Africa and Europe; and 3 evaluations from Latin America and the Caribbean. The country programs provided a reasonable representation of the population of countries being assisted by IFAD. Based on the information available, it was not possible to calculate the percentage of IFAD expenditures represented by the 32 evaluations.

The meta-synthesis of evaluation reports was supplemented by a review of IFAD and DFATD corporate documents (Appendix 5), as well as interviews with a small number of DFATD and IFAD personnel responsible for managing the relationship between DFATD and IFAD. Additional interviews were conducted with donor partners that work closely with Canada and IFAD. The interviews contextualized the results of the meta-synthesis of evaluations, and took account of advances that have taken place in recent years.

2.3   Criteria and Sub-Criteria

The methodology does not rely on a particular definition of development effectiveness. It focuses instead on some of the essential characteristics of developmentally effective multilateral organization programming, as described below:

1. Achievement of development objectives and expected results

• 1.1 Programs and projects achieve their stated development objectives and attain expected results.
• 1.2 Programs and projects have resulted in positive benefits for target group members.
• 1.3 Programs and projects made differences for a substantial number of beneficiaries and where appropriate contributed to national development goals.
• 1.4 Programs contributed to significant changes in national development policies and programs (policy impacts) and/or to needed system reforms.

2. Crosscutting themes

• 2.1 Extent to which multilateral organization supported activities effectively address the crosscutting issue of gender equality.
• 2.2 Extent to which changes are environmentally sustainable.

3. Sustainability of results and benefits

• 3.1 Benefits continuing or likely to continue after project or program.
• 3.2 Projects and programs are reported as sustainable in terms of institutional and/or community capacity.
• 3.3 Programming contributes to strengthening the enabling environment for development.

4. Relevance of interventions

• 4.1 Programs and projects are suited to the needs and/or priorities of the target group.
• 4.2 Programs and projects align with national development goals.
• 4.3 Effective partnerships with governments, bilateral and multilateral development and humanitarian organizations and NGOs for planning, coordination and implementation of support to development.

5. Efficiency

• 5.1 Program activities are evaluated as cost/resource efficient.
• 5.2 Implementation and objectives achieved on time.
• 5.3 Systems and procedures for project/program implementation and follow up are efficient (including systems for engaging staff, procuring project inputs, disbursing payment, logistical arrangements).

6. Use of evaluation and monitoring to improve development effectiveness

• Systems and process for evaluation are effective.
• Systems and processes for monitoring and reporting on program results are effective.
• Results-based management (RBM) systems are effective.
• Evaluation is used to improve development effectiveness.

2.4   Limitations

As with any meta-synthesis, there are a number of methodological challenges that limit the findings.

Sampling bias was mitigated by including almost all relevant evaluations published during the period. There was also an adequate coverage of the criteria since all 19 sub-criteria used to assess development effectiveness were covered in the evaluations reviewed (Appendix 2).

The review was not able to distinguish differences in effectiveness by type of program or by sector of disbursement. The 32 evaluations did not allow for comparative analysis of IFAD loans and grants by area of investment, country classification or other dimensions of programming.

Finally, the recent reforms of IFAD are not necessarily reflected in the findings of the review. As such, the review was strengthened by the inclusion of supplementary information from DFATD and IFAD corporate documents and interviews with a small number of DFATD and IFAD personnel and donor partners.

3.0  Findings on the Development Effectiveness of IFAD

This section presents the results of the meta-synthesis of evaluation findings for the six main criteria and 19 sub-criteria.

Table 2 below presents the percentages of "satisfactory" or better versus "unsatisfactory" or worse ratings for each sub-criterion across all evaluation reports (as assessed by the review team).

A summary of key findings is presented by major criterion in the following sections. This is accompanied by an explanation of factors contributing to those findings, taking into consideration the frequency with which they were reported in the evaluation reports.

Table 2 : Percentage of Evaluations Reporting Findings of "Satisfactory or better" and "Unsatisfactory or worse" for each Sub-Criterion

3.1   IFAD is Achieving Most of its Development Objectives and Expected Results

3.1.1   Key Findings

IFAD programs were found to be relatively effective in terms of achieving most development objectives and expected results (Figure 2). For sub-criterion 1.1 "Programs and projects achieve stated objectives", 17 of the 32 evaluations (53%) received a rating of "satisfactory", while 69% were rated "satisfactory" for sub-criterion 1.2 "Positive benefits for target group members". However, lower scores were noted for sub criteria 1.3 "Substantial numbers of beneficiaries/contribution to national development goals" and 1.4 "Significant changes in national development policies/programs", where 18 (56%) and 21 evaluations (66%) were rated "unsatisfactory" or worse, respectively.

Figure 2 : Achievement of Development Objectives and Expected Results (Findings as a percentage of the number of evaluations addressing the sub-criterion, n=32)

Among positive outcomes noted by the evaluations:

• Considerable effort was made to target poor geographic regions often specifying distinct poverty profiles (18 evaluations);
• Substantial contributions were made to positive changes in the targeted groups in terms of increased income and poverty reduction (23 evaluations);
• Relative increases in household income and/or assets were noted due to higher levels of agricultural productivity and investments in livestock in the targeted communities (15 evaluations); and
• Policy dialogue was successfully established in many countries under difficult circumstances (12 evaluations).

Highlight Box 1 illustrates that IFAD has a comparative advantage in focusing its efforts in agriculture and rural development.

3.1.2   Contributing Factors

Several factors were noted as either contributing to (positive factors) or detracting from (negative factors) IFAD's achievement of objectives and expected results. A summary is provided in Figure 3 below, where the numbers indicate how many evaluations reported on each factors.

Clearly, partnership arrangements and the quality of upfront program design are strong predictors of eventual effectiveness in results achievement. Setting clear strategic objectives and embedding them in a well thought-out logic model, which ensures strong logical links between activities, resources, and outcomes, was correlated with effectiveness. When initial design did not have these characteristics, results were weaker. As well, programs that showed strong collaborative partnerships with local stakeholders, early consultation on design, good targeting of beneficiaries, and respect for national priorities, tended to be more successful. One of the more notable predictors of poor results was inadequate funding and unreliable flows, which produced disruptive delays; these were especially noted in instances where joint funding was relied upon.

Figure 3 : Summary of Contributing Factors - Achievement of Objectives and Expected Results

3.2   IFAD Faces Challenges in Contributing to Gender Equality and Environmental Sustainability

3.2.1    Key Findings

Gender issues and environmental sustainability could have been addressed more effectively as illustrated in Figure 4. Nineteen (59%) of the 32 evaluations rated IFAD programs as "unsatisfactory" or worse for sub-criterion 2.1 "Effectively addressed the crosscutting issue of gender". Similarly, sub-criterion 2.2 "Changes are environmentally sustainable" was rated 'unsatisfactory" or worse in 20 evaluations (63%).

Figure 4 : Crosscutting Themes: Gender Equality and Environmental Sustainability (Findings as a percentage of the number of evaluations addressing each sub-criterion, n=32)

The findings suggest that IFAD is facing some challenges in addressing and reporting on gender equality and environmental sustainability of programming. The 2010 corporate-level evaluation that assessed IFAD's performance with regard to gender equality and women's empowerment noted as a crucial concern that most evaluation reports reviewed lacked a results framework for systematically measuring progress on gender equality. There was a lack of specific gender equality achievement indicators for evaluators to rely on. The evaluation also rated gender equality as unsatisfactory on the basis that IFAD programs "either lack gender objectives or achieved less than half of their stated gender equality objectives".

However, recent reports have noted some improvements in this area. The 2013 ARRI observed that IFAD operations are very good at promoting gender equality and women's empowerment but that there was still need to raise the bar so that a greater proportion of projects are considered satisfactory under this criterion.^{Footnote 24}

The Evaluation Cooperation Group's 2012 synthesis report on gender equality and development evaluation units, which compared the performance of a number of multilateral organizations, noted IFAD's clear instructions for considering gender-related results by means of the IOE's dedicated evaluation gender criterion that was created in 2010.^{Footnote 25}

The 2013 UN System Wide Action Plan Reporting, which provides an overview of promotion of gender equality and the empowerment of women for the UN system, noted IFAD's leadership in this area, particularly in program review, gender equality and the empowerment of women policies, performance management, monitoring and reporting, and organizational culture.^{Footnote 26} This is likely a result of the establishment of a new Policy on Gender Equality in 2012, which provides responsibility for gender mainstreaming across IFAD's systems.^{Footnote 27}

Experience in Vietnam illustrates that sustained positive results are possible when gender equality is addressed proactively in IFAD country strategies and policies (Highlight Box 2).

The 2013 ARRI has noted improvements in the areas of natural resources and the environment and climate change.^{Footnote 28} IFAD has launched an innovative Adaptation for Smallholder Agriculture Program, which provides financing to scale-up and integrate climate change adaptation across the organization's new investments.^{Footnote 29}

3.2.2   Contributing Factors

The review of contributing factors related to crosscutting themes of gender equality and environmental sustainability (Figure 5) suggest that more attention is needed at the program and project design stage to mainstreaming gender equality and environmental sustainability in IFAD programming. Clear understanding of target group members' needs, and appropriate and focused responsive strategies have also been identified as a prerequisite for success in this area. Quality of consultation and consensus building were also noted as important factors in effectively addressing crosscutting themes.

Figure 5: Summary of Contributing Factors - Crosscutting Issues

3.3   Sustainability of Benefits/Results is Generally Satisfactory

3.3.1   Key Findings

The evaluations reviewed reported broadly positive sustainability of benefits/results for IFAD programs (Figure 6). Eighteen (56%) of the 32 evaluations rated sub-criterion 3.1 "Benefits likely to continue after project or program completion" as "satisfactory" or better. Sub-criterion 3.2 "Projects and programs are reported as sustainable in terms of institutional and community capacity" was rated higher with 21 evaluations (66%) that scored "satisfactory" or better. There is some scope for improvement regarding sub-criterion 3.3 "Programming contributes to strengthening the enabling environment for development", which was rated "satisfactory" or better in 15 (47%) evaluations.

Figure 6: Sustainability of Results/Benefits (Findings as a percentage of the number of evaluations addressing each sub-criterion, n=32)

The evaluations reviewed noted the following positive results in this area:

• Strong active participation from the national governments and/or local partners. (17 evaluations); and
• Initiatives contributed to strengthening institutional and community capacity (14 evaluations).

3.3.2   Contributing Factors

A number of factors (Figure 7) were identified as having a bearing on sustainability. It was best assured when planned for and addressed early in the implementation of programs and projects. The support and endorsement of government was considered important, along with adequate and timely availability of funds and staff to ensure continuity of the benefits of the programs. Planning for sustainability requires early identification of, and working with, appropriate institutions or organizations to ensure a continuation of the benefits of a program beyond its lifetime.

Figure 7: Summary of Contributing Factors - Sustainability

The 2013 ARRI noted weaknesses in regards to sustainability and identified a number of factors that contributed to these challenges, namely: limited attention to early preparation of exit strategies; complex designs with multiple components; and weak institutional capacities in partner countries.^{Footnote 30}

3.4   IFAD Programs are Aligned and Relevant to Stakeholder Needs

3.4.1   Key Findings

Relevance was among the most highly rated of the six criteria in the evaluations reviewed (Figure 8). Twenty-six (81%) of the 32 evaluations were rated as "satisfactory" or better for sub-criterion 4.2 "Programs align with national development goals", while sub-criterion 4.1 "Programs and projects are suited to the needs and/or priorities of the target group" was rated "satisfactory or better" in 19 (59%) evaluations. Effective partnerships (sub-criteria 4.3) were noted in about half of the reports (56%). There is evidence that IFAD is able to deliver projects and programs in a politically challenging environment, including where there is civil unrest, (e.g. Yemen, Mali, Sudan, and Pakistan) to support the poorest communities.

Similarly, the recent ARRI and 2013 MOPAN assessment of IFAD also reported positive results in this area. The MOPAN assessment specifically noted that the country strategic opportunity programs sampled were well aligned with national development plans, although the link between results frameworks at the project/program and the country level was not always evident.^{Footnote 31}

Figure 8: Relevance to Stakeholder Needs and National Priorities (Findings as a % of the number of evaluations addressing each sub-criterion, n=32)

Findings for sub-criteria 4.1 and 4.2 are corroborated by the 2013 MOPAN assessment of IFAD where micro indicators 5.3 "Expected results consistent with national strategies" and 5.4 "Expected results developed in consultation with direct partners/beneficiaries" were rated as being "adequate".^{Footnote 32}

3.4.2   Contributing Factors

A number of factors were important in contributing to the achievement of program relevance (Figure 9). The direct effort to ensure that country assistance is aligned and integrated into national development programs and priorities during the design stage strengthened the relevance of IFAD programs. The establishment of close partnerships and regular consultations with stakeholders was noted as important to ensuring stakeholder buy-in and proper implementation. Relevance is also better assured when it is monitored and progress reported regularly throughout the life of the program.

Highlight Box 3 shows the process used by IFAD to ensure relevance of activities in Yemen through geographic targeting and participatory approaches.

Figure 9: Summary of Contributing Factors - Relevance

3.5   Evaluations Report Less Positive Results in Efficiency

3.5.1   Key Findings

The three sub-criteria related to efficiency showed low ratings in more than half of the evaluation reports (Figure 10). For sub-criterion 5.1 "Program activities are evaluated as cost/resource efficient", 17 (53%) of the 32 evaluations were rated "unsatisfactory" or worse, while sub-criterion 5.2 "Program implementation and objectives achieved on time" had a 78% (25 evaluations) "unsatisfactory" or worse rating. Sub-criterion 5.3 "Systems and procedures for program implementation and follow-up" was also weak with 25 evaluations (78%) receiving scores of "unsatisfactory" or worse.

Figure 10: Efficiency (Findings as a % of the number of evaluations addressing each sub criterion, n=32)

The 2013 corporate-level evaluation on IFAD's institutional efficiency and efficiency of its funded operations noted the following issues related to efficiency (see Highlight Boxes 4 and 5).

In response to efficiency-related proposals in IFAD's ninth replenishment report and recommendations of the 2013 corporate level evaluation, IFAD has prepared a Consolidated Action Plan to Enhance Operational and Institutional Efficiency.

3.5.2   Contributing Factors

Several factors contributing to the results achieved in regard to the efficiency of the programs were noted in the evaluations reviewed (Figure 11). Adequate planning, adequate and predictable funding, and staff capacity were associated with efficiency. The preparation of appropriate plans and schedules to ensure proper use of available resources and tracking of results was identified as an important step to ensure efficiency of IFAD programs.

Figure 11 : Summary of Contributing Factors - Efficiency

The recent ARRI also identified factors that detracted from efficiency. These included wide geographical coverage of projects, delays in deployment of project staff, relatively slow disbursements and inadequate targeting of poor rural people.^{Footnote 33}

3.6   Evaluations are Effective and Useful, but Weaknesses Exist in Monitoring, Reporting and Results-Based Management

3.6.1   Key Findings

Although evaluation is used by IFAD to improve development effectiveness, the systems and processes for monitoring and tracking progress on the achievement of results need to be improved (Figure 12). For sub-criterion 6.1 "Systems and processes for evaluation are effective", 17 (53%) of 32 evaluations were rated "satisfactory" or better. Similarly, 19 evaluations (59%) scored positively for sub-criterion 6.4 "Use of evaluation to improve development effectiveness". However, 21 (66%) and 22 (69%) evaluations were rated "unsatisfactory" or worse for sub-criteria 6.2 "Systems and processes for monitoring and reporting" and 6.3 "Result Based Management systems" respectively.

Evaluations reviewed made frequent mention of the inadequacy of the monitoring and reporting systems as impediments to the conduct of evaluations. Other sources of information were therefore sought. As noted in section 1.3.4, the general overall quality of the evaluation reports themselves was determined to be good following the quality assessment conducted for this review.

As highlighted in Box 6, there are constraints in gathering information for evaluation and monitoring to improve development effectiveness and expected results.

Figure 12: Using Evaluation and Monitoring to Improve Development Effectiveness (Findings as a % of the number of evaluations addressing each issue, n=32)

These findings are in line with those of the 2013 MOPAN assessment of IFAD where micro-indicator 18.3 "Quality of Evaluations" was rated as being "very strong for ensuring evaluation process and products" and micro-indicator 18.4 "Use of Evaluation findings to inform decisions" was rated as being adequate or better. The MOPAN assessment also noted that IFAD has implemented a number of recommendations from the 2008 Peer Review to improve its approach to evaluation.

3.6.2   Contributing Factors

Factors that contributed to the use of evaluations and monitoring to improve effectiveness are illustrated in Figure 13. Appropriate baseline data for projects, as recommended in the IFAD's Evaluation Manual, was noted. Target setting and establishment of key indicators and timeframes for planned deliverables was also identified as contributing positively in this area. One element that contributed to poor results was insufficient knowledge and capacity on the part of partnering governments for evaluation and monitoring.

Figure 13: Summary of Contributing Factors - Use of Evaluation and Monitoring Results to Improve Development Effectiveness

4.0  IFAD and Canada's Priorities in International Development

In May 2009, the Minister of International Cooperation announced Canada's intention to focus its development assistance on three thematic priorities: increasing food security, stimulating sustainable economic growth, and securing the future of children and youth (described in Highlight Box 7). This section first reviews Canada's relationships with IFAD, including management responsibility within DFATD, and then assesses the extent to which IFAD contributes to Canada's priorities in international development. It also assesses implementation of DFATD's strategic objectives for engagement with IFAD.

4.1   DFATD's Support for IFAD

Canada is a founding member of IFAD and has had a permanent seat on the Executive Board since the establishment of the organization in 1977.

For IFAD's 8th replenishment (2010-2012), Canada doubled its contribution from $37.5 million to $75 million, making it the fifth-largest contributor to IFAD's replenishment resources during this period.^{Footnote 34} Canada is providing $75 million to IFAD during its 9th replenishment period (2013-2015), and currently ranks as the fourth largest donor to IFAD's replenishment resources. Additionally, Canada has contributed $19.9 million to IFAD's new multi-donor Adaptation for Smallholder Agriculture Program and $5 million to IFAD's Nutrition-Sensitive Agriculture and Rural Development initiative. These contributions are being used by IFAD to improve rural food and nutrition security and to enable the rural poor through their work with smallholder farmers.

As shown in Figure 14, from 2006-2007 to 2011-2012, funding to IFAD has been relatively constant with two significant increases in 2009-2010 and 2011-2012, where long-term institutional and initiative-specific funding were increased, respectively. During the same period, 83% of the funding to IFAD was long-term institutional support while 15% was provided for specific initiatives. A small number (1%) was also funded under DFATD's bilateral programming (see Appendix 6 for a description of the types of funding and the organizational responsibilities for each type).

Figure 14 : DFATD Funding to IFAD: 2006-2007 to 2011-2012 ($ Canadian millions)

Source: DFATD, Chief Financial Officer Branch

4.2   Managing DFATD's Relationship with IFAD

The responsibility for managing DFATD's relationship with IFAD lies with the Global Issues and Development Branch, which is guided by DFATD's Institutional Strategy for IFAD.^{Footnote 35} The Global Issues and Development Branch coordinates closely with the Canadian Permanent Mission to the United Nations in Rome to ensure coherent and consistent messages are communicated to IFAD. The DFATD Strategy for Engagement with IFAD is an important tool for supporting the required consistency in DFATD's relationship with the organization.^{Footnote 36} It includes the following three strategic objectives:

• Encourage IFAD to invest in smallholder farmers, notably via increased productivity, improved nutrition and access to markets, with an emphasis on benefiting women and strengthening environmental sustainability;
• Work with IFAD to ensure its programs are coherent, coordinated and focused within the food security systems at the national, regional and international level; and  
• Ensure IFAD consolidates reforms for improved institutional and development effectiveness (via strengthened evaluation and results-based management practices), with improved delivery of high-quality programs, enhanced field presence and strengthened human resource management.

Canada has a de facto permanent seat on the Executive Board and is a member of the List A^{Footnote 37} donor countries (former Category I). Canada is a member of the Audit Committee and was previously involved in the Evaluation Committee.
In recent years, Canada has achieved the following:

• Led, with support from Norway and the Netherlands, a multi-donor review and assessment of the 8th Replenishment, which resulted to improvements in the strategic framework.
• Increased IFAD's focus on programs related to improved nutrition, climate change adaptation, partnerships with the private sector, result-based management and scaling-up of successful programs. It has done this through strategic consensus building and targeted funding for initiatives like nutrition and climate resilience.
• Though still a challenge, DFATD has had a positive impact on IFAD's gender policy and mainstreaming of gender equality operational policies. DFATD, with support from other donors, is promoting a gender-based budget and an increased focus on gender equality results.
• Developed a concise organizational strategy for IFAD (2011 to 2013), which is being updated with a view to providing more clarity on Canada's strategic goals and expectations.
• Advocated consistently for increased investment in systems for strengthening management for institutional and development efficiency and effectiveness, including increased decentralization, improved result-based management, evaluation, and results reporting especially at the outcome and impact levels.
• Increased focus on environmental sustainability and resilience (e.g., adaption to climate change) in small-scale agriculture through Canada's support to IFAD's Adaptation for Smallholder Agriculture Program.

4.3   Alignment with DFATD's Thematic Priorities

4.3.1   Increasing Food Security

IFAD's drive to work directly with smallholder farmers is closely aligned with DFATD's Food Security Strategy as both seek to deliver country-specific solutions, which involve increasing the rural poor's access to agricultural technologies, financial services, markets, land and other natural resources. IFAD's programming contributes directly to Canada's food security priority by addressing the needs of 500 million smallholder farms worldwide, on which one-third of the world's population depends for their livelihoods and nutrition. Canada has been working with IFAD to broaden and strengthen partnerships with the private sector. Canada, for example, is supporting IFAD directly in its programming with smallholder farmers and the private sector to improve the production of nutritious food and promote agricultural innovations.

4.3.2   Stimulating Sustainable Economic Growth

IFAD's overarching goal^{Footnote 38} and second objective, which focus on improving the capacity of the rural poor to raise their incomes, contributes to Canada's priority of sustainable economic growth. Though not part of the review, some of IFAD's initiatives are focused on forging stronger public and private sectors. IFAD is using public-private partnerships to accelerate poverty reduction and leverage the resources of the private sector to scale-up development interventions as highlighted in the box below. IFAD policies and strategic plans recognize that "under the right conditions, private sector investment can contribute to pro-poor development that raises incomes and strengthens food security".^{Footnote 39} This is a win-win situation as the private sector gains on the supply side, while smallholder farmers' benefit from the secure market access.

4.3.3   Securing the Future of Children and Youth

IFAD contributes indirectly to Canada's thematic priority of securing the future of children and youth. IFAD recognizes that rural young people are key actors in feeding a growing global population through sustainable agriculture. IFAD has therefore identified the creation of viable opportunities for rural youth as one of its eight principles for engagement. In line with this principle, IFAD have committed to: consider rural youth's needs in the design and implementation of its programs; implement measures and strategies that include rural youth in all initiatives aimed at developing small farm and non-farm enterprises in rural areas; provide training, support and advice to rural youth to take advantage of opportunities; ensure equitable access to services; seek greater inclusion of youth in decision-making; and work with partners to put young rural people top of the development agenda.^{Footnote 40}

5.0  Conclusions

The 32 evaluations reviewed provide reasonable coverage of IFAD's programming and offer adequate confidence for the findings reported. During the period under review (2007 to 2013), IFAD has developed and implemented a number of reforms to improve its operational effectiveness and programming. Many of these changes correspond to the findings of the evaluations used in this review.

Based on the findings and related contributing factors reported, this review concludes the following:

• IFAD has been effective in achieving its stated development objectives. Its programming is resulting in positive benefits for targeted groups, but there are some challenges in expanding gains on a wider scale and influencing national development policies/programs. There is scope for increasing dialogue to improve national policies and reforms as two thirds of the evaluations reported "unsatisfactory" or worse results on this issue. Partnership arrangements and the quality of program design have been noted as important factors contributing to effectiveness in results achievement.
• Relevance was among the most highly rated of the six criteria. IFAD programming was noted as relevant to the needs of beneficiary groups and aligned with national development goals. Effective partnerships were also formed with beneficiary groups, stakeholders and governments.
• IFAD faces challenges in addressing gender equality in its programming (also noted in the IOE's 2010 corporate-level evaluation on IFAD's performance with regard to gender equality and women's empowerment). Although most IFAD programming included measures to address the needs of poor rural women and men, there was a lack of results frameworks to systematically measure gender equality at the program outcome and impact level. Needs analysis was identified as a key contributing factor for mainstreaming gender equality in IFAD programming.
• Challenges were also noted in addressing environmental sustainability effectively. Only forty percent of the evaluations scored "satisfactory" or better. The evaluations reviewed indicate the need to increase focus on this crosscutting theme at both the design stage and throughout the implementation process.
• Sustainability of results was generally satisfactory, especially in regard to continuation of benefits after project completion and ensuring institutional and community capacity. However, there is scope for strengthening the enabling environment within which the agencies operate. Factors that contributed positively to sustainability included attention in the design stage to clearly stated objectives and expected results. There were some instances where the inadequacy and unpredictability of funding through the life of the projects, particularly where joint-funding was relied upon, adversely affected sustainability.
• Evaluations report less positive results for efficiency, especially with regard to efficient systems and procedures and on time implementation. Adequate planning, predictable funding, and staff capacity were key factors contributing to efficiency. IFAD's 2013 corporate-level evaluation on the organization's institutional efficiency and efficiency of its funded operations noted similar weaknesses in this area, highlighting the need for additional improvements to its operations, delivery model and internal processes.
• Evaluations were used to improve development effectiveness. However, challenges were noted regarding monitoring, reporting and results-based management systems. Factors that inhibited effective monitoring and reporting included poor identification of indicators at the design stage and weak capacity of partner and institutions to implement monitoring and evaluation systems.

6.0  Recommendations for DFATD

The following recommendations to DFATD are based on the findings and conclusions outlined above. Implementation will of course require engagement with other shareholders.

DFATD should:

• Encourage IFAD management to ensure that issues related to sustainability are addressed early in project cycles with clearly stated objectives, expected results exit strategies, and attention to adequate follow on resourcing.
• Emphasize the need to effectively address and promote gender equality throughout IFAD's programming. Gender-based analyses, results frameworks, and monitoring of gender equality results would contribute to strengthening the mainstreaming of this crosscutting theme.
• Continue to engage with IFAD to ensure that the environmental sustainability of its operations receives sufficient attention during the design phase and throughout the implementation process, and that results in this area are improved over time.
• Encourage IFAD to make significant efforts to improve the efficiency of its programming in line with IOE's 2013 evaluation on IFAD's institutional efficiency.
• Emphasize the need to strengthen systems for results-based management, including results monitoring and reporting, and the use of appropriate and focused indicators and associated targets.

Appendix 1: Evaluation Population

Appendix 2: Methodology

This annex provides a more thorough explanation of the key elements of the methodology used for the development effectiveness review of IFAD. It is structured around the sequence of tasks undertaken during the review: determining the rationale for the review; drawing the sample of evaluations; undertaking the process of review and controlling for quality during the analysis phase; and assessing the level of coverage provided by the development effectiveness review.

The meta-synthesis of evaluation reports was supplemented by a review of IFAD and DFATD corporate documents (Appendix 5), as well as interviews with a small number of DFATD and IFAD personnel responsible for managing the relationship between DFATD and IFAD. Additional interviews were conducted with donor partners that work closely with Canada and IFAD. The interviews contextualized the results of the meta-synthesis of evaluations, and took account of advances that have taken place in recent years.

Rationale

The methodology used in the review is a systematic and structured meta-synthesis of the findings of the thirty-two (32) evaluations that passed the quality screening and were published by the IOE between 2007 and 2013 as they relate to six main criteria and 19 sub-criteria that are considered to be essential elements of effective development (see Chapter 2.3).

The methodology offers a more rapid and cost effective way to assess the effectiveness rather than the more costly and time-consuming joint evaluation.^{Footnote 41} The methodology was developed to fill an information gap regarding the development effectiveness of multilateral organizations and originally established to be complementary to the assessment conducted by MOPAN. Although these organizations produce annual reports for their management and/or boards, bilateral shareholders were not receiving a comprehensive overview of the performance on the ground of multilateral organizations.

The methodology suggests conducting a review based on the organization's own evaluation reports when two specific conditions exist:^{Footnote 42}

• There is a need for field-tested and evidence-based information on the effectiveness of the multilateral organization; and
• The multilateral organization under review has an evaluation function that produces an adequate body of reliable and credible evaluation information that supports the use of a meta-evaluation methodology to synthesize an assessment of the organization's effectiveness.

The first condition is satisfied, as IFAD's reporting mechanism did provide sufficient, field-tested information on the organization's development effectiveness. The second condition is also satisfied, as IFAD's existing reporting mechanisms produced a sufficient number of robust evaluation reports (305 projects and 33 country evaluations from 2006 to 2013)^{Footnote 43} to support an assessment of the organization's development effectiveness.

IFAD's Evaluation Function (Quantity and Quality)

Quantity of IFAD Evaluations

IFAD's Independent Office of Evaluation (IOE) conducts different types of evaluations, including corporate level evaluations, country program evaluations and project evaluations (see Figure 1 for an overview of the different types).

IFAD's Evaluation website (ifad.org/evaluation) identifies 33 evaluation reports at the programming and corporate levels conducted by the IOE and published between 2007 and 2013 (see Appendix 1 for list). These were all managed by the IOE.

The evaluations published by IFAD since 2007 create a substantial pool of reports that is large and diverse enough to support a meta-synthesis approach for assessing the development effectiveness of the organization.

Quality of IFAD Evaluations

The 2008 Peer Review concluded that IFAD had created a credible, independent evaluation function and prepared a comprehensive Evaluation Manual. It also noted that independent evaluation is supported and valued in IFAD and that there has been some notable use of evaluations, with some affecting IFAD corporate policies and country strategies. However, the review panel recommended that IOE:

• harmonize its approach to evaluation with that of the Evaluation Cooperation Group good practice by basing its portfolio and project assessment more heavily on evidence drawn from validated project completion reports;
• further strengthen the use of evaluation findings, learning and the feedback loop; and
• identify ways to improve further the quality through use of a broader range of evaluation, approaches and methodologies.

The review team conducted its own quality review of the evaluations included in this review. The results were positive, with 87.5% of the reviewed evaluations scoring 31 points or more out of a possible total of 40 (see Appendix 3: Evaluation Quality Scoring Guide and Results). Given these results, 32 of the IOE's 33 evaluation reports completed within the scope of the evaluation were included in the analysis.

Selecting the Evaluation Sample

The IOE published thirty-three (33) evaluations at the programming and corporate levels between 2007 and 2013 (described in more detail in Appendix 1). During the screening process, four evaluations were to be removed from the universe of 33 evaluations for scoring too low on the three screening criteria for use of multiple lines of evidence, good evaluation design, and relevant/evidence based conclusions.^{Footnote 44} The methodology requires that each evaluation score about 10 points for these three criteria combined. Three of the four evaluations scored 9/10 while one scored 5/10. Without considering the score on these criteria, the three evaluations that scored 9/10 had relatively high overall scores for quality, so a decision was made to include these despite their slightly low scores on G,H and I. The final sample of 32 evaluations is comprised of 23 country program evaluations and 9 corporate-level evaluations.

Geographically, the distribution of the 23 country program evaluations in the sample was as follows: 6 evaluations from East and Southern Africa; 5 evaluations from Asia and the Pacific; 5 evaluations from West and Central Africa; 4 evaluations from Near East, North Africa and Europe; and 3 evaluations from Latin America and the Caribbean. The country programs provided a reasonable representation of the population of countries being assisted by IFAD. Based on the information available, it was not possible to calculate the percentage of IFAD expenditures represented by the 32 evaluations.

The following chart outlines the distribution of the 32 reports by year of publication and by type of evaluation

Challenges were encountered in attempting to determine the coverage of the evaluations with regards to the total IFAD development expenditures. This was particularly difficult corporate-level evaluations that cover multiple programming components or only specific theme as part of a program area.

Following a detailed analysis of the coverage of evaluations and a review of IFAD's recent ARRIs, the review team were able to conclude that the evaluations represented a reasonable representation of IFAD's programming.

While one could draw this conclusion, it would seem that there is a need for the IOE to track more explicitly the value of spending in each country covered by the evaluations and to prepare an annual summary of the significance of the country selection for evaluation each year.

Review Process and Quality Assurance

The review itself was conducted by a team of three analysts and a team leader. A training session was held for analysts to build a common understanding of the review criteria. Following, the team leader and analysts conducted a pre-test in order to independently review two evaluations. The team compared their ratings for these two evaluations and developed common agreement on the classification of results for all sub-criteria. This process helped to standardize classification decisions made by the analysts, thus maximizing inter-rater reliability.

During the review of evaluations, analysts conferred regularly over any classification issues that arose. A second test occurred at the approximate mid-point of the review period, with all analysts independently rating a third evaluation. As previously, the analysts compared their respective classification decisions and resolved discrepancies.

Once the reviews were completed, the team leader reviewed the coded findings and carefully examined the cited evidence and contributing factors. Based on this examination, the team leader made a small number of adjustments to the coded findings. The process of training, testing and monitoring the review process minimized any inter-rater reliability issues and controlled for bias on the part of any one reviewer.

Appendix 3: Evaluation Quality Scoring Guide and Results

A. Purpose of the evaluation:

• why the evaluation was done (1)
• what triggered the evaluation (including timing in the project/program cycle) (1)
• how evaluation is to be used (1)

Maximum Points: 3

B. Evaluation objectives

• evaluation objectives are clearly stated (1)
• objectives logically flow from purpose (1)

Maximum Points: 2

C. Organization of the evaluation

• logical structure to the organization of the evaluation (1)
• evaluation is well written (1)
• clear distinction between evidence, findings, conclusions, and recommendations (1)

Maximum Points: 3

D. Subject evaluated is clearly described

Evaluation describes:

• the activity/program being evaluated (1)
• the program’s expected achievements (1)
• how the program addresses the development problem (1)
• the implementation modalities used (1)

Maximum Points: 4

E. Scope of the evaluation

Evaluation defines the boundaries of the evaluation in terms of:

• time period covered (1)
• implementation phase under review (1)
• geographic area (1)
• dimensions of stakeholder involvement being examined (1)

Maximum Points: 4

F. Evaluation criteria

Evaluation criteria include:

• the achievement of development objectives and expected results (including impacts) (1)
• cross-cutting issues: inclusive development which is gender sensitive and environmentally sustainable (1)
• the sustainability of benefits and positive results achieved (1)
• the relevance of MO activities and supported projects and programs (1)
• the efficiency of MO operations in support of projects and programs (1)

Maximum Points: 5

G. Multiple lines of evidence

• one point (1) for each line of evidence used (case studies, surveys, site visits, interviews, etc.), up to a maximum of four points (4)

Maximum Points: 4

H. Evaluation design

Elements of a good evaluation design include:

• an explicit theory of how objectives and results were to be achieved (1)
• specification of the level of results achieved (output, outcome, impact) (1)
• baseline data (quantitative or qualitative) on conditions prior to program implementation (1)
• comparison of conditions after program delivery to those before (1)
• a qualitative or quantitative comparison of conditions among program participants and a control group (1)

Maximum Points: 5

I. Evaluation findings and conclusions are relevant and evidence based

Evaluation report includes:

• evaluation findings relevant to the assessment criteria (1)
• findings that are supported by the chosen methodology (1)
• a clear logical link between the evidence and the finding (1)
• conclusions which are linked to the evaluation findings as reported (1)

Maximum Points: 4

J. Evaluation limitations

• statement of the limitations of the methodology (1)
• impact of limitations on evaluation (1)
• remedies of limitations (1)

Maximum Points: 3

K. Evaluation Recommendations

• evaluation contains recommendations that flow from findings and conclusions (1)
• recommendations are directed to one or more organization (1)
• recommendations are aimed at improving development effectiveness (1)

Maximum Points: 3

Total (required to have a minimum of 25 points): 40

Total for Criteria G, H and I (required to have minimum of 10 points): 13

Evaluation Quality Scoring Results

During the Pilot Testing of the methodology, the Management Group of participating development agencies guiding the work on behalf of DAC-EVALNET suggested grouping quality score results for each evaluation into groups of five (in total score). This was seen as presenting the best level of "granularity" and transparency. It allows independent observers to reach their own conclusions on the distribution of quality scores.

Appendix 4: Guide for Classifying Evaluation Findings

Common Development Evaluation Assessment Criteria

1. Achievement of Development Objectives and Expected Results

1.1 MO supported programmes and projects achieve their stated development and/or humanitarian objectives and attain expected results.

(1) Highly Unsatisfactory

Less than half of stated output and outcome objectives have been achieved including one or more very important output and/or outcome level objectives.

(2) Unsatisfactory

Half or less than half of stated output and outcome level objectives are achieved.

(3) Satisfactory

MO supported programs and projects either achieve at least a majority of stated
output and outcome objectives (more than 50% if stated) or that the most important of stated output and outcome objectives are achieved.

(4) Highly Satisfactory

MO supported programs and projects achieve all or almost all significant development and/or humanitarian objectives at the output and outcome level.

1.2 MO supported programs and projects have resulted in positive benefits for target group members.

(1) Highly Unsatisfactory

Problems in the design or delivery of MO supported

activities mean that expected positive benefits for target group members have not occurred or are unlikely to occur.

(2) Unsatisfactory

MO supported projects and programs result in no or very few positive changes experienced by target group members. These benefits may include the avoidance or reduction of negative effects of a sudden onset or protracted

(3) Satisfactory

MO supported projects and programs have resulted in positive changes experienced by target group members (at the individual, household or community level). These benefits may include the avoidance or reduction of negative effects of a sudden onset or protracted emergency.

(4) Highly Satisfactory

MO supported projects and programs have resulted in widespread and significant positive changes experienced by target group members as measured using either quantitative or qualitative methods (possibly including comparison of impacts with non program participants). These benefits may include the avoidance or reduction of negative effects of a sudden onset or protracted emergency.

1.3 MO programs and projects made differences for a substantial number of beneficiaries and where appropriate contributed to national development goals.

(1) Highly Unsatisfactory

MO supported projects and programs have not contributed to positive changes in the lives of beneficiaries as measured quantitatively or qualitatively.

(2) Unsatisfactory

MO supported projects and programs have contributed to positive changes in the lives of only a small number of beneficiaries (when compared to project or program targets and local or national goals if established).

(3) Satisfactory

MO supported projects and programs have contributed to positive changes in the lives of substantial numbers of beneficiaries as measured quantitatively or qualitatively. These may result from development, relief, or protracted relief and rehabilitation operations and may include the avoidance of negative effects of emergencies.

(4) Highly Satisfactory

MO supported projects and programs have contributed to positive changes in the lives of substantial numbers of beneficiaries. Further, they have contributed to the achievement of specific national development goals or have contributed to meeting humanitarian relief objectives agreed to with the national government and/or national and international development and relief organizations.

1.4 MO activities contributed to significant changes in national development policies and programs (including for disaster preparedness, emergency response and rehabilitation) (policy impacts) and/or to needed system reforms.

(1) Highly Unsatisfactory

National policies and programs in a given sector or area of development (including disaster preparedness, emergency response and rehabilitation) were deficient and required strengthening but MO activities have not addressed these deficiencies.

(2) Unsatisfactory

MO activities have not made a significant contribution to the development of national policies and programs in  given sector or area of development, disaster preparedness, emergency response or rehabilitation. (Policy changes in humanitarian situations may include allowing access to the effected populations).

(3) Satisfactory

MO activities have made a substantial contribution to either re-orienting or sustaining effective national policies and programs in a given sector or area of development disaster preparedness, emergency response or rehabilitation.

(4) Highly Satisfactory

MO activities have made a substantial contribution to either re-orienting or sustaining effective national policies and programs in a given sector or area of development disaster preparedness, emergency response or rehabilitation. Further, the supported policies and program implementation modalities are expected to result in improved positive impacts for target group members.

2. Cross Cutting Themes: Inclusive Development Which can be Sustained

2.1 Extent MO supported activities effectively address the crosscutting issue of gender equality.

(1) Highly Unsatisfactory

MO supported activities are unlikely to contribute to gender equality or may in fact lead to increases in gender inequalities.

(2) Unsatisfactory

MO supported activities either lack gender equality objectives or achieve less than half of their stated gender equality objectives. (Note: where a program or activity is clearly gender-focused (maternal health programming for example) achievement of more than half its stated objectives warrants a satisfactory rating).

(3) Satisfactory

MO supported programs and projects achieve a majority (more than 50%) of their stated gender equality objectives.

(4) Highly Satisfactory

MO supported programs and projects achieve all or nearly all of their stated gender equality objectives.

2.2 Extent changes are environmentally sustainable

(1) Highly Unsatisfactory

MO supported programs and projects do not include planned activities or project design criteria intended to promote environmental sustainability. In addition changes resulting from MO supported programs and projects are not environmentally sustainable.

(2) Unsatisfactory

MO supported programs and projects do not include planned activities or project design criteria intended to promote environmental sustainability. There is, however, no direct indication that project or program results are not environmentally sustainable. OR MO supported programs and projects include planned activities or project design criteria intended to promote sustainability but these have not been successful.

(3) Satisfactory

MO supported programs and projects include some planned activities and project design criteria to ensure environmental sustainability. These activities are implemented successfully and the results are environmentally sustainable.

(4) Highly Satisfactory

MO supported programs and projects are specifically designed to be environmentally sustainable and include substantial planned activities and project design criteria to ensure environmental sustainability. These plans are implemented successfully and the results are environmentally sustainable.

3. Sustainability

3.1 Benefits continuing or likely to continue after project or program completion or there are effective measures to link the humanitarian relief operations, to rehabilitation, reconstructions and, eventually, to longer-term developmental results.

(1) Highly Unsatisfactory

There is a very low probability that the program/project will result in continued intended benefits for the target group after project completion. For humanitarian relief operations, the evaluation finds no strategic or operational measures to link relief, to rehabilitation, reconstruction and, eventually, to development

(2) Unsatisfactory

There is a low probability that the program/project will result in continued benefits for the target group after completion. For humanitarian relief operations, efforts to link the relief phase to rehabilitation, reconstruction and, eventually, to development are inadequate. (Note, in some circumstances such linkage may not be possible due to the context of the emergency. If this is stated in the evaluation, a rating of satisfactory can be given)

(3) Satisfactory

Likely that the program or project will result in continued benefits for the target group after completion. For humanitarian relief operations, the strategic and operational measures to link relief to rehabilitation, reconstruction and, eventually, development are credible.

(4) Highly Satisfactory

Highly likely that the program or project will result in continued benefits for the target group after completion. For humanitarian relief operations, the strategic and operational measures to link relief to rehabilitation, reconstruction and, eventually, development are credible. Further, they are likely to succeed in securing continuing benefits for target group members.

3.2 Extent MO supported projects and programs are reported as sustainable in terms of institutional and/or community capacity.

(1) Highly Unsatisfactory

The design of MO supported programs and projects failed to address the need to strengthen institutional and/or community capacity as required. In the case of humanitarian operations, the design of programs and projects failed to take account of identified needs to strengthen local capacities for delivery of relief operations and/or for managing the transition to rehabilitation and/or development.

(2) Unsatisfactory

MO programs and projects may have failed to contribute to strengthening institutional and/or community capacity or, where appropriate, to strengthen local capacities for delivery of relief operations and/or for managing the transition to rehabilitation and/or development.

(3) Satisfactory

MO programs and projects may have contributed to strengthening institutional and/or community capacity but with limited success.

(4) Highly Satisfactory

Either MO programs and projects have contributed to significantly strengthen institutional and/or community capacity as required or institutional partners and communities already had the required capacity to sustain program results.

3.3 Extent MO development programming contributes to strengthening the enabling environment for development.

(1) Highly Unsatisfactory

For development programs, there were important weaknesses in the enabling environment for development (the overall framework and process for national development planning; systems and processes for public consultation and for participation by civil society in development planning; governance structures and the rule of law; national and local mechanisms for accountability for public expenditures, service delivery and quality; and necessary improvements to supporting structures such as capital and labour markets). Further, the MO activities and support provided to programs and projects failed to address the identified weakness successfully, further limiting program results.

(2) Unsatisfactory

MO development activities and/or MO supported projects and programs have not made a notable contribution to changes in the enabling environment for development.

(3) Satisfactory

MO development activities and/or MO supported projects and programs have made a notable contribution to changes in the enabling environment for development including one or more of: the overall framework and process for national development planning; systems and processes for public consultation and for participation by civil society in development planning; governance structures and the rule of law; national and local mechanisms for accountability for public expenditures, service delivery and quality; and necessary improvements to supporting structures such as capital and labour markets.

(4) Highly Satisfactory

MO development activities and/or MO supported projects and programs have made a significant contribution to changes in the enabling environment for development including one or more of: the overall framework and process for national development planning; systems and processes for public consultation and for participation by civil society in development planning; governance structures and the rule of law; national and local mechanisms for accountability for public expenditures, service delivery and quality; and necessary improvements to supporting structures such as capital and labour markets. Further, these improvements in the enabling environment are leading to improved development outcomes.

4. Relevance

4.1 MO supported programs and projects are suited to the needs and/or priorities of the target group

(1) Highly Unsatisfactory

Substantial elements of program or project activities and outputs were unsuited to the needs and priorities of the target group.

(2) Unsatisfactory

No systematic analysis of target group needs and priorities took place during the design phase of developmental or relief and rehabilitation programming or there is some evident mismatch between program and project activities and outputs and the needs and priorities of the target group.

(3) Satisfactory

MO supported activity, program or project is designed taking into account the needs of the target group as identified through a process of situation or problem analysis (including needs assessment for relief operations) and the resulting activities are designed to meet the needs of the target group.

(4) Highly Satisfactory

Methods used in program and project development (including needs assessment for relief operations) to identify target group needs and priorities (including consultations with target group members) and the program and project takes those needs into account and is designed to meet those needs and priorities (whether or not it does so successfully).

4.2 MO supported development projects and programs align with national development goals:

(1) Highly Unsatisfactory

Significant elements of MO supported development program and project activity run counter to national development priorities with a resulting loss of effectiveness.

(2) Unsatisfactory

Significant portion (1/4 or more) of the MO supported development programs and projects are not aligned with national plans and priorities, but there is no evidence that they run counter to those priorities.

(3) Satisfactory

Most MO supported development programs and projects are fully aligned with national plans and priorities as expressed in national poverty eradication and sector plans and priorities. Wherever MO supported programs and projects are reported in the evaluation as not directly supportive of national plans and priorities they do not run

(4) Highly Satisfactory

All MO supported development projects and programs are reported in the evaluation to be fully aligned to national development goals as described in national and sector plans and priorities, especially including the national poverty eradication strategy and sector strategic priorities.

4.3 MO has developed an effective partnership with governments, bilateral and multilateral development and humanitarian organizations and NGOs for planning, coordination and implementation of support to development and/or emergency preparedness, humanitarian relief and rehabilitation efforts.

(1) Highly Unsatisfactory

MO experiences significant divergence in priorities from those of its (government, NGO or donor) partners and lacks a strategy or plan which will credibly address the divergence and which should result in strengthened partnership over time.

(2) Unsatisfactory

MO has experienced significant difficulties in developing an effective relationship with partners and that there has been significant divergence in the priorities of the MO and its partners.

(3) Satisfactory

MO has improved the effectiveness of its partnership relationship with partners over time during the evaluation period and that this partnership was effective at the time of the evaluation or was demonstrably improved.

(4) Highly Satisfactory

MO has consistently achieved a high level of partnership during the evaluation period.

5. Efficiency

5.1 Program activities are evaluated as cost/resource efficient:

(1) Highly Unsatisfactory

Credible information indicating that MO supported programs and projects (development, emergency preparedness, relief and rehabilitation) are not cost/resource efficient.

(2) Unsatisfactory

MO supported programs and projects under evaluation (development, Emergency preparedness, relief and rehabilitation) do not have credible, reliable information on the costs of activities and inputs and therefore the evaluation is not able to report on cost/resource efficiency. OR MO supported programs and projects under evaluation present mixed findings on the cost/resource efficiency of the inputs.

(3) Satisfactory

Level of program outputs achieved (development, emergency preparedness, relief and rehabilitation) when compared to the cost of program activities and inputs are appropriate even when the program design process did not directly consider alternative program delivery methods and their associated costs.

(4) Highly Satisfactory

MO supported (development, emergency preparedness, relief and rehabilitation) programs and projects are designed to include activities and inputs that produce outputs in the most cost/resource efficient manner available at the time.

5.2 Evaluation indicates implementation and objectives achieved on time (given the context, in the case of humanitarian programming)

(1) Highly Unsatisfactory

Less than half of stated output and outcome level objectives of MO supported programs and projects are achieved on time, there is no credible plan or legitimate explanation found by the evaluation which would suggest significant improvement in on-time objectives achievement in the future.

(2) Unsatisfactory

Less than half of stated output and outcome level objectives of MO supported programs and projects are achieved on time but the program or project design has been adjusted to take account of difficulties encountered and can be expected to improve the pace of objectives achievement in the future. In the case of humanitarian programming, there was a legitimate explanation for the delays.

(3) Satisfactory

More than half of stated output and outcome level objectives of MO supported programs and projects are achieved on time and that this level is appropriate to the context faced by the program during implementation, particularly for humanitarian programming.

(4) Highly Satisfactory

Nearly all stated output and outcome level objectives of MO supported programs and projects are achieved on time or, in the case of humanitarian programming, a legitimate explanation for delays in the achievement of some outputs/outcomes is provided.

5.3 Evaluation indicates that MO systems and procedures for project/program implementation and follow up are efficient (including systems for engaging staff, procuring project inputs, disbursing payment, logistical arrangements etc.)

(1) Highly Unsatisfactory

Serious deficiencies in agency systems and procedures for project/program implementation that result in significant delays in project start-up, implementation or completion and/or significant cost increases.

(2) Unsatisfactory

Some deficiencies in agency systems and procedures for project/program implementation but does not indicate that these have contributed to delays in achieving project/program objectives.

(3) Satisfactory

Agency systems and procedures for project implementation are reasonably efficient and have not resulted in significant delays or increased costs.

(4) Highly Satisfactory

Efficiency of agency systems and procedures for project implementation represent an important organizational strength in the implementation of the program under evaluation.

6. Using Evaluation and Monitoring to Improve Development Effectiveness

6.1 Systems and process for evaluation are effective.

(1) Highly Unsatisfactory

Evaluation practices in use for programs and projects of this type (development, emergency preparedness, relief and rehabilitation) are seriously deficient.

(2) Unsatisfactory

No indication that programs and projects of this type (development, emergency preparedness, relief and rehabilitation) are subject to systematic and regular evaluations

(3) Satisfactory

Program being evaluated is subject to systematic and regular evaluations or describes significant elements of such practice. No mention of policy and practice regarding similar programs and projects. This may include specialized evaluation methods and approaches to emergency preparedness, relief and rehabilitation programming.

(4) Highly Satisfactory

Program being evaluated (along with similar programs and projects) is subject to systematic regular evaluations or describes significant elements of such practice.

6.2 Systems and processes for monitoring and reporting on program results are effective

(1) Highly Unsatisfactory

Absence of monitoring and reporting systems for the development and humanitarian programming. This would include the absence of adequate monitoring of outputs during the implementation of humanitarian programming.

(2) Unsatisfactory

While monitoring and reporting systems for the development and humanitarian programming exist they either do not report on a regular basis or they are inadequate in frequency, coverage or reliability.

(3) Satisfactory

Monitoring and reporting systems for development and humanitarian programming as appropriate are well established and report regularly.

(4) Highly Satisfactory

Monitoring and reporting systems for the program are well – established and report regularly. The quality of regular reports is rated by the evaluation and results are reportedly used in the management of the program.

6.3 Results Based Management (RBM) systems are effective

(1) Highly Unsatisfactory

No evidence of the existence of an RBM system for the program and no system is being developed.

(2) Unsatisfactory

While an RBM system is in place, or being developed, it is unreliable and does not produce regular reports on program performance.

(3) Satisfactory

RBM system is in place and produces regular reports on program performance.

(4) Highly Satisfactory

RBM system is in place for the program and there is evidence noted in the evaluation that the system is used to make changes in the program to improve effectiveness.

6.4 MO makes use of evaluation to improve development/humanitarian effectiveness

(1) Highly Unsatisfactory

Report does not include a management response and does not have one appended to it or associated with it. There is no indication of how the evaluation results will be used. There is no indication that similar evaluations have been used to improve effectiveness in the past.

(2) Unsatisfactory

Report includes a management response (or has one attached or associated with it) but it does not indicate which recommendations have been accepted.
OR
There is some, non-specific indication that been used to improve program effectiveness in the past.

(3) Satisfactory

Report includes a management response (or has one attached or associated with it) that indicates which recommendations have been accepted.
OR
There is a clear indication that similar evaluations in the past have been used to make clearly identified improvements in program effectiveness.

(4) Highly Satisfactory

Report includes a management response (or has one attached or associated with it) describes a response to each major recommendation which is appropriate and likely to result in the organizational and programmatic changes needed to achieve their intent.

Appendix 5: Corporate Documents Reviewed

IFAD Documents

Annual Report 2012, IFAD, 2012

Annual Report 2011, IFAD, 2011

Annual Report 2010, IFAD, 2010

Annual Report 2009, IFAD, 2009

Annual Report 2008, IFAD, 2008

Annual Report 2007, IFAD, 2007

Annual Report on Results and Impact (ARRI) of IFAD Operations Evaluated in 2012 prepared by the Independent Office of Evaluation, 2013

Annual Report on Results and Impact (ARRI) of IFAD Operations Evaluated in 2011 prepared by the Independent Office of Evaluation, 2012

Annual Report on Results and Impact (ARRI) of IFAD Operations Evaluated in 2010 prepared by the Independent Office of Evaluation, 2011

Annual Report on Results and Impact (ARRI) of IFAD Operations Evaluated in 2009 prepared by the Office of Evaluation, 2010

Annual Report on Results and Impact (ARRI) of IFAD Operations Evaluated in 2008 prepared by the Office of Evaluation, 2009

Annual Report on Results and Impact (ARRI) of IFAD Operations Evaluated in 2007 prepared by the Office of Evaluation, 2008

Evaluation Policy, IFAD, May 2011

Evaluation Manual, Methodology and Processes, IFAD Office of Evaluation, April 2009

Portfolio Performance Report 2006/07, Volume I & II, Executive Board, December 2007

Report on IFAD's Development Effectiveness, Executive Board, December 2012

Report on IFAD's Development Effectiveness, Executive Board, December 2011

Report on IFAD's Development Effectiveness, Executive Board, December 2010

Report on IFAD's Development Effectiveness, Executive Board, December 2009
Report on IFAD's Development Effectiveness, Executive Board, December 2008

Other Documents

Development Effectiveness Review of the Asian Development Bank 2006 – 2010, ADB, March 2013

Development Effectiveness Review of the United Nations Development Programme 2005-2011 Synthase Report, Canadian International Development Agency, April 2012

Review of the World Food Programme's Humanitarian and Development Effectiveness 2006-2011, Synthesis Report, Canadian International Development Agency, March 2012

Development Effectiveness Review of the World Health Organization 2007–2010, Canadian International Development Agency, December 2012

Multilateral Development Banks' Common Performance Assessment System 2012, COMPAS 2012 Report, January 2014

Multilateral Development Banks' Common Performance Assessment System 2011, COMPAS 2011, Report, 2011

Multilateral Development Banks' Common Performance Assessment System 2010, COMPAS 2010 Report, 2010

Multilateral Development Banks' Common Performance Assessment System 2011, COMPAS 2009 Report, 2009

Multilateral Organisation Performance Assessment Network (MOPAN) Institutional Report International Fund for Agriculture (IFAD) 2013 Volume I & II, IFAD, November 2013

MOPAN Common Approach International Fund for Agricultural Development (IFAD) 2010 Volume I & II, MOPAN, January 2011

Appendix 6: DFATD Funding to Multilateral Development Organizations

Long-term Institutional Funding^{Footnote 45}

Long-term institutional funding (core funding) can be defined as un-earmarked funding to a multilateral organization in support of that organization's mandate. According to the Organization for Economic Co-operation and Development (OECD) there are currently 170 multilateral organizations active in development and eligible to receive aid funding. As of 2010/11, DFATD provided long-term institutional funding to 30 of these multilateral organizations. DFATD's funding was highly concentrated with nine multilateral organizations receiving 80% of its total long-term institutional funding from 2007/08 to 2010/11.

Funding to Specific Multilateral and Global Initiatives

Specific multilateral and global funding can be defined as funding to multilateral organizations in support of a key program or activity usually in a specific thematic area and often global in scope. Within this category, there are two sub-types: 1) humanitarian assistance and 2) other global initiatives programming.

Humanitarian assistance is provided based on need and usually in response to specific appeals issued by multilateral organizations with expertise in providing humanitarian assistance. The main multilateral partners involved in providing humanitarian assistance are WFP, United Nations High Commissioner for Refugees, the International Committee of the Red Cross and Office for the Coordination of Humanitarian Affairs. United Nations Children's Fund, although not primarily a humanitarian organization, also delivers humanitarian assistance with a specific emphasis on the needs of children.

The second sub-type of specific multilateral and global funding involves global initiatives in other sectors. These initiatives are in sectors that deal with issues, which transcend borders and thus lend themselves to a multilateral approach. The main sectors DFATD supports with this type of funding are health, environment and economic growth. The health sector is the most important of these, especially in light of the challenges of infectious diseases like AIDS and tuberculosis, which do not respect international borders. Bilateral programming in a single country is unlikely to succeed in meeting the challenges of infectious diseases in the absence of regional and global programs.

Funding to Multilateral Initiatives Delivered by other DFATD Branches

Multilateral initiatives can also receive funding from other DFATD branches, mostly through multi-bi funding from Geographic programs. Multi-bi funding refers to earmarked funding to a specific multilateral organization initiative by a DFATD geographic program to support a specific activity in a specific country or group of countries. It is considered "bilateral" assistance because it is funded through DFATD's geographic programs in the context of the program's country strategies or programming frameworks.

At the time of the evaluation, multi-bi funding accounted for a large and growing share of DFATD resources. It more than tripled in the five years from 2002/03 to 2007/08, mainly as a result of substantial funding to programs in fragile states. By 2007/08, DFATD multi-bi funding had reached $691 million, with 53% spent in fragile states including 37% of all multi-bi funding spent in Afghanistan.

In fragile states, where United Nations (UN) organizations and the World Bank are often assigned specific roles by member governments, use of multi-bi funding by DFATD can sometimes help the Agency to limit fiduciary risk and result in a reduced administrative burden on the very weak national institutions. The use of this type of funding is also consistent with Canada's commitment to the Paris Declaration principles of aid effectiveness, which includes a call for donors to harmonize their aid and use program based approaches where they can be effective.

It is important to note that DFATD's geographic programs manage multi-bi funding according to the same basic processes that govern all of the Department's geographic programming. For example, DFATD's geographic programs are responsible for monitoring and reporting on the effectiveness of funds used in this way. Country program evaluations, which examine DFATD's bilateral programs in a given country include in their remit programming delivered by multilateral organizations and supported by multi-bi funding.

Appendix 7: DFATD Management Response

By focusing on improving food security and nutrition, the International Fund for Agricultural Development (IFAD) seeks to move 80 million smallholder farmers out of poverty by 2015.  As a specialized agency of the United Nations, and an international financial institution, IFAD provides loans and grants to support smallholder farmers in the developing world to increase production and access markets, fostering more employment opportunities for rural men, women and youth. Support to IFAD contributes to the achievement of three of Canada's international development priorities: increasing food security (including nutrition), stimulating sustainable economic growth, and securing the future of children and youth, making it a key multilateral partner for DFATD in achieving Canada's development objectives in these priority areas.

DFATD agrees with the Review's recommendations for improving IFAD's performance in the areas of results-based management and results monitoring and reporting; institutional efficiency, especially at country level; mainstreaming cross-cutting issues of gender equality and environmental sustainability; and sustainability of benefits. The findings and recommendations of the Review will serve to guide DFATD's continued engagement with IFAD, including the prioritization of Canada's objectives for the tenth replenishment of IFAD (2016-2018). 

During its Ninth Replenishment Period (2013-2015), IFAD has initiated a number of actions in the following four areas: operational effectiveness, institutional effectiveness and efficiency, financial management, and results management. The findings of the mid-term review of IFAD9 indicate that the organization is on track to meet the majority of its commitments. The results for 2013, measured against the 2015 Results Measurement Framework indicators and targets,  reveal that good progress is being made by IFAD to improve its development impact, and raise its operational effectiveness and institutional efficiency, thereby demonstrating a positive trend towards addressing some of the shortcomings identified in this Review.

DFATD will take concrete actions as outlined in the table below to address the Review's recommendations, using the opportunities available through its representation in IFAD's Governing Bodies, notably the Governing Council and the Executive Board, and other structures such as the Audit Committee.  

Recommendation 1

Encourage IFAD management to ensure that issues related to sustainability, are addressed early in project cycles with clearly stated objectives and expected results, exit strategies, and attention to follow-on resourcing.

Commitments/measures

Agreed.

IFAD has significantly sharpened its focus on scaling-up successful interventions, and has increased its country presence to foster the sustainability of results.

1.1 Canada will press IFAD to act on the following recommendations to improve the sustainability of its investments and results:

1.1.1 ensure sufficient resources for scaling-up successful interventions;

1.1.2 build local capacities, including strengthening national monitoring and evaluation systems;

1.1.3 systematically assess the impacts of scaling-up successful innovations and initiatives.

Responsible

Food Security Division  (MSF), Global Issues and Development (MSM)

Completion date

February 2015.

Ongoing follow-up, including through mid-term and final reviews of IFAD10.

Recommendation 2

Emphasize the need to effectively address and promote gender equality throughout IFAD’s programming. Gender-based analyses, results frameworks, and monitoring of gender equality results would contribute to strengthening the mainstreaming of this crosscutting theme.

Commitments/measures

Agreed.

2.1 In October 2014, the IFAD10 Consultation will present an updated Results Measurement Framework (RMF). Canada will use this opportunity to recommend that the updated RMF includes clear and measurable indicators, baselines and targets to enable IFAD to mainstream gender in its operations at corporate and field levels, and to systematically track and report gender equality results.

2.2 Canada will follow-up on IFAD10 commitments and continue to press IFAD to improve GE mainstreaming and reporting of GE results, taking the annual Report on IFAD’s Development Effectiveness (RIDE), which includes a dedicated section on progress of its Policy on Gender Equality and Women’s Empowerment, as annual opportunities to pursue this objective.

2.3 DFATD will initiate a new Gender Equality Institutional Assessment (GEIA) of IFAD prior to the next replenishment period (2016-2018).

Responsible

2.1 Food Security Division (MSF), Global Issues and Development (MSM)

2.2 Food Security Division (MSF), Global Issues and Development (MSM)

2.3 Food Security Division (MSF), Global Issues and Development (MSM)

Completion date

2.1 Completed as of December 2014

2.2 Completed as of December 2014 and ongoing

2.3 December 2015

Recommendation 3

Continue to engage with IFAD to ensure that the environmental sustainability of its operations receives sufficient attention during the design phase and throughout the implementation process, and that results in this area are improved over time.

Commitments/measures

Agreed.

3.1 Canada will continue to advocate for improved performance on natural resource and environmental management through IFAD’s Executive Board, by engaging on:

• the results of IFAD’s forthcoming Evaluation Synthesis Report on Natural Resource and Environmental Management (to be completed in 2015) and related follow-up;
• the findings of IFAD’s mid-term review of ASAP, which will take place in 2015; and
• IFAD’s annual results reporting, which includes trends related to performance on natural resource and environmental management, and dedicated reporting on ASAP.

3.2 Canada will encourage IFAD to increase its resource base for this cross-cutting issue by leveraging additional resources for climate finance.

3.3 Canada will monitor the implementation of IFAD’s 10-point action plan to achieve its goal of mainstreaming climate change adaptation into all (100%) IFAD country strategies, project designs, corporate policies, communications, policy dialogue and implementation programs by the end of IFAD10 (2018).

Responsible

3.1 Food Security Division (MSF), Global Issues and Development (MSM)

3.2 Food Security Division (MSF), Global Issues and Development (MSM)

3.3 Food Security Division (MSF), Global Issues and Development (MSM)

Completion date

3.1 December 2015 and beyond through Executive Board participation

3.2 December 2015 and beyond through Executive Board participation

3.3 December 2015

Recommendation 4

Encourage IFAD to make significant efforts to improve the efficiency of its programming in line with IOE’s 2013 evaluation on IFAD’s institutional efficiency.

Commitments/measures

Agreed.

4.1 In line with IFAD9 commitments and in response to the recommendations of the 2013 Corporate Level Evaluation on Efficiency (CLEE), IFAD has revised and consolidated its action plan to enhance efficiency. Based on IFAD’s annual reporting to the Executive Board, Canada will encourage IFAD to accelerate progress in implementing management actions within the timelines specified in IFAD’s consolidated action plan to enhance its institutional efficiency.

Responsible

Food Security Division  (MSF), Global Issues and Development (MSM)

Completion date

December 2018

Recommendation 5

Emphasize the need to strengthen systems for results-based management, including results monitoring and reporting, and the use of appropriate and focused indicators and associated targets.

Commitments/measures

Agreed.

5.1 In October 2014, the IFAD10 Consultation will present an updated Results Measurement Framework (RMF). Canada will use this opportunity to recommend that the updated RMF includes clear and measurable indicators, baselines and targets; establishes clearer linkages between the different levels of the results chain; and has a well-articulated theory of change that credibly describes impact pathways through which IFAD’s programming contributes to development results.

5.2 Canada will also continue to reinforce these messages at the Executive Board sessions of IFAD, including the need to strengthen IFAD’s RBM systems and results monitoring and reporting practices, especially at country level. Canada will also track whether IFAD has enhanced its results reporting in future annual reports on IFAD’s Development Effectiveness (RIDE), starting with the 2014 RIDE that will be presented to the Executive Board in December 2014.

Responsible

5.1 Food Security Division (MSF), Global Issues and Development (MSM)

5.2 Food Security Division (MSF), Global Issues and Development (MSM)

Completion date

5.1 Completed as of December 2014

5.2 Completed as of December 2014 and ongoing

Appendix 8: IFAD Management Response

IFAD believes the Development Effectiveness Review (DER) of IFAD complements the assessments done in 2013 by MOPAN and DFID and contributes to a better understanding of IFAD's role and effectiveness. The endorsement of alignment with member countries' national development goals, the attainment of stated development objectives and the role of a well-prepared logic models in project design are all areas where IFAD has expended time and effort over recent years. We are pleased to see that Canada's DER has recognized this effort.

Nevertheless, we note the importance of extending the rigour of project design to both gender equality and environmental sustainability and believe that the emphasis made in the IFAD10 results management framework will begin to address these concerns. This is supplemented by our initiatives to track gender in the administrative budget and include climate indicators in the Results and Impact Management System (RIMS).

In terms of the broader goals of programme sustainability, we would like to highlight the government ownership of initiatives is part of project design (and the Financing Agreement) and subsequent implementation so there is , 'de facto', no transfer of responsibilities. The onus remains on Government to make the necessary budgetary provisions, and staffing, for any further activities. This also raises the issue of contribution (as opposed to alignment) to national development goals. This is an issue that was raised by the 2013 MOPAN report and we would similarly respond by highlighting that IFAD's interventions are at a programme/project level with, as noted, clear objectives that relate to specific activities. Attribution, given the level of programme financing available, to national goals is not easily correlated.

We recognise the role and importance of IFAD's Results Management Framework at the corporate level and the continued need to support results-based management at project level with the needed levels of IFAD Supervision and Government monitoring. The recent changes to IFAD's Results Measurement Framework (RMF) during the IFAD10 discussions, including the level 5 indicators for corporate efficiency should give us a basis to both manage and report on these issues. We look forward to further collaboration with Canada as we move forward on these important issues.