Development Effectiveness Review of the Inter-American Development Bank - 2008-2013 - Final Report

June 2014

Acknowledgments

The Development Evaluation Division wishes to recognise all who have contributed to this review.

Our thanks go first to the independent team at the consulting firm Universalia Management Group, comprised of Senior Evaluator Katrina Rojas and Consultant Elisabetta Micaro.

We would also like to thank Vaughn Lantz, Vincent Raiche, Mariette Maillet, and Laura Griggs from the Global Issues and Development Branch for their valuable support.  Alejandro Guerrero, from the IDB’s Office of Evaluation and Oversight (OVE), and Tamara Sorger, from the Canadian Executive Director’s Office in Washington, also provided useful and practical advice to the evaluators.

We would also like to acknowledge the engagement in this process of José Manuel Argiles Marín, Head of the Evaluation and Knowledge Management Division at the Spanish International Cooperation for Development Secretariat, and thank him for his comments on the approach and the report.

From the Development Evaluation Division, Gabrielle Biron Hudon managed the exercise, and Andrés Velez-Guerra oversaw it.

James Melanson
Head of Development Evaluation

List of Abbreviations

Executive Summary

Background

The Inter-American Development Bank (IDB) is the oldest development bank in Latin America and the Caribbean. Its mission is to reduce poverty and inequalities and to achieve sustainable growth in the region. It is the region’s largest financer of development projects. In 2013, it approved US $13.9 billion in loans, and US $251 million in concessional financing and grants to the most vulnerable countries in the region and provided US $1.8 billion in technical cooperation.

In 2010, the Bank launched a series of reforms to strengthen its strategic focus, development effectiveness, efficiency, and accountability for results. IDB has a strong independent evaluation function and a self-evaluation system that is promising but still being fine-tuned.

The IDB’s objectives closely align with Canada’s development priorities of stimulating sustainable economic growth, increasing food security, and securing the future of children and youth. IDB has also demonstrated commitment to the cross-cutting themes that should be integrated into all of Canada’s international development programs and policies (gender equality, environment and governance).

The Bank is one of Canada’s key partners in Latin America and the Caribbean (LAC) and is relevant to Canada’s 2012 Strategy for Engagement in the Americas. The strategy highlights three goals: increasing mutual economic opportunity; strengthening security and institutions; and fostering lasting relationships. Canada’s work with the IDB is an important avenue toward achieving these goals.

Purpose of review

This report presents the results of a review of the development effectiveness of the IDB, and an assessment of Canada’s engagement with the Bank.

To date Foreign Affairs, Trade and Development Canada (DFATD) and the Netherlands’ Ministry of Foreign Affairs have led development effectiveness reviews of seven multilateral organizations. They have applied a methodology that was developed under the guidance of the Organisation for Economic Co-operation and Development’s Development Assistance Committee (OECD-DAC) Network on Development Evaluation (EVALNET). The purpose of these reviews is to address a gap in the information available to bilateral development agencies on the development effectiveness of the multilateral organizations they support.

In accordance with the evaluation policies of Canada’s Treasury Board and DFATD, the review of the IDB’s development effectiveness has accountability and learning purposes, namely:

• to provide Canadians with an evidence-based neutral assessment of the IDB’s relevance and performance; and
• to support DFATD’s relationship with the IDB by identifying lessons for program and policy improvements.

Approach and Methodology

The review is based on a meta-synthesis of a sample of 33 IDB evaluation reports that were published between 2008 and 2013 on the website of the IDB’s Office for Evaluation and Oversight (OVE), and that met established criteria. The synthesis of evaluation findings was supplemented by a review of IDB corporate documents.

Key Findings and Conclusions

Review of development effectiveness of the IDB

The IDB has been delivering and reporting on results each year in its Development Effectiveness Overview. It is also clearly committed to improving its development effectiveness. The following conclusions nonetheless note several areas for improvement, as they emerged from the sample of evaluation reports. Where literature review and interviews have provided evidence that such improvements have in fact commenced, these have also been noted.

1. IDB programs are relevant to the needs of target groups and support the development plans and priorities of IDB borrowing member countries. The IDB developed effective relationships with governments. The Bank’s programs were seen as particularly relevant in countries where it applied a strategic approach that clearly identified its comparative advantage. In addition, its level of alignment with the borrowing country’s development priorities and its use of appropriate tools for program implementation contributed to the relevance of the Bank’s programs.

2. For many years, the IDB lacked evidence of progress toward its stated development objectives and results; however, this is beginning to change. The programs reviewed often did not provide evidence of progress towards their stated development objectives. This was due in part to a lack of information from systematic monitoring of project or program level results and to contextual challenges that hindered program implementation. However, where information was available, the Bank was comparatively more successful in countries where it was able to develop a clear strategic approach, a specific market niche, and a focus on a limited number of areas in which it had comparative advantages. As further noted below, the Bank has made significant progress since 2008 in its design of projects and country programs and in the importance it gives to evaluation and learning. The retrospective nature of this review did not fully capture that growing evidence base on results.

3. The IDB had shortcomings in project and program level planning and in monitoring and evaluation, which limited the availability of information on development effectiveness during the period covered. The evaluations reviewed consistently pointed to problems on how Bank projects and programs identified results, defined indicators, and monitored progress, which led to poor evaluability of individual projects and country programs. The mid-term evaluation of IDB-9 suggests, however, that the introduction of the Development Effectiveness Framework in 2008 appears to be promoting development effectiveness at the project level and that progress in this area is promising.^{Footnote 1}

4. The review did not find solid evidence of the extent to which the results of Bank- supported programs are sustainable. Only 36% of the evaluation reports reviewed suggested that the Bank had contributed to strengthening the enabling environment for development and 40% indicated that programs and projects were sustainable in terms of institutional capacity. Sustainability seems to be hampered by a poor enabling environment in many countries. Factors limiting sustainability include a lack of recurrent cost funding mechanisms for infrastructure and service delivery projects, a lack of country ownership and political will, and weak capacity of the Bank-supported institutions.

5. The IDB has a strong system for conducting country program evaluations, but there is limited evidence that evaluations were used to improve development effectiveness. The IDB has a strong independent evaluation function. Country program evaluations have been carried out regularly at the IDB. However, approximately two-thirds of the evaluation reports reviewed did not include a management response or other evidence that the Bank took into account the recommendations from these evaluations. The Bank has taken recent actions in this area and in 2013 introduced a platform to record, manage, and monitor OVE’s recommendations. In addition, the Bank is implementing an ambitious plan to carry out impact evaluations (more than 67 evaluations are currently underway) with the intent of using these strategically to make meaningful contributions to the development knowledge base.

6. The IDB has made a strong commitment to gender equality in recent years, but this review found that its programs and evaluations do not yet regularly integrate gender equality as a cross-cutting theme. The evaluation reports reviewed provided insufficient coverage of gender equality, and the absence of gender equality in IDB projects was noted in previous studies.^{Footnote 2} The IDB has begun to address this situation by introducing a Gender Equality Policy in 2010 and a Gender Action Plan that provide a clear focus on gender equality and women’s empowerment as integral to development. The integration of gender equality considerations in IDB projects, country strategies, and evaluations is still in early stages.

7. The Bank has an environment policy, but the sample of evaluations reviewed provided insufficient coverage of environmental sustainability to warrant presentation of results in this area. The IDB takes a two-pronged approach to implementing its commitment to environmental sustainability: a mainstreaming approach, which seeks to enhance environmental and social benefits; and a safeguards approach, which seeks to minimize and compensate for negative impacts.^{Footnote 3} Recent reports suggest that the mainstreaming of environmental sustainability is a work in progress and that it is still not effectively integrated into the Bank’s country strategies. The sample of evaluations reviewed did not report on the extent to which program and project results were environmentally sustainable.

8. The Bank has faced challenges in managing its portfolio of projects efficiently, often due to the operating context at the country level. Many of the evaluations noted delays in the start-up or during the implementation of IDB projects. Complex country processes (such as those required for Congressional ratification of loans), the varying capacities of executing agencies, transactional difficulties (including delays in procurement and contracting), and complexities in project design resulted in delays and additional costs in the project cycle.

Canada’s engagement with the IDB

The review of Canada’s engagement with the IDB during the period of 2008-2013 reflected on the engagement strategy delivered through the Canadian International Development Agency (CIDA) in its Strategy for Engagement with the IDB. The assessment concludes that:

1. The IDB remains a highly relevant partner for DFATD in view of Canada’s international development priorities and its strategy for the Americas region. The IDB’s objectives closely align with Canada’s thematic priorities for stimulating sustainable economic growth, increasing food security, and securing the future of children and youth. IDB has also demonstrated commitment to the cross-cutting themes (gender equality, environment, governance) that should be integrated into all of Canada’s international development programs and policies.

2. DFATD has made important contributions to the IDB over the past three years and is a valued partner for the Bank. DFATD has made progress towards the objectives articulated in the former CIDA strategy for engaging with IDB. The data available demonstrate that Canada has been able to facilitate internal reforms at IDB by helping to build consensus on difficult topics, such as the revised process for conducting Macroeconomic Sustainability Assessments, and by promoting certain agendas (i.e. results focus, gender equality) that have had an uptake in the IDB and the Multilateral Investment Fund (MIF). Both the IDB and MIF value Canada’s financial contributions as well as its technical cooperation and the opportunities it provides for engaging the Bank in an exchange of ideas.

3. Canada is recognized for the unique role that it plays in the Caribbean, and especially in Haiti. Canada is a lead donor in this region. The IDB and other donors respect its expertise and on-going support to meet development challenges, particularly in Haiti.  DFATD has been effective in monitoring the Bank’s commitments to Haiti, and in supporting the Bank in adapting its structure and approaches to Haiti’s more fragile context.^{Footnote 4}

4. Canada’s potential for influence at IDB is greater than its relative voting power as a member of the IDB Board of Directors.^{Footnote 5} Although it is a small shareholder and may sometimes influence only at the margins, Canada plays a critical role in working with other Borrowing and Non-Borrowing member countries and building consensus among regional and extra-regional members. As a result, stakeholders welcome/encourage Canada to have a strong voice on IDB’s critical reform agendas and to play a bridging role. One of the key reform agendas at the IDB today is private sector reform and, in this regard, Canada is well positioned to contribute.

5. CIDA’s strategy for engaging with the IDB has been developed as internal guidance. While objectives have been published on the website, they have not had high visibility with external stakeholders. Prior to amalgamation, objectives primarily reflected the perspective of the former CIDA, one of the Canadian departments that engaged with the IDB. The strategy and objectives in the past have lacked a clear performance measurement framework for reporting on results.

Recommendations for DFATD

The following recommendations are crafted with the intent of helping DFATD to strengthen its engagement with IDB.

As one of several shareholders working with IDB, DFATD is limited in the extent to which it alone can influence improvements on the development effectiveness of the organization.  Therefore, DFATD needs to continue to engage with like-minded shareholders to advocate for some of these recommendations.

1. DFATD should take the opportunity presented by its recent amalgamation to refresh its strategy for engagement with the IDB, taking into account whole of government priorities, and proposing a performance measurement framework.

2. DFATD should play an active role in influencing the Bank’s potential restructuring of its private sector financing windows and ensuring that it leads to concrete improvements in development impact. This will also provide an opportunity for the Department to communicate its vision of the private sector’s contribution to poverty reduction in the region.

3. DFATD should continue to emphasize and support the integration of gender equality and environmental sustainability in the operations of the IDB group.

4. DFATD should continue to play an important role in ensuring that the IDB engages effectively in the Caribbean, and particularly in Haiti.

5. DFATD should build on past efforts to have the IDB improve planning and monitoring of results, with clearer reporting on development effectiveness.

1.0 Background

1.1 Introduction

This report presents the results of a review of the development effectiveness of the Inter-American Development Bank (IDB). The review is based on a meta-synthesis of IDB evaluation reports published between 2008 and 2013 by the IDB’s Office for Evaluation and Oversight (OVE), and is supplemented with a review of IDB corporate documents.

In accordance with the evaluation policies of Canada’s Treasury Board and Foreign Affairs, Trade and Development Canada (DFATD), the review of the IDB’s development effectiveness has accountability and learning purposes, namely:

• to provide Canadians with an evidence-based neutral assessment of the IDB’s relevance and performance; and
• to support DFATD’s relationship with the IDB by identifying lessons for program and policy improvements.

The report is structured as follows:

• Section 1 provides an introduction to the review and a general description of the IDB as an organization;
• Section 2 presents a brief description of the approach and methodology used to carry out the review;
• Section 3 details the findings of the review in relation to six main criteria and 19 sub-criteria of effectiveness in development programming and humanitarian action;
• Section 4 assesses the effectiveness, relevance, and efficiency of Canada’s relationship with the IDB;
• Section 5 presents the conclusions of the development effectiveness review; and
• Section 6 provides recommendations to DFATD related to its relationship with the Bank. (Note: it does not make recommendations to IDB concerning its programs.)

1.2 Development Effectiveness Reviews – Purpose and Approach

To date Foreign Affairs, Trade and Development Canada and the Netherlands’ Ministry of Foreign Affairs have led development effectiveness reviews of seven multilateral organizations. The purpose of these reviews is to address a gap in the information available to bilateral development agencies on the development effectiveness of the multilateral organizations they support.

Although the Multilateral Organisation Performance Assessment Network (MOPAN) provides regular reports on the organizational effectiveness, progress towards results, and relevance of multilateral organizations, it has not yet fully addressed the need for field-tested evidence on the development effectiveness of multilateral organizations. Other approaches, such as large-scale, joint donor-funded evaluations of multilateral organizations, are time-consuming, costly, and result in significant management burdens for the organizations being evaluated.

A standard approach for assessing the development effectiveness of multilateral organisations was developed under the guidance of the Organisation for Economic Co-operation and Development’s Development Assistance Committee (OECD-DAC) Network on Development Evaluation (EVALNET). It is based on a common set of assessment criteria derived from the DAC’s evaluation criteria (Annex 1) to classify the findings of a sample of evaluation reports meeting established evaluation quality standards.

The methodology was pilot tested in 2010 as part of the Development Effectiveness Reviews of the Asian Development Bank and the World Health Organization. The overall approach and methodology were endorsed in June 2011 by the members of the DAC-EVALNET. It is intended to work in a coordinated way with initiatives such as the DAC-EVALNET/United Nations Evaluation Group Peer Reviews of United Nations evaluation functions. It also recognizes that multilateral organizations continue to make improvements and strengthen their reporting on development effectiveness.

1.3 IDB: A regional bank promoting economic growth and poverty reduction in Latin America and the Caribbean

1.3.1 Overview

Established in 1959, IDB is the oldest development bank in Latin America and the Caribbean and the region’s largest financer of development projects. It is composed of 48 member countries, 26 of which are borrowers. The Bank’s mission is to reduce poverty and inequality, and to achieve sustainable growth in the region.

Funding: In 2013, the IDB approved US$7.8 billion in sovereign guaranteed (SG) loans, US$4 billion in policy-based loans, and US$2.1 billion in non-sovereign guaranteed (NSG) loans to the private sector, including companies, financial institutions, and civil society organisations.^{Footnote 6} The IDB Fund for Special Operations (FSO) provides concessional financing to the most vulnerable countries in Latin America and the Caribbean (including Bolivia, Guyana, Haiti, Honduras, and Nicaragua), as well as grants to Haiti through the IDB Grant Facility. In 2013, FSO approvals totalled US$251 million, a 69% decrease from the previous year.

The IDB also provides non-reimbursable Technical Cooperation and at the end of 2013, this portfolio consisted of 1,530 operations with an approved amount of US$1.8 billion. In 2013, 36% of approved financing was allocated to institutional support for development, 34% to the infrastructure and environment sectors, 21% to social sector programs, and 9% to integration and trade programs. Sixty-three percent of total approved financing went to Argentina, Brazil, Mexico, Venezuela, Chile, Colombia, Peru (known as Group A and B countries).^{Footnote 7}

2013 IDB Approvals

IDB Group: The larger Inter-American Development Bank Group (IDB Group) is composed of the IDB; the Inter-American Investment Corporation (IIC), established in 1986; and the Multilateral Investment Fund (MIF), established in 1993.

The IIC promotes private sector development in Latin America and the Caribbean by providing financing to small and medium-sized enterprises in the form of equity investments, loans, and guarantees to assist companies in streamlining management processes. In 2013, the IIC approved US$415.4 million in investments and direct loans, in addition to US$197.5 million for syndicated loans and co-financing operations to leverage direct IIC funding.^{Footnote 8}

The MIF’s mission is to promote private sector development benefitting low-income households in Latin America and the Caribbean by providing technical assistance grants, equity investments, and loans to private partners (such as civil society organisations, industry associations, foundations, universities cooperatives, companies, and financial institutions). In 2013, the MIF approved 68 projects with an overall value of US$108.3 million.^{Footnote 9}

Although the IIC and MIF work in close collaboration with the IDB, both entities have independent strategic frameworks and reporting mechanisms.

Reforms and priorities: In 2010, the Board of Governors approved the Report on the Ninth General Increase in the Resources of the Inter-American Development Bank (IDB-9) and the Cancun Declaration, both of which tied IDB-9 to a series of reforms intended to strengthen the strategic focus, development effectiveness, and efficiency of the Bank to help it remain relevant and competitive in the years ahead.^{Footnote 10} IDB-9 proposed a corporate strategy based on five strategic priorities (subdivided into 13 thematic areas^{Footnote 11}). They are:

1. Social policy for equity and productivity;

2. Infrastructure for competitiveness and social welfare;

3. Institutions for growth and social welfare;

4. Competitive regional and global international integration; and

5. Protect the environment, respond to climate change, and enhance food security.

In addition, IDB-9 called for better accounting for results. Prior to 2010, the IDB did not have a comprehensive Corporate Results Framework (CRF) and monitoring at the corporate level was limited to tracking two lending targets.^{Footnote 12}

The IDB’s Corporate Results Framework is in line with corporate results frameworks used by other development banks and is composed of four tiers:

• tier I monitors regional development objectives;
• tier II monitors the Bank’s output contributions to regional goals;
• tier III monitors the Bank’s lending program priorities; and
• tier IV monitors the Bank’s operational effectiveness and efficiency.

Tiers I and II are structured around the Bank’s five strategic priorities. IDB-9 requires that data from the Corporate Results Framework be reported on annually in the Development Effectiveness Overview (DEO).

1.3.2 Reporting on Development Effectiveness

In 2010, the IDB’s Office of Strategic Planning and Development Effectiveness (SPD) presented its first DEO. According to the MOPAN 2011 assessment of the IDB, the DEO was a significant improvement over previous reporting mechanisms, which focused on the probability of achieving development objectives rather than on the results themselves.^{Footnote 13} However, the DEO does not assess the development effectiveness of the IIC or MIF as they have their own strategic frameworks and reporting mechanisms.^{Footnote 14}

In 2011, the AfDB financed 184 operations in regional member countries at a cost of UA 5.72 billion. Table 1 provides the percentage breakdown by type of financial instrument. Figure 1 provides an overview of financial allocations to regional member countries in 2011, by area of strategic focus.

While the IDB has made great strides in improving reporting on development effectiveness, the DEO still has limitations.

First, it does not address all issues and criteria of interest to bilateral donors, as outlined by the OECD-DAC Network on Development Evaluation. Information on relevance, sustainability, and crosscutting themes (especially gender equality) is notably absent. Other IDB reports provide insights into the issues/criteria lacking in the DEO and may be useful supplements. These reports include background papers prepared for the mid-term evaluation of IDB-9 and annual reports on the IDB’s contributions to sustainability.

Second, the DEO is not comprehensive as it does not report on 100% of the Bank’s operations or address all of the indicators outlined in the Corporate Results Framework. The Office of Evaluation and Oversight (OVE) has commented on the lack of a clear methodology used in the selection of Corporate Results Framework indicators and the data reported on in the DEO.^{Footnote 15} The IDB’s contribution to the 2012 Common Performance Assessment System (COMPAS) Report, a report of the members of the Multilateral Development Bank Working Group on Managing for Development Results, notes that the IDB’s corporate reporting is based on a sample of projects. In 2012, this reflected monitoring data from 65 out of 156 projects in the portfolio.^{Footnote 16}

Finally, weaknesses in the Corporate Results Framework, including a lack of baseline data, and inconsistencies regarding the types of evidence used to assess development effectiveness, may affect the quality of data presented in the DEO. In addition, in reporting on the results framework, the DEO relies on self-reported project data (project completion reports) that have not always been validated by the OVE. The 2013 DEO includes two useful additions: the reporting on lessons emerging from the impact evaluations of IDB projects (10 were completed in 2013) and a novel chapter focused specifically on the failure of approaches adopted in certain sectors (such as citizen security and urban development and housing).

The IDB has now completed its fourth cycle of development effectiveness reporting. At the time of the Mid-term Evaluation of IDB-9 Commitments (2013), the Bank had implemented many of the IDB-9 requirements related to the Development Effectiveness Framework (DEF). The DEF provides the policy framework to improve the effectiveness of the Bank’s projects and programs.^{Footnote 17} The DEF also provides a set of tools for learning and accountability for results that includes the Development Effectiveness Matrix, a Progress Monitoring Report, and a Project Completion Report. The implementation of all of the components of the DEF is crucial for addressing weaknesses in the DEO. However, implementation of this framework has been uneven.

• While its Development Effectiveness Matrix^{Footnote 18} had been successfully rolled out for Sovereign Guarantee (SG) operations^{Footnote 19}, the Development Effectiveness Matrix for Non-Sovereign Guarantee (NSG) operations^{Footnote 20} was still a work in progress.
• On the other hand, the Progress Monitoring Report for SG projects had been fully implemented and was being revised to address challenges identified in the first stage of its implementation.
• At the time of the Mid-term Evaluation of IDB-9 Commitments (2013), the Bank was about to launch its new Project Completion Report, which would enhance the Bank’s records of project accomplishments.^{Footnote 21}

1.3.3 Evaluation at the IDB

The evaluation function at the IDB was established in the late 1960s and has gone through several reforms to address issues of independence and quality, among others. In 1999, with resolution RE-238 Strengthening Oversight and Rebuilding Evaluation in the Bank, the Office of Evaluation and Oversight (OVE) was established as an independent body mandated to evaluate the Bank’s programs and systems. Since 1999, several efforts had been made to create a functioning self-evaluation system, which was eventually established in 2008 through the Development Effectiveness Framework.^{Footnote 22}

Evaluation at the Bank is a responsibility shared between Bank management, the OVE, and the Board of Executive Directors.

Management oversees the self-evaluation system and focuses on project evaluations and monitoring portfolio performance. Within management, the Office of Strategic Planning and Development Effectiveness (SPD) is responsible for ensuring that IDB’s actions are guided by the “achievement of results and an efficient management of risks”^{Footnote 23} In this capacity, it has an important role in providing guidance for the application of the DEF and strengthening the internal processes for monitoring and evaluation.

The OVE is structurally and operationally independent from management. It reports directly to the Board of Executive Directors through the Policy and Evaluation Committee (PEC). It also supports the evaluation efforts of the IIC and MIF on a contract basis. The Office assesses whether the IDB’s strategies, policies, and programs comply with the Bank's objectives and is mandated to undertake:

• oversight functions;
• country program evaluations;
• sector, thematic, and ex-post evaluations;
• policy and instrument evaluations; and
• evaluation capacity development.

OVE develops an annual work program and budget for planned evaluations in the coming year, and an indicative list of evaluations for the following year.^{Footnote 24} The OVE has established a cycle of country program evaluations (CPE) that provides full coverage of country programs. The main documents governing OVE’s role and activities are its founding document (Strengthening Oversight and Rebuilding Evaluation in the Bank – RE-238, 1999), the draft terms of reference for the Director of OVE (RE-366-11, 2011), and the protocol for the review and consideration of evaluations in the Administrative Manual (AM 140-1).

The Board of Executive Directors reviews relevant findings and recommendations arising from evaluations of the Bank’s country programs, strategies, policies, operations, and institutional processes.

Strengthening the IDB’s evaluation function has been an ongoing process. An external review was carried out by an independent panel in 2011. The aim was to strengthen the evaluation function in order to achieve greater development effectiveness. The review found that IDB had made substantial progress in evaluation since 1999, that its evaluation systems were more comprehensive, and that the function had good practice standards.^{Footnote 25} It also found, however, that the evaluation function’s effectiveness was limited by a lack of collaboration and integration among the parties involved in evaluation.^{Footnote 26}

The review noted that the self-evaluation function experienced early success with the implementation of the DEF, but that it was still facing challenges. With regard to the independent evaluation function, the review noted that OVE had produced a large number of quality reports but that it needed to improve the effectiveness, use and influence of independent evaluation, and reduce the isolation that had emerged as a result of its interpretation of independence.^{Footnote 27} Recommendations were provided to strengthen effectiveness of the evaluation system:

• Build a more collaborative learning environment that reduces tensions and fosters a learning culture.
• Increase the use of independent evaluation products and reform OVE’s culture of isolation.
• Fully implement self-evaluation and increase its use through an effective change management strategy and by providing the needed resources.
• Strengthen Board oversight of the evaluation system by empowering the Policy and Evaluation Committee to act as necessary to provide leadership during a period of change.
• Increase client country involvement in the evaluation process.
• Formulate a new evaluation policy that consolidates IDB’s approaches, addresses new realities, and reflects the findings of the review.

OVE’s work program (2013-2014) reflects the efforts it has made to respond to the findings and recommendations of the 2011 review. For example, it has made efforts to change the product mix; work with management to develop a new protocol for the handling of evaluations; developed approach papers for all evaluations (which are shared with management for comment); and developed a system for tracking evaluation recommendations.^{Footnote 28} The formulation of a new evaluation policy was one of the few recommendations that were not acted upon, and the Bank has not prepared a consolidated evaluation policy.

In sum, the IDB has a strong independent evaluation function, and a self-evaluation system that is promising, yet still being fine-tuned. As a result of the Independent Panel report, several improvements have been made in the evaluation architecture of the IDB. However, the sample for this review only includes evaluations published from 2008 to 2013. As such, the results from these efforts are not yet evident in the findings of the review.

2.0 Methodology

2.1 Overview

This section describes the methodology used in the Development Effectiveness Review of the IDB, including the universe of evaluations, sampling method, review criteria, review and analysis processes, and a discussion of other data collection methods. The section concludes with a discussion of the review’s limitations.

2.2 Evaluation Universe and Sample

Beginning with IDB evaluation reports available on the website of the Office of Evaluation and Oversight (OVE), a universe of 94 evaluation reports (the Review universe) was constructed. The process of finalizing the list involved several discussions and exchanges with the OVE and management units within the IDB.

As part of the preliminary review, the review team screened the evaluation reports by applying the criteria outlined in the DAC-EVALNET Evaluation Quality Screening Scoring Guide (further explained in Annexes 3 and 4).

The objective of the preliminary review was to determine which of the following scenarios would apply to the universe of evaluation reports and, subsequently, what steps should be taken to assess IDB development effectiveness given the available information.

• Scenario A: The multilateral organization’s own reporting on the development effectiveness criteria is rigorous and evidence-based, covering a large proportion of its investments and therefore the reporting may be relied upon by its bilateral shareholders to assess the organization’s development effectiveness.
• Scenario B: The multilateral organization’s reporting on development effectiveness is not sufficient, but the organization has an evaluation function that produces an adequate body of reliable and credible information that can be used to synthesize the organization’s performance on the development effectiveness criteria.
• Scenario C: The multilateral organization’s reporting on development effectiveness criteria is inadequate, and the body of evaluation information available from evaluations is either not adequate or is not credible or reliable to allow for a synthesis of the organization’s performance on the development effectiveness criteria.  

The preliminary review excluded 42 of the 94 reports (45%) that did not meet necessary criteria in the DAC-Evaluation Quality Screening Guide (i.e. they did not achieved a minimum score for the criteria listed in Annex 4). It identified 52 reports (i.e. 55%) that met the criteria and were therefore considered suitable for inclusion in the meta-synthesis.^{Footnote 29}

Subsequently, 19 of these 52 reports were also screened out – 12 were corporate evaluations (focused more on corporate business processes than results, relevance or sustainability of programs on the ground) that did not report on any of the six common development effectiveness assessment criteria,^{Footnote 30} and five were project and impact evaluations that were narrow in scope and did not always specify the IDB’s role.

Based on the preliminary review, Scenario B was applied to the IDB. The final sample consisted of 33 evaluation reports (see Annex 2).The key characteristics of the sample are described below:

• Geographic Coverage: The evaluation sample covers the majority of IDB’s borrowing member countries, including the five borrowing member countries receiving the highest levels of loan and guarantee approvals from the IDB during the period 2008-2013 (i.e. Argentina, Brazil, Mexico, Colombia, and Peru). The sample also includes countries from each of the Bank’s four country groupings, based on their relative level of development.^{Footnote 31}
• Coverage of the Bank’s concessional financing mechanisms: The sample includes evaluation reports from countries in the region receiving concessional financing from the IDB’s Fund for Special Operations (including Bolivia, Guatemala, Guyana, Haiti, Honduras, Nicaragua, and Paraguay) as well as grants through the IDB Grant Facility (in the case of Haiti).
• Broad Representation of IDB Operations (volume and type): The evaluation sample represents an important proportion (94%) of the IDB’s loan and guarantee approvals in the period 2007-2010.^{Footnote 32} In addition, the sample includes the most recent evaluation of the Multilateral Investment Fund, which approved 76 projects with an overall value of US $93.4 million in 2012.^{Footnote 33} Given the emphasis on Country Program Evaluations, the sample included performance information on different types of Bank products, including policy-based loans, lines of credit, investment loans, and technical cooperation, among others.
• Coverage of Development Effectiveness Criteria: The sampled reports all addressed development effectiveness criteria, although the extent to which individual criteria and sub-criteria were covered varied from one evaluation report to another.
• Timeframe: All reports in the evaluation sample were published between 2008 and 2013 and most focused on programming carried out between 2000 and 2010. Consequently, neither the universe of 94 evaluation reports nor the sample of 33 reports provide many insights into the results of IDB operations following its commitments outlined in IDB-9.

2.3 Criteria

The methodology used in this review does not rely on a particular definition of development or humanitarian effectiveness. Instead, it focuses on the following essential characteristics of effective multilateral organization programming derived from DAC evaluation criteria:

• Programming activities and outputs would be relevant to the needs of the target group and its members;
• Programming would contribute to the achievement of development objectives and expected development results at the national and local level;
• The benefits experienced by target group members and the development (and humanitarian) results achieved would be sustainable in the future;
• Programming would be delivered in a cost efficient manner;
• Programming would be inclusive, i.e. it would support gender equality and environmental sustainability, thereby not compromising future development prospects;
• Programming would enable effective development by allowing participating and supporting organizations to learn from experience and use performance management and accountability tools, such as evaluation and monitoring, to improve effectiveness over time.

The methodology, therefore, involves a systematic and structured review of the findings of IDB evaluations as they relate to the six main criteria and 19 sub-criteria considered essential elements of effective development and humanitarian programming (Annex 1).

2.4 Review Process and Data Analysis

The review team consisted of two members. Each reviewed a set of evaluations and, based on information in the evaluations and the standard review grid (Annex 5), provided a rating on each sub-criterion.^{Footnote 34} Team members substantiated their ratings by providing evidence from the evaluations.

Efforts were made to ensure consistency in ratings (inter-rater reliability): an initial session was held during which team members reviewed and compared their ratings for the same evaluation report. A second session, held at the mid-point of the review process, addressed any issues facing reviewers. Upon completion of the reviews and documentation of qualitative evidence to support reviewers’ ratings, the team leader examined all ratings to ensure that sufficient evidence was provided, that the available evidence was consistent with the ratings, and to identify the factors contributing to or detracting from the achievement of the criteria outlined by the DAC.

2.5 Other Data Collection

The review of IDB evaluation reports to inform the meta-synthesis was supplemented by a review of IDB corporate documents and evaluations. In particular, the 2013 background papers prepared by the OVE as part of the mid-term review of IDB-9 were important sources of information (See Annex 6).

2.6 Limitations

This review took place during a period of change at the IDB; the Bank is implementing a series of reforms aimed at increasing its effectiveness. In this context, there were several limitations to the Development Effectiveness Review.

• Retrospective Nature: The sample of evaluations used in this review covered IDB operational work dating from the late 1990s to 2012 (approximately) and are retrospective. The meta-synthesis may, therefore, identify challenges to development effectiveness that have already been partially or fully addressed by the IDB, particularly as a result of implementing its commitments under IDB-9. To the extent possible, the evaluation team integrated information from relevant IDB-9 mid-term review studies (published in 2013) to provide more time-sensitive findings and conclusions.
• Sample Composition: As most of the evaluations included in the sample were Country Program Evaluations (CPE), the analysis could draw on country-level evidence (which is key when considering the development effectiveness of multilateral organizations). However, the sample did not represent a broad set of sector/thematic and corporate studies.^{Footnote 35} While CPEs cover a diversity of programming in-country, they do not always make an overall judgment or statement about the achievement of objectives at the country level. Where the CPEs did not provide this overall judgment, the reviewers compiled evidence from each of the objective areas to make their own assessments.
• Data Availability: As noted above, there were relatively few sector or thematic evaluations in the universe of available documents and, therefore, in the sample of evaluations reviewed. Sector/thematic studies provide information about the Bank’s work in multiple countries and are therefore useful in strengthening the understanding of the Bank’s work in a particular subject area (for example, water and sanitation, energy, education, etc.)
• Inability to Provide an Analysis in Relation to the IDB’s Strategic Priorities: As noted above, the IDB-9 articulated five strategic priorities for the IDB. Although country programs have pillars that may mirror these priorities, the programs were designed prior to IDB-9 and were tailored to the demands of each client country. Therefore, it was not possible to aggregate and analyze data in a way that could provide an assessment of how the IDB is performing in each of these strategic priority areas.

3.0 Findings on the Development Effectiveness of IDB

3.1 Overview

This section presents the results of the review as they relate to the six major criteria and associated sub-criteria (Annex 1). Each section includes:

• Coverage: An analysis of coverage based on the number of evaluations in the sample of 33 that addressed the six major criteria and their associated sub-criteria.
• Key findings of the review, including quantitative data on the distribution of ratings (ranging from highly satisfactory to highly unsatisfactory) on each sub-criterion as reported in the evaluations reviewed.^{Footnote 36}
• Contributing factors: A qualitative analysis of factors that affected IDB’s results achievement on the different criteria. The figures presented refer to the frequency with which analysts in the review team found that these factors were reported across all reviewed evaluations. While these present an overall picture of the relative importance of these factors, they should not be considered an exhaustive census of the contributing/detracting factors.

3.2 Objectives achievement

3.2.1 Coverage

In the evaluation reports reviewed, the coverage of the sub-criteria related to objectives achievement was mixed. While the coverage was high for sub-criterion 1.1 on objectives achievement and for sub-criterion 1.4 on positive changes in national policies and programs, it was moderate for sub-criterion 1.2 on the benefits for target group members, and weak for sub-criterion 1.3 on program results reaching substantial numbers of beneficiaries.

3.1 Objectives achievement - number of evaluations that addressed sub-criteria (n=33)^{Footnote 37}

The lower levels of coverage for sub-criteria 1.2 and 1.3 can be largely explained by the limited availability of data on results achieved by programs supported by the Bank (most reports indicated lack of monitoring and evaluation (M&E) data to assess program effectiveness). If the review team did not find sufficient evidence of benefits or contributions to long-term impacts, it was noted as “not reported” in the review grid.

3.2.2 Findings

Of evaluations that addressed objectives achievement (sub-criterion 1.1), the majority (63%) indicated that IDB programs and projects were unable to fully achieve their stated development objectives, primarily due to contextual issues (political and economic environment) and/or insufficient data to demonstrate that objectives and results were achieved.

Results were more positive for the extent to which programs provided positive benefits for their target groups (sub-criterion 1.2). Of the evaluations that included data in this area, almost two-thirds (63%) reported that programs contributed to benefits for the target population.

There was insufficient coverage of contributions to making substantial differences for beneficiaries and contributing to national development goals to warrant presentation of results on sub-criterion 1.3.

Results were less positive for changes in national policies and programs (sub-criterion 1.4); in 64% of the evaluation reports reviewed there was limited evidence that the Bank’s programs had contributed to substantial policy change.

Figure 3.2 Objectives achievement - distribution of ratings (%) by sub-criteria^{Footnote 38}

3.2.3 Contributing factors

Evaluations pointed to the following factors influencing the success of country programs in meeting their stated objectives:

• The Bank was comparatively more successful in countries where it had been able to develop a clear strategic approach, a specific market niche, and a focus on a limited number of areas in which it was seen to have comparative advantages (10 of the 33 reports reviewed). For example, in Nicaragua the Bank was seen to have comparative advantages in transportation, energy, agriculture, health, and water.^{Footnote 39}
• The Bank’s partnership behaviour played an important role in the achievement of stated objectives (11 evaluations). For example, its level of contribution to policy dialogue, its communication practices, and its level of flexibility in terms of realigning program goals and resources when needed were noted in particular.
• Bank loans were more successful in countries where they were implemented by executing agencies that were more professional and more specialized than average (5 evaluations).

Evaluations also highlighted the following factors that limited the achievement of objectives:

• Projects and programs supported by the Bank had very weak monitoring and evaluation capacity and the overall level of project and program evaluability was low (21 evaluations). Consequently, most evaluation reports concluded that it was difficult, if not impossible, to measure the effectiveness of the Bank’s country level work due to factors such as lack of baseline data, lack of ex-post evaluations, inappropriate or weak project outcome indicators and/or targets, as well as imprecise and weakly defined development objectives. This issue was particularly pronounced in relation to the Bank’s Technical Cooperation work for which none of the evaluation reports were able to identify any meaningful outcomes or impact. The weakness in monitoring and evaluation systems made it very difficult for the Bank to demonstrate results.

• The Bank’s lack of clear strategic approach, combined with weaknesses in program and project design, negatively influenced the achievement of objectives (16 evaluations). In particular, evaluations noted limitations in the Bank’s capacity to develop a strategic approach in country strategies and a clear market niche. In some evaluations, the Bank was perceived as being more reactive than proactive. Other evaluations noted that the Bank developed a wide range of generic proposals for intervention in multiple sectors, without considering its comparative advantages. The importance of developing a clear strategic approach and building on comparative advantages was confirmed in a Background Paper on Country Programming produced by the OVE in 2013. The report concluded that the Bank’s Country Strategies rarely articulate a strategic approach for the Bank in key sectors or fully explore the Bank’s comparative advantages.^{Footnote 40}
• Challenging political contexts and a lack of political interest and country ownership of Bank-supported development projects and programs were limiting factors (13 evaluations). In many cases, these problems were further exacerbated by significant delays in project approval and a lack of interest on the part of subsequent borrowing member administrations. In particular, 8 evaluations noted that complex and ineffective national political processes prevented congressional approval of proposed projects, thus limiting the number of loans executed. While these external factors are difficult for the Bank to address, they seem to constitute an important limitation tor the effectiveness of Bank operations in certain countries.
• Institutional weaknesses (or capacity issues) and lack of experience on the part of executing units limited the achievement of objectives (6 evaluations).

3.3 Cross-Cutting Themes

3.3.1 Coverage

None of the evaluations reviewed analyzed the extent to which IDB-supported activities address gender equality and only five evaluations assessed the extent to which changes caused by IDB-supported projects are environmentally sustainable.

Figure 3.3 Cross-cutting themes - number of evaluations that addressed sub-criteria (n=33)

3.3.2 Findings

The evaluations reviewed did not provide sufficient information on either gender equality or environmental sustainability for the review team to make assessments.

It is important to note that the inclusion of evaluation criteria related to gender equality and environmental sustainability are not mandatory in the IDB’s Protocol for Country Program Evaluation. Rather, both gender equality and environmental sustainability are included as possible sub-criteria under “impacts” from which evaluators may draw.

Progress and challenges in gender equality and mainstreaming

As part of its IDB-9 mandate, the Bank adopted a Gender Equality Policy in 2010 and a Gender Action Plan to address concerns raised by the Independent Advisory Group (IAG) established in 2009.^{Footnote 41} According to the Office of Evaluation and Oversight (OVE)’s Mid-term Evaluation of IDB-9 Commitments,^{Footnote 42} the implementation of the Gender Action Plan is progressing well in most areas, although not all IAG recommendations have been fully implemented.

The Bank is working on a number of initiatives in this area, including the preparation of country gender policy and sector notes to supplement Country Strategies. The focus of these notes has varied from multi-sector or cross-cutting overviews to more focused analyses of gender issues in targeted areas comprising the core of IDB discussions within borrowing member countries.

However, according to the review, there are variations in the extent to which these notes have influenced the content of Country Strategies.

Other initiatives have included learning events for IDB staff and a pilot implementation of the gender safeguards launched in 2011-2012. At the time of the OVE mid-term evaluation (March 2013), about 140 projects had been screened for potential gender risks, and mitigation plans for seven projects had been completed and integrated into Environmental and Social Management Plans (ESMPs). The Bank has also conducted a number of social assessments that include a gender focus.

In spite of progress in the area of gender equality and mainstreaming, the OVE Mid-term Evaluation concluded that the mainstreaming of gender considerations remains a work in progress and that the Bank still lacks of a clear longer-term agenda for achieving this objective. The OVE mid-term evaluation did suggest that mainstreaming of gender in projects was increasing, with 19% of lending operations approved in the first three quarters of 2012 including a gender mainstreaming indicator in their results matrices. This was an increase from 9% in 2011. The OVE found, however, that the relevance and quality of gender indicators was uneven across results frameworks.^{Footnote 43}

Contributions to environmental sustainability

In 1979, the IDB became the first multilateral development bank to adopt an environment policy, which mandated that the organization support environmental projects and ensure the environmental quality of its operations. Since then, the policy has been updated several times, and in 2006 the Bank adopted its Environmental and Safeguards Compliance Policy, which takes a two-pronged mainstreaming and safeguards approachto implementing its commitment to environmental sustainability. The approach seeks to enhance environmental and social benefits and avoid, minimize, and compensate for negative impacts.^{Footnote 44}

In 2009, the Bank established an Independent Advisory Group (IAG) to review progress on implementing the policy and to provide advice on potential improvements. The IAG’s final report, presented to management in February 2011, suggested that the mainstreaming of environmental sustainability had improved, but that there were still significant challenges ahead. The mainstreaming policy had not received adequate priority and leadership and the “the Bank lacked a framework to ensure effective mainstreaming of sustainability concerns into its Country Strategies and operations.”^{Footnote 45} The Bank had however made some progress in implementing the commitments outlined in the 2006 policy. As a response to the Group’s recommendations, management created an internal Sustainability Working Group (SWG) with the mandate to develop an action plan structured around mainstreaming and safeguards.

According to the Mid-term Evaluation of IDB-9 Commitments, the Bank has prepared sector notes as inputs to several Country Strategies, but has not yet found an effective way to integrate environmental sustainability into its country programs and still lacks an overarching framework to guide mainstreaming.

Since 2005, the IDB has produced the annual Sustainability Report, a self-assessment that covers its annual progress and performance in relation to social and environmental sustainability. Although the report does not provide an independent review of the Bank’s performance in these areas, it gives a general overview of the Bank’s work in relation to environmental and social sustainability and highlights some of the most important environmental issues in the region.

3.4 Sustainability

3.4.1 Coverage

The coverage of sub-criteria related to sustainability varied in the sample of evaluation reports. Sub-criterion 3.1, “benefits continuing after program completion,” was covered by only 12 evaluation reports, while sub-criterion 3.2, “extent multilateral organization supported projects and programs are reported as sustainable,” was addressed by 20 evaluation reports. Sub-criterion 3.3, “Strengthening the enabling environment,” was addressed by 25 evaluation reports.

Program sustainability is one of the mandatory assessment criteria according to the IDB’s Protocol for Country Program Evaluation (CPE) 2008.^{Footnote 46} However, most evaluation reports commented on sustainability issues for only one or two projects, which made it difficult for the review team to make overall judgements about the sustainability of IDB programming. In these cases, the review team categorized the sub-criteria as “not addressed” in its review grid.

Figure 3.4 Sustainability - number of evaluations that addressed sub-criteria (n=33)

3.4.2 Findings

Overall, the results with regard to sustainability were mixed. In the sample of evaluation reports reviewed that addressed sub-criterion 3.2, 40% indicated that projects and programs are sustainable in terms of institutional and/or community capacity. Among the reports that addressed sub-criterion 3.3, 36% indicated development programming contributed to strengthening the enabling environment. The reports sampled provided little or no information on sub-criterion 3.1 (the likelihood of benefits continuing in the post-program period); therefore review team did not have enough information to make an assessment.

Figure 3.5 Sustainability - distribution of ratings (%) by sub-criteria^{Footnote 47}

3.4.3 Contributing factors

Reviewers found relatively few explicit references to factors that had positively contributed to the sustainability of IDB-supported projects and programs. Institutional capacity and political ownership were sometimes cited. The Country Program Evaluation (CPE) for Ecuador, for example, noted that: “The Bank’s successful experience with Ecuador’s Internal Revenue Service suggests that when there is political will within an institution - as appears to be the case in SENPLADES^{Footnote 48} - very successful and sustainable institution-strengthening outcomes can be delivered.”^{Footnote 49}

The reports reviewed noted the following factors limiting sustainability:

• The lack of recurrent funding mechanisms for infrastructure and service delivery projects was a limiting factor (6 evaluations). Several CPEs noted that road projects, in particular, suffered from insufficient funding and institutional capacity to ensure that both periodic and routine maintenance was carried out.
• The lack of country ownership and political will was noted as a limiting factor (5 evaluations). Examples include lack of government approval for road maintenance funds and changing government priorities due to long delays in authorization and execution of projects.
• The Bank did not sufficiently focus on institutional strengthening (5 evaluations) and in some cases, its programs had been too scattered; activities spread over a wide range of host institutions further limiting the potential for strengthening critically important institutions.^{Footnote 50}
• Weak capacity of local and state institutions (4 evaluations, most notably the CPE for Haiti 2007-2011).

3.5 Relevance

3.5.1 Coverage

There was strong coverage for all sub-criteria on relevance. Almost all evaluations addressed: program suitability to the target group (sub-criterion 4.1), alignment of programs to national development goals (sub-criterion 4.2), and effective partnerships with governments and other agencies (sub-criterion 4.3).

Figure 3.6 Relevance - number of evaluations that addressed sub-criteria (n=33)

3.5.2 Findings

Review findings suggest that IDB programs are generally relevant to their target groups and to the development priorities of borrowing member countries. Of the reports that addressed sub-criterion 4.1, 69% reported program suitability to target groups; and of those that addressed sub-criterion 4.2, 82% reported that the program aligned with country development goals. These findings reflect the fact that IDB country strategies are usually prepared in consultation with client governments.

The needs of two target groups are addressed in country programs: the government and other executing agencies involved in the Bank’s operations; and the beneficiaries of specific operations included in country portfolios. The Country Program Evaluations (CPE) reviewed tend to focus more on the former.^{Footnote 51}

Of the reports that addressed sub-criterion 4.3, 70% noted the presence of effective partnerships between the IDB and country governments, bilateral and multilateral development agencies, NGOs, and other partners.

Figure 3.7 Relevance - number of evaluations that addressed sub-criteria (n=33)^{Footnote 52}

3.5.3 Contributing factors

Several factors contributed to positive findings for IDB program relevance, including:

• A focused approach and identification of niche areas and sectors in which Bank programs have the potential to add value were noted as positive factors (9 evaluations). (See Highlight Box 6).
• Intentional program alignment with the borrowing country’s development priorities and the execution of projects clearly derived from government initiatives (8 evaluations).
• The use of appropriate tools for program implementation, including flexible or innovative lending instruments (i.e. policy based loans and conditional lines of credit) and use of technical cooperation to support investment loans or other lending (7 evaluations).
• The relative importance of the IDB as a source of financing compared to other (external or internal) sources (4 evaluations).^{Footnote 53}

However several factors can negatively affect the relevance of Bank programs:

• Country program strategies that lack focus, do not take a program approach, and do not clearly identify niche areas for the Bank’s work (11 evaluations). Examples include program strategies for Mexico (2001-2006) and Paraguay (2003-2008), which lacked a clear definition of the Bank’s comparative advantage.
• Poor selection of lending instruments, or instruments that were not sufficiently adapted to the context (6 evaluations).
• The inadequate quality of diagnostic or analytical work and poor integration into the country program’s objectives and lending program (5 evaluations). The limited availability or use of diagnostic studies was shown to limit relevance in the evaluations of Bank programs in Haiti (2007-2011), El Salvador (2004-2008), Paraguay (2003-2008), and Trinidad and Tobago (2000-2008), as well as in the corporate thematic evaluation of the Bank’s programming to address the problems of crime and violence in 11 countries in the Latin America and Caribbean region.^{Footnote 54}
• Political contexts in borrowing member countries that limit the ability of the Bank and country governments to reach agreements on strategic issues or on the details of project design and implementation (4 evaluations). The opinions of the Bank and country governments may diverge on specific projects.^{Footnote 55} Unstable political and/or institutional environments may also affect the ability of the Bank and country governments to form and maintain effective relationships (e.g. Ecuador 2000-2006 and Bolivia 2004-2007).

3.6 Efficiency

3.6.1 Coverage

While there was strong coverage for two of the sub-criteria on efficiency, the coverage for sub-criterion 5.1, which addresses the cost-efficiency of program activities, was weak in the sample of evaluation reports. This is not surprising given the challenges of assessing the efficiency of program activities.

Figure 3.8 Efficiency - number of evaluations that addressed sub-criteria (n = 33)

As noted in the IDB’s protocol for conducting Country Program Evaluations (CPE), “measuring efficiency is difficult at overall country program level because of the difficulty in estimating the combined benefit flows of various categories of Bank assistance (i.e., policy support, capacity building, aid coordination).”^{Footnote 56} Instead, CPEs typically draw on proxy indicators of the efficiency of the Bank’s support in comparison to cost. For the most part, the CPEs in the sample analyzed metrics on the IDB’s management of its portfolio. These metrics are developed for different stages of the project cycle and include:

• time from profile to approval (average project design time);
• time from approval to eligibility for first disbursements (indicative of the time required to actually “start-up” a project);
• disbursement performance (indicates the pace of implementation); and
• implementation and execution performance (including needs for extensions).

These metrics, which are regularly presented in CPEs, helped inform the assessment of sub-criteria 5.2 and 5.3. While the corporate and thematic evaluations reviewed also commented on efficiency issues related to sub-criteria 5.2 and 5.3, they did not address the cost-efficiency of the program activities reviewed. There was usually no direct data on individual projects or the extent to which project inputs and activities were efficiently producing outputs (schools built, kilometers of roads built, etc.), which may explain why there was weak coverage on sub-criterion 5.1.

3.6.2 Key findings

In the reports reviewed that addressed sub-criterion 5.2, 41% offered a positive assessment of the timeliness of program implementation. Of those reports that addressed sub-criterion 5.2, only 32% provided a positive assessment on efficiency of systems and procedures for implementation. There was insufficient coverage of sub-criterion 5.1 on whether program activities are cost/resource efficient to warrant presentation of results on this sub-criterion.

It is important to note that the IDB controls some stages of the project cycle more than others. IDB has more control over project design time (from project profile to approval) and less control over start-up (the time between approval and eligibility for first disbursements) as this requires ratification by Congress in most countries and the borrower must meet certain conditions. Project implementation can also be affected by borrower capacities and the capacities of executing agencies.

Figure 3.9 Efficiency - distribution of ratings (%) by sub-criteria^{Footnote 57}

3.6.3 Contributing factors

Evaluation reports rarely commented on factors that contributed positively to efficiency measures but when they did, two factors stood out:

• The right choice of lending instruments was cited as a factor contributing to stronger performance on key portfolio indicators (5 evaluations). For example, conditional credit lines for investment projects can be rapidly executed, disbursed more quickly, and contribute to favourable indicators. The incorporation of flexible lending products (that involve fewer transactions) contributed to the Bank’s efficiency indicators in Brazil, Colombia, Peru, and Mexico.
• Operational and administrative changes made by the Bank to strengthen its role during implementation of the country programs (5 evaluations). (See Highlight Box 7). In 2006, for example, the Bank introduced an organizational realignment in order to strengthen the Bank’s country focus, sector expertise, and achievement of results.^{Footnote 58} The positive effects of realignment of efforts on efficiency are noted in several reports.

In almost all of the evaluation reports reviewed there were references to the challenges that IDB has faced in efficiently executing its programs (28 evaluations). Key political and institutional factors that are part of the context in which the Bank works were frequently cited as affecting its efficiency, including:

• Issues related to the capacity of executing agencies in areas such as human resources (i.e. staff turnover), project management, and coordination affect project implementation (12 evaluations).
• Political dynamics within borrowing member countries (11 evaluations). In many countries, the executive and legislative branches of government do not coordinate their work. This causes delays in the early stages of the project cycle because IDB loans must be approved by the legislature. In addition, the efficiency of country project implementation is usually affected in times of political crisis (see Highlight Box 8) or following a change in government resulting in a significant policy shift (such as in Bolivia in 2005).

Other limiting factors affecting the timeliness and cost of implementing projects were noted with less frequency in the evaluation reports and included:

• Transactional difficulties, including delays in procurement and contracting (9 evaluations). These were referenced in 8 CPEs and in the thematic evaluation of IDB Action in Highway Development, 2010.^{Footnote 59}
• Project design flaws or complexity in project implementation (in terms of number of entities involved and breadth of desired objectives) were considered problematic. Overly complex projects had to be re-designed, thus causing delays and incurring additional administrative costs (6 evaluations).
• Capacity constraints of the IDB country office, including insufficient staff for project supervision. This was especially the case in contexts characterized by a proliferation of small operations in multiple sectors (6 evaluations). The reports also noted inconsistencies between portfolios and the complement of resident sector specialists in the Bank’s country offices.
• Shortfalls in the ability of borrowing member countries to provide the agreed counterpart funds (4 evaluations). This was often due to fiscal constraints and caused delays in project implementation.

3.7 Using evaluation and monitoring to improve development effectiveness

3.7.1 Coverage

Overall, coverage of the four sub-criteria related to evaluation and monitoring to improve development effectiveness was high. However, sub-criterion 6.3 (results-based management) had slightly lower coverage levels. The relatively lower coverage of this sub-criterion reflects the fact that not all evaluation reports explicitly commented on the effectiveness of the Bank’s results-based management systems and practices.

Figure 3.10 Evaluation and monitoring - number of evaluations that addressed sub-criteria (n=33)

3.7.2 Findings

Although the Bank’s systems and processes for evaluation were rated positively (sub-criterion 6.1), ratings for the other sub- criteria were mixed but generally unsatisfactory.

Figure 3.11 Evaluation and monitoring - distribution of ratings (%) by sub-criteria^{Footnote 60}

The positive rating for the effectiveness of systems and processes for evaluation (sub-criterion 6.1) mainly reflects the Bank’s practice of conducting regular Country Program Evaluations (CPE). In the reports reviewed that addressed the effectiveness of systems and processes for monitoring and reporting results, findings were mainly unsatisfactory (sub-criterion 6.2).  This was also the case for the effectiveness of RBM systems (sub-criterion 6.3). Results were mixed for using evaluation results to improve development effectiveness (sub-criterion 6.4), although most reports pointed to unsatisfactory ratings.

Almost all evaluation reports reviewed pointed to problems with the evaluability of country programs when using both top down (i.e. program level) and bottom up (i.e. project level) approaches.^{Footnote 61}  In addition, the CPEs reviewed often did not have many Project Completion Reports to draw on.

The Office of Evaluation and Oversight (OVE) observed that the evaluability of the Bank’s portfolio is critically low, thereby making it difficult to provide conclusive evidence about the Bank’s development effectiveness; this was a common theme in the CPEs reviewed.

The Independent Review Panel on Evaluation at the Inter-American Development Bank (2011) confirmed the difficulties that OVE had faced to pinpoint the “results of IDB-financed activities.”^{Footnote 62}

As noted earlier, this review of development effectiveness does not reflect recent improvements in the Bank’s monitoring and evaluation practices or the extent to which program and project evaluability may have increased.

According to the 2013 mid-term evaluation of the IDB-9 Commitments, the Bank has significantly improved its capacity to monitor and report on results. Among other improvements, the Bank has enhanced its Development Effectiveness Framework (DEF), strengthened its procedures and processes for improving project evaluability, and strengthened tools for project monitoring and results reporting. It also notes that not all of the IDB-9 commitments have yet been implemented. Furthermore, the IDB is implementing an ambitious program of impact evaluations (67 are ongoing) with the intent not only of measuring whether resources are used effectively and efficiently in development, but also learning about why programs work and how to make them better.^{Footnote 63} The expectation is that these evaluations will contribute to the knowledge base on development in the region and be used to improve development effectiveness.

3.7.3 Contributing factors

The reviewers found few examples of factors positively contributing to the effectiveness of the Bank’s monitoring and evaluation practices (only three evaluation reports provided examples of contributing factors).

Two evaluation reports mentioned country capacity as a factor clearly influencing the extent to which project results can be tracked. This was the case in Brazil which has strong institutional capacity in this area and in Mexico, where government programs supported by the IDB are subject to rigorous impact evaluation. One corporate evaluation also mentioned the strength of the data collection system of the Multilateral Investment Fund’s youth portfolio and its use of rigorous impact evaluations.^{Footnote 64}

The following factors limiting the Bank’s ability to demonstrate its contributions to development outcomes and impact were noted:

• Low evaluability of Bank-supported programs and projects (32 evaluations). The CPE for Brazil described this as a systemic problem for the Bank: “The low number of projects that collect information on progress indicates a serious problem with the Bank’s monitoring system. In general, independently of the borrower and sector, the older projects lack a clear differentiation between outcomes and outputs; few indicators are SMART^{Footnote 65}; and most projects lack baseline information and data collection mechanisms. Many older projects were retrofitted, as the Bank’s systems were evolving, especially Project Completion Reports (PCRs). There is now a general recognition that PCRs provide very limited information.”^{Footnote 66}
• Lack of evidence regarding the Bank’s use of evaluation recommendations (26 evaluations). Most evaluations reviewed did not include a management response and did not mention the extent to which the recommendations of previous evaluations had been used to improve the effectiveness of country programs. Some evaluations that did include a management response concluded that most of the recommendations outlined in previous evaluation reports remained valid and that little seemed to have been done in terms of their implementation.
• Generally weak and unreliable monitoring and evaluation systems and practices used in the assessment of Bank supported projects and programs (16 evaluations).

• Poor program and project results frameworks with weak and incomplete indicators and a lack of targets, baselines, and milestones were noted as a limiting factor (16 evaluations). Many of the evaluation reports reviewed indicated that poor program and project results frameworks constitute a major factor limiting the Bank’s capacity to systematically demonstrate results.
• Imprecisely defined development objectives impeded the Bank’s measurement of results and implementation of Results Based Management practices (9 evaluations).
• Weak data collection capacity in member countries was a major obstacle to developing sound M&E systems and impeded the Bank’s ability to collect quantitative results data (4 evaluations). In particular, weak data collection capacity was noted in Haiti, Guyana, and Trinidad and Tobago.

4.0 IDB and Canada

4.1 Introduction

The purpose of this chapter is to assess the engagement between DFATD and the IDB between 2010 and 2013 according to the criteria of relevance, effectiveness and efficiency. It fulfills DFATD and Government of Canada evaluation requirements, and provides DFATD with evidence-based guidance for its future engagement with the IDB.

The emphasis of the review is on Canada’s engagement with the IDB through its role as a member of the Executive board of IDB, and as a contributor to Multilateral Investment Fund (MIF) and multi-donor trust funds managed by the Global Issues and Development Branch of DFATD. The review does not directly encompass the investments made through bilateral channels, such as energy projects in Peru, the Complete Caribbean Program (which focuses on private sector growth in Caribbean countries), or infrastructure and education projects in Haiti.

The IDB Development Effectiveness Review was launched prior to the creation of DFATD in 2013 and therefore this chapter reflects on the strategy for engagement with the IDBdeveloped by the former Canadian International Development Agency (CIDA). The former CIDA is now known as DFATD.^{Footnote 67} In addition, this chapter was prepared while the amalgamation of the two previous entities was still ongoing, which may have future implications for how the relationship with the Bank is managed.

Following this introduction, section 4.2 provides a description of Canada’s financial support for and relationship with the IDB. Section 4.3 considers the relevance of the IDB’s work to Canada’s international development priorities and strategy for the Americas, and the extent to which the former CIDA strategy has been relevant to Canada’s priorities and the IDB context. Section 4.4 considers the extent to which DFATD effectively implemented its strategy for engagement with the IDB, and section 4.5 discusses the efficiency of that engagement.

Methodology

The assessment relies on document review and a limited number of interviews with DFATD and IDB staff, and other donors. Among the documents reviewed are descriptions of Canada’s overall foreign and development objectives and IDB corporate documents, including evaluation reports. A list of the documents reviewed and interviewees who contributed to this assessment is included in Annex 7.

The criterion of efficiency is challenging as there is limited data available to assess the ratio between the inputs (staff time and resources) and the outputs of the engagement between Canada and the IDB, which may often be intangible.  The emphasis is thus on the practices used in managing the relationship, including planning, monitoring, and communications. 

4.2 Canada's support for and relationship with IDB

Canada's support to IDB

Canada has demonstrated its commitment to Latin America and the Caribbean and to the IDB through its significant financial support to Bank operations since 1972.

Between 2008/2009 and 2012/2013, former CIDA disbursements to the IDB totalled $428.97 million. Of this amount, 74 % was managed through the Multilateral and Global Programs Branch (now called the Global Issues and Development Branch).

Figure 4.1 CIDA disbursements to IDB 2008-2013

Of the Bank’s 22 non-borrowing member countries, Canada is the third largest shareholder in the Bank, contributing just over 4% of IDB’s ordinary capital.^{Footnote 68} Canada is also a significant contributor to the Multilateral Investment Fund (MIF), which promotes inclusive growth in Latin America and the Caribbean through private sector development.

Canada’s relationship with the IDB

Canada’s engagement with the IDB takes place through various channels. DFATD, in close cooperation with Finance Canada, is the lead Canadian department overseeing the Bank’s activities. Within DFATD, the Global Issues and Development Branch takes the lead in coordinating the institutional strategy with Finance Canada.

Canada also plays a role in the governance structure of the IDB. The Board of Governors, comprising representatives from 48 member countries, is the Bank’s highest governing body. Canada’s representative on the Board of Governors is the Minister of Foreign Affairs, accountable for the International Development Assistance Act, and the alternate representative is the Assistant Deputy Minister of the International Trade and Finance Branch at Finance Canada.

The Bank’s second level of governance structure is its 14-member Board of Directors, which meets weekly and oversees the daily management of the Bank’s activities. Canada holds a permanent seat on the Board of Directors and its representative is the Executive Director for Canada; it does not represent other countries as part of its constituency. Canada is represented on all committees of the Board. Between 2010 and 2013, Canada’s representative chaired Board committees, working groups and task forces that were instrumental for the Ninth General Capital Increase and the Bank’s reform agenda.

The office of the Executive Director for Canada communicates with relevant Government of Canada agencies and departments and works with commercial interests, academics, and non-governmental organizations on issues related to the IDB and Canada’s interests in Latin America and the Caribbean. The responsibility for managing DFATD support for the IDB varies according to the type of funding provided (i.e. multilateral or bilateral channels of support). The Global Issues and Development Branch at DFATD manages Canada’s core contributions to the Bank and is responsible for the overall institutional relationship. This work also includes managing contributions to thematic trust funds. Canada also provides direct support to IDB for projects implemented in its Borrowing Member Countries. DFATD geographic branches manage the bilateral funding and relationships with IDB in these countries.

4.3 Relevance

4.3.1 Introduction

This section provides an assessment of three dimensions of relevance with regard to DFATD’s engagement with the IDB. It reviews the relevance of the IDB to Canada’s Strategy for Engagement with the Americas and to Canada’s international development priorities. It also considers the relevance of the former CIDA’s strategy for engagement with the IDB (i.e. whether the strategy addresses the right issues given Canada’s priorities).

4.3.2 Relevance of the IDB to Canada’s Strategy for Engagement in the Americas

The IDB is one of Canada’s key partners in Latin America and the Caribbean (LAC) and is relevant for its strategy of engaging with the Americas. In 2012, Canada renewed its Strategy for Engagement in the Americas and more than 30 departments, agencies, and Crown corporations now place strategic priority on engagement with the countries in the region to meet joint goals and further Canadian interests.^{Footnote 69} The 2012 strategy highlights three goals: increasing mutual economic opportunity; strengthening security and institutions; and fostering lasting relationships. Canada’s work with the IDB is an important avenue toward achieving these goals;^{Footnote 70} as the oldest and largest regional bank in the world, the IDB provides more financing to LAC countries than any other regional financial institution, it therefore has both credibility and influence in the region.

With respect to increasing mutual economic opportunity, one of Canada’s objectives is to encourage Canadian participation in public-private partnerships and with international and regional banks, including the IDB. The objective of leveraging Canada’s development assistance to advance sustainable economic growth is also in line with the IDB’s broader mandate to promote economic development in LAC.

In terms of strengthening security and institutions, the America’s strategy highlights the importance of sharing Canadian experience to strengthen institutions required for stability and growth. The strengthening of institutions and public sector capacity is one of the objectives guiding DFATD’s engagement with the IDB.

Finally, the America’s strategy is based around four instruments: reinforcing bilateral relationships; strengthening regional organizations; bolstering Canadian partnerships; and expanding Canada’s presence. The strategy refers explicitly to the IDB as one of the key regional organizations with which Canada can work to achieve these ends.^{Footnote 71}

4.3.3 Relevance of the IDB to Canada’s International Development Priorities

The IDB’s objectives closely align with Canada’s thematic priorities of stimulating sustainable economic growth, increasing food security, and securing the future of children and youth. IDB has also demonstrated commitment to the cross-cutting themes that should be integrated into all of Canada’s international development programs and policies (gender equality, environment and governance).

IDB’s strategy is centered on sustainable economic growth

With respect to Canada’s development priorities, the IDB has made evident contributions to stimulating economic growth. This is largely due to the Bank’s nature as a financial institution. The Bank’s institutional strategy (IDB-9) outlines two overarching objectives for Bank activities, one of which is achieving sustainable economic growth in LAC. In addition, IDB-9 calls for action along five sector priorities, one of which is strengthening institutions for growth and social welfare – which emphasizes access to financial markets, particularly for small and medium-sized enterprises (SMEs).^{Footnote 72} The IDB has a range of instruments for supporting sustainable economic growth, including Opportunities for the Majority, created in 2007 to promote and finance market-based, sustainable business models that engage private sector companies, local governments and communities in the development and delivery of quality products and services for people who are at the base of the pyramid in Latin America and the Caribbean.^{Footnote 73}

The Multilateral Investment Fund (MIF) was created to support economic growth and reduce poverty through encouraging private investment and advancing private sector development. It is the largest provider of technical assistance for private sector development in the region, providing financing in the form of grants, loans, guarantees, equity, and/or quasi-equity, as well as advisory services. According to a recent evaluation, the MIF has a clear comparative advantage in addressing the needs of poor and low-income populations and has worked where the private sector faces constraints.^{Footnote 74} It is also recognized as an innovation lab for the IDB.

Both the Opportunities for Majority initiative and the MIF are relevant instruments for addressing Canada’s priority of stimulating sustainable economic growth in the region.

The IDB Group’s role in financing development of the private sector is a key topic now as it pursues a private sector reform agenda that aims to merge the current private sector windows that include the non-sovereign guarantee lending of the IDB, the Multilateral Investment Fund, and the Inter-American Investment Corporation (IIC). The amalgamation will not only include organizational matters such as oversight and accountabilities for the new entity, but it will likely provoke discussions on the private sector’s role in development.

IDB projects and programs help to secure the future of children and youth

The IDB has contributed to improving the circumstances of children and youth in LAC through its investments in early childhood development, school to work transition, and improving the quality of educators. In addition, the MIF emphasizes skills training, with positive implications for children and youth in the region.

The IDB has been a long-time supporter of conditional cash transfer programs that provide cash directly to poor families (most notably in Mexico, Brazil, Colombia, and Guatemala). Its work in this area was highlighted positively in the country program evaluations considered in the review of development effectiveness of the IDB.

The Bank’s efforts to support member countries in developing effective and universal civil registration are also relevant to Canada’s priorities as strengthening civil registries would ensure that children have a legal identity and therefore, the ability to claim their rights and access the benefits to which they are entitled.

IDB support for agricultural development contributes to food security

A recent study produced by the IDB highlighted the importance of Latin America as the world’s breadbasket.^{Footnote 76} Thus, investments in productive and environmentally sustainable agricultural systems can contribute to global food security, as well as to the region’s development. One of the Bank’s four strategic priorities, as outlined in IDB-9, encompasses environmental protection, response to climate change, promotion of renewable energy, and enhancement of food security. The Bank’s primary means of improving food security is through its investments in agriculture, in particular agricultural production, infrastructure, and value chains. In 2008, the IDB implemented three initiatives to alleviate the effects of food price increases on the poor and to increase food production. It did this, in part, through agricultural infrastructure investments, such as irrigation systems.

IDB commitment to gender equality, environment and governance

The IDB has committed to mainstreaming gender equality, and, as noted in this review, began implementing a new Gender Policy (2010) and a Gender Action Plan. While these efforts are still in the early stages and the effects of these policies and plans are not yet clear, there is leadership and willingness to integrate gender considerations into Bank programs and projects.

The IDB has also made efforts to integrate environmental sustainability considerations into its programming. Its first Environment Policy in 1979 aimed to ensure the environmental quality of its operations and to support environmental projects within the LAC region. In 1994, the Eighth Replenishment of Resources identified the environment, poverty reduction, and social equity as areas of priority for Bank-funded projects and operations and included specific environmental mandates.^{Footnote 77}

The Board of Governors approved a strategy for Climate Change and Renewable Energy in 2011 that encompasses five areas of IDB engagement: knowledge for addressing mitigation and adaptation priorities; institutional and public and private capacity for climate change and action in the region; development of guidelines to mitigate the impact of IDB operations in certain sectors; identification and development of lending and technical cooperation in key sectors; and leveraging private sector investments for climate change initiatives and renewable energy.^{Footnote 78} Importantly, IDB’s 2010 Ninth General Capital Increase stated that, by 2015, 25% of the Bank’s lending program would address the areas of climate change, sustainable energy, and environmental sustainability.

IDB works with borrowing member countries on a wide range of issues related to good governance, including public sector strengthening and reform, decentralization, fiscal and economic issues. Its areas of emphasis are articulated in the Sector Strategy on Institutions for Growth and Social Welfare, approved in 2011. It has been promoting several initiatives of interest to Canada, including citizen security, transparency, and universal registry systems.

IDB role in countries of strategic priority to Canada

The Bank is active in the Caribbean, one of Canada’s regions of priority, and supports many of Canada’s countries of focus such as Honduras, Peru, Colombia, Bolivia, and Haiti. As part of the Ninth General Capital Increase (IDB-9), the Bank allocated $479 million to cancelling Haiti’s debt and mandated that the country’s undisbursed loans be converted to grants to facilitate recovery from the January 2010 earthquake.

A thorough review of Canada’s cooperation with the IDB through bilateral channels in these countries falls outside of the scope of this evaluation.  Nonetheless, respondents from the IDB as well as other donors recognize Canada’s efforts in continuously promoting and helping to strengthen the Bank’s work in the Caribbean, and especially in Haiti. Few other donor countries are as interested in or provide as strong a voice with respect to issues that arise in this sub-region.

4.3.4 Relevance of CIDA’s Strategy for Engagement with the IDB

CIDA’s strategy for engagement with the IDB outlined four objectives which were published on the DFATD website.^{Footnote 79} In general, these broad objectives were relevant to the issues that the IDB was facing at the time and aimed to engage the IDB in areas of priority to Canada. Canada signalled its clear support for the Bank in implementing the IDB-9 strategy and associated reforms.  CIDA explicitly committed to supporting the newly created Haiti Response Group, which was a necessary element in strengthening the Bank’s involvement in and support to post-earthquake Haiti. In addition, CIDA committed to support the Bank in strengthening its organizational effectiveness and results.

These objectives were primarily developed to guide the agency’s efforts and were not always well known by external stakeholders.  For the respondents who were familiar with the objectives, these were considered broad and encompassing a wide range of initiatives in response to many demands, which did not facilitate greater focus or selectivity in activities.

The strategy was conceived from the perspective of the former CIDA, and did not at the time reflectthe efforts of other government departments.  Respondents suggest that the relevance of a DFATD strategy can be enhanced if it encompasses all of the “tools” that the new Department has at its disposal, not only through the Global Issues and Development Branch, but through geographic programs, as well as through the Office of the Executive Director of Canada for the IDB.

4.4 Assessment of the achievement of former CIDA Strategic Objectives

This section of the report reviews Canada’s progress in achieving its four strategic objectives for its engagement with the IDB. CIDA did not develop a formal results framework aligned with these objectives. Thus, the section identifies key accomplishments or gaps that signal the extent of progress toward the overall objectives.

Objective 1 – Working with other IDB donors to implement the IDB’s new strategy as outlined in the Report on the Ninth General Increase in the Resources of the Inter-American Development Bank

The CIDA objective of supporting the implementation of the IDB institutional strategy as agreed in the Ninth General Capital Increase has been largely met. The objective outlined three areas in which Canada would pay particular attention: support for Haiti, focusing on the needs of the poorest, and monitoring of the IDB’s reform agenda.

• The mid-term evaluation of IDB-9 commitments carried out by the Office of Evaluation and Oversight (OVE) in 2013 indicated that the Bank had adopted policies to support the reforms indicated by IDB-9, but that there were still implementation challenges in some areas.^{Footnote 80} As an Executive Board member, Canada provided input to key Board discussions and key policy papers and is now monitoring the implementation of the reforms.
• In addition, Canada was recognized by several of those interviewed for playing an active and positive role in the Task Force that is addressing the problems encountered with the introduction of Macroeconomic Sustainability Assessments, one of the more challenging requirements of the IDB-9 commitments. The mid-term review of IDB- 9 included a recommendation to redesign this assessment process.^{Footnote 81}
• As part of its IDB-9 mandate, the Bank committed to forgiving Haiti’s debt incurred up to 2010 (US$479 million) and approving US$200 million in grant resources annually for the next 10 years. The Evaluation of IDB-9 Commitments (March 2013) confirms that these specific financial mandates for Haiti were largely met. It also confirms that the Bank has played an active role in catalyzing and enlisting resources from other bilateral agencies, coordinating its operations with its partners in the international donor community, and leading technical roundtables in its sectors of strategic focus.^{Footnote 82} The evaluation also confirms that achieving results and ensuring relevance in Haiti remains a major challenge for the IDB.

DFATD has been instrumental in monitoring the Bank’s commitments to Haiti, and in supporting the Bank in adapting its structure and approaches to Haiti’s more fragile context (including the establishment of the Haiti Response Group following the 2010 earthquake).

Objective 2 – Supporting the IDB’s efforts to strengthen institutions and public sector capacity, encourage private sector development, enhance trade regulations and frameworks, and support regional integration

Canada supported the Bank’s efforts to meet the goals outlined under this objective through its contribution of trust fund resources, ideas, and strategic guidance.

Trust Funds

• Canada is recognized for its contributions (CAD $10 million) to the Aid for Trade Strategic Thematic Fund, which helps countries in Latin America and the Caribbean expand trade by building the supply side capacity and trade-related infrastructure needed to implement and benefit from liberalized trade.^{Footnote 83} The grant-making fund has been one of the IDB’s tools in supporting integration and trade in the region. In addition, the Aid for Trade fund includes a grant-making window for technical assistance projects that address trade and gender, and trade and the environment, two of Canada’s cross-cutting themes.
• Canada has made a similar contribution (CAD $10 million) to the Regional Infrastructure Integration Fund (RIIF).
• In 2012, Canada launched the CAD $250 million Canadian Climate Fund for the Private Sector in the Americas, its largest partnership initiative with the Bank. The Climate Fund finances projects that focus on renewable energy, energy efficiency, and reducing greenhouse gas emissions and climate change vulnerabilities. Although it is still too early to see the results emerging from such an investment, the importance of this Fund for the Bank is noted. It was referred to as a potential “game changer” for private sector involvement in climate change.

Interviews to date suggest that by contributing to trust funds, Canada was able to influence strategic directions within the region, stay informed about regional dynamics, and build personal relationships with Bank staff to foster good will and cooperation.

Canada’s support for trust funds has also resulted in leveraging effects. First, there is the leveraging of IDB financing. Data suggest that between 1984 and 2013, Canada’s contributions to single donor trust funds and project specific grants in the region leveraged approximately another US$ 367 million in loans from the IDB.^{Footnote 84} Second, Canada’s investment in trust funds is often amplified by contributions from other countries. For example, in the Regional Infrastructure Integration Fund, every $1 contributed by Canada was leveraged by $1.20 from Spain, United States, Mexico and Colombia. Similar leverage effects have been reported with regard to its involvement in Aid for Trade.

In the area of private sector development, documents and interviews suggest that Canada has played a visible role in the MIF, during the period under review.  Canada is the sixth largest donor to the MIF and also engages the MIF on strategic issues such as its results focus and women’s economic empowerment. The MIF has been able to draw on Canada’s experience in development effectiveness metrics. DFATD results framework and project design tools have provided important reference in developing the MIF frameworks.

Objective 3 – Helping the IDB enhance its performance in terms of operations, development results, and focus on aid effectiveness.

And

Objective 4 – Working more intensively with IDB management to improve institutional effectiveness, including the development and use of an institution-wide, measureable results-based framework.

Results-based planning monitoring, and evaluation – Canada raises consistent messages about its expectations for planning, monitoring and evaluation that can help measure results. In addition to technical advice provided to the MIF (described above), Canada supported the IDB in its efforts to improve its Development Effectiveness Framework (DEF) and, in particular, strengthen its evaluation function. The Canadian Executive Director chaired the Policy and Evaluation Committee, which established the Terms of Reference and guided the work of an Independent Review Panel (2011) on the evaluation function at the Bank. Canada also played an important role in discussions regarding the OVE’s mid-term evaluation of IDB-9, largely because of the significance of the evaluation and also because it had chaired the Policy and Evaluation Committee during the period when many of the IDB-9 policies and issues were developed.

DFATD has also stressed managing for development results in the strategic discussions on trust funds through the IDB Office of Outreach and Partnerships. DFATD has provided technical advice to improve results frameworks and, according to interviewees this has been a positive contribution to the Bank and the spinoff effects of this work are noticeable as the quality of trust fund proposals and reporting increases.

Gender equality – As noted above, the IDB adopted an Operational Policy on Gender Equality and Development (2010) which is seen as a positive step forward. Its related action plan includes both proactive directives (direct investment and mainstreaming) and preventive directives (safeguards) for the policy.

As part of the former CIDA Strategy for Engagement with the Inter-American Development Bank (IDB) 2010-2013, Canada undertook a gender equality institutional assessment of the IDB. The assessment reportedly opened up a space for dialogue with the Bank’s Gender and Diversity Unit and had positive implications for the relationship between the gender unit and DFATD. Canada has been a contributor, along with other donors, to the Gender and Diversity Fund at the IDB, which aims to contribute to the equitable and culturally-appropriate development of IDB member countries by fostering gender equality, combating discrimination, and supporting development with identity. Canada is also recognized for asking questions concerning gender equality in Bank programming.

Other contributions

DFATD has often helped in facilitating the exchange of ideas on topics of mutual interest to the Bank and Canada through meetings held in Canada and Washington, D.C. Bank interviewees noted that these were useful in deepening engagement. In addition, there was secondment of a DFATD security specialist to the IDB. This type of support was seen to be invaluable for bringing new conceptual approaches and distinct experiences from Canada.

4.5 Management Practices: Assessing Canada’s efficiency with IDB

The assessment of efficiency generally requires data on the relationship between inputs and outputs. However, given data limitations in this regard, this section focuses on respondents’ perceptions of how DFATD managed the relationship with IDB, particularly with regard to time and resources, planning, communications and decision-making.  These management practices will be used as proxy indicators for efficiency.

In general, during the period under review, DFATD effectively managed its relationship with IDB.  Respondents suggested that DFATD was able to make important contributions to directing and influencing the IDB with the staff complement at hand.

Given that strategy setting has for the most part been an internal exercise, most interviewees outside the Department were not aware of DFATD’s objectives for engagement with the IDB. Some did see potential advantage in broader consultation and dissemination of strategic objectives in the future, with focus and clarity on such issues as the private sector and climate change.

Some interviewees noted that the 2013 amalgamation of DFAIT and CIDA created uncertainties about the appropriate channels for decisions and approvals, which have contributed to some delays and increased transaction time. Because of the changes, there are also uncertainties among external stakeholders (IDB and other donors) about whether there have been shifts in Canadian priorities for development and its engagement with the IDB.

5.0 Conclusions

Conclusions on the development effectiveness of the IDB

To the extent possible, the evaluation team considered the Bank’s recent progress as reported in the Office of Evaluation and Oversight’s mid-term evaluation of the implementation of IDB-9 commitments. However, an exhaustive analysis of the implications of recent reforms for the Bank’s development effectiveness was beyond the scope of this review.  Based on the key findings and related contributing factors reported, this review concludes the following:

1. IDB programs are relevant to the needs of target groups and support the development plans and priorities of IDB borrowing member countries. The IDB developed effective relationships with governments. The Bank’s programs were seen as particularly relevant in countries where it applied a strategic approach that clearly identified its comparative advantage. In addition, its level of alignment with the borrowing country’s development priorities and its use of appropriate tools for program implementation contributed to the relevance of the Bank’s programs.

2. For many years, the IDB lacked evidence of progress toward its stated development objectives and results; however, this is beginning to change. The programs reviewed often did not provide evidence of progress towards their stated development objectives. This was due in part to a lack of information from systematic monitoring of project or program level results and to contextual challenges that hindered program implementation. However, where information was available, the Bank was comparatively more successful in countries where it was able to develop a clear strategic approach, a specific market niche, and a focus on a limited number of areas in which it had comparative advantages. As further noted below, the Bank has made significant progress since 2008 in its design of projects and country programs and in the importance it gives to evaluation and learning. The retrospective nature of this review did not fully capture that growing evidence base on results.

3. The IDB had shortcomings in project and program level planning and in monitoring and evaluation, which limited the availability of information on development effectiveness during the period covered. The evaluations reviewed consistently pointed to problems on how Bank projects and programs identified results, defined indicators, and monitored progress, which led to poor evaluability of individual projects and country programs. The mid-term evaluation of IDB-9 suggests, however, that the introduction of the Development Effectiveness Framework in 2008 appears to be promoting development effectiveness at the project level and that progress in this area is promising.^{Footnote 85}

4. The review did not find solid evidence of the extent to which the results of Bank- supported programs are sustainable. Only 36% of the evaluation reports reviewed suggested that the Bank had contributed to strengthening the enabling environment for development and 40% indicated that programs and projects were sustainable in terms of institutional capacity. Sustainability seems to be hampered by a poor enabling environment in many countries. Factors limiting sustainability include a lack of recurrent cost funding mechanisms for infrastructure and service delivery projects, a lack of country ownership and political will, and weak capacity of the Bank-supported institutions.

5. The IDB has a strong system for conducting country program evaluations, but there is limited evidence that evaluations were used to improve development effectiveness. The IDB has a strong independent evaluation function. Country program evaluations have been carried out regularly at the IDB. However, approximately two-thirds of the evaluation reports reviewed did not include a management response or other evidence that the Bank took into account the recommendations from these evaluations. The Bank has taken recent actions in this area and in 2013 introduced a platform to record, manage, and monitor OVE’s recommendations. In addition, the Bank is implementing an ambitious plan to carry out impact evaluations (more than 67 are currently underway) with the intent of using these strategically to make meaningful contributions to the development knowledge base.

6. The IDB has made a strong commitment to gender equality in recent years, but this review found that its programs and evaluations do not yet regularly integrate gender equality as a cross-cutting theme. The evaluation reports reviewed provided insufficient coverage of gender equality, and the absence of gender equality in IDB projects was noted in previous studies.^{Footnote 86} The IDB has begun to address this situation by introducing a Gender Equality Policy in 2010 and a Gender Action Plan that provide a clear focus on gender equality and women’s empowerment as integral to development. The integration of gender equality considerations in IDB projects, country strategies, and evaluations is still in early stages.

7. The Bank has an environment policy, but the sample of evaluations reviewed provided insufficient coverage of environmental sustainability to warrant presentation of results in this area. The IDB takes a two-pronged approach to implementing its commitment to environmental sustainability: a mainstreaming approach, which seeks to enhance environmental and social benefits; and a safeguards approach, which seeks to minimize and compensate for negative impacts.^{Footnote 87} Recent reports suggest that the mainstreaming of environmental sustainability is a work in progress and that it is still not effectively integrated into the Bank’s country strategies. The sample of evaluations reviewed did not report on the extent to which program and project results were environmentally sustainable.

8. The Bank has faced challenges in managing its portfolio of projects efficiently, often due to the operating context at the country level. Many of the evaluations noted delays in the start-up or during the implementation of IDB projects. Complex country processes (such as those required for Congressional ratification of loans), the varying capacities of executing agencies, transactional difficulties (including delays in procurement and contracting), and complexities in project design resulted in delays and additional costs in the project cycle.

Conclusions on Canada’s engagement with the IDB

The review of Canada’s engagement with the IDB reflected on more specifically on CIDA’s engagement strategy. The assessment concludes that:

1. The IDB remains a highly relevant partner for DFATD in view of Canada’s international development priorities and its strategy for the Americas region. The IDB’s objectives closely align with Canada’s thematic priorities for stimulating sustainable economic growth, increasing food security, and securing the future of children and youth. IDB has also demonstrated commitment to the cross-cutting themes (gender equality, environment, governance) that should be integrated into all of Canada’s international development programs and policies.

2. DFATD has made important contributions to the IDB over the past three years and is a valued partner for the Bank. DFATD has made progress towards the objectives articulated in the former CIDA strategy for engaging with IDB. The data available demonstrate that Canada has been able to facilitate internal reforms at IDB by helping to build consensus on difficult topics such as the revised process for conducting Macroeconomic Sustainability Assessments and by promoting certain agendas (i.e. results focus, gender equality) that have had an uptake in the IDB and the Multilateral Investment Fund (MIF). Both the IDB and MIF value Canada’s financial contributions as well as its technical cooperation and the opportunities it provides for engaging the Bank in an exchange of ideas.

3. Canada is recognized for the unique role that it plays in the Caribbean, and especially in Haiti. Canada is a lead donor in this region. The IDB and other donors respect its expertise and on-going support to meet development challenges, particularly in Haiti.  DFATD has been effective in monitoring the Bank’s commitments to Haiti, and in supporting the Bank in adapting its structure and approaches to Haiti’s more fragile context.^{Footnote 88}

4. Canada’s potential for influence at IDB is greater than its relative voting power as a member of the IDB Board of Directors.^{Footnote 89} Although it is a small shareholder and may sometimes influence only at the margins, Canada plays a critical role in working with other Borrowing and Non-Borrowing member countries and building consensus among regional and extra-regional members. As a result, stakeholders welcome/encourage Canada to have a strong voice on IDB’s critical reform agendas and to play a bridging role. One of the key reform agendas at the IDB today is private sector reform and, in this regard, Canada is well positioned to contribute.

5. CIDA’s strategy for engaging with the IDB has been developed as internal guidance.  While objectives have been published on the website, they have not had high visibility with external stakeholders.  Prior to amalgamation, objectives primarily reflected the perspective of the former CIDA, one of the Canadian departments that engaged with the IDB.  The strategy and objectives in the past have lacked a clear performance measurement framework for reporting on results.

6. Recommendations for DFATD

This section contains recommendations to DFATD based on the review of IDB’s development effectiveness and the conclusions derived from the assessment of Canada’s engagement with IDB.

As one of several shareholders working with IDB, DFATD is limited in the extent to which it alone can influence improvements on the development effectiveness of the organization.  Therefore, DFATD needs to continue to engage with like-minded shareholders to advocate for some of these recommendations.

1. DFATD should take the opportunity presented by its recent amalgamation to refresh its strategy for engagement with the IDB, taking into account whole of government priorities, and proposing a performance measurement framework.

There are a number of good elements in place for setting objectives and guiding Canada’s engagement with the IDB.  These include published strategic objectives of the former CIDA; annual inter-departmental objective setting exercises; and formal statements at the Bank Annual Meeting which outline Canada’s views on issues confronting the Bank. The Department should set out how these will work together in setting clear medium term priorities for engagement with the Bank. This should reflect and communicate a whole of government approach to the Bank, integrating foreign affairs, trade, and development priorities.  It should include a focussed articulation of key results areas, and indicate how progress in these areas will be monitored and assessed.  It should be developed in consultation with and subsequently communicated to key stakeholders in the IDB in order to increase understanding of Canada’s key interests.

2. DFATD should play an active role in influencing the Bank’s potential restructuring of its private sector financing windows and ensuring that it leads to concrete improvements in development impact.  This will also provide an opportunity for the Department to communicate its vision of the private sector’s contribution to poverty reduction in the region.

Canada has stressed the important role that the private sector plays in driving inclusive economic growth.^{Footnote 90} At present consideration is being given to amalgamation of the IDB’s private sector financing windows.  Canada will have an important role to play in both positively influencing the structures and oversight adopted, as well as in articulating its vision of how the private sector can contribute to poverty reduction and sustainable development in the region.

3. DFATD should continue to emphasize and support the integration of gender equality and environmental sustainability in the operations of the IDB group.

DFATD has promoted integration of gender equality in the context of the MIF and in specific trust funds. It should continue to play a vocal role in this area, which has been a longstanding interest of Canada. In the area of environment, Canada has provided important trust funding for engaging the private sector in climate change. DFATD should clearly communicate Canadian priorities in these areas and encourage IDB reform efforts to strengthen environmental sustainability in its operations.

4. DFATD should continue to play an important role in ensuring that the IDB engages effectively in the Caribbean, and particularly in Haiti.

DFATD has a strong track record in the Caribbean. It is one of the leading bilateral donors in the region, and has been particularly valued for its knowledge and expertise in working in Haiti. DFATD should continue to encourage IDB’s work in the region, and in particular development of the necessary strategic and operational tools for working in the fragile context of Haiti.

5. DFATD should build on past efforts to have the IDB improve planning and monitoring of results, with clearer reporting on development effectiveness.

DFATD has been a source of technical advice and guidance in the area of results-based management.  Based on this experience, the Department should encourage the IDB to improve project results frameworks, increase use of evaluation evidence in corporate results reporting, and decision making, and demonstrate the contribution it is making to development in the region.

Annex 1: Effectiveness Criteria

Table 1: Common Development Effectiveness Assessment Criteria

Part One: Criteria for Assessing Development Effectiveness

Achieving Development Objectives and Expected Results

1.1 Programs and projects achieve their stated development objectives and attain expected results.

1.2 Programs and projects have resulted in positive benefits for target group members.

1.3 Programs and projects make a difference for a substantial number of beneficiaries and where appropriate contributed to national development goals.

1.4 Programs contributed to significant changes in national development policies and programs (including for disaster preparedness, emergency response and rehabilitation) (policy impacts) and/or to needed system reforms.

Cross-Cutting Themes – Inclusive Development which can be Sustained

2.1 Extent to which multilateral organization supported activities effectively address the cross-cutting issue of gender equality.

2.2 Extent to which changes are environmentally sustainable.

Sustainability of Results/Benefits

3.1 Benefits continuing or likely to continue after project or program completion or there are effective measures to link the humanitarian relief operations, to rehabilitation, reconstruction and, eventually, to longer term development results.

3.2 Projects and programs are reported as sustainable in terms of institutional and/or community capacity.

3.3 Programming contributes to strengthening the enabling environment for development.

Relevance of Interventions

4.1 Programs and projects are suited to the needs and/or priorities of the target group.

4.2 Projects and programs align with national development goals.

4.3 Effective partnerships with governments, bilateral and multilateral development and humanitarian organizations and NGOs for planning, coordination and implementation of support to development and/or emergency preparedness, humanitarian relief and rehabilitation efforts.

Efficiency

5.1 Program activities are evaluated as cost/resource efficient.

5.2 Implementation and objectives achieved on time (given the context, in the case of humanitarian programming).

5.3 Systems and procedures for project/program implementation and follow up are efficient (including systems for engaging staff, procuring project inputs, disbursing payment, logistical arrangements, etc.).

Using Evaluation and Monitoring to Improve Development Effectiveness

6.1 Systems and process for evaluation are effective.

6.2 Systems and processes for monitoring and reporting on program results are effective.

6.3 Results-based management (RBM) systems are effective.

6.4 Evaluation is used to improve development effectiveness.

Notes on Using the Criteria

In applying the above criteria to the review of evaluations concerning a given MO it is important to note the following guidelines:

• The findings of each evaluation as they relate to each of the 15 sub-criteria listed above (plus 4 relating to the use of evaluation) are to be classified using a four-point scale (Highly Satisfactory, Satisfactory, Unsatisfactory, and Highly Unsatisfactory).
• Analysis and reporting will be by sub-criteria and criteria only. There is no intention to combine criteria into a single index of development effectiveness.
• Specification of micro-indicators is not recommended because experience with similar evaluation reviews suggests that over-specification of the review criteria soon leads to very few evaluations being specific enough in their findings to address a significant number of micro indicators. In essence, this would involve over retro-fitting of external criteria to the evaluations in a way which distorts their original design. The experience of the pilot tests suggests that the level of specification illustrated is precise enough to allow for reliable classification of evaluation findings and still broad enough that most reasonably well designed evaluations will have findings which contribute to a credible analysis.

Annex 2: Evaluation Sample

Annex 3: Methodology

This annex provides a more detailed explanation of the population identification and sampling methodology used for the review of IDB development effectiveness.

It is structured around the sequence of tasks undertaken during the review: determining the rationale for the review; drawing the sample of evaluations; undertaking the process of review and controlling for quality during the analysis phase; and, assessing the level of coverage provided by the development effectiveness review.

The review of evaluation reports was supplemented by a review of IDB corporate documents. These were reviewed to expand the profile of the organization and its programming. A list of the documents consulted is provided in Annex 5.

Rationale for the Development Effectiveness Review

The term “common approach” describes the use of a standard methodology, as implemented in this review, to consistently assess the development effectiveness of multilateral organizations. It offers a rapid and cost effective way to assess the development effectiveness of the organization, relative to a more time consuming and costly joint evaluation.^{Footnote 91} The approach used in this review to assess the development effectiveness of multilateral organizations was developed to address a gap in the information available to bilateral development agencies.

Although multilateral organizations produce annual reports for their management and/or boards, member states were not receiving a comprehensive overview of the performance on the ground of multilateral organizations. This approach complements the assessments of the Multilateral Organization Performance Assessment Network (MOPAN). The approach suggests conducting a review based on the organization’s own evaluation reports when two specific conditions exist:^{Footnote 92}

• There is a need for field-tested and evidence-based information on the development effectiveness of the organization.
• The multilateral organization under review has an evaluation function that produces an adequate body of reliable and credible evaluation information that supports the use of a meta-synthesis methodology to synthesize an assessment of the organization’s development effectiveness.

Both conditions were satisfied for IDB, as explained in the following section.

Steps in the Process

The Review Team conducted the preliminary review of the information available to inform the Development Effectiveness Review of IDB. The preliminary review involved the following steps:

• Initiating the dialogue: DFATD conducted an initial briefing with IDB in Washington, with opportunities for the IDB to ask questions about the assessment methodology and for DFATD to confirm the universe of evaluation reports and to map out the various sources of evaluation products at the Bank. The Review Team also participated via telephone.
• Determining the universe of evaluations: DFATD compiled an initial list of evaluation reports published between 2008 and 2013, which was then verified and completed by the IDB. The finalization of the universe of evaluation reports involved several discussions and exchanges with OVE and management units in the IDB. The universe of evaluations published over this period was considered final on 21 December 2013 and included a total of 94 evaluation reports.
• Review of performance reporting and evaluation function: The Review Team carried out a desk review of IDB’s reporting on development effectiveness and on the evaluation function. The desk review of IDB’s reporting on development effectiveness was based on the criteria drawn from the methodological guidance from DAC-EVALNET for conducting such reviews to assess the organisation’s development effectiveness reporting.^{Footnote 93} The desk review of IDB’s evaluation function was based on the criteria of independence, coverage and evidence of use of evaluation.
• Quality screening of evaluation reports:^{Footnote 94} Each evaluation report in the universe was reviewed for quality according to 11 DAC-EVALNET evaluation quality criteria of the DAC- EVALNET Evaluation Quality Screening Scoring Guide listed in Annex 4. The guide includes two quality thresholds for each evaluation report – one based on total points received and another (hereafter referred to as the “methodological” threshold) based on the scores on the criteria related to (i) use of multiple lines of evidence, (ii) design of the evaluation, and (iii) findings and conclusions that are relevant and evidence based. The latter threshold focuses on the criteria that relate most directly to the quality of the information to be used for the assessment of development effectiveness since if an evaluation does not score well with respect to the design and implementation of the evaluation methodologies, then it is not likely to provide credible information on the MO’s effectiveness. An evaluation report must meet both thresholds in order to be considered for the Development Effectiveness Review. As a result of this exercise, a subset of 52 evaluation reports (corresponding to 55% of the entire universe of evaluation reports) was found suitable^{Footnote 95} for conducting the meta-synthesis that is foreseen under Scenario B of the DAC-EVALNET methodology.

Following the preliminary review, which led to the recommendation of applying Scenario B of the DAC-EVALNET methodology to the subset of 52 evaluation reports, a sample was drawn for the purposes of conducting the meta-synthesis. Nineteen reports were removed from the subset, among which twelve corporate evaluations since they did not report on any of the six common development effectiveness assessment criteria^{Footnote 96}, and the five project and impact evaluations, as these either were too narrow in scope (focusing only on a particular project), did not specify IDB’s role, or had been designed from a research perspective. As a result, the selected sample consisted of a total of 33 evaluation reports. The sample is provided in Annex 2.

Key characteristics of the evaluation sample

The sample of 33 evaluation reports that were considered for the development effectiveness review consisted of 29 country program evaluations, 2 corporate evaluations, and 2 thematic and sector evaluation. The key characteristics of the sample are described below.

• Geographic Coverage: The evaluation sample covers the majority of IDB’s borrowing member countries, including the five borrowing member countries receiving the highest levels of loan and guarantee approvals from the IDB from 2008-2013 (i.e. Argentina, Brazil, Mexico, Colombia, and Peru). The sample also includes countries from each of the Bank’s four country groupings, based on their relative level of development.^{Footnote 97}
• Coverage of the Bank’s concessional financing mechanisms : The sample includes evaluation reports from countries in the region receiving concessional financing from the IDB’s Fund for Special Operations (including Bolivia, Guatemala, Guyana, Haiti, Honduras, Nicaragua, and Paraguay) as well as grants through the IDB Grant Facility (in the case of Haiti).
• Broad Representation of IDB Operations (volume and type): The evaluation sample represents an important proportion (i.e. 94%) of the IDB’s loan and guarantee approvals in the period between 2007-2010.^{Footnote 98} In addition, the sample includes the most recent evaluation of the Multilateral Investment Fund, which approved 76 projects with an overall value of US $93.4 million in 2012.^{Footnote 99} Given the emphasis on Country Program Evaluations, the sample included performance information on different types of Bank products, including policy-based loans, lines of credit, investment loans, and technical cooperation, among others.
• Coverage of Development Effectiveness Criteria: Although the sampled reports all addressed development effectiveness criteria, the extent to which individual criteria and sub-criteria were covered varied from one evaluation report to another.
• Timeframe: All reports included in the evaluation sample were published between 2008 and 2013. Most evaluation reports focused on programming carried out between 2000 and 2010. As a consequence, neither the universe of evaluation reports nor the sample provide many insights into the results of IDB operations following its commitments outlined in IDB-9.

Table 4: Coverage of Effectiveness Criteria

Criteria: 1. Achievement of development objectives and expected results

Sub-criteria: 1.1 MO supported programs and projects achieve their stated development and/or humanitarian objectives and attain expected results.

Number of evaluations that addressed sub-criteria (a): 33
Coverage level: Strong

Sub-criteria: 1.2 MO supported programs and projects have resulted in positive benefits for target group members.

Number of evaluations that addressed sub-criteria (a): 19
Coverage level: Moderate

Sub-criteria: 1.3 MO programs and projects made differences for a substantial number of beneficiaries and where appropriate contributed to national development goals.

Number of evaluations that addressed sub-criteria (a): 13
Coverage level: Weak

Sub-criteria: 1.4 MO activities contributed to significant changes in national development policies and programs (including for disaster preparedness, emergency response and rehabilitation) (policy impacts) and/or to needed system reforms.

Number of evaluations that addressed sub-criteria (a): 25
Coverage level: Strong

Criteria: 2. Cross Cutting Themes: Inclusive Development Which Can Be Sustained

Sub-criteria: 2.1 Extent MO supported activities effectively address the cross-cutting issue of gender equality.

Number of evaluations that addressed sub-criteria (a): 0
Coverage level: Weak

Sub-criteria: 2.2 Extent changes are environmentally sustainable.

Number of evaluations that addressed sub-criteria (a): 5
Coverage level: Weak

Criteria: 3. Sustainability

Sub-criteria: 3.1 Benefits continuing or likely to continue after project or program completion or there are effective measures to link the humanitarian relief operations, to rehabilitation, reconstruction and, eventually, to longer-term developmental results.

Number of evaluations that addressed sub-criteria (a): 12
Coverage level: Weak

Sub-criteria: 3.2 Extent MO supported projects and programs are reported as sustainable in terms of institutional and/or community capacity.

Number of evaluations that addressed sub-criteria (a): 20
Coverage level: Moderate

Sub-criteria: 3.3 Extent MO development programming contributes to strengthening the enabling environment for development.

Number of evaluations that addressed sub-criteria (a): 25
Coverage level: Strong

Criteria: 4. Relevance

Sub-criteria: 4.1 MO supported programs and projects are suited to the needs and/or priorities of the target group

Number of evaluations that addressed sub-criteria (a): 29
Coverage level: Strong

Sub-criteria: 4.2 MO supported development projects and programs align with national development goals

Number of evaluations that addressed sub-criteria (a): 28
Coverage level: Strong

Sub-criteria: 4.3 MO has developed an effective partnership with governments, bilateral and multilateral development and humanitarian organizations and NGOs for planning, coordination and implementation of support to development and/or emergency preparedness, humanitarian relief and rehabilitation efforts.

Number of evaluations that addressed sub-criteria (a): 30
Coverage level: Strong

Criteria: 5. Efficiency

Sub-criteria: 5.1 Program activities are evaluated as cost/resource efficient

Number of evaluations that addressed sub-criteria (a): 13
Coverage level: Weak

Sub-criteria: 5.2 Evaluation indicates implementation and objectives achieved on time (given the context, in the case of humanitarian programming)

Number of evaluations that addressed sub-criteria (a): 31
Coverage level: Strong

Sub-criteria: 5.3 Evaluation indicates that MO systems and procedures for project/program implementation and follow up are efficient (including systems for engaging staff, procuring project inputs, disbursing payment, logistical arrangements etc.)

Number of evaluations that addressed sub-criteria (a): 28
Coverage level: Strong

Criteria: 6. Using Evaluation and Monitoring to Improve Development Effectiveness

Sub-criteria: 6.1 Systems and process for evaluation are effective

Number of evaluations that addressed sub-criteria (a): 32
Coverage level: Strong

Sub-criteria: 6.2 Systems and processes for monitoring and reporting on program results are effective

Number of evaluations that addressed sub-criteria (a): 32
Coverage level: Strong

Sub-criteria: 6.3 Results Based Management (RBM) systems are effective

Number of evaluations that addressed sub-criteria (a): 23
Coverage level: Moderate

Sub-criteria: 6.4 MO makes use of evaluation to improve development effectiveness

Number of evaluations that addressed sub-criteria (a): 30
Coverage level: Strong

Annex 4: Evaluation Quality Scoring Grid

Points for criteria scored

A. Purpose of the evaluation (Maximum Points = 3)

• why the evaluation was done (1)
• what triggered the evaluation (including timing in the project/program cycle) (1)
• how evaluation is to be used (1)

B. Evaluation objectives (Maximum Points = 2):

• evaluation objectives are clearly stated (1)
• objectives logically flow from purpose (1)

C. Organization of the evaluation (Maximum Points = 3)

• logical structure to the organization of the evaluation (1)
• evaluation is well written (1)
• clear distinction between evidence, findings, conclusions, and recommendations (1)

D. Subject evaluated is clearly described (Maximum Points = 4)

Evaluation describes:

• the activity/program being evaluated (1)
• the program’s expected achievements (1)
• how the program addresses the development problem (1)
• the implementation modalities used (1)

E. Scope of the evaluation (Maximum Points = 4)

Evaluation defines the boundaries of the evaluation in terms of:

• time period covered (1)
• implementation phase under review (1)
• geographic area (1)
• dimensions of stakeholder involvement being examined (1)

F. Evaluation criteria (Maximum Points = 5)

Evaluation criteria include:

• the achievement of development objectives and expected results (including impacts) (1)
• cross-cutting issues: inclusive development which is gender sensitive and environmentally sustainable (1)
• the sustainability of benefits and positive results achieved (1)
• the relevance of MO activities and supported projects and programs (1)
• the efficiency of MO operations in support of projects and programs (1)

G. Multiple lines of evidence (Maximum Points = 4)

• one point (1) for each line of evidence used (case studies, surveys, site visits, interviews, etc.), up to a maximum of four points (4)

H. Evaluation design (Maximum Points = 5)

Elements of a good evaluation design include:

• an explicit theory of how objectives and results were to be achieved (1)
• specification of the level of results achieved (output, outcome, impact) (1)
• baseline data (quantitative or qualitative) on conditions prior to program implementation (1)
• comparison of conditions after program delivery to those before (1)
• a qualitative or quantitative comparison of conditions among program participants and a control group (1)

I. Evaluation findings and conclusions are relevant and evidence based (Maximum Points = 4)

Evaluation report includes:

• evaluation findings relevant to the assessment criteria (1)
• findings that are supported by the chosen methodology (1)
• a clear logical link between the evidence and the finding (1)
• conclusions which are linked to the evaluation findings as reported (1)

J. Evaluation limitations (Maximum Points = 3)

• statement of the limitations of the methodology (1)
• impact of limitations on evaluation (1)
• remedies of limitations (1)

K. Evaluation Recommendations (Maximum Points = 3)

• evaluation contains recommendations that flow from findings and conclusions (1)
• recommendations are directed to one or more organization (1)
• recommendations are aimed at improving development effectiveness (1)

Total (required to have a minimum of 25 points) = 40

Total for Criteria G, H and I (required to have minimum of 10 points) = 13

Annex 5: Guide for Classifying Evaluation Finding^{Footnote 101}

Common Development Evaluation Assessment Criteria

1. Achievement of Development Objectives and Expected Results

1.1 MO supported programs and projects achieve their stated development and/or humanitarian objectives and attain expected results.

Criteria:

(1) Highly Unsatisfactory: Less than half of stated output and outcome objectives have been achieved including one or more very important output and/or outcome level objectives.

(2) Unsatisfactory: Half or less than half of stated output and outcome level objectives are achieved.

(3) Satisfactory: MO supported programs and projects either achieve at least a majority of stated output and outcome objectives (more than 50% if stated) or that the most important of stated output and outcome objectives are achieved.

(4) Highly Satisfactory: MO supported programs and projects achieve all or almost all significant development and/or humanitarian objectives at the output and outcome level.

1.2 MO supported programs and projects have resulted in positive benefits for target group members.

Criteria:

(1) Highly Unsatisfactory: Problems in the design or delivery of MO supported activities mean that expected positive benefits for target group members have not occurred or are unlikely to occur.

(2) Unsatisfactory: MO supported projects and programs result in no or very few positive changes experienced by target group members. These benefits may include the avoidance or reduction of negative effects of a sudden onset or protracted emergency.

(3) Satisfactory: MO supported projects and programs have resulted in positive changes experienced by target group members (at the individual, household or community level). These benefits may include the avoidance or reduction of negative effects of a sudden onset or protracted emergency.

(4) Highly Satisfactory: MO supported projects and programs have resulted in widespread and significant positive changes experienced by target group members as measured using either quantitative or qualitative methods (possibly including comparison of impacts with non-program participants). These benefits may include the avoidance or reduction of negative effects of a sudden onset or protracted emergency.

1.3 MO programs and projects made differences for a substantial number of beneficiaries and where appropriate contributed to national development goals.

Criteria:

(1) Highly Unsatisfactory: MO supported projects and programs have not contributed to positive changes in the lives of beneficiaries as measured quantitatively or qualitatively.

(2) Unsatisfactory: MO supported projects and programs have contributed to positive changes in the lives of only a small number of beneficiaries (when compared to project or program targets and local or national goals if established).

(3) Satisfactory: MO supported projects and programs have contributed to positive changes in the lives of substantial numbers of beneficiaries as measured quantitatively or qualitatively. These may result from development, relief, or protracted relief and rehabilitation operations and may include the avoidance of negative effects of emergencies.

(4) Highly Satisfactory: MO supported projects and programs have contributed to positive changes in the lives of substantial numbers of beneficiaries. Further, they have contributed to the achievement of specific national development goals or have contributed to meeting humanitarian relief objectives agreed to with the national government and/or national and international development and relief organizations.

1.4 MO activities contributed to significant changes in national development policies and programs (including for disaster preparedness, emergency response and rehabilitation) (policy impacts) and/or to needed system reforms.

Criteria:

(1) Highly Unsatisfactory: National policies and programs in a given sector or area of development (including disaster preparedness, emergency response and rehabilitation) were deficient and required strengthening but MO activities have not addressed these deficiencies.

(2) Unsatisfactory: MO activities have not made a significant contribution to the development of national policies and programs in a given sector or area of development, disaster preparedness, emergency response or rehabilitation. (Policy changes in humanitarian situations may include allowing access to the effected populations).

(3) Satisfactory: MO activities have made a substantial contribution to either re-orienting or sustaining effective national policies and programs in a given sector or area of development disaster preparedness, emergency response or rehabilitation.

(4) Highly Satisfactory: MO activities have made a substantial contribution to either re-orienting or sustaining effective national policies and programs in a given sector or area of development disaster preparedness, emergency response or rehabilitation. Further, the supported policies and program implementation modalities are expected to result in improved positive impacts for target group members.

2. Cross Cutting Themes: Inclusive Development Which can be Sustained

2.1 Extent MO supported activities effectively address the cross-cutting issue of gender equality.

Criteria:

(1) Highly Unsatisfactory: MO supported activities are unlikely to contribute to gender equality or may in fact lead to increases in gender inequalities.

(2) Unsatisfactory: MO supported activities either lack gender equality objectives or achieve less than half of their stated gender equality objectives. (Note: where a program or activity is clearly gender-focused (maternal health programming for example) achievement of more than half its stated objectives warrants a satisfactory rating).

(3) Satisfactory: MO supported programs and projects achieve a majority (more than 50%) of their stated gender equality objectives.

(4) Highly Satisfactory: MO supported programs and projects achieve all or nearly all of their stated gender equality objectives.

2.2 Extent changes are environmentally sustainable.

Criteria:

(1) Highly Unsatisfactory: MO supported programs and projects do not include planned activities or project design criteria intended to promote environmental sustainability. In addition changes resulting from MO supported programs and projects are not environmentally sustainable.

(2) Unsatisfactory: MO supported programs and projects do not include planned activities or project design criteria intended to promote environmental sustainability. There is, however, no direct indication that project or program results are not environmentally sustainable. OR MO supported programs and projects include planned activities or project design criteria intended to promote sustainability but these have not been successful.

(3) Satisfactory: MO supported programs and projects include some planned activities and project design criteria to ensure environmental sustainability. These activities are implemented successfully and the results are environmentally sustainable.

(4) Highly Satisfactory: MO supported programs and projects are specifically designed to be environmentally sustainable and include substantial planned activities and project design criteria to ensure environmental sustainability. These plans are implemented successfully and the results are environmentally sustainable.

3. Sustainability

3.1 Benefits continuing or likely to continue after project or program completion or there are effective measures to link the humanitarian relief operations, to rehabilitation, reconstructions and, eventually, to longer-term developmental results.

Criteria:

(1) Highly Unsatisfactory: There is a very low probability that the program/project will result in continued intended benefits for the target group after project completion. For humanitarian relief operations, the evaluation finds no strategic or operational measures to link relief, to rehabilitation, reconstruction and, eventually, to development.

(2) Unsatisfactory: There is a low probability that the program/project will result in continued benefits for the target group after completion. For humanitarian relief operations, efforts to link the relief phase to rehabilitation, reconstruction and, eventually, to development are inadequate. (Note, in some circumstances such linkage may not be possible due to the context of the emergency. If this is stated in the evaluation, a rating of satisfactory can be given)

(3) Satisfactory: Likely that the program or project will result in continued benefits for the target group after completion. For humanitarian relief operations, the strategic and operational measures to link relief to rehabilitation, reconstruction and, eventually, development are credible.

(4) Highly Satisfactory: Highly likely that the program or project will result in continued benefits for the target group after completion. For humanitarian relief operations, the strategic and operational measures to link relief to rehabilitation, reconstruction and, eventually, development are credible. Further, they are likely to succeed in securing continuing benefits for target group members.

3.2 Extent MO supported projects and programs are reported as sustainable in terms of institutional and/or community capacity.

Criteria:

(1) Highly Unsatisfactory: The design of MO supported programs and projects failed to address the need to strengthen institutional and/or community capacity as required. In the case of humanitarian operations, the design of programs and projects failed to take account of identified needs to strengthen local capacities for delivery of relief operations and/or for managing the transition to rehabilitation and/or development.

(2) Unsatisfactory: MO programs and projects may have failed to contribute to strengthening institutional and/or community capacity or, where appropriate, to strengthen local capacities for delivery of relief operations and/or for managing the transition to rehabilitation and/or development.

(3) Satisfactory: MO programs and projects may have contributed to strengthening institutional and/or community capacity but with limited success.

(4) Highly Satisfactory: Either MO programs and projects have contributed to significantly strengthen institutional and/or community capacity as required or institutional partners and communities already had the required capacity to sustain program results.

3.3 Extent MO development programming contributes to strengthening the enabling environment for development.

Criteria:

(1) Highly Unsatisfactory: For development programs, there were important weaknesses in the enabling environment for development (the overall framework and process for national development planning; systems and processes for public consultation and for participation by civil society in development planning; governance structures and the rule of law; national and local mechanisms for accountability for public expenditures, service delivery and quality; and necessary improvements to supporting structures such as capital and labour markets). Further, the MO activities and support provided to programs and projects failed to address the identified weakness successfully, further limiting program results.

(2) Unsatisfactory: MO development activities and/or MO supported projects and programs have not made a notable contribution to changes in the enabling environment for development.

(3) Satisfactory: MO development activities and/or MO supported projects and programs have made a notable contribution to changes in the enabling environment for development including one or more of: the overall framework and process for national development planning; systems and processes for public consultation and for participation by civil society in development planning; governance structures and the rule of law; national and local mechanisms for accountability for public expenditures, service delivery and quality; and necessary improvements to supporting structures such as capital and labour markets.

(4) Highly Satisfactory: MO development activities and/or MO supported projects and programs have made a significant contribution to changes in the enabling environment for development including one or more of: the overall framework and process for national development planning; systems and processes for public consultation and for participation by civil society in development planning; governance structures and the rule of law; national and local mechanisms for accountability for public expenditures, service delivery and quality; and necessary improvements to supporting structures such as capital and labour markets. Further, these improvements in the enabling environment are leading to improved development outcomes.

4. Relevance

4.1 MO supported programs and projects are suited to the needs and/or priorities of the target group.

Criteria:

(1) Highly Unsatisfactory: Substantial elements of program or project activities and outputs were unsuited to the needs and priorities of the target group.

(2) Unsatisfactory: No systematic analysis of target group needs and priorities took place during the design phase of developmental or relief and rehabilitation programming or there is some evident mismatch between program and project activities and outputs and the needs and priorities of the target group.

(3) Satisfactory: MO supported activity, program or project is designed taking into account the needs of the target group as identified through a process of situation or problem analysis (including needs assessment for relief operations) and the resulting activities are designed to meet the needs of the target group.

(4) Highly Satisfactory: Methods used in program and project development (including needs assessment for relief operations) to identify target group needs and priorities (including consultations with target group members) and the program and project takes those needs into account and is designed to meet those needs and priorities (whether or not it does so successfully).

4.2 MO supported development projects and programs align with national development goals

Criteria:

(1) Highly Unsatisfactory: Significant elements of MO supported development program and project activity run counter to national development priorities with a resulting loss of effectiveness.

(2) Unsatisfactory: Significant portion (1/4 or more) of the MO supported development programs and projects are not aligned with national plans and priorities, but there is no evidence that they run counter to those priorities.

(3) Satisfactory: Most MO supported development programs and projects are fully aligned with national plans and priorities as expressed in national poverty eradication and sector plans and priorities. Wherever MO supported programs and projects are reported in the evaluation as not directly supportive of national plans and priorities they do not run counter to those priorities.

(4) Highly Satisfactory: All MO supported development projects and programs are reported in the evaluation to be fully aligned to national development goals as described in national and sector plans and priorities, especially including the national poverty eradication strategy and sector strategic priorities.

4.3 MO has developed an effective partnership with governments, bilateral and multilateral development and humanitarian organizations and NGOs for planning, coordination and implementation of support to development and/or emergency preparedness, humanitarian relief and rehabilitation efforts.

Criteria:

(1) Highly Unsatisfactory: MO experiences significant divergence in priorities from those of its (government, NGO or donor) partners and lacks a strategy or plan which will credibly address the divergence and which should result in strengthened partnership over time.

(2) Unsatisfactory: MO has experienced significant difficulties in developing an effective relationship with partners and that there has been significant divergence in the priorities of the MO and its partners.

(3) Satisfactory: MO has improved the effectiveness of its partnership relationship with partners over time during the evaluation period and that this partnership was effective at the time of the evaluation or was demonstrably improved.

(4) Highly Satisfactory: MO has consistently achieved a high level of partnership during the evaluation period.

5. Efficiency

5.1 Program activities are evaluated as cost/resource efficient

Criteria:

(1) Highly Unsatisfactory: Credible information indicating that MO supported programs and projects (development, emergency preparedness, relief and rehabilitation) are not cost/resource efficient.

(2) Unsatisfactory: MO supported programs and projects under evaluation (development, emergency preparedness, relief and rehabilitation) do not have credible, reliable information on the costs of activities and inputs and therefore the evaluation is not able to report on cost/resource efficiency. OR MO supported programs and projects under evaluation present mixed findings on the cost/resource efficiency of the inputs.

(3) Satisfactory: Level of program outputs achieved (development, emergency preparedness, relief and rehabilitation) when compared to the cost of program activities and inputs are appropriate even when the program design process did not directly consider alternative program delivery methods and their associated costs.

(4) Highly Satisfactory: MO supported (development, emergency preparedness, relief and rehabilitation) programs and projects are designed to include activities and inputs that produce outputs in the most cost/resource efficient manner available at the time.

5.2 Evaluation indicates implementation and objectives achieved on time (given the context, in the case of humanitarian programming)

Criteria:

(1) Highly Unsatisfactory: Less than half of stated output and outcome level objectives of MO supported programs and projects are achieved on time, there is no credible plan or legitimate explanation found by the evaluation which would suggest significant improvement in on-time objectives achievement in the future.

(2) Unsatisfactory: Less than half of stated output and outcome level objectives of MO supported programs and projects are achieved on time but the program or project design has been adjusted to take account of difficulties encountered and can be expected to improve the pace of objectives achievement in the future. In the case of humanitarian programming, there was a legitimate explanation for the delays.

(3) Satisfactory: More than half of stated output and outcome level objectives of MO supported programs and projects are achieved on time and that this level is appropriate to the context faced by the program during implementation, particularly for humanitarian programming.

(4) Highly Satisfactory: Nearly all stated output and outcome level objectives of MO supported programs and projects are achieved on time or, in the case of humanitarian programming, a legitimate explanation for delays in the achievement of some outputs/outcomes is provided.

5.3 Evaluation indicates that MO systems and procedures for project/program implementation and follow up are efficient (including systems for engaging staff, procuring project inputs, disbursing payment, logistical arrangements etc.)

Criteria:

(1) Highly Unsatisfactory: Serious deficiencies in agency systems and procedures for project/program implementation that result in significant delays in project start-up, implementation or completion and/or significant cost increases.

(2) Unsatisfactory: Some deficiencies in agency systems and procedures for project/program implementation but does not indicate that these have contributed to delays in achieving project/program objectives.

(3) Satisfactory: Agency systems and procedures for project implementation are reasonably efficient and have not resulted in significant delays or increased costs.

(4) Highly Satisfactory: Efficiency of agency systems and procedures for project implementation represent an important organizational strength in the implementation of the program under evaluation.

6. Using Evaluation and Monitoring to Improve Development Effectiveness

6.1 Systems and process for evaluation are effective.

Criteria:

(1) Highly Unsatisfactory: Evaluation practices in use for programs and projects of this type (development, emergency preparedness, relief and rehabilitation) are seriously deficient.

(2) Unsatisfactory: No indication that programs and projects of this type (development, emergency preparedness, relief and rehabilitation) are subject to systematic and regular evaluations.

(3) Satisfactory: Program being evaluated is subject to systematic and regular evaluations or describes significant elements of such practice. No mention of policy and practice regarding similar programs and projects. This may include specialized evaluation methods and approaches to emergency preparedness, relief and rehabilitation programming.

(4) Highly Satisfactory: Program being evaluated (along with similar programs and projects) is subject to systematic regular evaluations or describes significant elements of such practice.

6.2 Systems and processes for monitoring and reporting on program results are effective

Criteria:

(1) Highly Unsatisfactory: Absence of monitoring and reporting systems for the development and humanitarian programming. This would include the absence of adequate monitoring of outputs during the implementation of humanitarian programming.

(2) Unsatisfactory: While monitoring and reporting systems for the development and humanitarian programming exist, they either do not report on a regular basis or they are inadequate in frequency, coverage or reliability.

(3) Satisfactory: Monitoring and reporting systems for development and humanitarian programming as appropriate are well-established and report regularly.

(4) Highly Satisfactory: Monitoring and reporting systems for the program are well-established and report regularly. The quality of regular reports is rated highly by the evaluation and results are reportedly used in the management of the program.

6.3 Results Based Management (RBM) systems are effective

Criteria:

(1) Highly Unsatisfactory: No evidence of the existence of an RBM system for the program and no system is being developed.

(2) Unsatisfactory: While an RBM system is in place, or being developed, it is unreliable and does not produce regular reports on program performance.

(3) Satisfactory: RBM system is in place and produces regular reports on program performance.

(4) Highly Satisfactory: RBM system is in place for the program and there is evidence noted in the evaluation that the system is used to make changes in the program to improve effectiveness.

6.4 MO makes use of evaluation to improve development/humanitarian effectiveness

Criteria:

(1) Highly Unsatisfactory: Report does not include a management response and does not have one appended to it or associated with it. There is no indication of how the evaluation results will be used. There is no indication that similar evaluations have been used to improve effectiveness in the past.

(2) Unsatisfactory: Report includes a management response (or has one attached or associated with it) but it does not indicate which recommendations have been accepted. OR There is some, non-specific indication that similar evaluations have been used to improve program effectiveness in the past.

(3) Satisfactory: Report includes a management response (or has one attached or associated with it) that indicates which recommendations have been accepted. OR There is a clear indication that similar evaluations in the past have been used to make clearly identified improvements in program effectiveness.

(4) Highly Satisfactory: Report includes a management response (or has one attached or associated with it) describes a response to each major recommendation which is appropriate and likely to result in the organizational and programmatic changes needed to achieve their intent.

Annex 6: Corporate Documents Reviewed

Annex 7: List of Sources for the Chapter on Canada’s Engagement with the IDB

Interviews

Nancy Lee, Manager and Yuri Soares, Development Effectiveness Unit, Multilateral Investment Fund

Ginya Truitt Nakata, Operations Sr. Specialist, Office of Outreach and Partnerships

Carol Nelder-Corvari, Alternate Executive Director, Office of the Executive Director for Canada

Jason Hollman, Manager, Partnerships for Development Innovation, Foreign Affairs, Trade, and Development Canada

Mariette Maillet, Senior Analyst, Global Issues and Development Branch, Foreign Affairs, Trade, and Development Canada

Marita Olson, Representative of Sweden

Ana Maria Rodríguez, Manager, Institutions for Development, Inter-American Development Bank

Johannes Hubertus Paulus Smeets, Representative of the Netherlands

Tamara Sorger, Counsellor, Office of the Executive Director for Canada

Documents and web sites

CIDA. CIDA Strategy for Engagement with the Inter-American Development Bank (IDB) 2010-2013. 2010.

CIDA. Gender Equality Institutional Assessment of the Inter-American Development Bank (IDB). 2011.

CIDA. 2011/2012 Report of the Office of the Executive Director for Canada, Inter-American Development Bank. 2012.

DFATD. Canada’s Strategy for Engagement in the Americas. 2014. http://www.international.gc.ca/americas-ameriques/stategy-stratege.aspx?lang=eng

DFATD Development Information System. IDB Disbursements By Year and By Channel. 2014.

DFATD. Evaluation of the Americas Strategy, Section 3.0, Relevance. 2011.

DFATD. Inter-American Development Bank. 2013. http://www.acdi-cida.gc.ca/acdi-cida/acdi-cida.nsf/eng/nad-27153647-s72

DFATD. 2012/2013 Report of the Office of the Executive Director for Canada, Inter-American Development Bank. 2013.

Foreign Affairs and International Trade Canada. Evaluation of the Americas Strategy. 2011.

IDB. Board of Governors. 2014. http://www.iadb.org/en/about-us/board-of-governors,1325.html

IDB Board of Governors. Report on the Ninth General Capital Increase in the Resources of the Inter-American Development Bank. 2010.

IDB. Civil Registration and Identity Management. 2014. http://www.iadb.org/en/topics/government/civil-registration-and-identity,4032.html

IDB Office of Strategic Planning and Development Effectiveness. Annual Business Review. 2012.

IDB. Public Consultations, Strategy for Institutions for Growth and Social Welfare. 2014. http://www.iadb.org/en/civil-society/public-consultations/strategy-for-institutions-for-growth-and-social-welfare-sigs/strategy-for-institutions-for-growth-and-social-welfare,1842.html

IDB. Strengthening Evaluation to Improve Development Results. 2011. http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=36931307

IDB, Global Harvest Initiative, The Next Global Breadbasket: How Latin America Can Feed the World: A Call to Action for Addressing Challenges and Developing Solutions. 2014.

Annex 8: Management Response

The Development Review of the Inter-American Development Bank, prepared by DFATD’s Evaluation Division, provides an overview of the development effectiveness of the Bank, an assessment of DFATD’s engagement with the Bank, and recommendations intended to help DFATD strengthen its relationship with IDB.

The evaluation correctly notes that the IDB is an important development partner for Canada in Latin America and the Caribbean.  As the main source of multilateral financing for economic, social and institutional development in the region, the IDB serves as an important vehicle for delivering Canadian development goals, and for enhancing Canada’s Strategy for Engagement in the Americas.  Canada is the third largest non-borrowing shareholder in the Bank, with a shareholding of 4%. We remain committed to working in partnership with the IDB to strengthen its developmental impact in the region.

DFATD broadly agrees with the conclusions related to the development effectiveness of the IDB as outlined in the Review.   The Review indicates that IDB programs are relevant to the needs of target groups, that the Bank has recently made significant progress in its approach to results based management, and that the IDB has a strong system for conducting evaluations.   However, further improvements are required in some key areas, in particular in ensuring the sustainability of results achieved and in continuing efforts to integrate gender equality and environmental sustainability in IDB operations. The evaluation rightly notes that the retrospective nature of its approach does not allow it to fully capture the growing evidence base on results or changes implemented by the Bank since GCI-9.  We agree with the Review’s findings that the Bank is already addressing some of the areas of concern identified and that progress is promising.

DFATD welcomes the findings related to Canada’s engagement with the IDB. As previously noted, we confirm the finding that IDB is a highly relevant partner for DFATD. We also appreciate the recognition of Canada’s key contributions to the IDB and its potential to influence and play a leadership role at the Bank.  We will seize the opportunities presented through amalgamation to further advance Canadian foreign, trade and development priorities through the IDB.

Looking forward, DFATD accepts the recommendations included in the report.  The recommendations call on DFATD to continue efforts related to:  1) adopting a whole of Government approach to our engagement with the Bank; 2) influencing the potential private sector re-organization; 3) supporting IDB’s engagement in the Caribbean and Haiti; 4) advancing the integration of gender equality and environmental sustainability in IDB operations; and 5) furthering results based management.  DFATD is already engaging in these areas, and will continue to do so both through our Executive Director’s office and directly during Annual Meetings and in discussions with Bank management and staff.  DFATD will also undertake specific actions as outlined in the detailed responses to the specific recommendations provided in the table below.

Recommendations: 1. DFATD should take the opportunity presented by its recent amalgamation to refresh its strategy for engagement with the IDB, taking into account whole of government priorities, and proposing a performance measurement framework.

Commitments/measures: Agree

1.1 DFATD will continue to work with Finance Canada in setting annual objectives for its engagement with the IDB. These performance objectives build on the commitments outlined in the Institutional Strategies, reflect whole of government priorities, and are designed as actionable staff-level objectives, enabling us to assess performance each year.

1.2 DFATD will update information on its website related to its engagement with IDB, including its strategic objectives.

Responsible: DFATD/Global Issues and Development

Completion date: March 2015

Status:

Recommendations: 2. DFATD should play an active role in influencing the Bank’s potential restructuring of its private sector financing windows and ensuring that it leads to concrete improvements in development impact. This will also provide an opportunity for the Department to communicate its vision of the private sector’s contribution to poverty reduction in the region.

Commitments/measures: Agree

2.1 DFATD will continue to actively engage in discussions regarding the Bank’s potential restructuring of its private sector financing windows through:

1. engagement through our Executive Director’s office to provide leadership in the restructuring process at the Board;
2. participation in upcoming Governors’ meetings; and
3. direct engagement with IDB/IIC staff, including at senior levels.

2.2 Efforts above will include:

1. continuing to seek strong synergies between public and private operations; and
2. active dialogue with the private sector gender working group to ensure gender equality is effectively mainstreamed in private sector operations.

2.3. During discussions with Bank officials, DFATD will communicate the Government of Canada’s private sector and development approach that seeks increased results through investment, partnerships and innovation.

2.4 DFATD will write a letter to senior management of the Bank, outlining the key findings of the review. This letter will reiterate Canada’s expectation that the potential restructuring of the private sector financing windows should lead to concrete improvements in development impact.

Responsible: DFATD/Global Issues and Development

Completion date: April 2016

Status:

Recommendations: 3. DFATD should continue to emphasize and support the integration of gender equality and environmental sustainability in the operations of the IDB group.

Commitments/measures: Agree

3.1 As per Section 2.4, DFATD will write a letter to senior management of the Bank, outlining the key findings and requesting the Bank to address the key areas of concern identified in a timely manner. In particular, DFATD will stress the importance of ensuring gender equality and environmental sustainability are included in accountability frameworks, evaluations, and country strategies.

3.2 DFATD will update its Gender Equality Institutional Assessment and Strategic Environmental Assessment for the IDB in order to further identify priority areas for improvement.

3.3 DFATD will also work with the Canadian Executive Director’s office at the IDB to continue to raise these issues:

1. during Board meetings;
2. at Annual Meetings; and
3. directly with IDB staff, including at senior levels, and to mobilize support from other likeminded chairs.

Responsible: DFATD/Global Issues and Development

Completion date: March 2016, March 2017 (3.3)

Status:

Recommendations: 4. DFATD should continue to play an important role in ensuring that the IDB engages effectively in the Caribbean, and particularly in Haiti.

Commitments/measures: Agree

4.1 DFATD/Global Issues and Development Branch will continue to work in close collaboration with DFATD/Americas Branch and our Executive Director’s office to monitor the Bank's overall commitments and results achieved in the Caribbean and Haiti.

4.2 DFATD/Americas through its bilateral program in Haiti will continue to collaborate with IDB as both a donor and project partner, while encouraging the IDB to continue to improve the quality and effectiveness of its programming in Haiti, in particular to intensify coordination efforts among donors, strengthen the GoH to provide effective leadership, and increase the IDB in-house capacity to design, implement and monitor programs that seek to deliver results over the long term. DFATD will continue to reach out to these countries, through the Executive Director's office and during the Annual Meeting to further our understanding of their priorities and engagement with the IDB.

Responsible: DFATD/Global Issues and Development

Completion date: March 2016

Status:

Recommendations: 5. DFATD should build on past efforts to have the IDB improve planning and monitoring of results with clearer reporting on development effectiveness.

Commitments/measures: Agree

5.1. As per section 2.4, DFATD will write a letter to senior management of the Bank, outlining the key findings and requesting the Bank to address the key areas of concern identified in a timely manner, including in the area of planning and monitoring of results.

5.2 DFATD will work with the Canadian Executive Director’s office at the IDB to continue to seek further improvements in planning and monitoring of results:

1. during Board meetings;
2. at Annual Meetings; and
3. directly with IDB staff, including at senior levels, and to mobilize support from other likeminded chairs.

5.3 DFATD will also work with the Canadian Executive Director’s Office to actively engage in current discussions to update the IDB’s corporate results framework as well as ongoing efforts to introduce results-based budgeting.

Responsible: DFATD/Global Issues and Development

Completion date: March 2016

Status: