Evaluation of Development Programming in former Countries of Modest Presence 2009-10 to 2013-14 – Synthesis Report

Acknowledgments

This evaluation was undertaken by a team of external consultants from Goss Gilroy Inc., which included Louise Mailloux (Team Leader), Catherine Gander, Ali Anwer and Lindsay Renault. The evaluation was overseen by Deborah McWhinney from the Development Evaluation Division at Global Affairs Canada. Support for the preparation of the country profiles was provided by Gabrielle Biron-Hudon.

We would like to thank the numerous people who have contributed to this evaluation, including Country Program staff at headquarters and in the field who provided the documentation, organised and participated in interviews, and reviewed the draft report. Many other stakeholders made themselves available for meetings, interviews and project site visits, including Government officials, donor representatives, implementing partners, numerous civil society organisations, and individual consultants who were generous with their time and provided the evaluation team with valuable insights into their programs and projects.

David Heath
Head of Development Evaluation
Global Affairs Canada

List of Acronyms and Abbreviations

Executive Summary

This report presents the findings, lessons and recommendations from the evaluation of programming in eight former countries of modest presence^{Footnote 1} over a 5-year period (FY 2009-10 to 2013-14). The evaluation assessed their performance (relevance, effectiveness, and sustainability) and efficiency of programming, with an emphasis on programming through the bilateral delivery channel, the largest delivery channel across the eight countries.

As the evaluation was complex, spanning eight countries over three continents, it relied largely on secondary data sources (qualitative and quantitative) from documents and administrative data collected by Global Affairs Canada. Primary data (primarily qualitative), collected by the evaluation team through interviews, was used to validate the secondary data. The evaluation team visited seven of the eight countries in the sample. Jordan was not visited; rather, interviews were conducted by telephone. A purposive sample of four to six key bilateral projects per country was selected in order to validate and examine reported Country Program outcomes.

Findings

Relevance

The issues of food security and sustainable economic growth were relevant to the defined priorities and needs of partner countries. They were addressed primarily through sustainable agriculture production and modernizing government policies to create an enabling environment,. In Jordan and Morocco, generating employment for youth was a key priority with a focus placed on improving vocational training. Canadian programming in this area was both innovative and responsive. Addressing governance issues was relevant in all countries. A focus on human rights issues was particularly relevant in Guatemala and Sri Lanka. Within the focus on one or two thematic priorities, programming in the former countries of modest presence was able to address governance issues with specific and targeted programming.

There were numerous examples of innovative approaches that were introduced in Canadian-supported initiatives, such as the Integrated Watershed Management approach in Nicaragua, which incorporated agricultural production, environmental education, community development and grassroots capacity building, that were then adopted and replicated at a regional or national level. Other examples included the development of value chains in Cuba, business development services in Egypt, the adoption of a competency-based approach to vocational training in Morocco and multi-lingual service provision in Sri Lanka.

Effectiveness

There was considerable evidence of the achievement of project and Program-level immediate outcomes in all eight countries. Evidence of the achievement of intermediate outcomes was limited.

Canada had a strategic niche in gender equality in Jordan and the Philippines and achieved significant gender-related results in other countries as well. Canada’s traditional strength on gender equality issues was generally acknowledged but, with the exception of the Philippines, the engagement on gender equality issues had waned over the evaluation period. This was due, in part, to the emphasis on certain thematic priorities as of 2009-10 and also due to the reduced availability of gender specialists or advisors to support efforts to mainstream gender in projects.

Development programming in six of the eight former countries of modest presence contributed to sustainable environmental practices for smallholder producers through targeted agriculture initiatives (agricultural was not an area of focus in Morocco or Jordan). Most projects made significant contributions to strengthening governance, such as assisting the Government of Sri Lanka to implement key components of its Official Languages Policy across the country or contributing to reducing impunity in Guatemala. Overall, programming in these former countries of modest presence assisted with the implementation of national policies and the organization of service delivery in agriculture (most countries), local governance (Philippines, Morocco), vocational training (Jordan, Morocco, Sri Lanka), as well as micro, small and medium business development (Egypt, Philippines).

Key enabling factors that enhanced effectiveness included:

• strategically focusing resources in certain themes and geographic areas;
• closely and strategically aligning programming with partner government priorities;
• implementing programming through long-term trusted partners and building on previous successful initiatives;
• using a variety of funding approaches (e.g. contracts, contribution agreements, grants, pooled funds, local funds); and,
• good coordination with other donors and strong policy dialogue.

Key hindrances to effectiveness included:

• the lack of institutional capacity of host government partners; and,
• inadequate support from specialists, particularly with regard to gender equality.

Sustainability

In general, the outcomes achieved in countries were more likely to be sustainable when institutionalized by country partners. There were examples of this in various countries. The sustainability of interventions in former countries of modest presence was enhanced when there were opportunities to work with and through government entities at various levels and with strong civil society partners. Global Affairs Canada used their financial contributions to leverage financial support from other donors, thereby enhancing the potential sustainability of initiatives.

Efficiency

In all eight countries, efficiency was enhanced by taking a “fewer, bigger” approach to projects, engaging in pooled funding, where possible. Programming through multilateral organisations reduced transaction costs for Global Affairs Canada but also led to reduced visibility and opportunities for Canada to engage in policy dialogue with certain government partners and monitor activities directly. Further, focusing time and energy on a reduced number of large projects also posed risks in the event that a project was not approved, given that there were fewer projects in the pipeline.

Some decision-making was informed by project evaluations and monitoring data but almost all Country Programs would have benefited from more frequent project monitoring, as well as more robust performance measurement - particularly to gather data on results at the intermediate or ultimate outcome level. While program indicators were often agreed upon by government partners, there was very limited use of national systems to monitor and report on results due to weak monitoring systems in many partner countries.

The operational efficiency of programming was compromised by key external factors, such as slow decision-making by partner countries or the occurrence of natural disasters. Internal factors that hindered efficiency were related to resource limitations, and the impact of decentralisation and amalgamation on processes and systems. These factors included: slow project approval processes; unclear strategic direction; limited support on corporate contracting mechanisms and inadequate access to corporate information systems. Reductions in staffing in former countries of modest presence, combined with the closure of Program Support Units, impacted efficiency as the remaining program staff were required to also complete administrative and logistical tasks.

With the exception of the Philippines, staff in these eight former countries of modest presence had difficulty responding to all programming requirements and corporate requests. When time constraints occurred, project monitoring and policy dialogue were often sacrificed. Program staff mitigated these challenges by hiring local consultants or choosing between various donor fora but program activities suffered from the lack of staff and resources.

Lessons Learned

Projects that are strategic but with low dollar value can have a major impact. Some of the most relevant and successful initiatives supported by Canada in former countries of modest presence were those that were a high priority for the partner government and had concrete buy-in from both government and other donors to scale-up successful initiatives.

Canadian development programming in former countries of modest presence achieved strategic results with relatively small budgets. To maximize the impact of those limited resources, it was more effective to build on previously effective programming and to seek synergies between projects and with other donor initiatives, where possible.

Reductions in staff and support available for programming led to a further emphasis on responsive programming and increased use of the multilaterals to implement projects in many countries. This reduced transaction costs and the need to monitor projects but also resulted in reduced Canadian involvement in policy dialogue with partner government, visibility and direct contact with partners and stakeholders.

Even though budgets were small for these former countries of modest presence, they were still required to develop and implement a range of projects, engage in policy dialogue and monitor and evaluate on-going initiatives. The smaller budgets did not lead to a reduced workload for development staff in the former countries of modest presence. As such, they require a minimum number of staff with an adequate mix of skills (professional, specialist and administrative) in order to manage development programming effectively and efficiently.

As with all countries where Global Affairs Canada undertakes development programming, it is crucial to have a robust performance measurement system that is able to track and report on results at both the project and program level. Knowledge management systems are necessary to ensure continuity and to off-set frequent staff turnover. Supporting national capacities for monitoring and evaluation would strengthen the evidence base and guide decision-making for future development initiatives.

Recommendations

In light of the findings, conclusions and lessons, and in the context of an amalgamated Department, the evaluation recommends that Global Affairs Canada:

1. Identify criteria that define development partner countries, including their resource requirements.

2. Ensure that adequate support is provided to staff in development partner countries to plan, implement, monitor and report on development results, including strengthening their capacity to integrate governance and gender equality into programming.

3. Ensure that development partner countries have the flexibility to manage a diversified portfolio using different programming modalities thereby allowing them to mitigate risk and respond to emerging priorities.

4. Strengthen its suite of program-level results-based management tools and increase the level of support for results-based program management, including evidence-based planning, monitoring and reporting by the programs.

5. Establish or strengthen systems of support to field staff in development partner countries to enable strategic choices with regard to policy dialogue and donor coordination.

Management Response

This management response reflects the departmental priorities as expressed at the end of the evaluation period. Going forward, management responses will reflect new Departmental priorities as they evolve.

Commitments and Actions (NMD and PCD)

NMD and PCD will :

Commitments and Actions (NMD)

Commitments and Actions (PCC)

Commitments and Actions (MGS)

Commitments and Actions (MED)

Commitments and Actions (NMD)

Commitments and Actions (PCC)

Commitments and Actions (PCC)

Commitments and Actions (NMD)

1.0 Introduction

This report presents the findings, lessons and recommendations from the evaluation of development programming in eight former countries of modest presence.

In June 2013, the Department of Foreign Affairs and International Trade and the Canadian International Development Agency (CIDA) were amalgamated into Foreign Affairs, Trade and Development Canada (DFATD). It has since been renamed Global Affairs Canada, and will be referred to as such in this report.

1.1 Evaluation Purpose and Objectives

The purpose of this evaluation was to enhance learning and fulfill accountability requirements in accordance with the Federal Administration Act and the Treasury Board Policy on Evaluation.

The objective of the evaluation was to assess Global Affairs Canada programming in eight countries across three regions: the Americas (Cuba, Guatemala, Nicaragua), Middle East and Northern Africa (Egypt, Jordan, Morocco), and Asia (Philippines, Sri Lanka). Until 2014, these countries were termed, “countries of modest presence”. They are referred to as “former countries of modest presence” for the purpose of this evaluation.^{Footnote 2}

The evaluation assessed the relevance and performance of development programming (including effectiveness, efficiency, and sustainability), as well as cross-cutting issues and performance management. It also examined how Global Affairs Canada developed and implemented programming across three thematic priorities – increasing food security, stimulating sustainable economic growth and securing the future of children and youth – and documented lessons learned, particularly as they related to innovative programming in these three thematic priorities.

The evaluation is primarily intended to provide Global Affairs Canada management and programming staff with lessons and recommendations to support evidence-based decision-making on policy, expenditure management and program improvement. The secondary audience includes Canada’s development partners and the Canadian public. It is meant to inform them of the relevance and performance of Canada’s developmental programming.

1.2 Evaluation Object and Scope

The object of this evaluation was development programming in eight middle-income countries where Canada had a modest presence. The scope was a 5-year period from FY 2009-10 to 2013-14 with an emphasis on the 2009-2013 programming cycle. Attention was given to the bilateral delivery channel, which represented the majority of disbursements to these countries over the period of the evaluation (with the exception of humanitarian assistance provided in certain years to Sri Lanka, the Philippines and Jordan).

2.0 Methodology

2.1 Approach

This summative evaluation took a theory-based^{Footnote 3} and utilisation-focused approach^{Footnote 4} in its examination of decision-making, accountability and learning. As such, the focus was on generating practical findings and recommendations that could be acted upon by Global Affairs Canada. The evaluation team adopted a balanced and systematic approach to answering the evaluation questions, weighing available qualitative and quantitative data across multiple sources of evidence.

The evaluation team made an effort to contextualize programming and results in light of the realities in the partner countries. This included internal factors, such as the changing policy context in Global Affairs Canada during the period under review, and external factors, such as humanitarian crisis (e.g. Jordan) and natural disasters (e.g. Philippines).

2.2 Evaluation Criteria and Questions

The evaluation was conducted according to the Treasury Board of Canada’s Evaluation Policy (2009) and in accordance with quality standards for development evaluation of the Organisation for Economic Co-operation and Development’s (OECD) Development Assistance Committee (DAC). Table 1 presents the criteria and questions that guided the evaluation after having been agreed upon by the Advisory Committee at the end of the inception phase of this evaluation. Please refer to Appendix B for the Evaluation Matrix.

Table 1: Evaluation Criteria and Questions

1. Relevance

   • 1.1 To what extent has Global Affairs Canada programming been relevant to the development priorities and development needs in countries of modest presence?

   • 1.2 What have been the most relevant innovations^{Footnote 5} and ‘best practices’ in the programming carried out in countries of modest presence?

2. Performance – Effectiveness

   • 2.1 To what extent have the planned results identified in the 2009 Country Strategies been achieved?

   • 2.2 What contributions have the Global Affairs Canada -funded programs made in these countries, including the cross-cutting themes of gender equality, governance and environmental sustainability?

   • 2.3 To what extent has Global Affairs Canada programming contributed to progress in the Canadian thematic area priorities?

   • 2.4 What were the factors enabling or limiting the achievement of results, e.g. resources, policy dialogue available, choice of and synergy/complementarity between delivery channels, programming approaches and/or initiatives?

3. Performance – Economy/Efficiency

   • 3.1 To what extent has operational efficiency at the Country Program level been maximized in countries of modest presence?

   • 3.2 To what extent have performance measurement frameworks (and/or logic models, theories of change) been used to ensure adequate monitoring and reporting on results achieved? If not, why not? Has performance data been used to make timely adjustments to projects or programs?

   • 3.3 To what extent is Global Affairs Canada’s monitoring and reporting approach linked to the countries’ own reporting systems?

4. Performance – Sustainability

   • 4.1 To what extent has sustainability of programming been achieved in countries of modest presence?

   • 4.2 What conditions have enabled or hindered sustainability?

2.3 Sampling

There were two levels of sampling undertaken in this evaluation. First, consideration was given to the sample countries to be included. Following a discussion with representatives of the programs in each of the 14 former countries of modest presence and thematic specialists, a grouping of countries by region was agreed upon for this evaluation as well as the second phase of the evaluation. As a result, this evaluation includes 8 countries from 3 regions – Latin America, Middle East and North Africa and Asia-Pacific.The second phase will include the remaining 6 countries, all of which are in Africa.

The second level of sampling was the selection of projects to assess as a way of evaluating the Country Programs. A purposive sample of 4 to 6 bilateral projects was selected per country in order to examine and validate reported Country Program outcomes. The criteria for the selection of the sample included:

1. the existence of recent project evaluations;
2. encompassing close to 50% of the disbursements made by the bilateral program; and,
3. representativeness of programming in relation to three priority thematic areas (sustainable economic growth, food security, and children and youth).

Appendix C provides the list of sample projects along with additional summary information.

2.4 Data Collection Methods

The evaluation employed various methods to collect data. As the evaluation was complex, spanning eight Country Programs over three continents and had relatively limited resources to collect primary data, the evaluation relied largely on secondary data sources (qualitative and quantitative) from documents and administrative data collected by Global Affairs Canada. Secondary data was validated with primary data collected by the evaluation team via interviews during field missions. The evaluation team visited seven of the eight countries. Jordan was not visited. Interviews were conducted via telephone.

Figure 1: Data collection methods

2.5 Data Analysis

The evaluation team took a systematic approach to data analysis by collating and comparing evidence within and across the lines of evidence to develop findings, conclusions and recommendations. Administrative data was analyzed in more detail for 3 sample countries (Nicaragua, Egypt, and Sri Lanka) and compared with 3 sample countries of focus to inform a more in-depth analysis of operational efficiency (see Section 4.4 on Efficiency and Appendix E for more information).

2.6 Challenges and Limitations

The evaluation team relied largely on information from Global Affairs Canada, both in terms of reporting and key informants. Generally, documentation of results at the program level was qualitative and descriptive in nature without a clear or detailed assessment of the extent to which program-level outcomes were achieved beyond the immediate outcome level for the period covered by Country Strategies (2009-10 to 2013-14). The amount and quality of documentation available for sample projects varied widely.

The extent to which there were key informants that could speak for the whole evaluation period varied. At times, there were few individuals available to speak to earlier years, both from Global Affairs Canada and programming partners, due to staff turnover and departures. The evaluation team sought to fill data gaps in one line of evidence by pursuing data in more depth from another source (e.g. ensuring to interview several project representatives when project documentation was sparse). However, because the former countries of modest presence were relatively small programs with a limited number of staff, the evaluation team, at times, had to rely on evidence from a few interviews. When there was insufficient evidence or there were issues of data quality that hindered the ability to adequately triangulate the data, conclusions were not made. This had an impact on the ability of the evaluation team to make conclusive statements about sustainability, in particular.

3.0 Canada’s Programming in Former Countries of Modest Presence

Canada disbursed a total of $565 million to the 8 countries covered by this evaluation. Table 2 summarizes key information about Global Affairs Canada development programming for the 2009-10 to 2013-14 period, including budget and thematic focus. While six of the eight countries remained non-focus countries as of 2014, Jordan and the Philippines became countries of focus in 2014-15.

Figure 2 provides an overview of disbursement to the eight countries by thematic priority. “Other” includes humanitarian assistance,^{Footnote 6} human rights and gender equality.

Figure 2: Disbursement in Countries of Modest Presence 2009-10 to 2013-14

Figure 2: Disbursement in Countries of Modest Presence 2009-10 to 2013-14 – Table

Figure 2: Disbursement in Countries of Modest Presence 2009-10 to 2013-14 – Numbers

Countries	Children and youth	Food security	Sustainable economic growth	Others
Source: Global Affairs Canada Data 11-15
Cuba	$307,056	$3,027,048	$13,395,721	$5,140,303
Guatemala	$4,364,688	$12,800,610	$14,055,285	$15,674,900
Nicaragua	$9,459,988	$12,874,422	$38,781,800	$7,563,989
Egypt	$14,835,579	$2,530,963	$20,033,910	$13,060,435
Jordan	$23,071,493	$16,974,016	$10,959,252	$34,259,028
Morocco	$10,294,488	$67,281	$17,220,363	$7,148,571
Philippines	$1,043,360	$20,912,353	$39,336,675	$53,641,261
Sri Lanka	$1,216,867	$23,143,106	$14,241,976	$31,528,031

4.0 Findings

The evaluation team gathered evidence from a range of sources, assessed the quality of data and triangulated it in order to arrive at the findings below. The findings are presented by criterion and respond to the evaluation questions.

4.1 Pertinence

Examples of this alignment included:

Cuba: A key priority of the Government of Cuba (GoC) over the evaluation period was to reduce Cuba’s dependence on food imports by increasing agricultural productivity. The 2011 GoC Guidelines reaffirmed food security and economic modernization as national priorities. Global Affairs Canada responded to these priorities and adapted projects to changing needs, such as recovery after extensive hurricane damage in 2012.

Guatemala: The leading national development priorities in the Government of Guatemala’s (GoG) sectoral development plans were food security, governance, human security and, to some extent, rural economic development within the context of fiscal management and economic competitiveness. Canada’s food security programming was particularly relevant in light of the country's priority development challenge, chronic child malnutrition. Canada's geographic focus in the Western Highlands region was also a priority for the GoG, being among the areas of highest need and where few donors were working.

Nicaragua: The Nicaragua Country Program responded to the priorities of the Government of Nicaragua (GoN) Human Development Plan by concentrating on sustainable economic growth, including energy improvements in the electricity sector and climate change, and food security. In response to a request from the GoN, Global Affairs Canada focused the majority of its initiatives in a region sensitive to environmental shocks, known as the “dry corridor”, which was stricken by elevated levels of extreme poverty and food insecurity.

Morocco: The Government of Morocco (GoM) strategic directions placed particular emphasis on training and productive employment in the rural sector, developing local skills and implementing the national employment pact. Education and vocational training were crucial for the GoM for decades. Canada’s governance programming was relevant to the GoM Regionalization Strategy-2010 reflecting the country’s pursuit of decentralization and fostering regional autonomy.

Jordan: Global Affairs Canada programming was relevant to Jordan’s national development priorities to improve the quality of education and develop human resources for a knowledge economy. The Country Program focused on improving education and skills for employment linked to host government development objectives outlined in the 2006-2015 National Agenda and the Education Reform for the Knowledge Economy initiative. During this time, Global Affairs Canada adjusted to changing needs and priorities in the country, addressing the growing pressure of conflict in Syria by increasing humanitarian and other bilateral programming.

Egypt: Although developed prior to the revolution of 2011, the focus by Global Affairs Canada on sustainable economic growth remained relevant to post-revolution socio-economic challenges and development priorities in Egypt. This included strengthening the enabling environment for micro, small and medium enterprises, providing skills for employment to marginalized populations and supporting cluster and sectorial business development services providers. The Program aligned with other national goals of poverty alleviation, job creation, as well as youth and women’s economic empowerment.

Philippines: The Country Program was relevant to key priorities of the Government of the Philippines (GoP) that focused on sublimating high economic growth into more inclusive growth through job creation, promoting more open and transparent government, and by increasing the competitiveness in such high-growth sectors as tourism, information technologies and agribusiness. This was consistent with the emphasis that the 2011-2016 Philippine Development Plan placed on micro, small and medium enterprise sectors as catalysts for economic growth.

Sri Lanka: Canadian programming aligned with the national development strategies of Sri Lanka (Mahinda Chintana and Lessons Learnt and Reconciliation Commission) and relevant to the needs of Sri Lankans affected by conflicts, as most projects addressed the economic needs of marginalized people or assisted in reconciliation efforts. However, unlike the Government of Sri Lanka’s focus on infrastructure development within its development strategies, Canada focused on training and capacity building of people and institutions. Improving access for marginalized rural populations, especially those on tea plantations, to vocational training was a priority. Addressing language rights was a pre-eminent post conflict issue that Canada was in a unique position to help remedy given its experience with two official languages.

The evaluation found evidence of several initiatives, approaches or models that were replicated or scaled up in many Country Programs. Some models were adopted by the partner country Government with its own budget or with funding leveraged from other donors. Other projects actively assisted the national partner country to leverage funding from other donors for expanded follow-up.

Cuba: An innovative value chain approach to agricultural production of Global Affairs Canada’s was scaled up through the Agrocadenas project funded by the European Union and the Swiss Agency for Development and Cooperation. Following Global Affairs Canada’s first international industrial certification program in Cuba, the Ministry of Industry has plans to replicate it in the eastern region of Cuba. Similarly, the piloting of a value chain and municipal management of dairy production (local micro-industries) was scaled up through a large European Union-funded project. The same project introduced an improved system for collecting and storing milk in local cooling centres, leading the Ministry of Agriculture to establish 741 milk cooling centres across the country.

Guatemala: Elements of two projects were replicated and scaled-up nationally and had the potential for further expansion: first, the national association of coffee growers (Anacafé) adopted a method of pruning and renewing coffee plantations, replicated it across the country and then sold this expertise to Jamaica; second, the Ministry of Agriculture adopted more environmentally friendly fertilizer practices around Lake Atitlán.

Nicaragua: DAFTD’s successful Integrated Watershed Management approach, which incorporated agricultural production, environmental education, community development and grassroots capacity building, helped the government to mobilize $33 million from the Inter-American Development Bank, the Global Environment Facility and the World Bank to implement the model in other environmentally sensitive regions of Nicaragua. The National Institute of Technology adapted and shortened the methodology used to train young farmers in remote areas to improve agriculture production through ecologically sound and modern techniques to deliver agricultural technical training to 31,000 youth in communities across Nicaragua.

PELNICA (Nicaragua Electrification Project) applied a national planning system for rural electrification developed through a prior Canada-funded project (PREICA). PELNICA served as the model for the “National Sustainable Electrification and Renewable Energy Program”, an Inter-American Development Bank-led US $418 million program of which US $80 million was dedicated to rural electrification. The management system for the collection of client usage data and fees based on a Canadian model was to be replicated in the next extension phase.

The methodology and approach used for the Fourth National Agricultural Census (CENAGRO IV), for which Canada was the leading donor, was particularly successful and innovative. It was the sole census in Latin America aggregating national data with external data sources to assess specific census coverage. Used to validate the progress made in the agriculture sector through Nicaragua’s Productive Rural Development Sector Program (PRORURAL), it led a $33.9 million grant from the Global Agriculture and Food Security Program to further improve productivity and competitiveness of smallholder farmers.

Morocco: The successful Canadian ‘competency-based approach’ for more effective vocational training was scaled up across the Moroccan education system with financial support from the European Union, USAID and Germany. One component of the Support to School Management project integrated activities around a common initiative to improve organisational efficiency and led the Ministry of Education to extend it across all existing 13,400 public schools, as well as new schools.

The Ministry of the Interior replicated a successful strategy developed in a Canadian-funded project to augment the capacity of local governments, modernize work methods and improve the performance of employees, structures and organisations in three regions of southern Morocco.

Philippines: The models and tools that the Philippines Commission on Women developed as part of the Gender Responsive Economic Actions for the Transformation of Women project (GREAT) to mainstream gender issues, such as its Economic Empowerment approach, gained national and international prominence. The Philippines Commission on Women was invited to present its models to the Asia-Pacific Economic Cooperation and at the Association of Southeast Asian Nations (ASEAN) Summit in 2015.

Initially, Local Economic Development was not a priority of Local Government Units (LGUs). However, the local governance and economic development project influenced the Department of the Interior and Local Government who became advocates for Local Economic and Investment Promotion Officers in LGUs and urged the Department to fund this position at the LGU level.

Sri Lanka: The National Languages Project led to innovations that the Government began to replicate as it expanded the piloting of front line services in ‘language of choice’ in seven local governments to approximately 13 sites. The Department of National Languages and Social Integration of the Ministry of Public Administration and Local Government planned to replicate this pilot project throughout the country.

Enabling and Hindering Factors for Replication and Scaling-up Innovation

Key enabling factors for the extension of Global Affairs Canada models and approaches were the strong engagement from national partners, as was the case in Cuba, Nicaragua, Egypt, Morocco and the Philippines, as well as a robust policy environment supported by national laws, decrees and strategies in key thematic areas. In all, the level of development of these countries facilitated the institutionalization and replication of new approaches and models by the government. However, some governments lacked the technical expertise or funds to maintain or scale-up initiatives. In many cases, partners relied on large donors to maintain and extend results. Donor coordination facilitated this process; Canada was present in such groups and in thematic working groups where information on successful initiatives was shared. This often led larger donors, like the European Union and regional development banks, to pick up successful initiatives or models.

4.2 Effectiveness

An analysis of data from performance measurement frameworks, evaluations of sample projects and stakeholder interviews showed that most immediate outcomes were met, and exceeded in several cases. However, with the exception of the Philippines and Cuba, performance measurement frameworks lacked solid evidence of the achievement of intermediate outcomes.

Overall, irrespective of Global Affairs Canada’s relatively small investment, stakeholders in all eight countries reported that Canadian development assistance achieved results.

Cuba: Global Affairs Canada projects made progress towards expected program-level intermediate outcomes, notably through increased diversification, availability and sustainability of food production in 10 local fund projects. Key project-level outcomes included:

• increased and diversified local agricultural production;
• improved management of production through adopting a value chain approach;
• strengthened capacity of municipal governments and cooperatives;
• greater participation of cooperatives and farmers in local decision-making; and,
• enhanced national policies for food security and agricultural development.

Global Affairs Canada also contributed to more effective, efficient and accountable public services delivery by introducing and institutionalizing internationally recognized public accounting practices and a project for “citizen-based” transportation services. Another project helped improve efficiency and productivity in the energy and mining sectors by establishing a more skilled and competitive workforce.

The Sectoral Support Fund in Cuba was labour intensive but proved to be an effective way for a small Country Program to respond both quickly and strategically to government and civil society players using modest amounts of money.

Guatemala: At the country-program level, Global Affairs Canada made significant contributions to improving food security and SEG in select regions in the Western Highlands and the dry corridor. The Program progressed toward its 2009 Country Strategy expected outcomes, and met or exceeded targets related to immediate outcomes. More specifically, it contributed to greater food security in seven departments by:

• increasing the agricultural productivity of small-holder farmers;
• fostering sustainable economic growth by increasing the incomes of small-holder farmers and small businesses; and,
• supporting their capacity to generate future income.

Although not among the three thematic priorities for 2009-2014, the Country Program improved the rule of law nationally by contributing to a reduction in the impunity in Guatemalan. The number of Canadian NGOs working in Guatemala dropped from 49 in 2004 to 12 in 2014; and 15 of those NGOs left between 2009-10 and 2013-14, limiting the number of partners for Global Affairs Canada to work with.^{Footnote 7}

Nicaragua: The Country Program attained all immediate Program outcomes, which provided enabling conditions for economic growth and led to increased food security. The Program assisted over 220,000 households to improve access to agriculture inputs and technical assistance through pooled funding of PRORURAL. This major national program also contributed to a 22% increase in agricultural production, which was cited as contributing to the reduction in chronic child malnutrition by 20% nationally and rural poverty by 10% since 2009. Other Canadian-funded initiatives directly assisted over 6,000 farming households, including 2,000 young farmers, to increase and diversify agricultural production in the Northern Region or “dry corridor” through training in improved production methods, the introduction of alternative crops and the implementation of an integrated watershed management institutionalized in 17 municipalities. The Country Program also contributed to increased access to electricity for over 120,000 people in the Northern Region, particularly in rural areas. Nationally, one million children in over 8,000 schools gained better access to food through Canada supported programming by the World Food Programme.

Egypt: Progress was made toward expected intermediate outcomes in Egypt. Global Affairs Canada projects directly supported the achievement of two of the immediate outcomes: an “improved access to financial and non-financial products and services that are demand-driven, sustainable and affordable for women and men, micro, small and medium enterprises (MSMEs) owners in selected areas and sectors”, as well as “supporting increased responsiveness of the policy and regulatory framework to meet the needs of MSMEs”. Concrete results included assisting with the establishment of a national strategy for MSMEs, operationalizing an SME Law and training of procurement managers from governmental bodies to collect baseline data on MSMEs and encourage MSME participation in the procurement process.

Morocco: Canadian innovative methods and approaches strengthened education and vocational training, as well as gender equality in Morocco. The relevance and effectiveness of these approaches led to increased donor investment. Although a small donor, the Canadian program was generally perceived as highly strategic and effective due to its innovative programming and pragmatic and leading edge approaches.

Jordan: The Country Program made progress towards its two intermediate outcomes of increasing the level of operational effectiveness of the management of the education system and improving the quality of operational strategies produced by the Ministry of Education as the educational planning process was decentralised from the national level to the field.

Progress towards immediate outcomes were also made as the number of directorates and schools using the new education model went from 780 schools and 7 directorates to 3,148 schools and 35 directorates during the life of the project supported by Canada.

Philippines: The Country Program met most expected program-level outcomes relating to “improving government effectiveness to stimulate public and private investment”, and anticipated results at the immediate level in the generation of employment. Specific results include:

• Approximately 75 local government units (LGUs) completed local economic development support management system and policy reforms, which helped launch 4,269 new businesses and 5,271 existing ones expand;
• Creation of new economic opportunities for more than 35,000 micro entrepreneurs, smallholder farmers and low-skilled workers, 71% of who were women;
• Policy development (98 local policies) in 45 assisted LGUs and the adoption by 13 national economic policies with specific provisions to empower women in the informal sector to increase access to services; and,
• Gender mainstreaming activities covering 21 Agencies.

Sri Lanka: The Sri Lanka Country Program achieved immediate outcomes in sustainable economic growth, with some progress towards intermediate outcomes on the issue of language rights. Vocational training included 4,236 individuals, which exceeded the target of 3,750. Graduates of the training saw monthly wage increases of 510% for male and 340% for female graduates, with a female participation rate of 30%, well above the national rate of 10%.

Contribution to the Thematic Priorities

Sustainable Economic Growth

As reported above, the contribution of the Nicaragua Country Program to PRORURAL led to a 22% overall increase in productivity of targeted crops and reductions in the level of chronic childhood malnutrition and rural poverty. In Sri Lanka, 18,000 returnee households were provided with support to increase food production, largely through inputs. In Guatemala, farming households over seven departments reported increased income from improved agriculture practices. In the Philippines, the Country Program helped bring new economic opportunities for more than 35,000 micro entrepreneurs, smallholder farmers and low-skilled workers, 71% of who were women.

In Egypt, Global Affairs Canada programming facilitated micro, small and medium enterprises access to loans from Egyptian banks valued at approximately $1.4 million, and helped to mobilize US$ 1.3 billion in complementary lines of credit from World Bank Group to improve access to finance, out of which US$ 410 million are investments in Egypt and Tunisia. Egypt Enterprise Development Project partners facilitated access to 170 loans totalling over $10 million. Programming strengthened the government systems in Jordan and Morocco to deliver programs aimed at boosting employment, including women, by enhancing the management effectiveness of responsible ministries, e.g. education, labour.

Food Security

Four Country Programs helped to improve food security through increased agricultural production and/or though World Food Programme initiatives.

In Cuba, the Support Program for the Modernization of the Agriculture Sector (PALMA) project supporting the modernisation of the agricultural sector significantly contributed to food security by piloting the value chain approach to agricultural production identified as a gap in a previous European Union-funded project.

As a small donor in Guatemala, Canada targeted aspects of the food security and sustainable growth sector and ‘punched above its weight’ during the first years of the 2009-2014 Program. Global Affairs Canada programming helped to improve food security in seven departments by increasing the agricultural productivity, capacity and income of 18,118 small-scale farmers.

In Nicaragua, the proportion of undernourished people was reduced from 23.2% in 2005-2007 to 16.6% in 2013-2015. This was influenced by the Government’s flagship program to increase production (PRORURAL) and the school meals program, both supported by donors, including Canada. Global Affairs Canada also contributed to increased agriculture production and assisted 6,000 Northern region households to diversify crops. The Country Program was instrumental in increasing access to electricity for over 126,000 people in 444 rural, northern communities.

In the Philippines, Canadian programming encouraged 35,000 households to increase and diversify agriculture production.

Children and Youth

In Jordan and Morocco, Canada contributed on a national scale to improving the quality of education for children and youth through reforms to increase the quality of primary education and by improving the relevance and quality of technical education and vocational training system. In Morocco, a competency-based approach to vocation training was adopted and, along with school restructuring, affected 9,550 school principals/managers, and reached more than 6 million primary and secondary school children. The endorsement of a competency-based approach fostered a new management culture at the school level that delegated more autonomy to school principals/managers, and enhanced their capacity to network. Finally, it enabled the creation of a Professional Practice Community of 500 school principals/managers to strengthen the flow of information and good practices within the educational system.

Cross Cutting Themes

Gender Equality

In the Philippines, the highly effective Gender Responsive Economic Actions for the Transformation of Women project implemented by the Philippine Commission of Women, along with the openness of national agencies, led to government partners taking great strides to integrate gender into national programs. In Jordan, Morocco and Guatemala, Canada contributed to enhancing the ability of partner government institutions and civil society organizations to address gender equality issues that deterred women’s participation in the economy.

During the 2009-2014 programming cycle there was a decline in Canada’s profile on gender issues. In Jordan and Guatemala, the 2009-2014 Country Programs no longer included gender-specific initiatives. In Jordan, Canada had a strategic niche in the areas of gender equality and education until 2011. It was seen as a committed and engaged donor willing to be the first to tackle challenging issues, such as gender equality institutional reform, until 2011. In Sri Lanka and Guatemala, staff received limited support on gender issues at the program level from field or headquarters-based specialists.

Environmental Sustainability

In general, Canada had a low profile on environmental issues in these countries, although all projects respected environmental laws and regulations. In the seven countries where agricultural production was a focus, all reported that ecologically sound practices were introduced. In Nicaragua, the Country Program contributed to sound watershed management practices in the Northern region, which were in turn replicated in all sensitive ecological zones. The Program also contributed to environmental sustainability through the promotion and adoption of sustainable practices in agriculture through targeted initiatives in Cuba, Guatemala, the Philippines and Sri Lanka.

Governance

Overall, Canadian programming contributed to governance in each of these countries. In Sri Lanka, the Country Program made a contribution to national reconciliation and human rights by assisting with the implementation of the Official Languages Policy. In Guatemala, the Country Program helped advance the rule of law by contributing to reducing the rate of impunity. Global Affairs Canada programming in Morocco and the Philippines contributed to governance by supporting decentralization and local governance processes. In the Philippines, decentralization contributed to local economic development and, even though not originally conceived as such, Canadian support convinced the Department of the Interior and Local Government agencies of the efficiency of a local economic development officer position. In Cuba, the value chain approach was a new form of management, which used to have fully centralized planning in the agricultural sector. In Egypt, Global Affairs Canada programming helped build the capacity of governmental partners on a national level. It supported governmental and semi-governmental entities, as well as civil society organisations and non-governmental organisations, playing an active role as micro, small and medium enterprise development enablers (e.g. including the Egyptian Banking Institute, the General Authority for Investment and the General Authority for Government Services) and implementers.

Enabling and Hindering Factors for Effectiveness

As the effectiveness and efficiency of development programs are closely related, some of the contributing factors in this section relate to efficiency and, as such, will also be discussed in the next section.

Enabling factors

Key enabling factors that supported the achievement of results included:

• strategic focusing of resources on select themes and geographic areas (“fewer, bigger”);
• implementing programming through long-term, trusted partners, including highly engaged partners in-country, that successfully leveraged funding from other national or international sources; and,
• strategic policy dialogue and appropriate coordination with other donors.

The buy-in and engagement from high-level officials in targeted ministries was a key factor that contributed to the achievement of results. Successful projects in Morocco were marked by the involvement of institutional partners at all stages of the project cycle, as early as the project design, planning, and selection of Canadian implementing agencies. The buy-in of local authorities and political will at the highest level led to the improvements in school management. The Government of Morocco invested more of its own funding to bring other donors to the table and mobilized new resources from existing donors, for example the European Union.

In Nicaragua, ministers or their deputies led or co-led a number of large projects and had a remarkable knowledge of the results achieved. In the Philippines, there was a high level of engagement from government partners, which helped yield positive results in most projects. In Sri Lanka, the outcomes achieved by the National Languages Project were due, in part, to the support from the Secretary and the Minister of National Languages and Social Integration, both of whom championed the issue and brought strong leadership to the project.

A common feature of success for all eight countries of modest presence in this evaluation was that they built on previous effective initiatives that had been delivered over the long-term with trusted government, NGO or multilateral partners. In some cases, the relationship had been built over decades, providing a solid foundation for new initiatives.

In Egypt, a key value added from Global Affairs Canada programming was its longstanding dedication to establish strong, diverse and strategic partnerships between donors, governmental bodies, and civil society. The Country Program held meetings for Canadian-funded projects between beneficiary MSMEs and donors that allowed for donor mapping exercises, gap identification and tangible collaboration. Convening development stakeholders was an approach also adopted by government representatives as a successful model for bilateral and multilateral coordination. In Nicaragua, focusing upon the Northern region created synergies between integrated watershed management, improved agricultural practices for young farmers and the electrification of more than 500 communities, to increase sustainable economic growth primarily through agriculture.

Channelling bilateral programming through multilateral agencies increased transactional efficiency and reduced the risks associated with implementing bilateral programming without the resources to adequately manage and monitor projects. There were two major multi-bi projects^{Footnote 8} in Guatemala – International Commission against Impunity in Guatemala and Purchase for Progress with the World Food Programme. The less favourable effects of working through multilateral agencies included reduced involvement in policy dialogue, reduced Canadian visibility and fewer direct contact opportunities with stakeholders.

Hindering Factors

External factors that hindered the achievement of results were threefold:

• Local partners’ weak institutional capacity;
• Unforeseen events, such as natural disasters, including drought in Nicaragua and hurricanes in Cuba,^{Footnote 9} the influx of Syrian refugees in Jordan and the revolution in Egypt; and,
• The lack of engagement from national government partners. In Sri Lanka, strained relations between the government and Western donors and restrictions on civil society organizations^{Footnote 10} hindered progress for the populations affected by conflict.

In addition to external factors, several internal factors also hindered the effectiveness (and efficiency) of programming in these eight countries. These related to:

• timelines for project approval processes;
• the capacity and number of human resources;
• challenges in communicating new policy directions with partners; and,
• lack of access to essential corporate IT systems for some Country Programs.

Project approval processes had an impact on the former countries of modest presence, which had little ability to shift programming if a project was not approved. In both Morocco, it took almost four years for PAGESM (Projet d’Appui à la Gestion des Établissements Scolaires au Maroc) and three years for REAPC (Projet d'appui à la réforme de l’Éducation Approche Par Compétence) to be approved. In Cuba, one request for proposal for a food security project took five years to be launched due to delays getting the approved from a contractual and administrative perspective internally, as well as obtaining the necessary approvals from the Cuban government.

In Morocco and Sri Lanka, high staff turnover rate, particularly at the Director and Program Manager levels, made both communication and the chain of command between the field and headquarters challenging.

The closure of Program Support Units negatively affected many Programs as this reduced Global Affairs Canada’s visible field presence, as well as the human resources available to monitor programming and respond to corporate requirements and requests.

Based on the “hub and spoke” model, staff in Honduras was supposed to provide support to other countries in the region, including Nicaragua and Guatemala. However, that model was not effective for the Nicaragua and Guatemala Programs, who received limited support from the hub.

Internal communication between HQ and the field related to strategic and financial decisions was considered weak by many of these eight Country Programs. In Jordan, country strategies and investment plans were not officially approved, requiring field staff to work with draft versions. In the case of Guatemala, the 2009 PMF was never approved or finalized. The Sri Lanka Program had seven Directors between 2009-10 and 2013-14, and the longest stay of Program Managers at headquarters was one and half years. This posed challenges for decision-making at senior levels.

Further, at the end of the evaluation period, former countries of modest presence received information about the new strategic direction of development programming. However, the clarity and transparency did not enable staff to work openly with partners.

Finally, in most programs, staff had difficulty accessing IT systems and finding documents following both decentralization and amalgamation. Decentralized programs in Central America did not have access to electronic IT systems with corporate information or project data. In Jordan, staff spent time managing a parallel filing system while waiting for headquarters to relay documents.

4.3 Sustainability

Many results achieved in Cuba by Global Affairs Canada-funded project were likely to be sustainable as they were continued with funding from other donors. The capacity developed remained as Cuban partners continued to deliver training courses on their own during a two-year gap between Global Affairs Canada projects. Equipment appropriate to the local context was procured and was still being used 10 to 15 years after it was delivered.

Global Affairs Canada programming in Guatemala achieved some sustainability with the projects having elements that supported it such as good practices in project design and strong ownership by numerous local stakeholders.

In Nicaragua, sustainability is highly likely, barring external factors such as drought or other natural disasters, as elements of projects have been incorporated into government systems. The strong engagement of government partners contributed to sustainability, as did the multi-pronged approach taken to build capacity of communities, civil society organizations, municipalities and ministries for the implementation of initiatives. However, in one key government program, PRORURAL, the approach could have been accompanied by more technical assistance to yield more sustainable outcomes in increased production.

Generally, it was premature to determine the extent that programming would be sustained in Jordan, as several projects in the sample remained ongoing. Some aspects of projects that were designed to use national systems were sustained beyond Global Affairs Canada involvement. However, many of the activities funded through the Gender Social Fund ended once Canadian support was no longer available. In the longer-term, the sustainability of results will depend on the national context, where challenges include the refugees from the conflict in Syria and capacity limitations.

Global Affairs Canada programming in Morocco achieved significant sustainability, with all projects reviewed either having elements already sustainable, or likely to be by the end of the project. Key success factors were projects designed with sustainability in mind, ownership by partners, and innovative programming approaches. In Egypt, several components of projects were sustained after they ended. Introducing the concept of sustainability in early project design phases, a high degree of ownership of partners and innovative programming approaches were enabling factors for sustainability.

In the Philippines, most Program outcomes are likely to be sustained as they were institutionalized within national agencies. For example, disaster risk reduction programming for business recovery centres should help mitigate losses from future natural disasters.

In Sri Lanka, the National Languages Project objectives were institutionalized and are sustainable provided the Government of Sri Lanka maintains a continued commitment to fund activities that advance national language policies. Sustainable economic growth programming outcomes were likely to be sustainable at the individual level but less certain at the organizational level, thereby limiting future benefits for target populations.

Enabling and Hindering Factors for Sustainability

Enabling Factors

Strong local ownership, capacity development and access to long-term resources contributed to the sustainability of results across Country Programs. Another key factor for sustainability was building upon past successful initiatives. In Cuba, Jordan, Nicaragua, Morocco and the Philippines, many Global Affairs Canada projects were designed to be implemented by or with national government partners, focusing on ensuring institutionalization from the start. In Egypt, selecting local implementing partners and Canadian agencies with strong capacities and a trusted level of expertise meant that many partners continued to fund some of the activities or engaged with other donors to continue what was started under Global Affairs Canada funding. The involvement of multiple stakeholders, including municipal government, local cooperatives, community organizations and the private sector was noted in key projects in Guatemala and Nicaragua.

Hindering Factors

Overall, the lack of institutional capacity to maintain results, high turnover in government and local organizations, and the lack of financial resources were the key factors that hindered sustainability across Country Programs. In some cases, project design did not sufficiently factor in the need to build capacity and was limited by an inability to track results at the intermediate outcome level (e.g. PRORURAL in Nicaragua and Rebuilding Agriculture Livelihoods Project in Sri Lanka). In some cases, external factors, threatened the achievement and sustainability of results, putting overall fiscal pressure on the Government of Jordan to shift resources away from longer-term development.

4.4 Economy/Efficiency

4.4.1 Extent that Operational Efficiency was Maximized

All the Country Programs reported gaining transactional efficiencies through the funding of fewer, larger projects as it required less staff time to manage them. The use of multilateral mechanisms also contributed to operational efficiency in two ways: i) Global Affairs Canada staff trusted their partners management and monitoring systems; and, ii) multi-bi grants were disbursed quickly, particularly when compared to the contract modality used to work through government partners. There were, however, disadvantages of using multilateral organisations, including reduced Canadian involvement in policy dialogue with government counterparts, visibility and direct contact with stakeholders. Of all eight countries, the Philippines allocated the greatest proportion of their funding through multilateral organizations at 60%.

Some Country Programs focused efforts in specific geographic areas, which improved efficiency. At the government’s request, both Guatemala and Nicaragua targeted particular regions in each country, which resulted in more visible outcomes for the Program. In Sri Lanka, much of the program was delivered in the Northern region long affected by conflicts.

Another factor that led to efficiencies was the use of local consultants as advisors in areas where there was little expertise among the team and where support from Global Affairs Canada specialists or advisors was difficult to access. The Philippines Program reported using private sector specialists to support the Program when internal resources were not available.

A good example of efficiencies was found in the Philippines Program, where all Country Program staff was decentralized. The Program was managed by two Canadians, two experienced locally engaged staff with full responsibility to develop and manage projects, three part-time local specialists in gender equality, economy and environment and one administrative assistant. Equally important was the high priority and considerable time given to planning new initiatives and supporting approval processes.

Factors that Hindered Efficiency

Changes in human resources: The evaluation period saw a reduction and turnover in human resources at headquarters and in the field. Fewer headquarters-based staff, including specialists, required that staff in the field navigate the fluid organizational environment that they were geographically distant from. This was compounded by frequent changes at headquarters of the part-time gender equality advisor assigned to Nicaragua and Guatemala (three different advisors in the space of 12 months during 2013 and 2014). The Sri Lanka Program had seven Directors from 2009-10 to 2013-14 and a series of Program Managers.

While a few employees for Program Support Units were hired to support Programs during the period, many existing employees departed. This, combined with fewer Canadian staff at headquarters and in the field, diminished the ability of the Program to engage in directive programming, monitor projects, ensure congruence with the local context, and maintain institutional memory to respond to departmental requirements and requests.

Delayed Decision-Making and Approval Processes: The process to obtain project approval was slow for all eight countries of modest presence during the evaluation period. In 2009, CIDA was testing new project design tools, which led to the development of the Authorised Programming Processes. There was considerable uncertainty about the tools to use, who to consult and the approval process. This was compounded in 2013 when the amalgamation of CIDA and the Department of Foreign Affairs and International Trade occurred as there were major organizational system changes.

The only country to have avoided delays in project approvals was the Philippines, which had major initiatives approved in 2012. The approval process was a key constraint for the Cuba, Guatemala, Morocco and Sri Lanka Country Programs. For most years during the evaluation period, the Cuba Program disbursed its allocated annual budget. However, in 2012-13 it under-spent approximately 20% of its budget due to the timing of project approval by the Canadian and Cuban governments. Similarly, the Sri Lanka Program lapsed approximately 50% of their budget from 2012 - 2014. Insufficient human resources to rapidly develop new programming, unclear communication on project selection criteria, and lack of support for contracting led to the emptying of the project pipeline in Sri Lanka for all of 2012. The Program did not advance any other projects for fourteen months pending the approval of one major initiative in the agricultural sector.

Whereas Country Programs maximized their operational efficiency by advancing fewer and bigger projects, the risks were high if the project underperformed or there were blocks in the project pipeline.

In Nicaragua, Canadian staff struggled to remain responsive and spent 40% of their time supporting the Guatemala Country Program after decentralisation. The focus on project management and oversight of the Guatemala Program resulted in reduced time for policy dialogue and project monitoring.

4.4.2 Extent that performance measurement frameworks (and/or logic models, theories of change) were used to ensure adequate monitoring and reporting on results achieved

Overall, the evaluation found that each of the eight Country Programs was guided by country strategies, logic models and performance measurement frameworks (PMF) at the project and program level. In several cases, the country strategy, logic model and PMF were developed rapidly at the beginning of the programming cycle with limited support from the corporate level. Consequently, these performance measurement tools did not align well with one another and had to be reviewed during the course of project implementation.

Global Affairs Canada’s corporate reporting format also changed over the evaluation period. For example, management summary reports were introduced as a management tool. The Country Programs lacked support to assist with implementing these changes.

In Cuba, Global Affairs Canada corporate changes, human resource capacity, and a limited results-based management culture in Cuba restricted the Program’s ability to manage for results. The fact that the results in the country strategy and logic model did not align was a challenge for corporate reporting. After the Program Support Unit closed in 2012, the Program’s capacity to plan and monitor projects diminished. Most projects reviewed in the evaluation sample reported mainly on outputs, and immediate outcomes in a few cases.

The Guatemala Program’s logic model and PMF were not finalized and baseline data was not collected. The program monitored projects results through reports and communication from implementation agencies and reported on program performance through annual country reports and management summary reports. After the Program Support Unit closed in 2010, the program had limited access to independent monitors with technical expertise. There was minimal support from sectoral specialists in headquarters and the Program Support Unit in Honduras. The limited resources with which to monitor was one of the reasons that the Guatemala Program favoured programming using grants, when possible.

Although not unique to the Nicaragua Program, their PMF was developed at the beginning of the 2009-2014 programming cycle but indicators, baselines and targets were not included and were not subsequently collected. Individual projects were asked to track and measure the results achieved through their interventions, which were then used by staff to report against the Program level logic model. This was possible for outputs and immediate level outcomes as the Program PMF integrated the outcomes of key interventions. However, intermediate outcomes were not measured.

Morocco developed a performance measurement framework at the Program level in 2010. However, there was no evidence that the same PMF was used to monitor program performance. There was also no consolidated logic model used to track progress on results at the program level rendering it difficult to assess performance other than by project. All projects had a monitoring and evaluation system integrated into the project design and Canadian implementing agencies reported regularly to Global Affairs Canada on results achieved. Since the program was largely executed through projects, most program indicators were included in these frameworks. However, in several cases, the indicators and targets in the project-level PMFs did not align with those at the Program-level, again making it difficult to obtain a consolidated performance picture.

In the Egypt Country Program, logic models, PMFs and other results-based management principles were integrated into project design and used to report on results achieved at the mid-term and end-of project period. Although project level reporting was undertaken regularly, there was no clear comprehensive monitoring system at the program level. The data on the intermediate program outcomes were traceable only through project level evidence, scattered data sources and validations of partners and counterparts, and were not quantified against intermediate outputs. Extracting key data on program progress was done manually by a local employee who had a corporate memory of the program.

The Jordan Program did not use results-based management tools to monitor and manage performance. There was no clear overall assessment of which targets were met, or of the cumulative program-level results achieved toward the Country Strategy. The PMF was finalized in 2010 and contained baseline data and targets for some, but not all, indicators. Annual reports provided a qualitative assessment of the extent to which targets for each outcome were met. The ability of Global Affairs Canada to effectively monitor projects was affected by the low capacity of local partner organizations. There was a consensus among interviewees that performance measurement should have received closer consideration at the project and program levels.

In the Philippines, performance measurement frameworks were adequate to measure progress towards outcomes at the project and Program levels and to make timely changes to projects. Indicators and targets were used to measure outcomes at all levels. The Program team indicated that it spent considerable time at the start of the programming cycle to consolidate the Program logic model and the performance measurement framework into a coherent whole. The initiatives that supported the immediate and intermediate outcomes were well integrated and congruent. However, a key challenge was gathering data from a large number of partners with different mandates and spread out over many locations.

The Sri Lanka Program had a well-integrated performance measurement framework with clear linkages between the projects and the Program. Although immediate and intermediate outcomes were measured at the project level, they were not assessed at the Program level. The sample projects reviewed each had a logic model and a PMF and they were linked to the Country Program logic model and PMF. At the Country Program level, baselines were present at all levels but targets were not measured beyond the output level, which originated from project partners PMFs. Consequently, the overall reported level of results achievement at the immediate and intermediate levels in the cumulative 2013-14 Annual Country Report was inconclusive. Several factors contributed to the absence of program level monitoring of PMF indicators, including limited human resources and a lack of Global Affairs Canada demand for such information.

4.4.3 Extent that Global Affairs Canada’s monitoring and reporting approach linked to the countries’ own reporting systems

In general, the monitoring frameworks for projects and programs in Guatemala and Cuba were not linked to national reporting systems. In Guatemala’s case, this was a conscious decision related to concerns around management systems and transparency. In Nicaragua, links were made to a very limited extent (e.g. with the Ministry of Agriculture for PRORUAL).

Some projects implemented with government partners linked their performance indicators to Morocco’s monitoring system. Some projects included specific indicators in their PMF that were tracked by the Ministry of Education or Interior, but no linkages were made at the Program level.

The Jordan Country Program did not use results based management tools to regularly monitor and manage performance. For some projects delivered in partnership with the Government of Jordan (e.g. School and Directorate Improvement project, School Feeding), project monitoring and reporting systems were developed to be managed by the Government of Jordan as part of its own national reporting systems.

In Egypt, performance measurement tools and reporting were not linked to national data. Most projects had weak or limited monitoring and evaluation systems in place.

The Philippines Country Program and partners agreed on a set of measurement indicators that partners were expected to integrate as part of project logic models and annual work plans and then link to resources. These indicators were used as part of monitoring and reporting systems, progress reports and evaluations. Indicators and targets were used to measure outcomes at all levels.

As the Sri Lanka Country Program had a single initiative implemented with the Government of Sri Lanka, there was limited use of government systems to monitor and report outcomes. The Official Languages Policy project implemented with the Ministry of National Languages and Social Integration used data generated from government administrative systems but reporting was based on the project PMF.

5.0 Conclusions

Relevance

The issues of food security and sustainable economic growth, addressed primarily through sustainable agriculture production and modernizing government policies to create an enabling environment, were relevant to the needs of partner countries. This programming responded to countries’ national priorities and was adapted to the countries’ strategies where already established. In several countries, generating employment for youth was a key priority, with the strategies of the Jordan and Morocco governments focusing on improving vocational training, to which Canadian programming was responsive. Addressing governance issues was relevant in all countries but programming on human rights issues was particularly relevant in Guatemala and Sri Lanka. While focused on one or two of Global Affairs Canada’s thematic priorities, programming was sufficiently flexible to address governance issues with specific and targeted programming.

A strong signal of relevance was the scaling-up of many Canadian-supported initiatives across the eight countries. Each Country Program introduced innovations that were adopted and replicated at a regional or national level. Notable examples include the development of value chains in Cuba, business development services in Egypt, a comprehensive watershed management system and a field-based sustainable agriculture training program for rural youth in Nicaragua, the adoption of the competency-based approach in Morocco and multi-lingual public service provision in Sri Lanka. Successful programming and innovations were generally scaled up with the support of other donors. A major limitation to replicating or scaling-up projects was the lack of institutional capacity and resources among governments (national/regional/local) and local organizations.

Effectiveness

Notwithstanding the shortage of program indicator measurement at the intermediate level in several programs, there was robust evidence of the achievement of immediate outcomes at the project level, exceeding targets in some cases.

Cross Cutting Issues

Canada had a strategic niche in gender equality in Jordan and the Philippines and many significant gender-related results were achieved in other countries. Canada’s strength on gender equality issues was generally acknowledged in other countries; however, with the exception of the Philippines, it waned over the evaluation period. This was partly due to limited number of gender analysts and the absence of specific gender equality initiatives.

With the exception of Morocco and Jordan, where agriculture was not a thematic area of focus, programming contributed to sustainable environmental practices for small producers through targeted agriculture initiatives. Most projects made significant contributions towards governance in the sectors of focus, particularly at the local level. Some projects focused on governance results, such as helping the Government of Sri Lanka implement key components of its Official Languages Policy across the country, or reducing impunity in Guatemala’s courts. Overall, programming in these former countries of modest presence helped to strengthen the implementation of national policies and the organization of service delivery in agriculture (most countries), local governance (Philippines, Morocco), vocational training (Jordan, Morocco, Sri Lanka), as well as micro, small and medium business development (Egypt, Philippines).

Key enabling factors included strategically focusing resources on certain themes and in certain geographic areas; closely and strategically aligning programming with partner government priorities; implementing programming through long-term trusted partners and building on previous successful initiatives; using a mix of funding approaches; good coordination with other donors; and, strategic policy dialogue. Key hindrances to effectiveness included the lack of institutional capacity of host government partners, the reduced support from Global Affairs Canada specialists at the program level, particularly with regards to gender equality and environment, limited time to engage in regular project monitoring and broad policy dialogue in many countries.

Sustainability

Overall, the sustainability of results achieved varied across the eight countries in this evaluation. Many of the outcomes achieved were institutionalised by country partners and, therefore, likely to be sustainable. To that end, ensuring alignment with national priorities and working to strengthen the capacity of government entities, in particular, were key enabling factors for sustainability. Factors hindering sustainability included the lack of government capacity (financial or human) to maintain results over time; however, in some countries, this was mitigated by securing financial support from other donors.

Efficiency

Operational efficiency of programming in countries of modest presence was enhanced with the focus on fewer and bigger projects. While there were advantages in transaction efficiencies for many former countries of modest presence, some were negatively affected by having fewer projects in the pipeline as the non-approval of one large project led to an empty pipeline.

The efficiency of programming in these eight countries was also affected by process and system changes at the corporate level and, to a lesser extent, by external factors, such as partner country decision making processes or natural disasters.

Some Program decision-making was informed by project evaluations and monitoring, but almost all Country Programs would have benefited from closer and more frequent project monitoring, as well as more robust performance measurement at the intermediate outcome level. In the eight countries, efficiency was enhanced by managing fewer and relatively larger projects, using pooled funding and programming through multilateral organizations so as to reduce transaction costs. However, high-impact policy changes were also achieved through strategic projects with small budgets in some countries. Many projects were able to leverage significant complementary and/or counterpart funding from national or international partners.

Key internal processes that hindered efficiency in these former countries of modest presence included project approval processes, inadequate support from the “hub”, limited support from corporate contracting mechanisms and inadequate access to corporate information systems.

Reductions in overall staffing in countries of modest presence, combined with the closure of PSUs, led to reduced efficiencies. While a few Project Support Units employees were hired to support Country Programs following their closure, the departure of most, combined with the reduction of Canadian staff at headquarters and in the field, diminished the ability of the remaining field-based staff to undertake directive programming, monitor projects, ensure congruence with the local context, or maintain institutional memory to bridge transition periods and respond to Corporate requests.

Further, with the exception of the Philippines, field-based staff had difficulty responding adequately to programming and corporate needs. In several cases, there were limited local sources of expertise, particularly on gender equality and environmental sustainability. Program staff found ways to mitigate some of these problems, such as hiring external consultants or ensuring a presence in a key donor forum; nonetheless, program design and oversight suffered from the lack of staff and resources.

6.0 Lessons Learned

This section sets out the lessons that emerged across the eight Country Programs, which in turn inform the recommendations.

• Projects that are strategic but with low dollar values^{Footnote 11} can have a major impact. Some of the most relevant and successful initiatives supported by Canada in former countries of modest presence were those that were a high priority for the partner government and had concrete buy-in from both the partner government and other donors to scale-up successful initiatives.

• Canadian development programming in former countries of modest presence achieved strategic results with relatively small budgets. To maximize the impact of those limited resources, it was more effective to build on previously effective programming and to seek synergies between projects and with other donor initiatives, where possible.

• In countries where there was strong government engagement or a willingness to build the capacity of government institutions, results achieved were more likely to be sustained.

• Reductions in staff and support for programming led to a further emphasis on responsive programming and increased use of the multilaterals to implement projects in many countries. This reduced transaction costs and the need to closely monitor projects. However, itreduced Canadian involvement in policy dialogue with partner governments, visibility and direct contact with partners and stakeholders.

• Even though budgets were small for these former countries of modest presence (average $5 million/year), they were still required to develop and implement a range of projects, engage in policy dialogue and monitor and evaluate on-going initiatives. As such, they required a minimum number of staff with an adequate mix of skills (professional, specialist and administrative) in order to manage development programming effectively and efficiently.

• As with all countries where Global Affairs Canada undertakes development programming, it is crucial to have a robust performance measurement system that is able to track and report on results at both the project and program level. Knowledge management systems are necessary to ensure continuity and to off-set frequent staff turnover. Supporting national capacities for monitoring and evaluation would strengthen the evidence base and guide decision-making for future development initiatives.

7.0 Recommendations

In light of the findings, conclusions and lessons, and in the context of an amalgamated Department, the evaluation recommends that Global Affairs Canada consider:

1. Identifying criteria that define development partner countries, including their resource requirements.

2. Ensuring that adequate support is provided to staff in development partner countries to plan, implement, monitor and report on development results, including strengthening their capacity to integrate governance and gender equality into programming.

3. Ensuring that development partner countries have the flexibility to manage a diversified portfolio using different programming modalities thereby allowing them to mitigate risk and respond to emerging priorities.

4. Strengthening its suite of program-level results-based management tools and increase the level of support for results-based program management, including evidence-based planning, monitoring and reporting by the programs.

5. Establishing or strengthening systems of support to field staff in development partner countries to enable strategic choices with regard to policy dialogue and donor coordination