Lessons from Development Evaluations - 2013

What gets measured gets done.
If you don't measure results, you can't tell success from failure.
If you can't see success, you can't reward it.
If you can't reward success, you're probably rewarding failure.
If you can't see success, you can't learn from it. If you can't recognize failure, you can't correct it.
If you can demonstrate results, you can win public support.^{Footnote 1}

Introduction

Evaluation serves various purposes - accountability and learning, most notably. It is not only a requirement that we take the time to evaluate our projects and programs, it is good practice to do so as it allows us to gather lessons from the successes and challenges faced in implementing development activities.

About this report

This report draws out key lessons from project and program evaluations of various kinds (formative, mid-term, summative) related to the Department of Foreign Affairs, Trade and Development (DFATD)'s development efforts completed between April 1, 2012 and March 31, 2013. The report provides a summary of each of the projects or programs evaluated, including key findings from the evaluation.

Learning what works

Lessons drawn from evaluations can improve one's understanding of what constitutes good development - in particular, what works for whom, in what conditions and why. Key users include DFATD staff and partners who design and implement international development programs and projects. Incorporating these lessons, many of which are components of good project management, into future operations will also give Canadian taxpayers better value for their tax dollars.

What the report reviewed

A sample of 20 evaluations from among the 58 completed in FY 2012/13 was used for this report. In this sample, 17 covered bilateral programming, 2 multilateral programming and 1 DFATD's Partnerships for Development Innovation program. The most common themes addressed were governance (9 evaluations) and economic growth (6 evaluations); some evaluations made reference to both (5 evaluations).

This year's main lessons

The main lessons from this year's evaluations are the importance of:

1. nurturing effective partnerships;
2. supporting capacity development; and
3. improving performance management.

1. Nurture effective partnerships: listen, coordinate, leverage and provide technical assistance to enhance effectiveness

Development initiatives face a multitude of challenges to success - this year's lessons from evaluations indicate that those projects that build mechanisms to listen, coordinate, leverage and engage strategically with multiple stakeholders are more effective.

Mechanisms to listen and coordinate

Projects in Afghanistan and in Burmese refugee camps created mechanisms to solicit individual concerns, even in difficult situations. These same mechanisms were also found to provide opportunities to consult with a range of groups, build capacity and develop broader social cohesion and trust.

The formative evaluation of Camp Management in the Burmese Refugee Camps in Thailand considered the camp management policies and practices to be a good example of comprehensive community-based refugee management; effective consultation and coordination mechanisms were provided to nine refugee camps and approximately 140,000 refugees in 2011. Camp management was overseen by three major stakeholder groups:

1. refugees and camp service providers;
2. the Thai Government; and
3. donors, each requiring separate coordination. Camp-level coordination facilitated good information sharing, but has not always proven effective in program planning and priority setting. The evaluation highlighted the importance of a shared vision and mutual trust for effective camp management, and recommended instituting mechanisms, such as direct minority representation, minority advisory bodies, or a camp public forum, to ensure that the voices of ethnic and religious minorities, as well as youth and women, were heard and considered.

Clarifying roles and responsibilities for effective coordination

The Food Security, Sustainable Agricultural Development project (FASDEP) evaluation in Ghana explored the effectiveness of DFATD's budgetary assistance to the Government of Ghana, including technical assistance. The evaluation noted that, as a result of this technical assistance, coordination between civil society organizations improved; organizations mapped all food security initiatives in the north and participated in bi-monthly meetings to improve the coordination of these activities. At the same time, the evaluation recommended clarifying the roles of technical experts in order to maximise the use of their skills.

The Support to the Nicaragua Productive Rural Development Sector Program (PRORURAL) evaluation highlighted how overcoming challenges of inter-departmental and territorial coordination required senior management attention. These coordination challenges could have been avoided by institutionalizing coordination mechanisms, and clarifying roles and responsibilities of the organizations in an iterative manner.

Similarly, the evaluation of the Regional Human Rights and Peace Building Fund Project in the Middle East noted that success required coordination of action at different levels. For instance, the local provision of essential services to victims of abuse should be combined with national advocacy using specific cases as references for broader awareness.

Maximise strategic partnerships

The Capacity Building Technical Assistance Facility Project in South Africa consisted of two components:

1. funded learning exchanges (primarily between South African civil society organizations and Canadian counterparts) to promote pro-poor service delivery and enhance transparency and accountability; and,
2. targeted capacity building support to government in order to achieve timely, adequate, gender-sensitive, pro-poor service delivery. This second component provided support to the Technical Assistance Unit within the National Treasury to facilitate broader capacity development of public institutions. This strategic institutional support resulted in strengthened national government institutions and greater local capacity to address public administration issues over the longer-term. The project evaluation hailed this as a positive example of complementary strategic and grass roots interventions.

Leverage complementary initiatives

Several evaluations noted the importance of linking diverse stakeholders to ensure success. The Partnerships with Canadians Governance Program evaluation noted that successful projects involved mature partnerships with mutually agreed outcomes, a focus on a specific area or level of government, and diverse stakeholder involvement. Non-governmental organizations were able to act as catalysts in this process by linking different levels of government while understanding local realities. This evaluation recommended that, where appropriate, small projects or 'downstream' initiatives in communities or with local governments are linked 'upstream' to national governments in order to amplify results.

The ELLONA project in Afghanistan embarked on a cost-sharing exercise with cooperatives to establish a potato cold storage facility, the key investment required to deliver the agricultural product to a wider market.

In Pakistan, the Pathways and Purse-strings project evaluation noted the critical importance of producer groups as a way of linking small amounts of produce to their larger markets, as well as to stimulate producers to improve product quality, improve alignment with market forces, and provide products on a timelier basis in order to fetch higher prices. In addition, the evaluation found that buyers were able to better relate to the producer groups, resulting in a win-win situation.

The PRORURAL project evaluation in Nicaragua flagged the importance of a partnership between multiple stakeholders to achieve growth across the agricultural sector and reduce poverty. These partnerships included the private sector, non-governmental organizations, local government, financial and commercial agents, and agribusinesses.

Consult the right experts

It is often necessary to recognise limitations and solicit external support. For instance, the PELNICA Nicaragua Rural Electrification project was designed to include technical assistance for the Government of Nicaragua - specifically, to support rural electrification and small business development. Despite various successes, the project was not able to establish a microfinance organization; the evaluation recommended that an external expert be consulted to address this particular challenge.

The Caribbean regional program evaluation recommended increasing the use of professional resources in the field to better identify strategic opportunities and integrate environmental sustainability as a cross-cutting theme.

2. Support capacity development: dynamic, participatory approaches have proven success

Several lessons from the evaluation reports focus on capacity building efforts. They highlight different elements of capacity, including the manner in which social structures can help to develop capacity and allow communities to work and learn together.

The ELLONA project evaluation noted that capacity building is a process that, unlike infrastructure building projects, lacks fixed "start-finish" boundaries. "Capacity" means that "there are processes in place through which individuals (men, women, girls and boys), organizations (institutions, governments, etc.) and societies can obtain, strengthen and maintain resources and competencies to set and achieve their own development objectives."^{Footnote 2} As such, capacity building is often a dynamic, iterative and multidimensional process.

The results of successful capacity building are described in the evaluation of the Pathways and Purse-strings project in Pakistan: improved market access and production techniques; higher prices for products; and, most importantly, significant increases in income. In this project, the implementing agency supported capacity development at all levels of the value chain - among suppliers, producers and buyers.

The project used a broad range of capacity development approaches, all in an interactive and participatory manner, which allowed project implementers and beneficiaries to overcome 'stumbling blocks'.

The evaluation of the Local Governance Support for Local Economic Development project in the Philippines noted mixed results on capacity development efforts; educational workshops and forums that permitted participants to share and learn from each other's experiences were rated well, while other formal training, in spite of high-quality trainers and training material, was considered too lengthy, inflexible and too classroom-focused.

3. Improve performance management to enhance the achievement of results

The lessons from the evaluation reports sampled speak to the importance of integrating performance management, including measurable indicators, into project implementation. This is a particular challenge in governance and human rights interventions, where, unlike economic growth for instance, it is more difficult to find and agree on measurable indicators.

Integrate performance management into project implementation

Results-based management (RBM), used by DFATD Development for decades, is a life-cycle approach to management focused on achieving outcomes, implementing performance measurement, learning, adapting, and reporting on performance. DFATD's tools include logic models, performance measurement frameworks and risk registers.

The evaluation of the Local Governance Support for Local Economic Development project in the Philippines noted that while RBM tools are appropriate, their complexity and detail renders them difficult for some partners and clients to use. The evaluation report suggested appointing an external monitor for the project to provide timely advice as the project or program takes shape. Ideally, this would occur during the preparation or submission of the program implementation plan.

The PRORURAL evaluation recommended that monitoring feed into day-to-day activities, with a participatory system that shares implementation responsibility and allows feedback from partners to influence performance.

Anticipate and manage risks

Although a possible change in the political landscape of Mali was identified as a program risk, Canada did not predict the extent of political change that took place in 2012.

The evaluation suggested that regular updates to the program risk analysis would be required. It's also noted that low cost, post-project support can help sustain the results of an investment, especially in high-risk environments.

The Colombia country program evaluation considered risk in the context of community members' vulnerability and the importance of the "do no harm" principle (for effective engagement in fragile states) to ensure that development efforts did not place beneficiaries at risk.

Defining targets and measures clearly

The mid-term review of the Canada Investment Fund for Africa concluded that the Fund achieved its objectives and was both a catalytic and innovative public-private equity fund. However, the review was not able to measure the Fund's performance in the absence of appropriate benchmarks; only anecdotal information was available on the achievement of development outcomes.

The review noted that the Fund was designed to operate independently, an approach that limited the Government of Canada's exposure to risks and was effective in leveraging third-party investors. For future agreements, the review suggested a clearer articulation of expected results, definition of related performance indicators, specification of data collection mechanisms, and inclusion of contractual provisions for ongoing financial oversight.

Governance and capacity development projects face challenges in effectively identifying and tracking relevant indicators. For instance, the evaluation of the Regional Mechanisms to Protect the Human Rights of Women and Girls in Southeast Asia described the challenges faced in effectively tracking changes in attitudes and behaviour regarding human rights.

DFATD's governance toolkit provides relevant resources, including outcomes and indicators. International partners also provide resources, such as the United Nations Development Programme's discussion paper on measuring democracy and democratic governance interventions, and the Inter-American Development Bank's interactive governance indicators database.

The World Bank Institute's Capacity Development and Results framework provides a strategic and results-oriented approach to learning for capacity development.

Annex: Evaluation Highlights

Corporate Evaluations^{Footnote 3}

Geographic Branches

• Country Program Evaluation - Mali
• Country Program Evaluation - Colombia
• Caribbean Regional Program Evaluation
• Evaluation of the Canadian Investment Fund for Africa (CIFA)

Global Issues and Development Branch

• African Development Bank Multilateral Review
• Asian Development Bank Multilateral Review

Partnership for Development Innovation Branch

• Governance Programming Evaluation

Evaluations led by Program Branches

Geographic Branches

• Pathways and Pursestrings: Enhancing Market Access for Women Producers in Pakistan
• Support to Ghana's Food and Agricultural Sector Development Policy (S-FASDEP)
• Productive Rural Development Sector Program (PRORURAL Incluyente)
• Nicaragua Electrification Project (PELNICA)
• Local Governance Support Program for Local Economic Development in the Philippines
• Evaluation of Support to Establish the ASEAN Commission on Women and Children
• Burma Border Assistance Program: Building Social Capital
• Capacity Building Technical Assistance Facility Project (Component Two: Technical Assistance Unit)
• Ukraine Horticulture Development Project
• Renewal Program (ACRP)-II Support to the Foundation for Civil Societ

Country Program Evaluation: Mali - 2006-2011

The Mali Country Program disbursed $486.5 million between 2006 and 2011 to help alleviate poverty and promote development in Mali, one of the world's poorest countries. The program placed particular emphasis on children and youth, economic development, and public sector reform.

Why conduct this evaluation?

The purpose of this evaluation was to assess results and management performance, and to document lessons learned.

What did the evaluation find?

Effectiveness: Long-term investments in education, health, and food security achieved good results in addressing the root causes of poverty. In education, the program increased the number of textbooks available in schools, and through local procurement, developed a Malian textbook industry. It also facilitated the decentralization process of Mali's education sector.

Relevance: The program supported the Government of Mali in meeting its commitments to promote redistributive growth and poverty reduction by boosting productive sectors and consolidating public sector reforms. The program also reflected Government of Canada priorities and maintained a focus on key long-term themes.

Sustainability: Long-term engagement allowed time for development gains to mature. At the same time, more robust risk analysis could have strengthened sustainability planning.

Gender equality and environmental sustainability: The program achieved progress in gender equality mainly through policy dialogue, but paid less attention to matters of environmental sustainability.

Coherence: There was excellent coordination of aid-delivery mechanisms with other donors and the Government of Mali. One challenge was to ensure effective collaboration among Canadian and local partners in the field.

Efficiency: There was an improved balance between management costs and investments from the last evaluation period (2000-2005). The program has one of the lowest ratios of field staff to total disbursements.

Performance management: The application of results-based management and risk-analysis tools improved over the review period.

What is next for the Mali program?

There is an opportunity to improve the program's performance by:

• maintaining up-to-date risk-assessment and mitigation strategies;
• undertaking more systematic monitoring and evaluation as informed by risk assessment; and,
• increasing efforts to integrate gender equality as a crosscutting theme, and placing increased emphasis on environmental sustainability.

Country Program Evaluation: Colombia - 2006-2011

The Colombia Country Program disbursed nearly $65 million from 2006 to 2011. There was a strong focus on children's rights, civilian protection, peacebuilding, sustainable economic growth and private sector development, and food security. By 2011 the program had focused its programming in the conflict-affected department (province) of Nariño.

Why conduct this evaluation?

This evaluation was intended to assess results and management performance, and to document lessons learned.

What did the evaluation find?

Effectiveness: Among other results of the program, the Government of Colombia created policies to entrench child human rights protection and fostered youth-led peacebuilding. The department of Tolima passed legislation to eliminate the worst forms of child labour.

Relevance: The program's highly relevant initiatives for the socially vulnerable-in particular children's rights, access to early childhood education, and prevention of youth involvement in violent and illicit activities-reflected real needs in this conflict-affected country.

Sustainability: The program designed and implemented projects that Colombians can sustain in the long term. Although many donors have announced reductions in funding, Canada's designation of Colombia as a country of focus has led to predictable funding.

Crosscutting themes: Mixed results in addressing gender equality and environmental sustainability indicate a need for improvements on the part of the program and partners.

Coherence: There was strong alignment with Canadian foreign policy priorities. The program also played a leadership role in donor coordination, particularly in human rights programming.

Efficiency: The focus on a key strategic issue-child rights-maximized impact. Using local organizations to implement projects proved cost effective.

Management principles: Ownership by the Government of Colombia and donor harmonization have significantly improved.

Performance management: There was consistency and leadership in the use of results-based management and risk analyses. Better baselines, gender-disaggregated data, and clear targets for selected indicators would further improve reporting.

What is next for the Colombia Country Program?

• Enhance policy dialogue on gender equality and environmental sustainability with the Government of Colombia and other donors.
• Ensure crosscutting themes are appropriately integrated into the design, implementation, and monitoring of the program's interventions.
• Strengthen attention to performance measurement, including the identification of baseline data and the establishment of targets for certain indicators.
• Seek further guidance for conflict analysis and programming in situations of fragility.

Regional Program Evaluation: Caribbean - 2006-2011

The Caribbean Program covers 11 island states (Antigua and Barbuda, the Bahamas, Barbados, Dominica, Grenada, Jamaica, Montserrat, Saint Kitts and Nevis, Saint Lucia, Saint Vincent and the Grenadines, and Trinidad and Tobago) and 3 continental ones (Belize, Guyana, and Suriname). Canada is a major contributor of official development assistance to the region.

Why conduct this evaluation?

The evaluation was conducted to assess the Caribbean Program's performance and results in 2006-2011, during which disbursements totalled $194.7 million.

What did the evaluation find?

Effectiveness: The program achieved valuable results in terms of capacity development. The use of country-specific programming in the earlier part of the evaluation period, in addition to regional projects, contributed to strengthened country capacity for participation in regional activities.

Relevance: There was a good correlation, among many of the projects, with regional and country needs. There were common strands and a progressively narrower focus on trade, economic activities, and vocational training over the five-year period.

Sustainability: Where the program worked with regional institutions and partnerships, sustainability increased because these relationships remained even when there was staff turnover.

Gender equality and environmental sustainability: Performance on gender equality as a crosscutting theme was mixed, and environmental sustainability was not effectively pursued.

Coherence: Policy dialogue was reasonably effective in some instances, but there is room for improvement. Caribbean officials responsible for aid coordination at the national and regional levels are less aware of substantial Canadian interventions.

Efficiency: Changes in the programming strategy, the decentralization of the program, and the introduction of new reporting mechanisms at the corporate level affected efficiency.

Management Principles: There are good examples of donor coordination and consistency with aid effectiveness principles.

Performance management: Program staff would benefit from increased support on the application, at the corporate level, of new monitoring and reporting tools to improve results management.

What is next for the Caribbean program?

• Strengthening emphasis on capacity building in selected areas of governance and regional integration.
• Strengthening capacity and effectiveness in supporting environmental sustainability.
• Capitalizing on Canada's comparative advantage and leadership on gender equality in the region.
• Strengthening the tracking and communication of the program's performance on regional integration.
• Increasing flexibility of projects.
• Strengthening monitoring and reporting.
• Leading the effort, at the corporate level, for a communications plan regarding Canada's 2007 commitment of $600 million in development assistance to the Caribbean region.

Canadian Investment Fund for Africa (CIFA) - 2005-2008

Purpose and Approach of the Evaluation

The purpose of this mid-term review was to provide independent and evidence-based information on the relevance, performance (i.e., effectiveness and efficiency), and the design and delivery of the CIFA initiative. It was also intended to provide lessons learned regarding the appropriateness of the CIFA model. The findings would also be useful to other Government of Canada departments and agencies and key stakeholders, such as CIFA Fund Managers, Actis LLP (United Kingdom) and Cordiant Capital Inc. (Montreal).

The program was designed to run between June 2005 and December 2013 with the possibility of extension for up to 2 years.

Intervention Description

The Canada Investment Fund for Africa is a Government of Canada initiative created to provide risk capital for investment in Africa; as such, it is an important component of Canada's contribution to private sector development in Africa. The Government of Canada investment of $100 million has leveraged an additional $160 million for investments in fifteen individual African companies in the oil and gas, mining, consumer goods, financial services, agribusiness, manufacturing, and logistics sectors, as well as two regional equity funds targeting small and medium-sized enterprises.

Intervention Sectors

• Private sector development
  • Industrial development: 30%
  • Mineral prospection and exploration: 30%
  • Privatization: 20%
  • Formal sector financial intermediaries: 10%
  • Small and medium-sized enterprises development: 10%

Partners and Stakeholders

The CIFA Fund Managers are Actis LLP (United Kingdom) and Cordiant Capital Inc. (Montreal). Other stakeholders include the Government of Canada, private sector investors, and beneficiaries in Africa.

Key Findings

• The full performance of the Fund in terms of the rate of return could not be judged, given that the average life of the investments was just 2.5 years and the global economy was in a pronounced downturn. However, the mid-term review helped identify some encouraging results.
• The decision to establish a private equity fund as a key element of the Canada Fund for Africa was well justified; given that financial market development in Africa was seriously lagging behind.
• The process for identifying the best mechanism for implementing the initiative, namely the private equity Limited Partnership model embodied in CIFA and for selecting the Fund Managers, was well conducted and achieved with dispatch.
• The intention to help stimulate private equity financing was an appropriate initiative at the time and was well aligned with Government of Canada international development assistance policies and priorities. The Canadian partner in the Fund Management Team was able to expand its private equity activities in Africa.
• The limited Partnership model chosen for CIFA allowed for hiring of a qualified Fund Manager; to catalyze fundraising from private investors and to help strengthen the ability of African countries to attract private sector investment.
• It was difficult to evaluate the overall performance of the funds, because of the limited number of similar private/public equity funds in Africa and the limited number of years on implementation.
• The performance of most portfolio companies has been negatively affected by the global financial crisis and subsequent economic downturn, particularly in more open markets. Fund Managers are taking measures to mitigate the long-term effects on performance while continuing to advocate corporate social responsibility and compliance with local taxation laws and regulations.
• The summative evaluation should be able to draw more conclusions on achievement of development outcomes and CIFA's contribution to deepening capital markets, use of new technologies, skills development and employment gains or losses.
• While the independence of the Fund Manager enabled the Fund to attract third party investors, it also limited the ability of the Government to modify terms and conditions during the life of the Fund.
• Branding CIFA as a unique Canadian Fund was designed to give Canada greater visibility internationally. This comes at a higher prices than relying on the fund-of-funds approach and is not consistent with the interest of Fund Managers to market their own brand;

Key Recommendations

• To ensure that Government of Canada has sufficient information for future investments of similar nature, the articulation of the expected results, the definition of related performance indicators, and the data collection systems to be put into place need to be improved. Reporting requirements must be understood and accepted by all parties involved and associated resource requirements must be provided to fulfill the obligations. In future negotiations, the Government of Canada could explore the option to include provisions for ongoing oversight by the Government of Canada/Investment Panel of the quality of reporting by the Fund Manager, and provisions for a mid-term review and/or summative evaluation.
• Currency exchange rate fluctuations should be taken into consideration when establishing International Rate of Return targets to ensure that the underlying assumptions for holding Fund Managers accountable for the achievement of expected results are made explicit in advance.
• This review was based on a limited number of indicators, including the assessment of International Rate of Return. Consideration should be given to a greater diversity of indicators to avoid the constraints encountered throughout the process of this mid-term review.
• Throughout the design and implementation phases of CIFA, the nature of its distinct legal and financial framework has been confounded with Canada's more commonly used development mechanisms. The distinct model of promoting and investing in a private equity fund needs to be more clearly in view when the final evaluation is undertaken.
• CIFA and similar models should not be assessed through the logic of Government's grants and contributions; rather it should be based on corporate financing modalities.

A Review of Program Evaluation Reporting: African Development Bank - 2007-2012

The African Development Bank (AfDB) contributes to the sustainable economic development and social progress of African countries. It provides development assistance to regional member countries in the form of financial instruments, policy advice, and technical assistance through the public and private sectors.

Why conduct this evaluation?

The original purpose of the review was to provide an independent assessment of the development effectiveness of the AfDB, building from the AfDB's own body of evaluative evidence. However, a preliminary assessment of the AfDB's evaluations for the period 2007-2012 noted weaknesses in program-level evaluations. The review therefore focused on presenting recommendations that would support the AfDB's ongoing efforts to strengthen its evaluation function.

What did the review find?

• The AfDB has not produced an adequate number of program-level evaluations and results reports over the review period (2007-2012) to enable an assessment of its development effectiveness using a synthesis approach.
• Program evaluations before 2012 provided inadequate coverage of results in the AfDB's priority areas and strategic objectives.
• The usefulness of reporting to decision making was sometimes impeded by the time lag between the year an evaluation was conducted and the year it was finalized and published.
• Some evaluation designs were not well articulated in the program-level evaluation reports reviewed.
• The AfDB has demonstrated a clear commitment to addressing some of the challenges mentioned above. In particular, the AfDB revised its Independent Evaluation Policy and prepared a first Independent Evaluation Strategy (2013-2017), which has already started to improve the quality and coverage of program evaluations.

What is next for DFATD and the AfDB?

• The AfDB has stated its commitment to strengthening its independent evaluation function by increasing evaluation coverage and improving evaluation designs as a means of better assessing its own program-level development effectiveness.
• Part of this commitment is the intention to engage in a peer review of its evaluation function.
• Canada, in concert with other donors and AfDB board members, should be ready to support the AfDB's efforts to strengthen its evaluation function.

Development Effectiveness Review of the Asian Development Bank - 2006-2010

The Asian Development Bank (ADB) is a major source of development financing for countries in Asia and the Pacific. Headquartered in Manila, ADB had, in 2011, US$21.7 billion in financing and 2,958 employees from 59 countries. Its mandate is to reduce poverty and to improve the lives of the poor in the region by fostering economic growth and regional cooperation.

Why conduct this evaluation?

This review provides an independent, evidence-based assessment of ADB's development effectiveness with a view to guiding Canada's future engagement with ADB and sharing with stakeholders. The review assessed 45 ADB evaluations conducted by its Independent Evaluation Department (IED) between 2006 and 2010.

What did the review find?

Relevance: ADB programs were relevant to the needs of the target groups and developing-country governments. However, a more robust analysis of the needs of target groups would improve the suitability of some program components.

Performance: Most ADB programs achieved their objectives and expected results thanks to the high level of program ownership by national governments and the high performance of most government and non-government partners.

Sustainability: The long-term sustainability of ADB programming results requires improvement, particularly through increased investment in the operation and maintenance of infrastructure.

Efficiency: Delays in program start-up and implementation contributed, in some cases, to increased costs.

Gender equality: ADB needs to ensure that more evaluation reports deal with this important issue. In evaluations that discussed gender equality, 75% of ADB programs effectively addressed it.

Environmental sustainability: Most ADB programs supported environmental sustainability effectively but they could benefit from investing in program elements designed to mitigate negative environmental impacts.

Monitoring and evaluation systems: ADB's independent evaluation systems were strong and the bank used them effectively. However, local systems for monitoring and reporting results need strengthening.

What is next for CIDA?

• Emphasize the need for the ADB to address gender equality in future evaluations.
• Engage with ADB to ensure that environmental sustainability of investments in infrastructure is highlighted in future programming.
• Identify the sustainability of ADB programming as a priority.
• Place emphasis on improving the timeliness of ADB program implementation as a means of increasing efficiency.
• Emphasize the need to support strengthened systems for results-based management, including monitoring and reporting, at the local level.

Partnerships with Canadians Program Evaluation: Governance - 2008-2012

Why conduct this evaluation?

The purpose of this evaluation was to examine the results achieved by the Partnerships with Canadians Governance Program from 2008 to 2012 and provide recommendations to improve its performance. The evaluation looked at 77 governance projects conducted with Canadian partner organizations in 81 countries around the world, covering $170 million in spending. The program is aimed at advancing Canada's commitment to freedom, human rights and the rule of law.

What did the evaluation find?

Effectiveness: Canada's partners demonstrated positive achievements in building accountable public institutions and promoting freedom and democracy. There are mixed results in the area of human rights programming, and limited programming exists on the rule of law.

Relevance: The Governance Program addresses needs identified by local organizations. Canadian organizations generally have strong and trusting relationships with local partners.

Sustainability: Improving governance is a long-term challenge requiring sustained investments to achieve results. Most governance projects lack information on sustainability considerations and plans for when funding agreements will end.

Gender Equality: Projects that paid more attention to local needs and context displayed greater success in promoting gender equality. However, many governance projects did not sufficiently integrate gender equality into planning and management.

Coherence: Most Canadian partners do not coordinate their governance activities with other Canadian organizations, although some examples exist. Coordination between Canadian organizations could add substantial value to work in this area.

Performance Management: Canadian partners make good use of performance-management tools to inform decision-making.

What is next for the program?

• Where appropriate, program staff should encourage collaboration between Canadian partners and other organizations and institutions in the countries in which they are working.
• The program should allow flexibility to extend or increase support to projects that demonstrate exceptional potential, since achieving results in governance sometimes requires long and uncertain timeframes.
• The program should promote training for Canadian partners on gender equality tools and analysis to improve gender data collection and outcomes.

Pathways and Pursestrings: Enhancing Market Access for Women Producers in Pakistan - 2007-2012

Purpose and Approach of the Evaluation

This evaluation examined the results that the project has achieved related to three specific areas: production capacity, improved access to market and the strengthened capacity of networked Key Facilitating Partners in terms of effectiveness, efficiency, relevance and sustainability criteria. Furthermore, it also assessed the project's performance in terms of local ownership and mutual accountability, its harmonization with other development efforts at the local level, and gender equality.

The evaluation relied on a document review, key informants interviews in Canada and Parkistan, and site visits in Pakistan.

Intervention Description

The goal of this project is to empower women in hard-to-reach rural areas, increase their household incomes, and build the capacity of Pakistani institutions to facilitate women's access to viable markets with appropriate products.

The project aims to provide homebound and isolated rural women with sustainable market linkages, product information, quality control, skills development, improved market supplies, and other support that enables them to access growing and dynamic markets with the products demanded by contemporary consumers.

To achieve the above, the Mennonite Economic Development Associates (MEDA) works with Pakistani civil society organizations and district governments that are already working on income generation with poor rural women producers, to develop their capacity to undertake successful market development programming. MEDA expects to provide technical assistance and training on collaborative value chain analysis and program design to ensure sustainable market development approaches that effectively integrate women and provide them with a fair return on their labour.

Intervention Sectors

• Democratic governance
  • Democratic participation and civil society: 30%
• Private sector development
  • Business support services and institutions: 50%
  • Cottage industries and handicrafts: 20%

Partners and Stakeholders

The project took place in the provinces of Sindh, Khyber Pakhtunkhwa, Punjab and Balochistan, and was executed by the Mennonite Economic Development Associates (MEDA). The lead Pakistani partner has been the Entrepreneurship and Community Development Institute, with support of four key facilitating partners: Pakistan Social Welfare Society, Haleeb Foods Limited, Kaarvan Craft Foundation, Sarhad Rural Support Program.

Key Conclusions

• The project lines up well with the whole range of DFATD policies, both at the departmental and sector levels as well as at country level and with the government of Pakistan's 2011 New Growth Framework, the goal of which is to generate opportunities in employment and entrepreneurship.
• The women that the evaluation team interviewed said that their participation in the Pathways and Pursestrings project had given them more confidence. Moreover, the project had helped them to connect both suppliers and buyers.
• Compared to other models of enterprise development, the value chain model is relatively cost effective.
• There is a reasonable chance that the changes that the project has inspired are likely to be sustained without the support of their key facilitating partners.
• The project has increased women's freedom and their participation in family decision-making. Women have increased their income and reduced their dependence on husbands or male relatives.

Key Recommendations

• DFATD should continue funding value chain development to benefit homebound women in Pakistan.
• MEDA should hold further high-level talks with stakeholders in the project's milk value chain might help save the investment.
• A bottom up approach to association building and clustering producer groups into chains, might enhance their ability to engage in policy advocacy.
• MEDA should consider how it intends to use the structures created for women producers as a platform on which to advocate for gender equality. MEDA should consider how it could position itself to become the lead implementing agency for several donor funded value chain projects.

The conclusions and recommendations above are those of the Consultant and do not necessarily reflect the views of DFATD or the Government of Canada. DFATD does not guarantee the accuracy of the information provided in this report.

Support to Ghana's Food and Agricultural Sector Development Policy (S-FASDEP) - 2009-2012

Purpose and Approach of the Evaluation

DFATD, (formerly CIDA) has provided sector budget support to the Ministry of Food and Agriculture (MoFA) since 2004. DFATD's first sector budget support programme covered 2004-08 (entitled Food and Agricultural Budget Support or FABS), valued at $85 M followed by a bridging year, 2009, worth $20M. The second budget support programme, valued at $110M covers the period 2009-13. As a means to ensure that its current and future investments and interventions are coherent and consistent, DFATD contracted an external evaluator to assess Canada's support to MoFA in 2012. This mid-term assessment was intended to evaluate the results, effectiveness and efficiency of budget support thus far while providing guidance on how to more effectively implement support programs within the agricultural sector.

This report provides a detailed assessment of sector-specific results achieved, implementation of budget support programs, institutional development results, effectiveness of budget support programs, and provides key recommendations and lessons learned.

The assessment relies on document review and interviews with key partners and stakeholders.

Intervention Description

This program is the second component of the Ghana Growth and Poverty Reduction Support Program. Through this component, Canada is providing sector budget support to the Government of Ghana to enhance its efforts in promoting sustainable rural development through food and agriculture As with all budget support initiatives, Canada is working in close cooperation with other donors and the Government of Ghana to strengthen aid effectiveness, by focussing on effective, transparent, and accountable country systems; increasing donor coordination and harmonization; and strengthening mutual accountability. Budgetary support also fosters greater policy dialogue between the Canadian government, the Ghanaian government, and partners, thus helping to strengthen efforts for effective, focused aid, as well as long-term development results. This initiative is continuously monitored and evaluated in coordination with other donors.

Intervention Sectors

• Improving health
  • Basic nutrition: 5%
• Private sector development
  • Agricultural development: 10%
  • Agricultural education/training: 10%
  • Agricultural extension: 10%
  • Agricultural policy and administrative management: 10%
  • Agricultural services: 10%
  • Food crop production: 10%
  • Agricultural inputs: 5%
  • Agricultural land resources: 5%
  • Agricultural research: 5%
  • Agricultural water resources: 5%
  • Industrial crops/export crops: 5%
  • Livestock: 5%
  • Plant and post-harvest protection and pest control: 5%

Partners and Stakeholders

MoFA is the government body responsible for the overall development of agriculture in Ghana and has been the main partner for this program. MoFEP, the Ministry of Finance and Economic Planning, now called the Ministry of Finance, has also been deeply involved. The Agriculture Sector Working Group (ASWG), which includes other development partners, other Government of Ghana departments, private sector and civil society organizations have also been involved.

Key Findings

• Canada's budget support has increased harmonization and alignment of aid. Canada had coordinated its effort with the World Bank and DFID in the sector to design its budget support programs.
• Coordination among donors and the government reduced transaction costs among donors (for instance DFID asked Canada to include additional triggers on its list of triggers to avoid having a separate list for DFID support).
• MoFA carried out a number of initiatives with deep involvement from Canada, to ensure harmonization and alignment of development aid. These include:
  1. The Agricultural Sector Working Group (ASWAG),
  2. The Joint Sector Review (JSR), and the scoping study on an agricultural sector-wide approach (SWAP)
  3. The FABS and SFASDEP budget support programs are fully aligned with MOFA strategic priorities.
• Canada has supported the overall MOFA strategy for the sector. Moreover, Canada is using the country M&E.
• Public policies and processes have been improved since the beginning of the sector budget support programs. The main reasons for this improvement include:
  1. the level of funding to MoFA as a percentage of the GoG discretionary budget is at least as high as in the previous year;
  2. MoFA designs an annual work plan and budget for the current year of budget execution, based on MoFA's 3-year Strategic Plan; and
  3. an agreed disbursement forecast for the budget execution year is done between MoFEP and MoFA. However, the three year work plan and budget request to MOFEP by MOFA need to be improved.
• Canada is providing technical assistance through three Food security advisors (FSA), consultants and its own staff. However, the 3 FSAs are under-utilized by MoFA mainly due to organization of MoFA and because their time is shared between MOFA and CIDA.
• Food security improved in the country as a result of increased production of all the major staple food crops with the exception of cocoyam and sorghum. The main food staple crops registered positive average annual production growth between 2000 and 2010.
• MoFA officially adopted the Agricultural Land Management Strategy and Action Plan (2009-2015) in 2009. Sustainable Land Management (SLM) activities had been carried out in selected districts as part of promoting an SLM Strategy and Action Plan. However, implementation of the Action Plan has been less than satisfactory since 2006 due to lack of a committed schedule of MoFA budget releases to support planned activities.
• Sensitization activities carried out to date have sought to enhance sustainable agriculture and improved productivity, and have resulted in an increasing awareness among beneficiary farmers of the negative effects of bad farming practices on the environment. More farmers have been adopting improved technologies to avert such negative environmental outcomes.
• Results of a study commissioned by MoFA showed that MoFA needs to intensify awareness of improved technologies along the value chain, which is a prerequisite for adoption. The pace and coverage of the dissemination of processing technologies by agricultural extension agents also needs to be expanded for greater effectiveness. Closer collaboration with credible civil society organizations and focusing on functional farmer based organizations will help increase the level of awareness and accelerate the rate of adoption of improved agricultural technologies in the country .
• There is a need to re-strategize with respect to extension methodologies and approaches to make extension delivery more effective and productive in MoFA's pursuit of its vision to modernize Ghana's agriculture. Canada's continued support through sector budget support in this area will be much needed, especially by providing increased funding for demonstrations to farmers, using the farmer field school concept, and the provision of appropriate equipment for farmer based organizations to increase the adoption of value addition technologies.
• Canada's sector budget support contributed to the mainstreaming of gender issues into MoFA policies, programs and projects through the implementation of Gender in Agricultural Development Strategy (GADS) in 2009 and the adoption of an accountability framework. Nonetheless, while previous reports have acknowledged improvement in the implementation of GADS, they have also lamented the slow pace and called for its speedy implementation.

Key Recommendations

• MoFA should develop a credible and realistic rolling three-year budget with the approved budget serving as the first year of the budget.
• For future budget support, a detailed logical framework, results matrix and a realistic set of triggers should be included with the contribution arrangement to be signed by both parties.
• Canada should clarify with MoFA how they want to use technical experts (TEs), with a clear role and function through an improved communication channel. Instead of sharing time, Canada should divide the TEs between itself and MoFA to clarify division of labour.
• Gender mainstreaming activities should be scaled up in all MoFA projects and closely monitored to assess outcomes and impacts. The capacity of Women in Agricultural Development (WIAD) needs to be strengthened at all levels and backed by increased budgetary allocations.
• If Canada plans to end all financing, discussions should be held immediately to help the government to find other partners or find ways to make self-sustaining investments in the sector. Should Canada decide to continue its sector budget support, then it should take advantage of its experience and design a larger sector budget support program.
• MoFA needs to deepen ownership of implementing the agricultural sustainable land management strategy and commit necessary resources to sustain good practice.
• MoFA should rigorously pursue evidence-based reporting through continued strengthening of monitoring and evaluation capacities.

The conclusions and recommendations listed above are those of the Consultant and do not necessarily reflect the views of DFATD or the Government of Canada, nor the Government of Ghana. DFATD does not guarantee the accuracy of the information provided in this report.

Productive Rural Development Sector Program (PRORURAL Incluyente) - 2010-2012

Purpose and Approach of the Evaluation

The evaluation aimed to support the effective management of the Productive Rural Development Sector Program in achieving its outcome-level objectives targeted for late 2014.

The evaluation assessed alignment with national policies, the contribution of results to three related national programs, and the management systems and institutional capacity for program implementation.

The evaluation relied on a document review, including analysis of the program's logic model, key informant interviews, and participatory workshops. Consulting decision makers, technical staff, program beneficiaries, donors and other private sector actors such as cooperatives, the evaluation provided a broad perspective of the program at the national and local levels.

Intervention Description

This project contributes to strengthening the agriculture sector in Nicaragua. It helps the Government of Nicaragua implement its Productive Rural Development Sector Program (known by its Spanish acronym PRORURAL). The goal of PRORURAL is to increase sustainable economic growth in rural areas and increase access to safe, nutritious food for poor rural women and men. PRORURAL helps farmers to increase their productivity and the quality of the food they produce; helps them to diversify their crops and transform their produce into higher-value products; and promotes sustainable use of forestry resources and improved management of the forestry sector.

As with all sector support initiatives, Canada is working in close cooperation with other donors and the Government of Nicaragua to promote effective, transparent, and accountable country systems; increase donor coordination and harmonization; and strengthen mutual accountability. This initiative is continuously monitored and evaluated in coordination with other donors.

Intervention Sectors

• Private sector development:
  • Rural development: 100%

Partners and Stakeholders

Target beneficiaries include rural families and agricultural producers, as well as citizens participating to reduce deforestation.

The institutions implementing PRORURAL are the Ministry of Agriculture and Forestry (MAGFOR), the Nicaraguan Agricultural Technology Institute (INTA), the Ministry of Family, Community, Cooperative and Associative Economy (MEFCCA) and the National Forestry Institute (INAFOR). MAGFOR is the lead organization responsible for the program within the government.

International donors include Austria, Finland, Norway, Spain and Switzerland as well as Canada.

Key Conclusions

The evaluation noted significant achievements, including:

• A good sector strategy and added value from PRORURAL,
• Ongoing adjustments by the Government, supported by donors, to align the program with national development goals, incorporate relevant experience, and strengthen management processes,
• Positive trends with respect to national goals and overall progress towards meeting sectoral and institutional goals, with opportunities for further gains through 2014, and
• Improved opportunities for dialogue between the Government and the private sector.

Major challenges that remain include:

• Ensuring a common understanding of the sector-wide approach adopted by PRORURAL;
• Establishing and using an integrated and coherent outcome and impact-level results framework;
• In the context of a new Ministry being created in 2012 (MEFCCA), formalizing inter-departmental coordination mechanisms and processes to ensure complementarities and reduce duplication;
• Structuring organizational arrangements and processes to work effectively in a sector-wide approach; and
• Addressing various resource management challenges to support the achievement of program objectives: estimating the gap in needed funding, focusing on efficiency, and strategically managing resources.

Key Recommendations

• The need for the sector-wide approach adopted by PRORURAL should be better articulated and explicitly described to build understanding, ownership and institutionalization.
• A coherent and integrated results framework should be developed and implemented.
• The Ministry of Family, Community, Cooperative and Associative Economy (MEFCCA), created in 2012, should prepare a strategic plan to clarify the roles of public institutions involved in PRORURAL.
• Organizational structures and processes should be improved to implement PRORURAL more efficiently and economically.
• Processes should be formulated to estimate costs and the funding gap for 2014, and an action plan and strategy should be formulated to improve efficiency and manage financial resources after 2014.

The conclusions and recommendations above are those of the Consultant and do not necessarily reflect the views of DFATD or the Government of Canada. DFATD does not guarantee the accuracy of the information provided in this report.

Nicaragua Electrification Project (PELNICA) - 2009-2012

Purpose and Approach of the Evaluation

The purpose of the evaluation was to determine if the project is on target for achieving expected outcomes. The evaluation was also undertaken for learning purposes.

The evaluation examined results and assessed the effectiveness, efficiency and relevance of the project. It also drew lessons for future design and implementation of an anticipated subsequent phase.

The evaluation relied on a document review, key informant interviews in Canada and Managua, and data collection in representative samples of electrified communities in the project's region. Focus groups and interviews focused on four result areas: development of the Synergy Network, Gender Equality, creation/ development of MSMEs and user satisfaction.

Intervention Description

This project aims to provide access to electricity for approximately 102,000 individuals in over 650 communities, thereby increasing their quality of life and improving their economic opportunities. The project helps communities in going ahead with productive activities that are possible due to having access to electricity.

The project is implemented by the Government of Nicaragua's Ministry of Energy and Mines, the electrification arm of the Ministry having acquired the needed skills through earlier CIDA projects. The Ministry has been given the mandate to efficiently manage funds to universalize access to electricity in Nicaragua. The Government of Nicaragua gives priority to projects that promote economic development, reduce poverty, and protect the environment.

Intervention Sectors

• Democratic governance
  • Public sector policy and administrative management: 20%
• Private sector development
  • Small and medium-sized enterprises development: 20%
• Other
  • Electrical transmission/distribution: 60%

Partners and Stakeholders

The government of Nicaragua's Ministry of Energy and Mines (MEM) identified communities to be electrified, based on socio-economic and geographic criteria, purchased materials and was responsible for construction, supervision, and provision of administrative and logistical support services. MEM-FODIEN (Fondo para el Desarrollo de la Industria Eléctrica Nacional) is the ministry's unit responsible for rural electrification. DISNORTE (the electricity distributor for the North of Nicaragua) was contracted by MEM to carry out electrification works. DISNORTE subcontracted construction companies. Starting with the second part of the 2011-2012 yearly program, ENATREL (Empresa Nacional de Transmisión Eléctrica) has been contracted to carry out the works for the rest of the project.

Key Conclusions

• Rewards of persistence: Continuing to support rural electrification in Nicaragua is deemed to have been a wise decision, rewarded by an achievement which constitutes a showcase of effective public development aid.
• Stable development partners: Since the beginning of CIDA's support to the process of rural electrification, there has been a remarkable continuity on the Nicaraguan side in terms of methodologies, procedures used, and key technical and management staff. In spite of government changes and reorganizations, many of the staff involved in the early projects are still present and continue to follow the same sound electrification policies and apply the same time proven approaches with great success. Dealing with a partner offering such consistency is deemed to have been a great advantage for CIDA. In general, PELNICA is a very successful project: the central expected objective of increasing the number of men and women with sustainable access to electricity has exceeded its targets in terms of the number of people and households with new access to electricity.

Key Recommendations

• Monitoring of the evolution of unit costs should be given the same importance as the other key parameters of the project.
• Through a joint task force or consultative working group, involving relevant technical staff from MEM, ENATREL and DISNORTE, continued joint efforts should be conducted to find ways and means to save on the costs of constructing the grid extensions.
• In spite of the many factors that hinder the improvement of service quality in rural areas, consultations between MEM-FODIEN and DISNORTE should continue towards finding ways in which the public sector (MEM-FODIEN, ENATREL, the municipalities), and eventually village leaders and the private firm, DISNORTE, can work together towards reducing the frequency and duration of service incidents and improving the overall quality of services.
• Unconnected groups of houses within electrified neighbourhoods should be surveyed by MEM's inspector and carefully reported upon, together with the reasons for leaving them aside. This information should be passed to the planning department of the electricity distributor for the North so that such groups of unconnected houses continue to be candidates for electrification in subsequent projects or campaigns.
• The team implementing the project's Rural Development and Gender Equality (RDGE)^{Footnote 4} component can be strengthened by the addition of a senior small enterprise development specialist. Because of the need to plan and coordinate the work of a team that would now consist of six people, consideration should be given to naming one of the members of the strengthened team as Head of the team or team Coordinator.

The conclusions and recommendations above are those of the Consultant and do not necessarily reflect the views of DFATD or the Government of Canada. DFATD does not guarantee the accuracy of the information provided in this report.

Local Governance Support Program for Local Economic Development in the Philippines - 2008-2012

Purpose and Approach of the Evaluation

The Mid-Term Assessment has the following objectives:

• Assess progress made towards results at the to date, including an assessment of how effective the cross cutting themes have been integrated
• Assess the performance in terms of:
  1. Development Results and
  2. Success Factors
• Provide recommendations for adjustments and refinements for future years and identify lessons for Canada's Program in terms of Sustainable Economic Growth programming in the Philippines.
• The evaluation relies on document review, and interviews with key stakeholders in Canada and the Philippines.

Intervention Description

The program supports national agencies in developing and improving policies, laws, programs, and activities in order to enhance local governance and economic development and strengthen relevant coordinating mechanisms (horizontal and vertical) at the national level. In addition, it supports the strengthening of umbrella organizations of local government units (LGUs) (e.g., alliances of provinces, cities, and municipalities) in order to increase their capacity to advocate for national-level measures to create an enabling environment for LGUs. The program also aims to strengthen the capacity of LGUs to improve the local business climate for increased investments, and to formulate and monitor policies for local economic development. The program supports the meaningful participation of civil society and the private sector in local development processes and strengthens resource generation and management capacities of LGUs. To improve conditions for local economic development, it also supports the strengthening of LGU/private sector/civil society partnerships and the development of supportive policies, incentives, and regulatory frameworks, and it helps to increase transparency and fairness in government systems and processes.

Intervention Sectors

• Democratic governance
  • Public sector policy and administrative management: 50%
  • Public finance management: 25%
• Private sector development
  • Business support services and institutions: 25%

Partners and Stakeholders

LGSD-LED is implemented by a consortium of the Canadian Urban Institute (CUI) and the Association of Canadian Community Colleges acting as the Canadian Executing Agency. The Canadian Executing Agency is working in partnership with the Philippine Department of Interior and Local Government (DILG). At the local level, LGSP-LED aims to build the capacity of Local Government Units/Alliances to develop innovative LED strategies that can produce important job creation and economic results.

Key Conclusions

• The Local Governance Support Program for Local Economic Development has made good progress towards the achievement of outputs and immediate outcomes and shows some progress in achieving intermediate outcomes. Progress is more evident in governance related areas and less clear in economic development ones.
• Lack of Private Sector perspective in the program has led to misunderstanding of the private and public sectors roles in economic development; emphasis on supply rather than on demand aspects in local level initiatives; confusion regarding what constitutes an investment and how to attract investors; and a tendency to support small interventions without properly integrating them in a market value chain.
• The expected role of DILG in is not clear to all stakeholders, including DILG staff, project staff, other National Government Agencies, and particularly local government units.
• Exposure to Canadian Public and Private Sectors partnership can provide local chief executives the vision and inspiration to foster it in their own constituencies.

Key Recommendations

• The External Monitor should be appointed as early as possible, ideally during the preparation/submission of the Program Implementation Plan to provide timely advice as the project takes shape.
• Simple and pragmatic monitoring systems should be developed for local interventions, and used consistently and in a participatory manner.
• Stakeholders' expectations can be better managed by clearly explaining the rules of the game.
• The Program and its partners should jointly review the Performance Measurement Framework - or come up with alternative or complementary tools to measure Program progress. A revamped system would help all partners better understand the present status of projects and Program, provide clear targets for the next four years, and serve to track progress and fine-tune strategies.
• The overall Program should be re-examined to increase the Private Sector perspective, including components, activities, strategies, approaches, and human resources.
• A new wave of interventions (Batch 3) should be introduced, following a different approach from the one used in the first two batches. The recommendation is to test a variety of interventions, hand-picking promising opportunities using transparent processes. Initiatives that provide interesting and replicable role models; put the private sector at the centre, with the support of the corresponding local government units and other relevant government and non-government partners; would immediately energize the corresponding value chain; and aim to achieve specific economic development targets over the short-to medium term would be selected.
• The Sustainability Strategy should address the subject of access to Program resources, and the proposed mechanisms to transfer greater control to DILG.
• Incorporate Financial Management analysis to the existing management tools.
• Keep the Local Economic Development Information Programme simple, relevant, and run by a group that has vested interest in its relevance.
• Review staff's roles, responsibilities, and level to effort to maximize Team's effectiveness.

The conclusions and recommendations listed above are those of the Consultant and do not necessarily reflect the views of DFATD or the Government of Canada. DFATD does not guarantee the accuracy of the information provided in this report.

Evaluation of Support to Establish the ASEAN Commission on Women and Children - 2010-2011

Purpose and Approach of the Evaluation

This mid-term review sought to assess progress made on planned results and to validate the appropriateness of the Performance Monitoring Framework (PMF).

The review explored achievement of results against indicators, relevance of results and appropriateness of methodology, cost-effectiveness of results, sustainability, partnership, appropriateness of resource utilization, informed and timely action, visibility of the project and lessons learned.

These issues are assessed with the objective of contributing to better-informed decision making and fostering an environment of learning.

The evaluation relied on document review and independent information sources, interviews with stakeholders and personnel in the field, questionnaires, and field visits to Thailand, Indonesia, and Cambodia.

Intervention Description

The project aims to ensure better protection of women's and children's rights in member countries of the Association of South East Asian Nations (ASEAN) through effective human rights mechanisms such as the ASEAN Intergovernmental Commission for Human Rights and the ASEAN Commission on Women and Children. The project develops the institutional capacity, functions, and structures of both Commissions. At the same time, it enhances the capacity of organizations and networks in the ASEAN region that interact with these two Commissions and that have a critical role in ensuring they both fulfil their mandates to promote gender equality and protect women's and children's rights.

Activities include: supporting civil society dialogues with the ASEAN committees and other relevant bodies; undertaking a study on the use and the impact of existing regional mechanisms on the promotion and protection of women's rights; and supporting the consultative workshops of relevant ASEAN bodies on the terms of reference of the ASEAN human rights commissions.

Intervention Sectors

• Democratic governance
  • Human rights: 50%
  • Public sector policy and administrative management: 50%

Partners and Stakeholders

Members of the ASEAN Human Rights bodies ACWC and the AIHCR benefited directly. Representatives from civil society organizations and National Human Rights Institutions (NHRIs) in the ASEAN regions also participated in policy dialogue.

Key Conclusions

• UN Women have proven to be a patient, responsive partner while still offering leadership to assist in planning project activities with a wide range of project partners. Remaining flexible and creative to prevent tensions among stakeholders has meant UN Women have been able to apply project funds in a catalytic manner so far.
• This project illustrates several key elements of a successful network, and areas of capacity required for effectiveness. Examples of indicators that could be used to encourage the development of these elements in other projects can be drawn from the project's reports.

Key Recommendations

• It is important to remain committed to a flexible project implementation while remaining within the principles of the logic model.
• Staffing shortages are at the root of some of the delays in project activities. Re-staffing procedures appear to be very slow within UN Women and all PMT members are fully stretched. Consideration of how to avoid the effects of this risk should be incorporated into project design and work planning.
• This project faces familiar challenges in identifying indicators that can track changes in attitudes and behaviours regarding human rights. This project's staff are aware of the need to modify some of the PMF indicators to assist in tracking such changes. Any effective examples should be passed on to other projects supported by Canada.
• Assumptions that collaboration with the logical UN partners will be easily facilitated need to be taken into account in future planning exercises. Leadership from one UN Agency does not mean others will be able to participate easily in joint activities.
• The proportion of time dedicated by PMT staff to technical support to several project partners is not necessarily reflected in the budget as linked to the work plan.
• The selection of APWLD^{Footnote 5} (and IWRAW-AP)^{Footnote 6} to manage the Women's Caucus component of the project illustrates an alternative approach to supporting one result area. This has not been possible with the partners within ASEAN as their administrative arrangements are still unsettled, but consideration of how technical support might be delivered to such partners should be taken into account in future project management arrangements.

The conclusions and recommendations listed above are those of the Consultant and do not necessarily reflect the views of DFATD or the Government of Canada. DFATD does not guarantee the accuracy of the information provided in this report.

Burma Border Assistance Program: Building Social Capital - 2010-2011

Purpose and Approach of the Evaluation

The purposes of this joint formative evaluation were to facilitate a constructive dialogue among stakeholders on the issue of camp management in refugee camps situated on the Thai-Burma border, describe the current camp management model that is in place, and identify areas where improvements and changes should be initiated.

It also assessed the effectiveness of the coverage of its responsibilities, the extent to which the work of the management structures in compliance with international standards, and fostered dialogue between partners about the model, based on documented evidence.

In addition to document review and key-person interviews, data was collected in the nine camps through a series of workshops and focus groups with a large sample of randomly selected refugees, including discussion groups for minorities, women and youth. In total, 1,060 stakeholders were engaged.

The main value of undertaking a joint evaluation with AusAID and Act for Peace lie in the amplified weight its recommendations carried among stakeholders.

Intervention Description

This project, implemented by Inter Pares, contributes to the development of civil society organizations and to building their capacity to identify needs, deliver services, and promote the interests of vulnerable and displaced people. An emphasis is placed on the participation of ethnic nationalities and women. Activities include: providing humanitarian assistance to Burmese refugees living in refugee camps in Thailand; the delivery of basic primary health care (including maternal and child health care) to displaced people in Burma's border areas; and civil society research, advocacy, and capacity-building on the issues of gender and violence against women, environmental degradation, forced displacement, human rights, drug abuse, and independent media.

Intervention Sectors

• Democratic governance
  • Human rights: 7%
  • Democratic participation and civil society: 6%
  • Media and free flow of information: 5%
• Improving health
  • Food aid/food security programs: 25%
  • Basic health care: 10%
  • Reproductive health care: 5%
• Environment
  • Environmental education/training: 3%
• Emergency assistance
  • Emergency food aid: 6%
  • Material relief assistance and services: 6%
• Relief co-ordination; protection and support
  • services: 6%
• Peace and security
  • Civilian peace-building, conflict prevention and resolution: 5%

Partners and Stakeholders

The evaluated component of the project supports the Border Consortium (TBC), an organization based in Thailand. Stakeholders include the Royal Thai Government, Refugee Committees, the United Nations High Commissioner for Refugees, the International Rescue Committee, the Committee for the Coordination of Services to Displaced Persons in Thailand, and the Donor Humanitarian Actors Working Group. The project was implemented by Inter Pares.

Key Conclusions

• There is deep potential for self-governance and self-management in refugee communities. Enabling refugees to exert control over their own lives and livelihoods through self-management is an important affirmation of the essential humanity of refugee populations.
• Shared values and vision, and mutual trust, form the foundation of effective refugee camp management. Transparency, mutual respect and trust, critical values that contributed to establishing the camp management model, have eroded. Timely improvements and changes to the system are required to maintain and strengthen a system valued by refugees, government and other development actors.
• Camp management and governance skills and experience may promote nation-building in the repatriation effort. Camp management structures have functioned as "public administration schools" and, for the broader refugee population, animated an experience of citizenship.

Key Recommendations

• All NGO service providers working in the camps should consult fully with refugee management structures on important decisions.
• All agencies working in the camps should participate in the monthly coordination meetings at the camp level and strengthen the strategic role of these meetings.
• UNHCR should actively advocate that the Royal Thai government recognize Refugee and Camp Committees as legitimate governance and management structures with clear and explicit the responsibilities and authority.
• The Refugee Committees and Camp Committees should ensure that all adults in the camps be given the right to vote in the 2013 elections and ensure that the minimum quota of 33% women on camp management structures is met and, preferably, surpassed.
• Refugee and Camp Committees should institute mechanisms to ensure that the voices of ethnic and religious minorities and youth are heard and their needs are given due consideration.
• UNHCR, the Royal Thai Government and the International Rescue Committee, in collaboration with the Refugee Committees should support a clearer identification and delineation of the roles and responsibilities of the various parties with respect to protection and access to justice.
• Refugee Committees should continue to receive support in revising and rolling out an updated set of camp rules and regulations, including a process of public education about the nature and purpose of rules.
• The effectiveness of the Donor Humanitarian Actors Working Group could be enhanced by establishing a small secretariat to provide the donor community with support, coordination, information and independent analysis.
• The Refugee Committees should be formally recognized as the legitimate representatives of refugees in all camps and be formally involved in the planning and priority setting processes of the Donor Humanitarian Actors Working Group.
• UNHCR should take on a leadership role on behalf of the donor community in developing a coordinated approach in preparation for the repatriation of Burmese refugees.
• The Donor Humanitarian Actors Working Group should commission a more detailed strategic analysis of ways in which the camp management model can make an optimum contribution to Burma's nation-building efforts.

The conclusions and recommendations listed above are those of the Consultant and do not necessarily reflect the views of DFATD or the Government of Canada. DFATD does not guarantee the accuracy of the information provided in this report.

Capacity Building Technical Assistance Facility Project (Component Two: Technical Assistance Unit) - 2007-2013

Purpose and Approach of the Evaluation

The evaluation investigated the extent to which the Capacity Building Technical Assistance Facility Project (Component Two) was executed efficiently and effectively and whether the stated results were achieved, using evaluation criteria including effectiveness, efficiency, sustainability, appropriateness of design, impact, and lessons learned.

The evaluation relied on interviews, focus group discussions, and review of sub-project documentation. Additionally, case studies for impact analysis were developed following site visits.

Intervention Description

The Capacity Building Technical Assistance Fund (CBTAF) supports South African initiatives to improve governance. More specifically, it helps build accountable public institutions and improve the capacity of government departments to deliver services.

The goal of this project was to contribute to improved service delivery, particularly for poor and vulnerable populations in South Africa in support of the South African Government's Program of Action. The project provided direct support to the Technical Assistance Unit of the South African National Treasury. The mandate of the Technical Assistance Unit is to improve the sustainable management of public finances through the provision of technical assistance and management support to other government departments (national, provincial and local).

Intervention Sectors

• Democratic governance
  • Public sector policy and administrative management: 100%

Partners and Stakeholders

Major stakeholders include International Development Cooperation at the National Treasury, the Department of Public Service and Administration, Public Administration Leadership and Management Academy and the Technical Assistance Unit as well as other national, provincial and local government departments.

Key Conclusions

• CBTAF Two enabled the Technical Assistance Unit to design initiatives including tools and methodologies that promoted gender equality.
• Through Canada's assistance, the Technical Assistance Unit contributed to a number of major reforms in Government including the implementation of the results-based management approach in Government.
• Clients supported are generally very satisfied with the expertise, professionalism and capacity building provided by the Technical Assistance Unit officials they engaged with.
• Baseline data is not being systematically collected during diagnostic reviews in sub-projects to allow measuring the "before and after" impact/change as intended by the CBTAF Two project during final evaluation.
• The design of the CBTAF Two project effectively contributed to the Aid Effectiveness agenda by strengthening government mechanisms that deliver services and by providing a sense of mutual accountability and ownership through joint funding arrangements.

Key Recommendations

• Canada should support the expansion of a "Knowledge Bank", for Knowledge sharing in Government and beyond.
• Canada should continue building the Technical Assistance Unit's capacity internally through institutional strengthening initiatives within the unit as it merges with other units into the Government Technical Advisory Centre.
• Roles of stakeholders should be redefined to strengthen accountability and participation in future programmes.
• Canada should continue to support the Technical Assistance Unit in developing mechanisms that will enable it to check/review progress towards achievement of the required change in sub-projects.
• Canada should support the Technical Assistance Unit to speed up progress in the development of Best Practice (BPR) Tools and methodologies.
• A basket fund should be established to target capacity building and institutional strengthening to improve coordination.

The conclusions and recommendations listed above are those of the Consultant and do not necessarily reflect the views of DFATD or the Government of Canada. DFATD does not guarantee the accuracy of the information provided in this report.

Ukraine Horticulture Development Project - 2008-2013

Purpose and Approach of the Evaluation

This evaluation assesses progress to date of the Ukraine Horticulture Development project and provides recommendations on the design of a new phase of the project.

The evaluation assesses the effectiveness, efficiency, relevance and sustainability of the project's interventions in the horticulture sector, and provides conclusions and recommendations to draw out lessons for a potential future phase of the project.

The evaluation relied on interviews with key informants, as well as project documents and reports.

Intervention Description

This project aims to strengthen the capacity of 5,000 smallholder farmers in the regions of Crimea and Zaporizhzhia to engage profitably, competitively, and sustainably in the production, processing, and marketing of high-value horticulture commodities. Farmers in these two regions face a range of constraints that inhibit their access to markets and lead to a significant percentage of the population living in poverty. The project involves partnering with lead farmers, local farmer associations and agriculture extension services to deliver technical training and other agriculture support services designed to improve production and facilitate access to markets for smallholder farmers.

Intervention Sectors

• Private sector development
  • Food crop production: 30%
  • Agricultural extension: 20%
  • Agricultural services: 20%
  • Agricultural co-operatives: 10%
  • Agricultural financial services: 10%
  • Agricultural inputs: 10%

Partners and Stakeholders

The project is being implemented by Canadian partners, the Mennonite Economic Development Associates and Ukrainian partners, the Association of Farmers and Land Owners in Crimea (AFLOC) and Agro-Tavria, a nonprofit organization providing extension services in the Zaporizhzhia Oblast.

Key Conclusions

• The project has applied a grant scheme for gender component. As a result 298 women have received grants from the project which have helped them to improve their businesses.
• The official Project Partners have limited authority and therefore not enough ownership on the project compared to similar Canada-funded projects.
• The consolidation of sales model is gaining encouraging results. Informal cooperation through this model has laid the foundation for cooperatives development. Further assistance will be vital to support those initiative groups which are ready for joint legal operations.
• There is a great need to support the development of extension services in Crimea in a view to sustain and further assist smallholder farmers in horticulture, producing 95% of fruits and vegetables in Crimea.

Key Recommendations

• Establish clearly defined criteria for selecting beneficiaries for direct project support.
• Use more complex designs to encourage beneficiaries to independently scale up their commercial agricultural activities, while sharing the cost of investments and the benefits amongst themselves.
• Pay particular attention to the most vulnerable smallholder producers among the economically active segment. Helping them to consolidate their activities and linking them to larger farmers groups will ensure increased participation of smaller producers in the value chain.
• Strengthen capacity building in economic literacy for the project clients. This has been shown to have widespread benefits.
• Increase practical, on-farm trainings and demonstrations for the project clients.
• For phase 2 of the project, place a particular emphasis on the linkages between environmental improvements and economic effectiveness, and develop corresponding indicators to measure progress.
• Project planners should consider closer cooperation with regional and national level authorities. Further cooperation can be focused on joint development of relevant programs and potential to contribute to their implementation.
• Build into the existing curriculum of local agrarian universities in Zaporizhzhia and Crimea a technical training component on modern approaches to horticulture production and marketing.
• Develop a capacity building program on cooperative legal literacy, service cooperative management and administration for farmers' initiative groups, existing or registered cooperatives, and head of village councils.

The conclusions and recommendations listed above are those of the Consultant and do not necessarily reflect the views of DFATD or the Government of Canada. DFATD does not guarantee the accuracy of the information provided in this report.

Regional Human Rights and Peacebuilding Fund (RHRPF) Project - 2007-2012

Purpose and Approach of the Evaluation

The objective of this End-of-Project Assessment was to ascertain progress and results of the Regional Human Rights Peacebuilding Fund in the Middle East and North Africa region as well as its overall performance.

The assessment focused on three main points:

1. Development Results;
2. Quality of Partnership; and
3. Success Factors.

The Assessment also identified some lessons learned and suggested a few recommendations to improve the project.

The assessment also looked at the following issues: the quality of designs and partnership, some best practices, appropriateness of resource utilizations, risk management, etc.

The evaluation relied on a questionnaire administered to civil society organizations, a field mission, interviews with key stakeholders, and extensive document review.

Intervention Description

This $5 M fund aims to help establish an environment for peace and stability that is rooted in the values of the Middle East and that respects the international principles and standards for human rights, including the rights of women. CIDA's contribution provides support to selected local and regional non-governmental organizations to implement projects which contribute to reducing human rights abuses and increasing women's influence on promoting human rights and peace building.

Intervention Sectors

• Democratic governance
  • Human rights: 40%
  • Democratic participation and civil society: 30%
• Peace and security
  • Civilian peace-building, conflict prevention and resolution: 30%

Partners and Stakeholders

Initially, the Fund was to operate in Iraq, Lebanon and WestBank. The events of the "Arab Spring" (2011), prompted CIDA Representatives to widen the geographic scope of the project to include sub-projects from Morocco, Tunisia and the Project Management Unit is based in Amman, Jordan. The project focused on collaboration with selected civil society organizations in the region.

Key Conclusions

• The program was highly relevant - as much in its intentions than its achievements. The fact of having given access to CSOs from North Africa introduced a new dimension to the program and provided a unique learning opportunity.
• This evaluation shows that the project was met with resounding success. It managed to produce very useful results for thousands of recipients - mostly women, youth and children - in terms of respect for their fundamental rights and peaceful conflict resolution in local communities.
• The project managed to implement and operate in difficult socio-political environments through informed collaboration with civil society organizations and still with a low administrative ratio.
• In the final leg, the RHRPF acquired a high absorption capacity and developed trusted relationships with major civil society organizations that are major drivers of change in promoting Human Rights and Peace and who embody the ideals of the "Arab Spring".
• In terms of Human Rights and Peaceful Conflict Resolution, partnership with civil society is the most functional way to channel ODA resources - particularly in fragile socio-political contexts.
• Prior knowledge of local issues relating to Human Rights and Conflict Resolution is a key factor of success.
• Provision of essential services to individuals must be accompanied by communication actions, awareness and advocacy so that concrete cases become references for larger scale action.
• The success of such a program is based on different levels of action that must be well coordinated: work at the local level, training, voluntary action, communication, learning, working with media, and mobilization at the national level on specific issues.
• Horizontal communication, networking of CSOs from different countries and continuing education are all essential elements of such a program.
• The regional dimension of such a program is well founded as the experiences of one country feeds the other's experiences.

Key Recommendations

• The field of Human Rights and Peacebuilding is a key field for both the welfare of people in countries affected by the transition to democracy put forward by the "Arab Spring" as well as for the security of all countries.
• Canada should develop more programs that meet the different needs of civil society organizations from the countries of the region, national advocacy for the establishment of a society of law, citizen participation in the democratic processes.
• Canada should find a partner CSO (or consortium of CSOs) at the regional level for the program. In a participatory manner, an institutional diagnostic of this partner should be supported as well as medium term strategic planning (over a period of 5 to 10 years).
• Capacity of this regional partner should be developed to take part in dialogue on development policy through high level interventions free of partisanship.
• The promotion of Women's Rights and Children's Rights should be prioritized.
• The capacity of all civil society partners should be developed to mobilize resources from other donors for their programs and to manage them.

The conclusions and recommendations listed above are those of the Consultant and do not necessarily reflect the views of DFATD or the Government of Canada. DFATD does not guarantee the accuracy of the information provided in this report.

Support to the Foundation for Civil Society - 2009-2011

Purpose and Approach of the Evaluation

The specific objectives of the evaluation were to establish the impacts of the interventions funded by the Foundation for Civil Society (FCS) in line with its Strategic Plan (2009-2013) and log frame, and to measure the effectiveness of the Foundation interventions relating to the first three key result areas: policy engagement, governance and accountability and civil society capacity strengthening.

The midterm impact assessment assessed the effectiveness and impact of the FCS funded activities of civil society organizations (CSOs) in meeting the intended beneficiaries' needs, the extent to which communities were involved in the activities of the CSOs and the impact on them, and the extent to which citizens have been empowered to realize their rights, engage in policy dialogue and impact on their communities and their lives.

The evaluation relied on desk review of documents and reports, in-depth interviews with key stakeholders, fieldwork visits for data collection and physical verifications/assessments.

Intervention Description

The Foundation for Civil Society is one of the largest support mechanisms for capacity building of civil society in Tanzania. It is an umbrella organization offering grants and training to support the work of CSOs in Tanzania. The FCS is a Tanzanian non-profit organization funded by a group of like-minded donor countries and governed by an independent Board of Directors. To enhance civil society's contribution to poverty reduction, FCS grants aim to improve organizational, financial, technical and managerial capacity of civil society organizations (CSOs); develop a more conducive environment for policy dialogue with the Tanzanian government; encourage and strengthen CSOs networking and, build a positive image of the role of civil society.

FCS provides grants to CSOs engaged in four key areas: policy dialogue with the Government of Tanzania; supporting and advocating for the needs of the vulnerable members of society; strengthening governance and advocacy; and strengthening CSO networks.

Intervention Sectors

• Democratic governance
  • Democratic participation and civil society: 90%
  • Human rights: 10%

Partners and Stakeholders

FCS was the main partner, while civil society organizations receiving grants and support from FCS, Local government authorities at district, ward and village levels, public expenditure tracking committees, and national level ministries were stakeholders and partners.

Key Conclusions

• Overall, there is considerable engagement in policy processes, a high level of citizens' awareness on rights and increasing demand for accountability from public resources. Also, FCS-supported CSOs have been strengthened in terms of effectiveness, sustainability and accountability.
• Results show that awareness and involvement in key government policy processes among FCS-supported CSOs increased from 50% in 2010 to 67% in 2011.
• Results also show that while there is some tension, relationships between CSOs and government leaders both at local and regional levels are positively evolving.
• Although FCS support has enabled CSOs to increase the number of their personnel in the accounts section, financial capacity was relatively poor. Specifically, 45% of the CSOs visited had "acceptable" or "below standard" financial capacity.
• Managerial capacity of the CSOs visited was relatively high. Fifty-five percent of the CSOs had excellent or good organizational set up. However, employee skills were relatively low. However, more educated and trained people are joining the civil society sector as a result of support extended to those institutions, so the skill level found in those organisations should improve.
• Improvement in capacity was realized in all grantees regardless of the grant size, geographical disparities and duration of operation. One of the most important impacts of the FCS was related to the increase of the knowledge base of those CSOs and stakeholders that were directly related to implementation of FCS-supported activities.
• Although FCS intended to promote inter-CSO collaboration, the organization did not succeed in opening spaces for collaboration among different CSOs it supports.
• By all standards, local citizens have been empowered and have relatively high capacity to demand accountability from their local leaders.
• It appears that the impact of FCS funding on governance has manifested in many levels, including the partners and citizens levels.

Key Recommendations

• The FCS should increase its efforts to include civic education through continuous efforts of mobilization and training of all key stakeholders including local government authorities at district, ward and village levels.
• While it continues to fund the CSOs on various interventions areas, FCS should extend its support to persuading the local government authorities to set aside resources for providing civic education to citizens in partnership with CSOs.
• There is a need to provide capacity building to the grantees in training local government elected leaders such as village chairpersons and ward councillors to build familiarity on the basic rights and responsibilities of their people.
• CSOs should use the opportunities gained from FCS funding to further develop their organizational, managerial, financial and advocacy skills to attract more funding. In this regard, FCS should expand its capacity building activities to include resources mobilisation.
• For FCS to evaluate its impact and measure the milestones these CSOs may have achieved after a given timeframe, FCS should monitor a sample of its interventions more closely.
• It was noted that few CSOs have follow-up plans after FCS support ends. It is recommended that CSOs receiving funds from the FCS show some evidence of being proactive in terms of designing sound strategic plans and actions for sustainability of their activities and organizational survival.

The conclusions and recommendations listed above are those of the Consultant and do not necessarily reflect the views of DFATD or the Government of Canada. DFATD does not guarantee the accuracy of the information provided in this report.