Formative evaluation of the Expert Deployment Mechanism (EDM) project – Executive summary

October 2023

Table of contents

• Rationale, purpose and specific objectives
• Context of the development intervention
• Evaluation object and scope
• Logic of intervention
• Evaluation approach and methodology
• Key findings
• Conclusions
• Recommendations

Rationale, purpose and specific objectives

This formative evaluation for the Expert Deployment Mechanism (EDM) project was commissioned at the mid-term of the project to provide an assessment of project implementation and achievements at a point when changes can be made to the project to enhance achievement of results, resolve any project challenges and ensure that project resources will be used for intended purposes, including building trade policy and institutional capacity for developing countries with which Canada has trade relationships.

The purpose of the evaluation is to inform evaluation users and stakeholders of:

• how the project is being implemented and results achieved to date
• what barriers and facilitators have influenced the achievement of results to date
• what is the level and extent of stakeholder participation (governments, private sector, civil society and other beneficiary groups) in project implementation
• what lessons have been learned from project implementation to date and what opportunities are being sought to strengthen capacity going forward to ensure the sustainability of results

The specific objectives of the evaluation are to:

• assess the status of implementation of the EDM project and progress toward the achievement of expected results
• assess the relevance of results to the needs and priorities of project stakeholders (governments, private sector, civil society and other beneficiary groups)
• assess the conditions created to promote the sustainability of results beyond the lifespan of the project
• assess the opportunities and constraints inherent in designing and delivering technical assistance activities through regional Association of Southeast Asian Nations (ASEAN) settings
• identify lessons learned and best practices in designing intervention strategies and implementations at country and regional levels
• assess the degree to which the cross-cutting issues of gender equality, social inclusion and environmental sustainability have been integrated into the project’s implementation
• provide findings, conclusions and recommendations to fulfil the stated evaluation purposes

The intended end users of this evaluation are Global Affairs Canada (GAC) and the consortium of Cowater International and the Institute of Public Administration of Canada, the project implementing organizations.

Context of the development intervention

International trade and investment are an essential driver of sustainable economic growth and poverty reduction. However, many developing countries face barriers that limit their ability to benefit from trade, including lack of institutional, legal and regulatory environments conducive to accessing global markets. Trade-related technical assistance for developing countries supports the development of their policy and institutional capacity to foster international trade that stimulates growth, while building mutual prosperity through deeper commercial relations with trading partners.

The project supports the deployment of Canadian and international technical experts to assist countries with negotiating, implementing, adapting to and benefiting from bilateral or regional free trade agreements (FTAs) and foreign investment promotion and protection agreements (FIPAs) with Canada. Technical assistance may include advisory support, training, study tours, policy research, stakeholder engagement and institutional development and is available to governments, the private sector and civil society. Technical assistance initiatives are aligned with the Feminist International Assistance Policy action areas, with gender equality, social inclusion and environment as cross-cutting themes.

Evaluation object and scope

EDM is a 7-year (2018 to 2025) initiative with a global scope and budget of $16.5 million ($6 million disbursed as of March 2023). At the time of this evaluation, the project was being implemented in 4 countries (Colombia, Paraguay, Peru and Vietnam) and the ASEAN region, while work was also underway to expand project activities to Indonesia, Malaysia and Thailand. Key project activities include the following:

1. providing technical assistance to government officials on engaging in discussions and/or negotiations on inclusive trade or investment agreements
2. supporting ministries, departments and agencies to implement inclusive trade and investment agreements with Canada
3. supporting private sector organizations on how to adapt to and benefit from expanded trade and investment agreements with Canada

The scope of the formative evaluation covers the entire development intervention including Colombia, Paraguay, Peru, Vietnam and the ASEAN region. It focuses on project implementation, the achievement of immediate outcomes and progress toward achieving intermediate outcomes.

Logic of intervention

Ultimate Outcome: Improved economic well-being (m/f) in developing countries where Canada engages in trade or investment agreements and that are targeted by EDM.

Intermediate Outcome 1100: More inclusive trade and investment exploratory discussions, negotiations and agreements between Canada and partner country governments.

Intermediate Outcome 1200: Improved development of inclusive legislation, regulations, policies or standards related to the implementation of trade and investment agreements with Canada by partner and country governments.

Intermediate Outcome 1300: Improved employment and entrepreneurship opportunities in EDM partner countries for individuals, particularly women and women-led small and medium-sized enterprises (SMEs), with special attention to poor popula­tions, especially women and girls and other underrepresented groups, such as SMEs and Indigenous peoples.

Immediate Outcomes:

1110: Enhanced ability of partner country governments to engage in exploratory discussions, negotiate, and finalize inclusive trade and investment agreements that consider the perspectives of women’s organizations, environmental groups and other underrepresented groups, such as SMEs and Indigenous peoples.

1120: Enhanced ability of non-government actors, especially women’s organizations, environmental groups and underrepresented groups, to contribute to partners country dialogues in relevant areas of trade and investment agreements.

1210: Enhanced ability of partner country governments to strengthen, implement and measure legislation, regulations, policies and/or standards that support inclusive trade and investment agreements with Canada.

1220: Increased capacity of partner country governments to provide their populations access to employment and entrepreneurship opportunities and business development services for individuals, particularly women and girls.

1310: Increased knowledge and awareness of partner country governments regarding how populations, especially women and girls and other underrepresented groups, such as SMEs and Indigenous peoples, can adapt to and benefit from expanded trade and investment with Canada.

1320: Increased awareness and skills of non-government actors, especially women and girls and other underrepresented groups, such as SMEs and Indigenous peoples, in taking advantage of trade and investment opportunities arising from trade and investment with Canada.

Evaluation approach and methodology

The EDM evaluation approach used mixed methods to identify and measure the changes that occurred as a result of the Cowater International and Institute of Public Administration of Canada implementation of the EDM project. Using the evaluation questions provided in the statement of work, the evaluation team assessed whether and to what extent results were achieved. The evaluation matrix was used to obtain answers to each evaluation question, ensure consistency, and support data collection and analysis. Data collected from GAC and the Canadian executing agency (CEA) included project-based literature, project planning and progress documents. An online survey was distributed to EDM beneficiaries, and key informant interviews were conducted online with partner country representatives and line ministries, and GAC and CEA staff, including Canadian missions overseas.

Key findings

1. Relevance: EDM has responded appropriately to the needs and priorities of developing country partners. Further outreach is required to attract stakeholders outside of national government partners and line ministries.
2. Effectiveness: EDM has achieved success in building capacity with government bodies but has made limited progress toward building capacity with the private sector and civil society organizations. Building capacity for negotiating free trade agreements has been slow.
3. Efficiency: While response to technical assistance requests has been timely, the request process is slower than expected.
4. Sustainability: EDM’s approach to technical assistance contributes to a lasting transfer of knowledge; however, issues of ownership and inclusion of private sector and civil society organizations must be addressed for project results to be sustained.
5. Cross-cutting themes: Gender-responsive trade policies have not yet been developed and implemented in EDM partner countries. EDM stakeholders have not yet exhibited capacity to develop environmental policies or chapters in free trade agreements.

Conclusions

1. EDM’s design, processes and structures have been appropriate to EDM’s objectives and to meeting the needs of partner countries as expressed in their Country Needs Assessment Plans (CNAPs).
2. Stakeholder participation in EDM design and implementation was extensive among partner country government bodies, but the private sector, including SMEs, and civil society needed to be more closely involved in the CNAP planning to help shape the partner country EDM strategy and make it relevant to all beneficiaries identified.
3. The CNAP and resulting technical assistance requests (TARs) and technical assistance activities (TAAs) have been effective tools in addressing capacity gaps in EDM partner countries. However, additional approaches are needed to strengthen local capacities, including those that support beneficiary learning by doing and learning from each other.
4. The EDM has not made sufficient progress toward its intermediate outcomes to contribute to the economic well-being of the population of beneficiary countries. It is unlikely, based on performance to date, that the project will be able to contribute to demonstrable benefits for populations by 2025.
5. Apart from small efficiencies such as the use of online workshops, operational efficiencies in EDM have been limited. This is primarily due to external factors, including delays in submitting TARs, limitations to travel, and fewer TAAs and deployments than planned.
6. Ongoing delays in the submission of TARs by partner countries continue to have a negative effect on the efficiency of resource use in EDM. Until EDM develops a strategy for addressing this major roadblock, the project will not be able to plan or deliver TAAs in a timely manner, thus posing a critical risk to the achievement of immediate and intermediate outcomes.
7. Sustainability of project results has not been addressed by either the CEA or partner countries. The assumption included in the EDM project implementation plan’s (PIP’s) theory of change that “the participants who receive the technical assistance have the means and authority to apply the knowledge and skills to their jobs” has not been tested and there has been no recorded action on the part of governments to move toward negotiating and implementing trade agreements. Sustainability of EDM results is in question unless the learning provided by the TAAs is put into practice.
8. The potential for achieving EDM’s intermediate outcomes is limited given a lack of gender-responsive policy implementation by the beneficiaries. The CEA has done an admirable job of demonstrating commitment to gender equality (GE) and integrating it into TAAs. However, there is no evidence yet of behaviour change on the part of beneficiaries that would lead to more inclusive trade. Without gender-responsive trade policy development and implementation that includes identifying barriers to women’s participation and consideration of the possible adverse effects of trade on women in developing countries, trade policies and practices will not be addressed.
9. The EDM has effectively addressed learning about human rights, through its introduction of women’s rights, and labour rights. However, the project has not yet addressed the potential for an increased risk of human rights violations as a result of expanded trade.
10. Beneficiaries’ capacity to develop environmental policies or environmental chapters within trade agreements is limited to their own sector or specialty and does not extend to broader knowledge about the trade and environment nexus that would enable them to formulate or negotiate trade agreements.

Recommendations

1. The Canadian executing agency (CEA) should take steps to ensure that project activities are relevant to all stakeholders, and particularly to private sector and civil society organizations (PSOs and CSOs).
2. The CEA should work to strengthen a sense of ownership of the project among partner country governments, which are the main interlocutors in implementing project processes. This will necessarily involve further consultation and messaging with key partners, including Canadian missions abroad, about the importance of PSO and CSO involvement in trade negotiation and implementation, and of gender equality, social inclusion, human rights and environmental sustainability in EDM to generate technical assistance activities (TAAs) that are inclusive.
3. It is essential for EDM to accelerate project activity. The CEA should conduct increased outreach to, and consultation with, governments, PSOs and CSOs to generate a sufficient number of new technical assistance requests (TARs) to contribute to EDM intermediate results. GAC should prioritize, to the extent possible, identification of new country partners for implementation of EDM and introduction of the project to key national government partners.
4. The CEA should undertake a revision of its current budget allocations so that unspent funds can be used to support such activities as increased project representation and promotion, Country Needs Assessment Plan revision, further investigation of country contexts and contextual issues such as the trade/development nexus. Funding should also be allocated to develop more TARs on environmental sustainability and gender equality and social inclusion (GESI) issues. Sufficient resources to reach CSOs and SMEs must also be allocated and justified to achieve results.
5. As part of the revised management and budget for the remainder of the project, the CEA should develop a sustainability/exit strategy and plan that includes targets for sustainability. The sustainability strategy should be participatory, designed with input from partners and beneficiaries and carefully monitored through the remainder of the project.
6. The CEA should build on gender equality knowledge amongst consultants and beneficiaries to directly address barriers that prevent women from participating in international trade.
7. The CEA needs to consider a more holistic approach to human rights in TAAs that recognizes and addresses the adverse effects of increased international trade on women, Indigenous peoples, racialized communities, children and adolescents, as well as other vulnerable groups.
8. Further capacity building is needed by the CEA to address country and sectoral environmental sustainability issues to develop mutually supportive trade and environment policies.

* The findings, conclusions and recommendations listed above are those of the consultant and do not necessarily reflect the views of the department or the Government of Canada. The department does not guarantee the accuracy of the information provided in this report.