Formative evaluation of Canada’s response to the Venezuelan refugee and migrant crisis in Latin America and the Caribbean region

December 23, 2023

Table of contents

• Acronyms and Abbreviations
• Executive summary
  • Overview, rationale, purpose and objectives
  • Summary of the intervention
  • Scope of the evaluation
  • Methodology used
  • Main findings
  • Conclusions
  • Recommendations
  • Lessons learned from the evaluation

Acronyms and Abbreviations

ACCBP

DFATD’s Anti-Crime Capacity-Building Program

ECHO

European Community Humanitarian Aid Office

ECW

Education Cannot Wait

EDM

Evaluation Design Matrix

EQ

Evaluation Question

ET

Evaluation Team

ETL

Evaluation Team Leader

DFATD

Department of Foreign Affairs, Trade and Development

DTM

Displacement Tracking Matrix

FGD

Focus Group Discussion

FIAP

Feminist International Assistance Policy

GBA+

Gender-Based Analysis-Plus

GBV

Gender-Based Violence

GOC

Government of Canada

IDB

Inter-American Development Bank

ILO

International Labour Organization

IP

Implementing Partner

IOM

International Organization for Migration

IRCC

Immigration, Refugees and Citizenship Canada

KII

Key Informant Interview

KPIs

Key Performance Indicators

LAC

Latin America and the Caribbean

LM

Logic Model

MSI

Management Statement of Intent

NLX

DFATD’s Venezuela Task Force

OAS

Organization of American States

OECD

Organisation for Economic Co-operation and Development

PMF

Performance Measurement Framework

PRM

Population, Refugees, and Migration

PTA

Project Technical Authority (DFATD lead for the evaluation)

QA

Quality Assurance

R4V

Regional Inter-Agency Coordination Platform for Refugees and Migrants from Venezuela

RBM

Results Based Monitoring

RMRP

Refugee and Migrant Response Plan

SRH

Sexual and Reproductive Health

SoW

Statement of Work

TA

Technical Activity

TB

Treasury Board

TIP

Trafficking in Persons

ToC

Theory of Change

TVET

Technical and Vocational Education Training

UNEG

United Nations Evaluation Group

UNFPA

United Nations Population Fund

UNHCR

United Nations High Commissioner for Refugees

UNICEF

United Nations Children's Fund

UNODC

United Nations Office on Drugs and Crime

VRM

Venezuelan Refugees and Migrants

WASH

Water Sanitation and Hygiene

WB

World Bank

Executive Summary

Overview, rationale, purpose and objectives

This is the final report of the evaluation of Canada’s Response to the Venezuelan Refugee and Migrant Crisis in Latin America and the Caribbean Region, a $59.9 million program managed by DFATD and IRCC. The commissioning agent for the evaluation was DFATD’s Americas Branch. Salasan Consulting Inc. was the Canadian consulting firm contracted to design and implement the evaluation and prepare this report.

Implementation of projects under this 2-year program began in March 2022. Thus, the evaluation took place roughly half-way through the program’s life-cycle. As the crisis continues to unfold, Canada is reflecting on the difference this program has made to date, thus this evaluation. The purpose of the evaluation is to provide an independent assessment of progress made, and results achieved. More specifically, the evaluation objectives are to:

• Determine the extent to which the program interventions made progress towards results (within the limited implementation timeframe of the program) and what factors supported or hindered the achievement of results;
• Identify innovative approaches applied by partners in addressing specific challenges faced by women and girls who are refugees and migrants, and in host communities;
• Identify lessons learned and distil good practices stemming from the program, with a particular focus on gender equality; and,
• Assess the program’s management modalities and identify aspects that facilitated the program’s delivery and aspects that acted as a barrier.

Summary of the intervention

The program includes a portfolio of 18 approved projects spanning nine countries (Argentina, Brazil, Chile Colombia, Ecuador, Guyana, Panama, Peru and Trinidad and Tobago) with most activity and targeted beneficiaries in Colombia, Ecuador, and Peru. All but five of the projects are being implemented across more than one country – typically two or three – and a third of the projects are regional in focus. Direct beneficiaries include Venezuelan refugees and migrants, as well as members of host communities in the nine countries of implementation.

Program interventions are diverse. Most projects have an original average duration of 15-20 months but by December 2023, according to Type 1 KIs, “most” had been extended. Project interventions are generally in the areas of employment; education, including comprehensive sexual education; health, including sexual, reproductive health and rights (SRHR) as a focused intervention; protection; human trafficking; migrant regularization; and concessional financing. The program supports the response to the development needs, security risks, and migration concerns stemming from the Venezuelan refugee and migrant crisis. Ultimately, the program seeks to improve the quality of life and resilience of Venezuelan refugees and migrants and host communities, particularly of women and girls, as well as improve stability and security in the wider Latin America and Caribbean region.

The program’s projects are implemented by five different types of partners: UN Agencies, International Finance Institutions, International NGOs, the Organization of American States, and local CSOs through DFATD’s Canadian Fund for Local Initiatives. DFATD and IRCC are the exclusive donors to the program.^{Footnote 1}

The program has nine expected Immediate Outcomes which are in turn expected to contribute to three Intermediate Outcomes aligned with three pillars of intervention: socioeconomic integration, mobility, and security. Specifically, the program’s three intermediate outcomes are as follows:

• 1100 - Enhanced economic resilience of Venezuelan refugees, migrants, and host community members, particularly women, as well as those experiencing marginalisation and/or vulnerability in targeted host communities in the Latin America region.
• 1200 - Reduced irregular migration impacting the region and Canada.
• 1300 - Enhanced mitigation and/or response by governments, institutions, and international organisations to reduce security threats stemming from the Venezuela crisis.

Organizationally the program is a whole-of-government response by Canada to the crisis in Venezuela. Committed funds are jointly managed by four different GoC institutional entities:

• DFATD – Venezuela Task Force (NLX) directly manages 11 projects ($48.8 million),
• DFATD – Caribbean Regional Program, directly manages 1 project ($2.25 million),
• DFATD – Anti-Crime Capacity Building Program (ACCBP) directly manages 2 projects ($4 million),
• IRCC – Immigration, Refugees and Citizenship Canada, directly manages 4 projects ($3.5 million).

Scope of the evaluation

The object of the evaluation is Canada’s Response to the Venezuelan Refugee and Migrant Crisis in Latin America and the Caribbean Region, a two-year, $59.9 million program response to the Venezuelan displacement crisis. The scope of the evaluation is this program in its entirety, from its inception and including activities undertaken up to the onset of data collection by the evaluation team (ET). The program’s original two-year implementation time frame is from April 1, 2022, to March 31, 2024, although at the time of writing this evaluation report (December 2023), DFATD informed the ET that “a number of projects had been extended beyond this time frame”.

Methodology used

An evaluability assessment took place before the evaluation’s work plan was approved. The evaluation used a non-experimental approach which relied on both quantitative and qualitative data and was primarily qualitative in its findings and conclusions. Analysis was strengthened through triangulation, that is, comparison across different evidence streams. The ET considered multiple variables simultaneously and worked collaboratively with key stakeholder representatives to analyse potential outcomes, outputs, activities, and the contextual factors that influenced project interventions and results.

The main data collection methods used included: document collection and review, project mapping (see Volume 2 of this report), key informant interviews (KIIs), focus group discussions (FGDs), country-specific case studies with site visits, and validation sessions. While the scope of the evaluand included all 18 projects in the program portfolio, a sample of eight projects helped to focus the evaluation. Interviews with targeted key informants were used to collect primary data at global, regional, country and project level. A sample of 116 KIs, with women well represented (67 percent of total), were interviewed.

Five country-specific case studies (three of which involved country and site visits) were used to examine program context and results at country level. Three of these (Colombia, Peru and Guyana) were “deep dives” in the sense that they included country and site visits by core members of the ET and were further supported by capable local consultants. Guyana was the first case study and took place soon after the work plan was approved. It served as a pilot to test the evaluation methodology and tools. Based on lessons learned from the pilot, work plans for the remaining case studies were adjusted.

The ET organised FGDs of stakeholder representatives in the three deep-dive countries. The FGDs provided a voice for local project sub-grantee implementors, a voice for end-beneficiaries of project activities, and a differentiated voice for women beneficiaries thus adding to sex-disaggregated data. The local consultants within the ET led the planning for the FGDs. In total, 21 FGDs took place involving 151 participants, 70 percent of them women.

Main findings

The main findings of the evaluation are as follows:

1. The program was explicitly designed to respond to the identified needs and priorities of vulnerable beneficiaries, namely Venezuelan refugees and migrants, as well as members of host communities.
2. The program was dispersed, with resources spread across eighteen projects, plus many additional sub-projects, and further diffused geographically across nine countries.
3. Assessing achievement of stated program-level results is challenged by the logic of its design which is focused on the performance of individual projects and less so on synergy, coherence and integration across initiatives.
4. Within a sample of nine projects, six were on track for contributing to expected results; two projects had no evidence yet of activities starting; and one project (00856-003, CUSO) required more time to realistically achieve its ambitious expected reach in Peru and Ecuador.
5. The CFLI initiatives, though small in scope, assured that program resources went directly to proven CSOs that already had established relationships within host communities, and first-hand knowledge of Venezuelan refugees and migrants (VRMs).
6. Program funding is providing opportunities to support effective organizational capacity building of established CSOs that are well-positioned to continue offering community-based best-practise services.
7. The program purposefully, and usually successfully, selected and approved projects that were designed to ensure that the more vulnerable, particularly women and girls, were the primary targeted beneficiaries. The exceptions were projects that instead more indirectly targeted institutions and government officers.
8. With some notable exceptions, most of the sample projects examined provided evidence of adaptive, context-specific implementation to assure that vulnerable VRMs, and particularly women and girls, were being reached.
9. Projects within the program portfolio had a wide range of differentiated interventions based on the specific contextual needs of women and girls.
10. The program is managed by DFATD and IRCC headquarters in Canada, with project team leaders (PTLs) for most projects also based in Canada. This makes it difficult at times for projects to manage effectively for results in the face of changing local priorities and needs.
11. Projects that had an in-country PTL, and close local engagement with their implementing partner, had more visibility. Projects that were managed in country by Canadian embassies through the CFLI had the most direct structure to facilitate project delivery, results, and local visibility.
12. Program-wide knowledge management processes, and more specifically meetings to share experiences between the implementing partners within this diverse program were largely absent.
13. Many projects had a regional scope, and regional coordination between country-specific project teams was evident during the design stage and across some training activities. However, only limited initiatives across these projects actively promoted coordination among the governments of the countries involved.
14. At a macro level, the program helped foster regional coordination between host country governments, by being a significant part of Canada’s broader diplomatic and humanitarian strategy for Latin America and the Caribbean.
15. The projects were of short duration, and approved project budgets and stated expected results often did not align with the 2-year program timeframe.
16. The limitation of the short program cycle was partially mitigated because most of the projects were continuations of broader, ongoing efforts by the implementing partners (IPs).
17. Absorptive capacity of available budget was not an issue for IPs and their local CSO partners.
18. Key factors that have facilitated the program’s ability to achieve results include: i) established partnerships with trusted IPs, ii) experienced CSOs as IP sub-contractors, iii) ability to extend already existing programs, iv) existence of an established coordination space, v) DFATD and IRCC’s use of grant agreements to contract projects, and vi) supportive elements of host government policy and legislation.
19. Key factors that have constrained project ability to achieve results include: i) overly ambitious intermediate outcomes paired with relatively small budgets, ii) a short project cycle, iii) limited budgets trickling down to CSOs, and iv) unstable political context and rise in insecurity in the region.

Conclusions

Most of the projects funded by the program were responding effectively to the needs and priorities of vulnerable Venezuelan migrants and host community members. The underlying theory of change for these projects was typically a mix of direct service delivery and case management with a strong GE commitment. For a smaller proportion of projects, the design instead targeted state agencies and departments. For these projects, the theory of change focused more on capacity building of government officers and government processes. For these projects it was more difficult to confirm effectiveness.

Implementing partners who focused on direct service delivery and case management, were trusted with some of the largest projects. These projects were mostly on track for achieving expected immediate results. Given the short program timeline and the wide dispersion of available resources, the extent of program contribution to ambitious intermediate results was less certain.

Projects were mostly extensions and expansions of already existing multi-year programs being implemented by well-established IPs. This sometimes limited the visibility of Canada’s contribution. Although the program was significant in size (close to $60 million), it was not easy to establish the specific contributions it is making. Perhaps this is the most unexpected outcome of the program: Canada’s contribution has limited visibility in any one country and cannot easily claim attribution to higher-level results because the program is dispersed. The smallest projects were those implemented through the CFLI using local CSOs as IPs. While the expected reach and results of the CFLI sub-projects were moderate because of the small budgets and short timeline, a high proportion of funds available were directly channelled to the most vulnerable.

CSOs’ partners typically had capacity to accept more funds that could have been used to directly channel DFATD support to the most vulnerable. CSOs were found to be well-positioned to offer community-based best-practice services. Funding for CSOs, both as implementing partners and as organizations could have been more explicitly included in grant agreements.

More obvious than the program’s LM-related outcomes were the process outcomes achieved: 1) the quick disbursement of Canada’s commitment by DFATD and IRCC, 2) alignment of Canada’s response with the ongoing needs assessments and monitoring led by the R4V platform to assure overall regional and country-specific coordination and collaboration, 3) the largely successful use of trusted IPs to quickly scale Canada’s contribution, and 4) the positive impression of Canada as a credible interlocuter as the LAC countries and civil society, and external donors, continue to respond to the complex needs caused by the crisis in Venezuela. It can be said with confidence that the program is contributing urgently needed resources to a much larger and ongoing multi-lateral response plan.

Most of the projects examined involved adaptive, context-specific implementation to assure that vulnerable VRMs, and particularly women and girls, were being reached. For the most part the IPs selected by DFATD and IRCC, and the projects approved, consistently applied a gender lens to assure that programming was focused on the most vulnerable. Projects that targeted institutions and government officers were less successful in putting the needs of women front-and-centre.

The program supported a wide range of approaches to address the differentiated needs of women and girls. These differentiated approaches, based on established best practise, were being implemented – largely successfully – by well-respected IPs, and typically as part of broader multi-year strategies.

A lack of program-wide knowledge management processes, and more specifically, the limited opportunities for sharing of experiences between implementing partners characterized a funding envelope rather than a cohesive, result-oriented program. The program did not have an overarching theory of change. Instead, projects selected were autonomous and independent and not linked or expected to work together to create synergy towards sustainable program-level outcomes.

Project team leaders (PTLs) were mostly based in Canada, far from the frontline of the crisis which added to the challenge of managing projects in the face of rapidly changing local contexts. A knowledgeable in-country PTL supported closer engagement with IPs and host country government contacts. At a smaller scale, this was mimicked by the CFLI sub-projects that had the most direct structure to facilitate project delivery, results, and local visibility.

While regional coordination between country-specific project teams was evident during the design stage and across some project training activities, only limited initiatives actively promoted coordination among the nine governments targeted by the program. The program did support regional coordination between host country governments by being part of Canada’s broader diplomatic, peace, humanitarian, and development efforts in the LAC region.

The expectation of program-level achievement is limited by its short timeline. The program initiatives were considered developmental rather than humanitarian and, as such, expectations for higher level results would have benefited from longer planning and implementation timelines. The limitation of the short program cycle was partially mitigated because most of the projects were continuations of broader, ongoing efforts by already established IPs. The established partnerships with trusted IPs that DFATD and IRCC relied on, available CSOs as sub-contractors, and the already existing programs that were added to, were all key success factors.

Recommendations

The evaluation made 9 recommendations as summarized below:

1. The Venezuelan crisis continues to negatively impact the region and beyond. The complex nature of the crisis i.e., the interlinkages between humanitarian, development and peace imperatives, and the need for effective socio-economic integration of migrants to help mitigate further mass movements of people within and out of the region, means that Canada should sustain its involvement and leadership, guided by the needs assessment and response plan laid out in the latest UN RMRP.
2. For future response in the region, DFATD and IRCC should streamline funding to reach fewer countries, fewer projects, and fewer sectors so that the overall commitment is more cohesive, more results-oriented, easier to manage, and provides greater visibility for Canada.
3. Not withstanding Recommendation 2, an additional small number of potentially valuable partners can be considered. However, DFATD and IRCC should continue collaborating with well-performing implementing partners already engaged in the sectors of SRHR, GBV, primary education, and youth skills and employment, and avoid creating a wholly new slate of partners and sectors.
4. If there is another tranche of development support, Canada should look for approaches that encompass the following sector-specific best practices:
   1. SRHR – SRHR programming that provides direct services to refugees, migrants and host communities, raises individual and community awareness of rights and responsibilities, and supports CSOs, especially women and youth led, in undertaking action related to SRHR, including influencing government policy.
   2. GBV – Programming that supports changing power dynamics between couples through gender dialogue, safe spaces, child advocacy centres, confidential helplines for survivors of GBV, and individualized, professional case management.
   3. Primary education – Programming that includes meaningful participation, influence, and leadership by local actors (CSOs, parents, and teachers) to support quality, inclusive education for vulnerable refugee and migrant children and adolescents from early childhood through secondary education.
   4. Youth skills and employment – Partnering with appropriate private-sector companies, job-specific training, and individualized case management approach to improve rate of job placement success among migrant and host community populations.
5. Given the protracted state of the crises, DFATD and IRCC should frame future programming as support for longer-term, sustained development:
   1. Supported by appropriate due diligence, timeline extension requests from IPs for existing projects should be approved to support achievement of higher-level outcomes.
   2. For new programing, design projects with timelines of three or more years (rather than two years) consistent with longer-term development goals.
   3. Ensure future program performance targets are designed to align realistically with the specifics of the implementation context in LAC countries.
6. Future DFATD and IRCC program development in support of Venezuelan refugees and migrants, as well as broader approaches to migration in the Americas, should prioritize management structures that bring Canadian project oversight and decision making as close as possible to the countries hosting the assistance.
7. Within the framework of projects funded by Canada, country-based DFATD focal points should continue to promote programmatic roundtables purposefully designed for IPs to participate in, together with state actors, where cooperation and knowledge sharing can be fostered.
8. Within approved projects, including new or expanded grant agreement with UN organizations, DFATD and IRCC should seek specific funding lines that support the organizational growth (administrative, operational, personnel capabilities, etc.) of involved CSOs, including those led by VRMs, and design projects to channel a larger percentage of the available resources in this direction.
9. To support effective local CSOs and directly reach the most vulnerable VRMs, DFATD should channel a larger portion of its response through the existing, country-led CFLI, striving to use a full two-year implementing cycle for that fund.
10. Explore extending the offer of the grant mechanism to other partners as a modality that offers greater flexibility and management efficiency

Lessons learned from the evaluation

The key lessons learned from undertaking this evaluation are summarized below:

1. When responding to a regional crisis, informed by a multi-stakeholder response plan (e.g., the RMRP), humanitarian and development goals may conflict. Development goals, and especially those that include GE and a commitment to the most vulnerable, require a longer timeline. When a program is spread widely across many countries and sectors and includes a broad portfolio of projects with an array of implementing partners and sub-contractors, its potential to contribute to sustained development outcomes becomes diffused, management oversight is dispersed, and Canada’s visibility within individual host countries is muted.
2. A whole-of-government programming approach to crisis response can bring different Canadian agencies under the same program umbrella. In this case: diplomatic, development, migration and border security expertise were combined in a single program with one LM and PMF to guide its management. This design approach complicated and diffused program management and made it difficult to achieve program coherence and outcomes that could be clearly attributed to Canada. To be effective, the whole-of-government approach to development programming requires well funded activities that support synergy and coordination across departments.
3. The evaluation re-emphasised the lesson that projects improved their performance in reaching vulnerable VRM by working directly with and through host communities. A correlated lesson is that when IPs work through CSOs that have a proven GE commitment, and already have established relationships with local leaders and institutional partners, assistance can be quickly scaled up, and successful targeting of the most vulnerable is easier. External assistance to migrants and refugees is inherently political and therefore host communities must be directly involved and included as direct beneficiaries to mitigate tension within affected communities.
4. Good practices that worked included a) IPs that continued GE analysis during project implementation, b) programming flexibility in response to contextual specifics and the needs identified for individual vulnerable cases, and c) working with local CSOs who already had established reach to the most vulnerable. Providing help lines, safe places for vulnerable women and adolescent girls, one-stop child advocacy centres to support abused girls, livelihood training and support geared to young women, and direct support to families and teachers to improve access to quality primary education are types of initiatives that could be further replicated and scaled up.
5. Migration and refugee crises such as this one in the LAC region take place in a dynamic, complex and constantly evolving context. Insecurity, political instability, changes in migrant flows, country-specific immigration policy changes, xenophobia and even rumours spread by social media can change the specifics of development challenges on the ground. DFATD and IRCC were able to give trusted IPs the flexibility to integrate Canada’s assistance into already active and proven programs. By predominantly using a grant modality for development aid disbursement (rather than more complex and demanding contribution agreements), DFATD and IRCC supported the need for flexibility.
6. Development project management structures, in which decision making is concentrated at DFATD and IRCC headquarters in Canada, makes it difficult for individual project managers to be nimble. DFATD and IRCC PTLs that are working from Ottawa, and with limited ability to travel to project implementing sites, are disadvantaged by being far from the frontline.