Evaluation of the Caribbean regional development program, 2011-12 to 2016-17

Final report

International assistance evaluation division (PRA)
Global Affairs Canada
April, 2018

Acronyms and abbreviations

AusAID

Australian Agency for International

CARICOM

Caribbean Community

CARILED

Caribbean Local Economic Development

CDB

Caribbean Development Bank

CDM-HIP

Comprehensive Disaster Management Harmonized Implementation Programme

C-EFE

Caribbean Education for Employment

CIDA

Former Canadian International Development Agency

CRDR

Caribbean Reducing Disaster Risk

CSJP

Citizen Security and Justice Programme

CSOs

Civil society organizations

DFAIT

Former Canadian Department of Foreign Affairs and International Trade

DFATD

Department of Foreign Affairs, International Trade and Development

DFID

United Kingdom Department for International Development

DMAS

Debt Management Advisory Service

DRR

Disaster Risk Reduction

EPIC

Entrepreneurship Program for Innovation in the Caribbean

GDP

Gross domestic product

OECD

Organisation for Economic Co-operation and Development

OECD-DAC

Development Assistance Committee (of OECD)

PAHO

Pan American Health Organization

PRASC

Project for the Regional Advancement of Statistics in the Caribbean

PROPEL

Promotion of Regional Opportunities for produce through Enterprises and Linkages

PTLs

Project team leaders

SEMCAR

Supporting Economic Management in the Caribbean

TVET

Technical Vocational Education Training

UN

United Nations

UNDP

United Nations Development Programme

USAID

United States Agency for International Development

UWI

University of the West Indies

Table of contents

Executive summary

This evaluation of the Caribbean Regional Development Program assessed the relevance, effectiveness, efficiency and sustainability of development programming results for the period of 2011/12 to 2016/17. Its purpose is to inform decision-making and to support policy and program improvements. This report presents the evaluation findings, conclusions, and recommendations.  Considerations for future international assistance programming are also identified.

For the projects examined, the evaluation found that Global Affairs Canada’s programming was relevant, as it aligned with the region’s needs and priorities, as well as with departmental policies. As a partner, Canada was in a position to provide support and expertise through its programming, thereby enabling mutually beneficial collaboration with the region. Canada was also a key partner for supporting gender equality, which is a complex issue in the region. Partners and other donors perceived Global Affairs Canada as a reliable and flexible development partner. Despite corporate changes, programming was recognized for its stability and consistency in supporting the same sectors throughout the 10-year strategy, which increased the potential for development results. At the same time, the program’s visibility across the region could have been strengthened.

The program was strategically designed to include complementary initiatives across three sectors, while also promoting regional collaboration and communication, knowledge sharing, and synergy among projects. At the same time, the regional approach limited the program’s flexibility to adapt to country-specific needs. In some cases, more lasting results could have been fostered by complementing regional initiatives with a targeted selection of country-based initiatives.

Canadian international assistance contributed to increased capacity in the region through the delivery of professional and technical expertise. Although some results were achieved across all programming sectors, more time was needed to achieve consistent longer‑term results. The program sought to integrate gender equality into its programming; however, obstacles led to uneven implementation and monitoring across projects. Projects in the sample aimed to build capacity and contribute to the program’s potential reach and sustainable results, but challenges to sustainability remained, such as the need for longer-term funding and further capacity building.  

Although the program efficiently managed its financial resources, challenges remained with regard to its efficiency in addressing other operational areas such as communications, the management matrix organizational structure, corporate constraints, information management resources, and access to technical expertise.

Summary of Recommendations:

1. Complement regional programming with country-specific investments and include national governments in regional projects.
2. Develop a short plan of action articulating how the program intends to promote knowledge sharing, visibility, policy dialogue and capacity building, to better integrate these into daily operations, and ensure common understanding of these concepts among staff and partners.
3. Develop a short plan of action articulating how the program intends to promote and integrate gender equality into daily operations, and ensure common understanding of gender equality among staff and partners.
4. Review the current management matrix structure to improve communication, coordination, and cohesion among staff.

Table of contents

Program background

Caribbean regional context

 

The Caribbean Regional Development Program comprises 11 island states (Antigua and Barbuda, Bahamas, Barbados, Dominica, Grenada, Jamaica, Montserrat, St. Lucia, St. Kitts and Nevis, St. Vincent and the Grenadines, as well as Trinidad and Tobago) and 3 continental states (Belize, Guyana and Suriname) with a total population estimated at 7 million.

Of the 14 countries covered by the regional program, 4 (Bahamas, Barbados, Trinidad and Tobago and St. Kitts and Nevis) are classified as high-income and are therefore not included in the Organisation for Economic Co‑operation and Development’s (OECD’s) Development Assistance Committee (DAC) list of countries eligible for official development assistance support. To the extent possible, the program has focused its activities on low and middle‑income countries while allowing high-income countries to participate to promote regional cooperation.

The Caribbean region comprises 25 countries, 15 of which are members and 5 are associate members of the Caribbean Community (CARICOM), the region’s initiative for economic and political union, which was established in 1973. Of the 15 CARICOM member countries, 14 are covered under Global Affairs Canada’s Caribbean Regional Development Program; Haiti is covered under a separate program.

Countries in the region can be classified into two categories: tourism‑ and service‑dependent economies, and commodity‑exporting countries. Many are small islands and low-lying coastal countries that face similar sustainable development challenges such as small populations, limited economic diversification, high debt levels, vulnerability to natural disasters, and human resource deficits that are accentuated by emigration of skilled and professional workers.

The 2008/09 global financial crisis worsened the already high debt burdens of most Caribbean countries. Public sector debt across the region averaged 64.6% of GDP in 2017 down from 79% of GDP in 2012^{Footnote 1}. Most countries could not benefit from debt relief because of their income status, and few were able to access concessional financing from international financial institutions.

The small size of some of the developing island states poses several capacity challenges and underscores the need for collaboration.  Caribbean states have been moving toward regional integration for three decades, and despite its promised gains, governments have held firmly to their individual sovereignty in many aspects.

Although CARICOM countries generally have shared strong democratic traditions, their public institutions have needed strengthening in policy‑making and regulation; financial transparency and accountability; and in-service delivery capacity.

With some exceptions, the region has remained peaceful. In the last two decades it has also made strides toward gender equality.  However, exclusion and violence were still highly prevalent in the region during the evaluation period. Further, the rise in unemployment and poverty across the region in recent years has been linked to increased social inequality and economic insecurity. Female-headed households also had a higher incidence of poverty and remained more vulnerable to economic decline.

Global Affairs Canada programming

The program traces its origins to 2006/07, when the Canadian government committed $600 million over a 12‑year period (2008/09 to 2019/20) to the Caribbean region. At the same time, the decision was made to replace previous bilateral programming with individual Caribbean countries (with the exception of Haiti) in favour of a regional approach in order to build on commonalities and address shared challenges across the region. In addition, the program managed both regional and national projects in Jamaica and Guyana.

Over the evaluation period, from 2011/12 to 2016/17, the Caribbean Regional Development Program was guided by three successive strategies and a draft development program framework, all of which maintained a focus on sustainable economic growth, advancing democracy, and disaster risk management. At the same time, gender equality and environmental sustainability were cross-cutting themes.

Programming focused on three main program sectors: Sustainable Economic Growth; Governance and Advancing Democracy; and Risk Resilience and Climate Change Adaptation. The Risk Resilience and Climate Change Adaptation sector was supported by Sustainable Economic Growth and International Humanitarian Assistance funds.

The Caribbean Regional Development Program has been managed through the Americas Branch at Global Affairs Canada. The department oversaw additional programming in the region through other branches. Canada also has delivered additional programming in the region through other federal departments, such as the Royal Canadian Mounted Police and the Department of National Defence. Between 2011/12 and 2016/17,  bilateral aid disbursements from Global Affairs Canada to the regional program totaled $336 million, with annual disbursements ranging from $43 million to $106 million and an annual average of $56 million^{Footnote 2}.

• Caribbean Community Strategic Programming Framework (2007/08 to 2017/18)
• Caribbean Strategy 2.0 (2009)
• Caribbean Development Programming Framework 2010-2015
• Draft Canada's Regional Development Strategy with the Caribbean (2014–2019)

Programming has been managed through a decentralized organizational structure. A director general at headquarters has authority over the long-term programming strategy for the Americas Branch. At the same time, management of the Caribbean Regional Development Program is delegated to a senior director based in Barbados, which is considered the regional hub for managing the decentralized regional program.

The senior director collaborates as a peer with five heads of mission, who are based at the high commissions in the region, and who also have the mandate of overseeing both the trade and the development programs. The senior director reports to two director generals, one in relation to the regional program, and one in relation to the Caribbean Development Bank.  The senior director plays a leadership role as a member of the Caribbean Development Bank board of directors.

Five deputy directors report to the senior director, four of which are based in the field and one is based at headquarters. The team in Ottawa oversees many of the tasks related to the coordination of corporate requests and compliance with department requirements.

Departmental changes between 2011/12 and 2016/17:

A number of changes occurred in the department during the evaluation period:

Amalgamation

• Before 2013, development assistance programming was delivered through the Canadian International Development Agency (CIDA). In 2013, CIDA was amalgamated with the Department of Foreign Affairs and International Trade (DFAIT) to form the Department of Foreign Affairs, International Trade and Development (DFATD).
• DFATD became Global Affairs Canada in 2015.

Shift toward regional programming

• The Caribbean program’s shift toward regional programming started as early as 1994, and was further emphasized mid‑2008, flowing from the Government of Canada’s 2008 announcement of a multi‑year funding commitment to the region.

Table of contents

Evaluation scope and methodology

Evaluation scope

• The evaluation covered the period from 2011/12 to 2016/17.
• The assessment focused on bilateral Americas Branch projects. (The Program Logic Model can be found in Appendix 1.)
• The evaluation sample consisted of 16 projects representing 40.9% ($137 million)^{Footnote 3} of total bilateral aid disbursements ($336 million)^{Footnote 4} over the evaluation period. (Details on the sampling approach can be found in Appendix 2.)
• The scope considered Global Affairs Canada’s position in the development and policy space in the Caribbean, along with the relationship of non-project activities (e.g., policy dialogue, donor coordination) to the effectiveness of the Caribbean Regional Development Program.

Evaluation questions

Relevance

1. To what extent were Global Affairs Canada’s Caribbean regional development programming strategy and priorities aligned with the region’s development needs and priorities?

Effectiveness

2. To what extent did Global Affairs Canada’s programming contribute to achieving intended immediate, intermediate and ultimate outcomes?
3.  What factors influenced the effectiveness of the Caribbean Regional Development Program?

Sustainability

4. To what extent have, or will, the results and benefits of Global Affairs Canada’s programming continue beyond its development assistance

Efficiency

5. Are there any opportunities to improve the efficiency of Global Affairs Canada’s programming?

Cross-cutting theme

6. To what extent did Global Affairs Canada’s programming in the Caribbean region integrate and achieve results related to gender equality in its development programming?

Methodology

Global Affairs Canada’s International Assistance Evaluation Division (PRA) conducted the evaluation using a traditional mixed‑methods approach, with a focus on qualitative methods. Primary and secondary data was collected, in Canada and in the Caribbean, through a combination of sources to provide multiple lines of evidence in support of findings and conclusions. A data collection mission was conducted in the first quarter of 2017/18 in countries selected for the evaluation sample (Barbados, Guyana, Trinidad and Tobago, Jamaica, St. Kitts and Antigua).Additional details on methodology and data limitations can be found in appendices 2 and 3, respectively.

• Key-stakeholder interviews n=177
  • Conducted in person and by phone
  • Current and former Global Affairs Canada program employees and sector specialists at headquarters and in the Caribbean
  • Implementing partners based in the Caribbean and in North America, including officials from the Government of Canada, international banks, and humanitarian organizations
  • Locally engaged international donor partner officials
  • Program sector monitors
  • Government officials in the Caribbean (at municipal and national levels)
  • Beneficiaries
• Document review
  • Review of 224 program and project documents:
    • Global Affairs Canada regional, sector and program strategies
    • Program logic models and performance measurement frameworks
    • Annual reports
    • Evaluation reports
    • Project-specific documents for 16 selected projects in the evaluation sample
• Financial analysis
  • Review of planned and actual financial disbursements:
    • Financial data source: Chief Financial Officer, Global Affairs Canada, 2017.
    • Review of disbursements to CARICOM and Caribbean regional programming; focus on bilateral aid disbursement through the geographic branch
• Site visits n=5
  • Primary data was collected through visits to implementation sites for four projects in the evaluation sample. Five site visits were carried out: Guyana (n=2), Jamaica (n=3).
• Group interviews n=2
  • Two group discussions with beneficiaries of one project in the sample were held in Jamaica and Guyana.
  • Group interviews involved 45 participants (22 beneficiaries, 20 implementing partner staff members and 3 local government program officials).
• Literature review
  • Review of 35 international literature documents
    • Academic journals
    • Grey literature
    • International aid agency website.

Table of contents

Findings

Relevance findings

Programming was aligned with the region’s needs and priorities, as well as with Canada’s policies.

The program was aligned with Canada’s foreign policy objectives, as outlined in Canada’s Strategy for Engagement in the Americas, which focused on increasing prosperity and addressing insecurity. It was also aligned with CARICOM goals and the United Nations Development Programme’s analysis of the region’s needs for resilient policy approaches known to reduce human vulnerability, remove barriers to inclusive growth, and help build environmental sustainability.^{Footnote 5}

Canada’s support and expertise enabled mutually beneficial collaboration in a number of areas. For instance, collaborations in security included countering financial and organized crime activities, which could transit between the Caribbean region and Canada. In addition, in the area of finance, there was a high potential for failed states in the Caribbean, which could have led Canada to disburse even more funds in the region but with little developmental impact. In regard to health, the rising incidence of communicable diseases in the Caribbean^{Footnote 6} (HIV/AIDS, tuberculosis, dengue, hepatitis, Zika^{Footnote 7}, Chikungunya^{Footnote 8}), inadequate monitoring and the volume of traffic in both directions has made the region a dangerous disease vector, and could have presented potential risks for Canada.^{Footnote 9}

Canada was a key partner in supporting gender equality. This is a complex issue in the region, where attitudes toward gender equality    vary, and where violence against women exists alongside the marginalization of vulnerable men and discrimination against lesbian, gay, bisexual, transgender and queer communities.

Canada’s foreign policy priorities for engagement in Latin America and the Caribbean^{Footnote 10}:

• Encourage inclusive economic growth and sustainable development
• Support poverty eradication
• Promote and defend human rights
• Strengthen democracy
• Support climate change mitigation and adaptation
• Improve regional security
• Increase opportunities for marginalized groups, in particular women and girls, and Indigenous peoples

Canada’s aid program was perceived positively and recognized for its stability and consistency in supporting the same sectors over the years, thus contributing to the program’s potential for development impact.

Canada was perceived as a reliable and flexible development partner, and both the program and its staff were found to have strong reputations across the region. The program was affected by corporate changes during the evaluation period (decentralization, amalgamation and pressure to quickly shift the focus to regional operations). Despite these changes, the program’s coherent vision was reflected across its three strategies and its framework, and it supported consistent investment of resources in the same sectors over 10 years. The program was designed to foster synergies among projects and between sectors and to support activities at regional, national and community levels. The program sought the right combination of projects and the right complementarity between the initiatives.

Limited visibility restrained the potential for public recognition of Canada’s commitment and contributions to international development in the region.

Various stakeholders in the region emphasized that there was little awareness of Canadian projects beyond that of direct participants. This low degree of awareness could have arisen in part from the program’s shift to a more regional approach and to an emphasis on channelling funds through multilateral and regional organizations, both of which diluted Canada’s visibility as a donor.

In general, high program visibility can be a catalyst for development because it attracts political attention to development policy and projects in Canada and abroad. It can also support greater effectiveness of operations. By revealing good and bad practices, visibility can help maintain peer pressure and, consequently, performance.^{Footnote 11} Efforts to enhance visibility would have facilitated policy dialogue.

The program sought to provide the region with:

• the capacity to mobilize stakeholders and decision makers for policy dialogue and to share knowledge
• promotion of regional networks and partnerships that give people a voice
• supporting synergies among projects and between sectors
• an approach that addresses regional challenges to reduce the amount of work that takes place in silos across countries

The program was perceived as the glue that facilitated regional operations and collaboration. 

The regional approach to the program design was relevant and offered value, but some challenges remained.

Programming in the Caribbean was intended to promote regional cooperation, networking, knowledge generation and sharing, through multilateral organizations, regional institutions and other partnerships. The program was inherently coherent because it was strategically designed to foster synergies among projects and between sectors. It included complementary initiatives that rolled out in one or multiple countries. In its design, the program successfully avoided building up a large portfolio of small projects or a series of independent, standalone multi‑country projects. The program’s vision of a regional approach went beyond regional funds covering several countries and addressing common issues. Instead, there was effort to create a link between the various initiatives and to ensure that everyone involved was aware of each other’s work. Many donors confirmed that Global Affairs Canada’s approach was relevant in every sector in which the program had invested. They also reported that Canada’s decision to maintain bilateral projects in some of the Caribbean countries was strategic.

On the other hand, although this approach allowed for rapid disbursement of large volumes of funding on time, the design of the program limited its flexibility to adapt to country‑specific needs. In support of a mixed approach, a previous evaluation of regional programming found that the more a regional program was attuned to country-based activities, the more likely regional and country programs could reinforce each other.^{Footnote 12}

Project beneficiaries and donors indicated that a number of the program’s regional‑scale projects were not consistently adapted to the diversity of the Caribbean. This suggests that, in some cases, regional initiatives complemented by a targeted selection of country-based initiatives could have fostered more lasting results.

Through a review of regional programming by other donors, the evaluation found that the benefits of addressing shared challenges collectively made regional approaches an important complement to bilateral programs.^{Footnote 13} Other donors’ experiences also suggested that, when working with small states, neither regional investments nor bilateral projects are effective and sustainable on their own. Instead, regional investments complemented by country‑based initiatives can have better chances of success.

Caribbean regional development program approach overview

The design of the program led to several inherent strengths and weaknesses:

• Strengths
  • Promoted policy dialogue and provided a means to foster regional communication and collaboration to address common problems by using common approaches;
  • Promoted networking, knowledge sharing, and south-south cooperation;
  • Allowed, in principle, for synergies among projects and between sectors.
• Weaknesses
  • Pressures from the government at the time to rapidly shift from a bilateral to a regional design, and to prioritize rapid disbursements of funds;
  • Complex to manage internal coordination and communication;
  • Time required to achieve results at a regional scale;
  • Measuring results on a regional scale;
  • Adaptation to the diversity of the Caribbean and to the local context and needs of specific countries. Expectations of homogenous results across the region for a given project were not realistic; the degree of progress toward achieving results tended to vary among countries;
  • Potential risk to bilateral relations; the program could have missed bilateral opportunities, and this may have lessened its influence with national governments;
  • The regional approach assumed that there was a commitment for regional cooperation among government, policy makers and stakeholders. Moreover, some countries remained wary of potential costs, encroachments on sovereignty and the uncertainty of potential benefits from the regional approach.

Staff, partners and other aid workers observed that the program, as designed, allowed Global Affairs Canada to engage with countries that were not eligible for official development assistance funding; fostered regional policy dialogue; and promoted regional functional cooperation, which was a more achievable goal than regional economic and political integration. They also observed that it allowed Global Affairs Canada to get more mileage by working with existing regional and national networks, which in turn increased the program’s capacity to influence on both regional and national issues, and facilitated cross-fertilization among projects

Effectiveness findings

For the projects examined, the Caribbean Regional Development Program contributed to increased capacity across the region and achieved positive outcomes across all sectors. However, more time would be required to achieve consistent intermediate results.

Achieving results in the Caribbean has been challenging, partly because of the non-cohesiveness of the countries as a group, weak policies and legal frameworks, lack of management and institutional capacity, and other bureaucratic barriers in the region.  

Result achieved: Sustainable economic growth

projects achieved results at the immediate and intermediate outcome levels, but more time was required to achieve results evenly across all levels to support improved business development and increased trade and economic activities.

At the immediate outcome level, small and medium‑sized firms increased their capacity to compete in markets in projects such as PROPEL, COMPETE and EPIC. Outcomes were also positive with regard to projects addressing public financial management and debt management. However, the level of political support required to foster change varied across countries.

At the intermediate outcome level, projects contributed to increased business development and to increased trade and economic activities.

At the same time, of the seven projects in the sample that focused on Sustainable Economic Growth, limited synergies were noted between at least two projects that had been initially designed to be complementary. A few other projects in the area of debt management were found to compete with each other.

Result achieved: Governance and advancing democracy

Projects in the Justice sector achieved the immediate outcome of supporting increased capacity in the judicial system. More time was required to achieve results over the longer term to support an improved system of justice for women and men, as well as more effective leadership and management of accountable and gender-sensitive public institutions. Among the projects in this sector, the JURIST project demonstrated results in support of increased capacity in the justice system to address gender equality, thus moving toward an improved system of justice for men and women.

Result achieved: Risk resilience and climate change adaptation

projects achieved expected results at both the immediate and intermediate outcome levels, but more time is required to achieve results evenly across the sector in support of more effective disaster risk management.

Appendix 4 contains additional information on project-specific results.

Opportunities were missed to leverage synergies and maximize results.

Overall, findings revealed that there were varying levels of synergy across the projects examined. On paper, projects were linked. In reality, for historical reasons, many were phased in at different times and with different partners. Projects also had different mandates and ways of working. A greater emphasis on ownership of regional projects and on policy dialogue and knowledge sharing could have contributed to better results. Other donors with experience in implementing a regional approach have highlighted these interrelated concepts as being key to the success of regional programming.^{Footnote 14} Missed opportunities and efforts to strengthen and maintain links among these concepts may have led to missed opportunities to achieve regional results.

Ownership of regional projects

Based on other donors’ experience, ownership of regional programming is facilitated when program objectives are country‑specific.^{Footnote 15} Although program design can be regional, a better balance in implementation between regional and country-specific activities is preferable.^{Footnote 16} Moreover, respondents identified political acceptance from national governments and social acceptance from direct beneficiaries as being key supporting factors to effective regional project implementation. However, the sense of ownership varied across projects, among project beneficiaries and among partners (especially among national governments). According to respondents, engaging with national governments early in the implementation of projects and timely collaboration to ensure ownership were essential to exit planning and sustainability.

Policy dialogue

Policy dialogue can be defined as an: “organized deliberation between two or more actors on the allocation of values that is likely to result in new policies or modification of existing ones. In this context, policy dialogue has the possibility of transformational outcomes rather than transactional.” (Source: AUSAID, Thinking and Working Politically: An Evaluation of Policy Dialogue in AUSAID, 2013)

On one hand, donors and partners recognized the program’s role in policy dialogue. For instance, on the board of directors of the Caribbean Development Bank, Canada was known for its effectiveness in sustaining policy dialogue, which in turn enabled the program to address questions linked to its priorities in the region, in addition to fostering visibility for Canada. There was evidence of constructive policy dialogue on other regional forums and with national authorities.

On the other hand, dialogue with national governments was uneven in content and frequency. In particular, the evaluation found some confusion among program staff in differentiating the concepts policy dialogue and donor coordination. At times, this confusion prevented some staff from carrying out the specific activities separately and strategically. Although the lack of strong interlocutors remained one of the main challenges to sustained policy dialogue in the Caribbean, a number of other factors also shaped the process. For instance, there was a perceived lack of evidence of Global Affairs Canada’s recognition of the importance of policy dialogue in the program. Findings revealed that some staff members had a well-grounded understanding of the political economy of the region, which enabled them to recognize opportunities for engagement. However, staff reported that they did not have the time, space and support required to engage in substantive policy dialogue. In interviews, staff said that the move to a regional programming approach and multilateral delivery methods had led to a focus on operations management.

The last evaluation, in 2013, included a similar observation: “Policy dialogue with beneficiaries and other donors has been reasonably effective, but there was room for improvement. While the Program’s resources have increased, senior officials responsible for aid coordination at both the national and regional levels are less aware of substantial Canadian interventions.” Note that there was no recommendation on policy dialogue in this evaluation.

Knowledge sharing could have been more consistent among projects

The regional approach addressed some important challenges to sustainable development in the Caribbean: promoting cooperation and integration among countries in the region; and sharing knowledge, experience and best practices.

In many instances, however, knowledge sharing within projects was conducted informally. On current projects, knowledge requirements for training and technical assistance were not always properly identified, and staff did not seem to spontaneously know where to find the knowledge they were looking for. More effective knowledge sharing would require project participants to further interact and influence each other’s behaviour. On some projects, this interaction was not observed. Projects would have benefited from a structured knowledge‑sharing process.

Gender equality

Efforts were made to address gender equality, yet greater access to technical expertise and a stronger analytical framework would have been required to add more value to program operations.

Gender equality issues in the region are complex. Although the program included references to gender equality in most of the approved logic models for projects, its integration, treatment, monitoring, and resource allocation was uneven across projects. Indeed, of the 45 projects that were active during the evaluation period, none specifically focused on gender. Over half of the projects included at least one gender equality outcome for women and girls at the intermediate or immediate levels, and 42% had no planned gender outcomes (Figure 1).

Global Affairs Canada identified gender integration on a scale based on the project’s planned results, institutional capacity, or core focus.

Classification of gender equality integration has Specific as the highest ranking, followed by Integrated, Limited, and None

Although the program was moving toward a gender‑sensitive project design, results relating to gender equality were in the early stages for most projects in the sample. For some projects, late integration and inconsistent use of appropriate gender‑related performance indicators prevented accurate monitoring and reporting on progress of gender‑equality results. In addition, in some instances, guidance from the program to implementing partners on integrating gender equality was unclear, which contributed to uneven results. In one instance, partners were required to report on gender activities that were not included in the original design of the project.

Figure 1. Gender integration in projects 2011/12 to 2016/17 (n = 45)

Data for Figure 1

None – 42%, limited – 16%, integrated – 42%, specific – 0%

Since the previous evaluation in 2013, the program has involved experienced gender specialists in various stages of programming (design, implementation, monitoring) on a number of projects. Their involvement has benefitted the program by raising awareness, educating, and challenging mindsets about gender equality among organizations and staff.

Findings also revealed obstacles such as limited access to gender specialists both at headquarters and in the region, and budget constraints that prevented the program from hiring its own specialist. Although gender specialists should ideally be involved throughout the project cycle from design to monitoring, gender specialists at headquarters had to simultaneously oversee various programs, which limited their ability to be dedicated to all the projects in a single program.

The following project activities were considered to have potential to generate results in support of gender equality:

• making connections between family, school and communities (JUST – Jamaica)
• focusing on empowerment and vulnerability for both women and men (JURIST)
• specialized training for women entrepreneurs (EPIC)
• gender-sensitive budgeting management (CARTAC)
• gender analysis and sex disaggregated data (PRASC)*
• involvement of Global Affairs Canada’s gender specialist in the field to provide training and assistance to improve project design

*Project was not part of the sample, but several interviewees mentioned this practice.

Sustainability finding

The program’s focus on capacity building and the adoption of promising practices across the region contributed to the program’s reach and sustainable results.

Overall, the program was designed to foster sustainability, and individual projects were expected to contribute to the sustainability of other projects. Most projects in the sample aimed to build capacity through training, technical assistance, sharing of knowledge and skills, and ultimately, through changing attitudes, behaviours and practices to ensure that systems were sustainable. Several project practices were adopted across the region, which expanded the program’s reach. The program’s direct collaboration with regional institutions, synergies across projects, and building technical and leadership capacity at the national, regional and community levels were deemed key to increasing the probability of sustainable results. Promising practices include “train the trainer,” building leadership and governance. The program worked in collaboration with local leaders and actors to foster the local engagement needed to help bring about change. Despite the ever‑changing nature of institutions, collaboration through regional networks remained and thereby enhanced project sustainability.

Sustainability remains a significant challenge in the region. Among the factors that affected the program’s sustainability:

1. Demographics, including the shortage of well-trained people to run governments, in particular in countries in the Organisation of Eastern Caribbean States;
2. Little to no economic growth, high debt burden and weak debt management;
3. Distinctiveness among countries;
4. Inadequate national and regional policies and, in some instances, a lack of political will to foster reforms;
5. Weak legal frameworks;
6. Limited absorption and management capacity of regional institutions, and weak regional partners in some instances;
7. Sustainability requirements vary among countries, which makes the management of regional projects challenging;
8. Vulnerability to climate change and natural disasters.

At the same time, variable efforts were placed on sustainability across projects. Challenges to sustainability remain: project complexity and breadth, lack of flexibility to adjust to specific country contexts, length of support, and limited synergies across some projects.

The potential for sustainability varied across the program. This could in part be explained by the staff and partners’ varied levels of understanding of the steps required to achieve sustainability and by the political pressure to disburse funds rapidly. In addition, the complexity of the coordination and the breadth of regional projects caused some staff to focus on daily operations at the expense of managing for longer‑term results. In some cases, work on the details of the sustainability component of the development strategy began well after project implementation had started.

Although the regional approach was not always adapted to the country context, training and technical assistance were adapted to local needs. The short-term impact of training and technical assistance, however, was not always sufficient to foster behavioural change within organizations. Strong corporate cultures and systems would have been required to further support organizational change.

Advisory services and policy advice, which would have been necessary to support institutional strengthening, were unevenly applied across projects. For example, the EPIC project demonstrated effective mentoring, but other projects such as SEMCAR and DMAS expressed longer‑term needs for Canadian expertise and technical mentoring to support sustained results.

However, the Risk Resilience and Climate Change sector had valuable experiences in working directly with countries. This experience could be adopted by projects from other sectors. For instance, several of the capacity‑building initiatives that were linked to disaster risk management were community‑driven and were in line with country needs. The CDM-HIP, for example, worked with several national offices, and PAHO worked with each national ministry of health, with an emphasis on institutional strengthening.

Efficiency findings

The program was generally well administered and efficient in terms of financial resource management.

For the projects in the sample, management and fiduciary risk analyses were completed according to Global Affairs Canada’s rules and procedures, and projects generally benefited from timely disbursements (planned versus actual disbursements). Although there were delays and disruptions in the implementation of operations for several projects, most projects were on track to be completed on time.

The program provided a strong project management function and addressed project management problems in a timely manner. Division of work by sector allowed staff to develop the necessary expertise to advise and respond quickly to partners’ needs. Moreover, the program was seen as responsive to observations and recommendations from sector monitors.

Program assets in relation to efficiency included its work with a variety of implementing agencies, as well as its ability to influence the design and outcomes of projects, including with multilaterals in the economic sector. Program staff positively influenced the management of projects.

Interviews with staff and partners indicated that decentralization has contributed to a greater understanding of the region. The field presence of project team leaders has enhanced the quality of partnerships. Locally engaged staff’s sector and country expertise added value to the program.

Overall, projects were deemed as technically sound, well respected and appreciated.

At the same time, challenges to operational efficiency remained: the management matrix structure, communications, the emphasis on rapid and large financial disbursements, corporate constraints, information management resources, and access to technical expertise.

Matrix management structure

The program has a complex organizational structure, and most staff are located across five countries (Barbados, Guyana, Jamaica and Trinidad and Tobago, plus Canada). Given the nature of the regional program, the use of a matrix organizational structure was deemed most appropriate. In a management matrix structure, staff can find themselves reporting to several managers at once, depending on the projects assigned to them.

During the evaluation period, the matrix structure was confusing because roles and responsibilities were not always clear. Indeed, the program’s management matrix structure was perceived by all levels of staff as contributing to a lack of clarity of program resources, staff allocation and responsibility across sectors, as well as reporting lines and authorities. The management matrix structure contributed to confusion around the difference between the role and authority of sector managers and sector team leaders on specific projects. Moreover, some staff were based in a different country from their managers and the projects they were responsible for, which limited opportunities for efficiency and accountability.

The difficulties with understanding the complex management structure also extended to some partners, who reported challenges in determining who their main program contact should be. For instance, some partners reported receiving competing directives from various staff members at headquarters and in the field.

Communications

The vast majority of respondents reported a lack of communication and coordination among some projects, as well as the absence of a formal coordination mechanism among implementing partners. These gaps led to duplication of efforts on at least two projects in the sample and to confusion about the mandates of two projects in the economic sector.

Some opportunities were missed with regard to internal communication and corporate knowledge transfer. Staff reported that a lack of clear top‑down information sharing from management limited their understanding of the rationale for decisions and of the program’s current situation and future direction.

Emphasis on rapid and large disbursements

Following the former government’s announcement in 2008 of $600 million for Canada’s aid program in the Caribbean, the program was pressured to quickly design a regional program and focus existing and new projects accordingly. This pressure had a negative influence on some of the projects over the evaluation period, including a lack of clear objectives and targets; a lack of ownership of regional projects from national authorities in a few cases; the need to refocus resources mid‑project in other instances; and finally, a few older projects were reframed to fit under the regional approach, which destabilized operations.

Corporate constraints

Despite the many benefits of decentralization, it had limited influence on the improvement of the field’s ability to respond quickly through its programming operations. This was mainly because, in practice, most decision‑making took place at headquarters, which resulted in operational delays. As well, over the evaluation period, the program adjusted to significant cuts in human resources. Increased workload was further intensified by competing requests and changes in priorities originating from headquarters. Staff underlined some persisting post‑amalgamation issues that affected program efficiency. These issues included synchronization issues between the project and reporting cycles, which caused the program to struggle to provide expected input for the Departmental Performance Report before the corporate annual reports on the project were completed. Financial management through an agency in the United States also led to unforeseen delays.

Additional corporate constraints and challenges included numerous requests and decisions communicated to the program from the department with little explanation or guidance, slow planning with little opportunity for analysis, a lengthy and labour-intensive planning process for new projects that was not conducive to flexible and quick cooperation in response to the region’s needs, and heavy bureaucratic procedures for contracting and approval processes. As a result, staff experienced confusion regarding the programming’s direction, which led them to focus on managing processes and reporting to the detriment of the qualitative analysis necessary for optimal project planning and management.

Information management

Inadequate or absent information management tools and process inefficiencies in the missions meant that there was less time to focus on the actual content of the programming. Lack of access to information management systems meant that staff in the region had to ask headquarters colleagues for help to access information and documents that they could not access, which reduced efficiency in the region and at headquarters. This point has been raised in the past, as well as in recent country program evaluations.

Technical expertise

Another issue that influenced program operational efficiency was monitoring and access to technical expertise at the sector level. Although monitoring reports provided useful analysis at the project level, few provided valuable analysis at the sector level. Such reporting would have helped ensure that projects were aligned with changes in a given sector, but requests for sector-level analysis were inconsistent across the program. The clarity of guidance provided to sector monitors about mandates and deliverables varied and was limited in some cases.

Finally, other frequently mentioned challenges to the program’s operations included limited access to training for locally engaged staff, limited travel budget for staff and management, and limited access to the expertise of sector specialists from headquarters.

Table of contents

Conclusions

Relevance

Global Affairs Canada’s Caribbean regional development programming strategy and priorities were aligned with the region’s needs and priorities. As underlined in the 2004 evaluation, a regional approach to development programming was appropriate for the Commonwealth Caribbean region, and this was still the case during the evaluation period. The program embraced the complex, multi-stakeholder reality of the region, which was key for the type of innovation necessary for sustainable development. The program has benefitted the region through the way it collaborated with its partners and provided expertise. At the same time, adding more country-specific projects to the regional program design could have facilitated the achievement of results.

Effectiveness

Global Affairs Canada’s programming contributed to achieving intended immediate outcomes. The program cycle was too short to allow the achievement of most of the intermediate outcomes. Achieving those outcomes would have usually required longer cycles than expected because of the nature of international development. Evidence indicated that projects supported by Global Affairs Canada achieved results during the evaluation period, but more time was required to see significant transformations.

One factor that influenced the effectiveness of the program was the regional approach. The regional approach could help achieve even greater results if the program were to further reinforce support for the following: ownership (political acceptance from governments and social acceptance from direct beneficiaries, country participation and coordination), knowledge sharing and policy dialogue. Donors’ experiences confirm that, when working with small island states, neither regional investments nor bilateral projects can be sustainable on their own; regional investments complemented by country‑based initiatives have better chances of success.

Although project team leaders (PTLs) played a pivotal role in the program, variances in individual approaches and work methods had an impact on operational effectiveness. PTLs differed in their perception, definition and promotion of the concepts of gender equality, partnership, capacity building, policy dialogue, and knowledge sharing.

Addressing Global Affairs Canada’s gender‑equality policy objectives would have required an in-depth analysis and understanding of gender norms and relations and how they contributed to inequalities between women and men in the Caribbean. Because of government priorities at the time, the program could not devote the necessary resources to conduct this analysis. Beyond individual projects, the program’s objectives with regard to reducing gender inequalities were unclear. The lack of a clear plan articulating how the program supported the achievement of gender‑equality outcomes in the region meant that programming efforts related to addressing gender inequality were fragmented and reactive, rather than coherent and strategic.

Sustainability

The program’s focus on capacity building combined with the regional approach, in some instances, resulted in the adoption of promising practices across the region, as well as collaboration of regional networks thereby contributing to sustainable results. That said, the development pace is slow in the Caribbean; therefore, sustainability tends to take longer to achieve. The region is complex and culturally and politically diverse. International cooperation in the region is a multi‑faceted, risky and long-term endeavour, but the program could have increased the likelihood of reaping continued benefits from its portfolio of development investments.  Specifically, the program could have undertaken more sustainability analysis at the project level. It could also have addressed how capacity building was conceived and treated across projects.

Efficiency

The efficiency of the Caribbean Regional Development Program in the field was limited in part by corporate constraints over which it had limited control.

The selection and design of a regional programming approach was initially intended to increase efficiency, because it facilitated the process of disbursing large volumes of funding through multilateral organizations. However, external pressure at the time to shift from a bilateral to a regional program design in a short period of time and to prioritize rapid disbursements of funds created challenges for efficient use of program resources within projects and for achieving results

The nature of the regional approach means that regional projects need to have increased influence on national enabling environments. This in turn requires more targeted dialogue, beyond the board of directors of the Caribbean Development Bank and beyond project operations.

Based on the consensus of staff interviewed, the program should review some of its current practices related to internal communications and the management matrix to improve efficiency.

Table of contents

Recommendations and management response

Recommendation 1

Complement regional programming with country-specific investments and include national governments in regional projects. (Relevance)

Agreed: The Caribbean Regional Development Programme recognizes the importance of addressing regional objectives and issues by working closely with Caribbean countries to respond to their needs and guide investments. Tailored country-specific investments may be necessary to respond most effectively to specific needs, for example in the context of reconstruction after a disaster.

Recommendation 2

Develop a short plan of action articulating how the program intends to promote knowledge sharing, visibility, policy dialogue and capacity building, to better integrate these into daily operations, and ensure common understanding of these concepts among staff and partners. (Effectiveness - Sustainability)

Agreed: The programme will develop a short Knowledge and Policy Development Action Plan that includes the promotion of knowledge sharing, visibility, policy dialogue, and capacity building. This will be complemented by a Communications Strategy which seeks to increase the visibility and understanding of Canada's international assistance contributions in the region and at the individual country level. 

Recommendation 3

Develop a short plan of action articulating how the program intends to promote and integrate gender equality into daily operations, and ensure common understanding of gender equality among staff and partners. (Effectiveness)

Agreed: In keeping with Canada's Feminist International Assistance Policy, the Caribbean Regional Development Programme will develop a Gender Action Plan which indicates how international assistance investments will be aligned to meet the GE targets of the policy. The GE Action Plan will include GE integration in daily operations, including GE analysis to inform project design, consultation, as well as monitoring and implementation. The GE Action Plan will also include Policy Dialogue and Communications, indicating how the programme will engage government, civil society, and regional partners on GE. Canada's approach to international assistance will also be included, and how it will link to the GE communications and public diplomacy efforts of Canada's Heads of Mission in the region.

Recommendation 4

Review the current management matrix structure to improve communication, coordination, and cohesion among staff. (Efficiency)

Agreed and completed: The Caribbean Regional Development Programme team has reviewed and updated its matrix management structure to ensure greater efficiency (including better communication, coordination, and cohesion among staff). The modified matrix management structure has been fully consulted with programme. It more clearly elaborates staff roles and responsibilities, as well as the reporting relationships of staff to managers. It will be further updated to include aspects of the 2018 re-organization of the Central America and Caribbean Bureau team at headquarters.

Table of contents

Considerations for future programming

Considerations for regional programming in geographic branches

1. Given that countries in a region are not all equally motivated by regional cooperation, programs may consider regional programming with select countries that are more supportive of this approach.
2. To inform the program design, facilitate coordination and avoid duplication, regional programs could map current and planned initiatives in the region including funding from all donors.
3. Simplicity of project design at the national and regional level can allow for better accommodation of the limited institutional capacity of implementing partners. It also helps facilitate coordination with multiple stakeholders, as well as project monitoring and evaluation.

Considerations for international assistance programming

1. Decentralized programs that provide field-based teams in partner countries with advisory and managerial capacity, and with program and financial authority, may be better equipped to provide aid efficiently^{Footnote 17}.
2. The presence of sector experts in the field may help programs deliver on commitments, including those related to the action areas set out in the Feminist International Assistance Policy, such as gender equality and empowerment of women and girls.
3. Access to corporate information management tools and resources in the field is essential for decentralized programming to function effectively and efficiently.

Table of contents

Appendices

Appendix 1: Programming frameworks

Appendix 2: Methodology – Sampling approach

A purposive sampling approach was used to ensure representation of thematic sectors, delivery channels, and investment types. In consultation with representatives from the program, 16 projects were selected out of the overall portfolio of 45 projects. The evaluation project sample represents 40.9% ($137 million)^{Footnote 18} of Global Affairs Canada’s total bilateral aid disbursements ($336 million)^{Footnote 19} toward development in the Caribbean between 2011/12 and 2016/17.

Appendix 3: Limitations

1) Logistical constraints in the Caribbean region limited the size and representativeness of the evaluation project sample, as well as access to data from primary sources

Program size, the geographic spread of countries in the region, time, travel, and budget constraints limited the possibility of site visits and the conduct of interviews with key stakeholders in all 14 participating countries.

Mitigation strategy: A sample of 16 projects out of the 45 was selected, and 6 countries were selected to be visited for data collection and key‑stakeholder interviews. Care must be taken, however, when drawing inferences at the program level for all projects based on the sample.

2) Challenges were encountered with the program’s tracking of documents at the project and program level

The evaluation reviewed over 200 documents and reports at the program and project level that were shared by GAC staff, partners and sector monitors. Some data gaps encountered could be attributed to various causes such as the absence of a report, or obstacles in locating and accessing information in the program’s document repository. In addition, comprehensive project‑level data was not always available because of the limited number of monitoring and evaluation reports at the project, sector, and program levels.

Mitigation strategy: When the available data was insufficient, the evaluation relied heavily on interview data collected during the field visit.

3) Data availability was a challenge in the region

Attribution of results from regional projects largely depended on quality and availability of data. In the Caribbean context, gathering data to represent the entire region was a challenge, not only because of the number of countries but also because of limited statistical capacity, especially in the case of small states. This lack of capacity could have had adverse impacts on data availability and reliability.

Mitigation strategy: The evaluation strived to triangulate evidence as much as possible.

4) Limited number of previous evaluations

In addition to limited reliable data sources, there was a limited number of previous regional and project evaluations, especially for small states, where statistical capacity was low to begin with.

5) Availability of performance measurement framework

No program‑level performance measurement framework was available for the timeframe being evaluated.

Mitigation strategy: The evaluation relied on logic models available at the project and program levels, interviewed more stakeholders (n=177), and reviewed more program documents (n=224) than initially planned.

6) Lack of data and resources required to conduct a cost‑benefit analysis

Baseline data was not available to assess the benefits of the regional approach.

Mitigation strategy: The evaluation strived to triangulate evidence as much as possible.

Appendix 4: Select sample of results achieved in each sector

Table of contents