Evaluation of the Anti-Crime and Counter-Terrorism Capacity Building Programs, 2015-16 to 2021-22

Evaluation Report

Prepared by the Evaluation Division (PRA)
Global Affairs Canada 
April 25, 2023

Acronyms and abbreviations

ACCBP

Anti-Crime Capacity Building Program

ADM

Assistant Deputy Minister

AVCs

Annual Voluntary Contributions

COM-B

Capability, Opportunity, Motivation model of Behaviour change

CTCBP

Counter-Terrorism Capacity Building Program

DG

Director General

DND

Department of National Defence

FIAP

Feminist International Assistance Policy

GAC

Global Affairs Canada

GE

Gender Equality

GoC

Government of Canada

Gs&Cs

Grants and Contributions

HOM

Head of Mission

IAE

International Assistance Envelope

ICC

Counter-Terrorism and Anti-Crime Capacity Building Programs’ Directorate

ICD

Bureau for International Crime and Counter-Terrorism

IFM

International Security and Political Affairs Branch

ISC

Interdepartmental Steering Committee

JCLEC

Jakarta Center for Law Enforcement Cooperation

MES

Middle East Engagement Strategy

NAP WPS

National Action Plan on Women, Peace and Security

OAS

Organization of American States

OGDs

Other federal government departments and agencies

PRC

Priority Review Committee

PSD

Public Security Directorate

PSF

Peace and Stabilization Fund

PSOPs

Peace and Stabilization Operations Program

RCMP

Royal Canadian Mounted Police

TAP

Technical Assistance Partnership Initiative

SALW

Smalls Arms and Light Weapons

UK

The United Kingdom of Great Britain and Northern Ireland

UNODC

United Nations Office on Drugs and Crime

Executive summary

This evaluation examined Global Affairs Canada’s Anti-Crime and Counter-Terrorism Capacity Building Programs during the period 2015-16 to 2021-22. The objective was to determine the extent to which these ongoing Programs were fit for purpose in enabling responsive and agile security capacity building. This report presents findings, conclusions, recommendations, and considerations for departmental learning and support program improvements.

The evaluation found that the Programs were an important part of the Government of Canada’s security assistance portfolio. They added credibility and visibility to Canadian international engagement and contributed to strengthening global security. The Programs were uniquely and appropriately designed as a horizontal initiative to pool Canadian security expertise and influence across federal organizations under one interdepartmental governance structure. The Programs’ original setup supported internal and external coherence, helped initiate and sustain relationships in sensitive contexts and profiled Canadian values. However, the effectiveness and efficiency of this horizontality waned over time, as federal partners engaged less in guiding program direction and accessing program funding. The Programs focused increasingly on grants and contributions. This decreased the Programs’ ability to make strategic and coordinated investments aligned with Canada’s overall national security priorities. The Programs also expanded their reach beyond security response capacity to addressing broader root causes of terrorism and crime with a development assistance focus, which led to spreading resources across multiple thematic and geographic areas and target groups, and hence moving away from their original niche.

Over the evaluation period, the Programs underwent significant organizational instability and staffing turnover, and their operating environment became more complex and administratively demanding. The Programs grew in funding and engaged actively in contributing to new government priorities, with heavier administrative workloads decreasing the ability of staff to take a strategic focus to programming. The Programs did not have sufficient flexibility to quickly disburse or realign funds, which led to funding lapses. Since 2021-22, program management has focused on stabilizing the capacity building team, re-engaging federal partners and limiting lapses. 

The Programs responded to a wide variety of capacity building needs, although their relevance to potential security threats to Canada and Canadian interests or development assistance objectives was not always evident. The Programs achieved strong results at the immediate outcome of building operational skills, increasing knowledge and awareness, supporting domestic and regional coordination and knowledge sharing. Program investments led to some notable examples of change in institutional practices, which added value to existing security operations in partner countries and contributed to reducing terrorism and criminal threats. The Programs enabled implementing partners to provide gender-responsive capacity building and to advance global knowledge on integrating gender in the traditionally male-dominated security sector. 

Background

Key Program Features

Figure 1:

The Anti-Crime Capacity Building Program (ACCBP) and the Counter-Terrorism Capacity Building Program (CTCBP) (the Programs) are a part of Canada’s whole-of-government effort to promote international security, and to protect the safety of Canadians and Canadian interests at home and abroad. 

The Programs recognize that the security of Canada is linked to the security of other states and provide a means to identify and address terrorism and criminal threats at the source. Both Programs provide assistance to foreign states that have the political will but lack the resources or expertise to prevent and respond to security threats posed by transnational organized crime and terrorism. The assistance includes training, equipment, technical and legal assistance and other capacity building measures that enable source and transit countries to reduce the negative impacts of security threats.

The Government of Canada established CTCBP in 2005 and ACCBP in 2009 as international security assistance programs. These two complementary and mutually supportive Programs are ongoing and are managed centrally at Global Affairs Canada (GAC), under one Director General in the Bureau for International Crime and Counter-Terrorism. The Bureau also has responsibilities for leading the development and coordination of Canadian foreign policy on international dimensions of crime and terrorism. 

ACCBP and CTCBP share one set of Terms and Conditions and a common interdepartmental governance structure that leverages subject matter expertise from across federal organizations to improve domestic and international coordination for anti-crime and counter-terrorism assistance (Annex 1). The Programs provide transfer payments to a wide range of eligible recipients for various types of capacity building measures that include technical instruction, advice, research, awareness raising, sharing of practices, provision of equipment among others. They support other government departments’ international capacity building engagements by reimbursing the incremental costs related to the assistance provided. 

The Programs implement projects aimed at several thematic areas, which are subject to periodic review and re-assessment by the Interdepartmental Steering Committee. From 2015 to 2022, these included the following:

Program resources

Figure 2:

2021-22 Programming Budget, by Funding Envelope and Vote

AVCs: Annual Voluntary Contributions

HSE:  Human Smuggling Envelope

MES: Middle East Engagement Strategy, which included Vote 10 funding to support Canadian Armed Forces’ capacity building activities in the Middle East under Operation IMPACT.

Budgets and Envelope Structure

The Programs were initially set up with one budget allocation each, funded under the Government of Canada’s International Assistance Envelope (IAE) as non-Official Development Assistance. ACCBP subsequently included funding dedicated to human smuggling (2010-present) and the Canadian Initiative for Security in Central America (2012-2017), while CTCBP added funding for the Sahel region (ongoing) and the Middle East Engagement strategy (2016-present). In addition, the Programs managed Canada’s ongoing annual voluntary contributions to the Organization of American States (OAS) and the United Nations Office on Drugs and Crime (UNODC), and the Afghanistan Counter-Narcotics Program (2007-2017). Between 2015-16 and 2021-22, the Programs disbursed $420M^{Footnote 1}, of which 71% came from the International Assistance Envelope (IAE).

Geographic Focus

The Programs had a global mandate, with distinct geographic areas of importance to address threats posed to Canadians and Canadian interests. Between 2015-16 and 2021-22, ACCBP focused its disbursements on the Americas (73%) and Asia (19%), while CTCBP focused on the Middle East (56%), Africa (24%) and Asia (17%).

Vote 1 and Vote 10 Programming

ACCBP and CTCBP funded programming through different categories of Global Affairs Canada’s parliamentary-approved expenditures, called votes. This included grants and contributions implemented by eligible implementing organizations from the Vote 10 authority, and the transfer of funds to federal organizations through the Vote 1 authority.

The Programs disbursed $391M^{Footnote 2} in Vote 10 grants and contributions between 2015-16 and 2021-22 ($248M for CTCBP and $143M for ACCBP). The Programs funded over 70 organizations, including multilateral and non-governmental organizations, third-donor governments, cooperation platforms, research institutions, consultancy firms and others. One-half of disbursements were regional or multi-country and the other were bilateral.

Other government departments and agencies accessed the Programs’ funding through the Vote 1 programming mechanism. While these funds were housed within GAC’s Vote 1 operating funds, they were not considered operating expenditures and were used to reimburse federal partners for incremental costs such as salary, travel and expenses directly related to capacity building work. Between 2015-16 and 2021-22, the Programs provided $29M^{Footnote 3} in Vote 1 funding ($11M for ACCBP and $18M for CTCBP), of which 54% supported bilateral engagements. The funding excluded equipment, venue rental and other support provided through Vote 10.

Evaluation Scope and Methodology

Evaluation Scope and Objectives

Evaluation Objectives

The objectives of this evaluation were to:

1. determine the extent to which the design and implementation of the Anti-Crime and Counter-Terrorism Capacity Building Programs supported Government of Canada priorities and commitments and enabled a responsive and agile approach to international security capacity building;
2. examine the relevance, performance, sustainability and coherence of programming interventions, and;
3. identify lessons and recommendations for improved program management and delivery.

Evaluation Scope

The scope of the evaluation focused on capacity building funded by the Programs through Vote 1 and Vote 10 funding authorities between 2015-16 and 2021-22. The evaluation examined the larger fit-for-purpose and results issues of the Programs and not performance within any individual funding envelope or thematic area of anti-crime or counter-terrorism. The evaluation used one common methodological approach for both Programs in response to their shared Performance Information Profile, Logic Model and Performance Measurement Framework.

Evaluation Approach

The evaluation adopted a utilization-focused approach to address the operation needs and information gaps of the evaluation users. It was conducted by the Global Affairs Canada’s Evaluation Division (PRA). The core evaluation team was supported by a methodological advisor specializing in contribution analysis and by local consultants in Jordan and Niger to support field data collection in these case study countries.

The evaluation applied a mixed-methods design using qualitative and quantitative methods. Data from different lines of inquiry were triangulated to validate findings and draw conclusions. To gather data on capacity building outcomes, which the 2016 evaluation showed to be limited, this evaluation employed contribution analysis grounded in behaviour change theory to examine how and why capacity building measures made contributions to the development of the partner country’s capacity. The application of the Capability-Opportunity-Motivation model of Behaviour (COM-B) allowed the evaluation team to build a theory of change behind each capacity building project. This enabled the evaluation team to understand the mechanism through which projects brought about change in skills, awareness and/or capacity at the behavioural, organizational and systems level (Annex 2). The COM-B model purposefully identified and gathered data on factors and conditions that supported or inhibited the achievement of targeted change. Finally, it allowed better assessment of the change in quality of targeted outcomes, as external factors can have a significant impact on quantitative security outcomes.

Evaluation Questions

Methodology

Global Affairs Canada and Government of Canada Document Review

The evaluation included a review of relevant departmental and GC documents:

• GoC strategies, policies and national plans
• Documents on departmental strategies, processes and systems relevant to the Programs
• ACCBP and CTCBP program-level strategic and operational documents (for example, Terms and Conditions, Priority Review, Standard Operating Procedures, memos, templates, project selection rubrics, meeting minutes, reports)
• Evaluations and audits of Government of Canada security strategies and programming relevant to capacity building.

Key Informant Interviews 

A total of 63 semi-structured individual and small group interviews were conducted for the evaluation:

• 28 interviews with former and current Bureau and Program management and staff
• 5 interviews with representatives from Global Affairs Canada
• 10 interviews with representatives from other government departments (Vote 1 partners)
• 20 interviews with representatives from implementing partner organizations (Vote 10 partners).

All interviews were conducted using virtual platforms and a standardized consent protocol for the collection, storage and use of information.

Country Case Studies

Five country-level case studies (Guatemala, Jamaica, Jordan, Niger, Thailand) were used to examine Programs’ relevance and performance using the COM-B model. Case studies included 45 projects and 134 individual and group interviews, and were chosen based on a sequenced, purposeful sample. In-depth case study assessments involved the following:

• detailed project file review
• development of project-level theory of change for each targeted beneficiary group
• field visits and direct observations
• interviews with programming beneficiaries, staff from implementing partners, Canada’s missions, other government departments, and others.

Scan of Donor Capacity Building Practices

An environmental scan of donors active in security capacity building was used to identify good practice and compare the Programs’ delivery approach. The selection of donors was informed by the case study visits. The scan focused on Australia, Denmark, France, Germany, Netherlands and the United Kingdom and included the following:

• literature review of reports, evaluations and other publicly-available information of donor capacity building programming in these countries
• follow-up interviews with donor representatives.

Literature Review

The evaluation included a literature review on the effectiveness of security capacity building measures and key international sources of data on global crime and terrorism. The review included:

• Published research and analyses of the outcomes of capacity building efforts
• Best practice guidelines on capacity building
• United Nations resolutions and reports specific to counter-terrorism and anti-crime measures
• Data and analyses from the Organized Crime Index, Global Terrorism Index, Sustainable Security Index and the Fragile States Index.

Global Affairs Canada Statistical & Financial Analysis

Departmental statistical, human resource and financial data through Power Business Intelligence platforms on Program disbursements and human resource capacity were used to:

• profile program investments over the evaluation period
• identify the extent to which program budgets were utilized
• examine human resource management.

Evaluation Limitations and Mitigation Measures

Findings: Relevance

Anti-Crime and Counter-Terrorism Capacity Building Programs directly and indirectly contributed to meeting Canada’s objectives in several security and foreign policies and whole-of-government strategies between 2015-16 and 2021-22.

Figure 3:

The Programs’ original set-up as flexible horizontal initiatives to lead and coordinate Canada’s capacity building expertise and influence was relevant.

The Programs were set up as an agile and flexible means to reduce terrorism and criminal threats before they impacted the security of Canadians and Canadian interests. The literature review and interviews confirmed the overall value and relevance of security capacity building assistance and its potential to support local ownership of capacity building outputs, even if little empirical evidence of its long-term impacts has been available to date. Interviews and case studies also supported the continued relevance of the Programs’ original design, which allowed the Government of Canada to bring together its capacity building expertise to achieve program objectives and to support other government priorities through one secretariat. The Programs’ Terms and Conditions provided for significant flexibility in administering program resources, which was viewed by interviewed staff and implementing partners as essential to respond and adjust to evolving criminal and terrorism threats. 

The Canadian approach was consistent with other donors’ anti-crime and counter-terrorism capacity building that similarly adopted a whole-of-government approach aligned with national interests and had interdepartmental governance, although some were positioned within broader peace and stabilization funds (Annex 3).

The Programs contributed to increasing Canada’s visibility and credibility internationally and allowed for relationship building in different contexts.

The Programs added visibility and credibility to Canada on the ground. According to case study analysis, program beneficiaries were aware of Canada’s capacity building contributions. They perceived Canadian support positively, as having a neutral policy stance, and expressed interest in continued collaboration to build their capacity.

The evaluation found that Global Affairs Canada and other government departments leveraged the Programs’ links to partner countries’ institutions and networks. Officials from other federal departments based in Canadian embassies took advantage of capacity building programming to further operational interactions and gain cooperation in the field. The Programs provided an important entry point and a way to sustain relationships for federal partners, and also served as vehicles to deliver equipment and other support for their international operations. For example, ACCBP supported the long-standing cooperation between Canadian and Jamaican defence forces that led to the establishment of several centres of excellence, expanding anti-crime and counter-terrorism capacity in Jamaica and the Caribbean. Departmental staff at some missions leveraged the Programs to increase Canada’s visibility and engagement with partner countries. While there were examples of the Programs’ contribution to Canada’s diplomatic priorities and influence, case studies and interviews showed that ACCBP and CTCBP were not leveraged fully due to missions’ limited knowledge of program priorities and funded projects.

Figure 4:

ACCBP and CTCBP programming coverage between 2015-16 and 2021-22

Source: Chief Financial Officer data, July 2022. Programs’ validated list for Vote 1 project disbursements.

The Programs did not have an articulated strategy to translate thematic priorities into project investments, which left implementing partners, de facto, guiding the programming focus.

The Programs’ thematic priorities remained largely consistent since their inception and addressed a range of threats and factors contributing to terrorism and transnational crime. Following the 2016 evaluation recommendation to use limited budgets strategically by concentrating on fewer thematic priorities and regions with longer-term investment to increase impact, the Programs adopted a new approach that set priorities for 3 years rather than annually. The 2017-18 priority review document reaffirmed broad thematic priorities in each region of focus, but did not include prioritization, identification of specific objectives, size of programming, target actors, or criteria for phase out. The next priority review, undertaken for 2022-2025, had similar limitations.

Interviewed Interdepartmental Steering and Priority Review Committee members spoke about a missing element in translating the Programs’ broad thematic priorities into projects investments and that the Programs did not leverage their subject matter expertise and intelligence to steer programming. During the evaluation period, the roles of the governance committees in defining program direction were reduced. Interviews and a document review showed that the Interdepartmental Steering Committee did not function from 2016-2017 until 2021, with the Programs relying on a less formal annual planning process. The Programs’ revival of the Steering Committee in the fall of 2021 was positively received by interviewed federal partners, but deemed insufficient without wider opportunities to inform programming and learn from past investments. The Priority Review Committee transformed from a forum to prioritize proposals several times per year, to an electronic review of single proposals with limited insight into overall programming, lacking purpose and perceived as a “red flags” exercise.

The Programs’ annual planning process relied on available unsolicited proposals, received by individual officers. Interviewed program staff noted that such a process worked well with limited funding available to support new programming and reduced administrative burden. However, it resulted in implementing partners’ proposals having a large place in leading the direction of program investments and spreading resources across multiple thematic and geographic areas, with some programming reflecting development assistance approaches. In contrast, the majority of interviewed implementing partners reported not knowing which issues or regions the Programs sought to support, making it difficult for them to align proposals with Canada’s priorities. Vote 10 grants and contributions partners reported submitting ideas to see “what sticks” or prioritizing new phases of existing projects in the absence of clear direction. The Programs used calls for proposal with the introduction of the Middle East Envelope in 2016 and the Human Smuggling Envelope renewal in 2018. Analysis of Priority Review Committee comments on the 2018 call for proposals showed that partners commonly submitted proposals in areas with limited relevance to Canada’s security interests. The Programs began to take steps to have more strategic engagement with larger Vote 10 implementing partners in 2021-22.

Figure 5:

Programs’ Vote 1 projects with other government departments and agencies and funding lapses

Note: The number of Vote 1 projects excludes those with the Canadian Commercial Corporation.

Source: Financial Status Reports for data on funding lapses, extracted in January 2023; Programs’ validated list for Vote 1 project disbursements

Median project disbursement, 2015-16 – 2021-22

• Other Government Departments - $0.07M
• Grants and Contributions - $0.67M

Decreased federal partner involvement in governance and delivery of capacity building resulted in missed opportunities for the Programs to support coordinated Canadian security assistance.

The Programs’ setup allowed the Government of Canada to improve domestic and international cooperation and coordination for capacity building assistance. However, interviews and case studies identified that the Programs’ potential to support coordinated Canadian security assistance was not fully tapped into and was affected by staffing constraints. There were limited examples of joint needs assessment missions and interdepartmental programming, and was further affected by decreased federal partner access to program funds.

Access to program funding by other government departments (OGDs) began to decline prior to and during the COVID-19 pandemic. OGD interviewees confirmed decreased interest in accessing program funds, citing heavy bureaucratic requirements relative to small project amounts, misalignment with Global Affairs Canada’s priorities, and negative impacts of funding delays on relationships with partner countries. Interviewees observed that the Programs’ overall framework for OGD engagement was inadequate in relation to program objectives, as capacity building activities took staff away from operational duties with limited reimbursement for associated costs, limiting their ability to release staff and requiring a Vote 10 partner for equipment and other expenses. On the other hand, the Programs lacked accountability mechanisms for OGD project implementation, with cancellations occurring late in the fiscal year. The Programs were also affected by changes in departmental interpretations of eligible Vote 1 expenses. Recently, several OGDs pursued expanding their Vote 10 authorities to support international engagement, moving away form the original intent of the Programs.

The Programs did not define their role and value-added in supporting immediate security response capacity versus long-term development-focused institutional and societal change.

The Programs initially focused on building security response capacity of partner countries and international institutions, but later expanded to tackle broader issues and root causes of international terrorism and crime. Interviews with staff, implementing partners, and a document review demonstrated that the Programs’ role became less clear over time. The focus broadened from immediate security goals and solutions to programming across the security – development continuum, including community resilience and civil society engagement.

The evaluation found that there was room to better define and balance contributions between supporting short-term security outcomes and long-term capacity development, by leveraging the flexibility of the Programs’ Terms and Conditions and linking with relevant development assistance work. The previous evaluation and literature review showed that short-term efforts were ineffective when they lacked institutional depth and long-term political engagement. On the other hand, the Programs’ long-term efforts were hampered by a limited budget, lack of articulated investment priorities, and limited field and policy connection required for sustainable change.

Findings: Coherence

A whole-of-government strategy provided the impetus and mechanisms for greater alignment and coordination of Canadian engagement.

Figure 6:

*Source: Institute for Economics and Peace public data on Global Terrorism Index (2022). The index is measured on a scale from 0 to 10, with 10 representing the highest measurable impact of terrorism.

The Programs lacked mechanisms to enable coherence thinking and collaboration with other Global Affairs Canada’s international assistance programming working in complementary areas.

The 2016 evaluation of the Programs identified a need for ACCBP and CTCBP to define and communicate their strategic niche in relation to complementary security and development programming and to bring departmental efforts into a “whole of department” approach.

This evaluation found that efforts and mechanisms to facilitate coherence thinking and collaboration remained limited as the Programs moved further away from their security focus and comparative advantage of flexible and rapid funding towards more development-focused objectives. The Programs primarily involved other international assistance programming teams and missions for feedback on project proposals at the working level. Interviewed management and staff identified the creation of integrated country frameworks, integration of explicit security programming in the Triple Nexus Working Group, and identification of dedicated security coordinators at missions as means to facilitate coherence.

Coherence of the Programs’ investments with other departmental assistance and diplomatic efforts was strong when included as part of a government-wide strategy.

The evaluation found that coherence within other international assistance engagements was notably stronger in the context of the Middle East Strategy, which had a significant security focus with clearly outlined priorities, common planning and reporting documents and a dedicated coordination resource in the field. The Jordan case study provided a strong example of the Programs’ value added in their coordinating role under a common framework of objectives for Canadian engagement. Through the Programs’ only dedicated regional coordinator, CTCBP effectively bridged efforts of mission staff, federal partners, grants and contributions partners, and other donors, and shared good practices among stakeholders.

In contrast, the Niger case study identified that positive project results had little prospect to continue in the country’s fragile context affected by regional insecurity. Security capacity-building literature underscored that countries most in need of security assistance were those least able to benefit from it as a result of complex drivers of instability, insecurity and poverty. There was also consensus within academic and implementing partner work that security assistance in fragile contexts should be part of holistic efforts that include security, equitable economic development and others. This was also reflected in the OECD/DAC principles for engagement in fragile states. While there was little opportunity to expand coherence due to limited development programming in the country and lack of an overarching strategy for the region, new priorities were emerging at the time of the evaluation led by Global Affairs Canada’s geographic bureau and the International Coalition for the Sahel.

Figure 7:

2020-21 Sector disbursements, by Branch

Source: Statistical Report on International Assistance 2020-21; Chief Financial Officer data, July.

The Programs took steps to increase coherence with other security assistance programs, with efforts largely focused on administrative processes and project proposal reviews.

The Programs were designed to complement and be delivered coherently with other security assistance programming due to common support to beneficiary states and shared implementing partners. The previous evaluation noted little coherence within Global Affairs Canada’s security assistance programs, which were designed independently in support of different Government of Canada commitments, and were guided by separate terms and conditions, standard operating procedures, project management and reporting tools. ACCBP and CTCBP were the only security programming at Global Affairs Canada that adopted a Fiduciary Risk Evaluation Tool (FRET) in 2019 as part of their risk management practices.

As a result of common implementing partners, the Programs initiated the Security Branch coherence programming group in 2019, with a mandate to align administrative processes and project templates, which created difficulties for the common partners. Efforts to coordinate at the strategic and relationship-building levels between security assistance programs lacked established mechanisms, which resulted in no follow-through. For example, the Programs’ lacked policy capacity and support as they were primarily staffed by program officers. The Integrated Peace and Security Plans were only available for Peace and Stabilization Operations Program’s focus countries and served as an inventory rather than a strategic planning tool. Collaboration opportunities for senior management’s on strategic issues in the Security Branch were also reported to be limited. Much of the coordination happened through feedback on proposals at the working level. 

The Programs made efforts to engage with other donors and partner states through monitoring visits, joint programming and various coordination fora.

The evaluation identified notable examples of engagement with donors and beneficiary states by individual program staff, who made good use of monitoring visits to meet with partner country institutions, implementing partners and other donors. This was further demonstrated through the work of the CTCBP regional coordinator in Jordan, whose efforts helped identify priorities for Canada’s contribution and ensured alignment of interests and knowledge sharing with other donors (including by leading donor coordination fora in security sectors). The Programs funded joint donor programming led by the United Kingdom, the United States and France, and the Programs’ presence at some key global counter-terrorism fora supported dialogue with external partners.

Overall, the evaluation found room to more systematically engage with external partners and share knowledge. Canada’s Indo-Pacific Strategy presents an opportunity to strengthen donor and beneficiary coordination, as many donors also identified Indo-Pacific, and cybersecurity capacity building in particular, as their new priority.

Findings: Design & Efficiency

Figure 8:  

Evolution of the Programs’ funding since 2015-16 (ongoing funding is highlighted in bold).

The Programs’ complex funding structure created difficulties in setting a programming strategy, managing project succession, and funding projects across ACCBP and CTCBP topics.

ACCBP and CTCBP budgets included ongoing funding dedicated to emerging Government of Canada priorities. New funding often addressed protracted crises or issues and was renewed. This led to a complicated management of multiple program budgets, with different ceilings, effective and sunset dates, and Vote 1 and Vote 10 allocations. The proportion of Vote 1 allocations within funding envelopes ranged from as high as 30% for the Human Smuggling Envelope to 7% for the ACCBP Global envelope. Available financial data and interviews highlighted that funding renewals required significant administrative investment and posed difficulties when funding became available later in the fiscal year as business processes did not enable quick disbursement. Project file analysis found that project approval processes averaged 6 months from a Priority Review Committee recommendation to the signed instrument. Implementing partners indicated that delays with funding renewals posed difficulties in staff retention and managing relationships with beneficiaries. Several partner and staff interviewees also commented on the difficulty in funding proposals that crossed ACCBP and CTCBP envelopes, which caused missed opportunities to bridge anti-crime and counter-terrorism efforts.

The Programs did not have adequate flexibility in their programming processes to support agile project implementation, which contributed to yearly funding lapses.

Previous program evaluations and a scan of donor capacity building practices highlighted the need for program agility, with sufficient funding flexibility across years and investment activities to address the uncertainty in the context of terrorism and criminal threats and the associated project implementation delays. However, the evaluation found that programming processes grew in complexity and length over the evaluation period and no quick disbursement mechanism was available outside of project funding. The fiduciary risk assessment process added time for Vote 10 project approval and contributed to a greater proportion of contribution agreements (peaking at 91% in 2019-20), including with multilateral partners such as INTERPOL and the International Organization for Migration (IOM). In 2020, the Programs aligned application requirements for other government departments (OGDs) with those for external partners, which added administrative complexities for OGDs that frequently requested small funding amounts and dis-incentivized access to program funds. The Programs also required confirmation of Vote 1 project implementation by September each year, which was difficult due to uncertainty in beneficiary availability. 

The evaluation found that lack of programming process flexibility contributed to yearly funding lapses. Between 2015-16 and 2021-22, funding lapses totalled 40% for Vote 1 projects; 17% for the Vote 10 Human Smuggling envelope and 21% for the Vote 10 Sahel envelope. Funding lapses were not possible to calculate for all Vote 10 envelopes due to data limitations.

Figure 9:

Proportion of Term and Casual Employees at the Senior Officer Level (PM-5)

Source: HRMS (Power BI), January 2023.

Average Senior Program Officer Workload, per officer per year

• 2015-16 - 2019-20 - 15 projects disbursements
• 2020-21 - 2021-22 - 7 projects disbursement

The Programs underwent a period of significant instability due to organizational changes, staffing turnover and discontinuity in leadership that led to a loss of corporate memory.

Between 2015-16 and 2021-22, the Programs moved across 3 groups within the department (Non-proliferation and Security Threat Reduction Bureau; Counter-Terrorism, Crime and Intelligence Bureau; International Crime and Counter-Terrorism Bureau). The frequent change in group mandates was deepened by the churn of 5 program directors. Staffing was a major challenge for the Programs with heavy reliance on short-term staffing options, particularly at the program officer level, and on acting opportunities. On average, 36% of staff were placed in acting positions over the evaluation period. The Programs’ organizational structure limited career progression for program officers as it included entry-level and senior officer positions without any positions in between. It also had limited staffing in support of policy and corporate activities that existed within other security programs. Previous program evaluations similarly observed significant workload and turnover issues that limited the ability of staff to support strategic programming and affected the implementation of prior evaluation recommendations.

Staff turnover and a general lack of follow-through on documentation handover led to information management gaps and a loss of corporate knowledge around partner strengths, their relationships with beneficiary countries and project selection rationale. Partner and staff interviewees noted a lack of opportunities for staff to expand their subject matter expertise in a high rotation and high project churn environment. Partners also noted differing advice provided by staff and examples of important project knowledge not communicated during officer succession. The new Bureau established in 2020, prioritized addressing staffing challenges, professionalizing the team, and moderating officer workload by focusing on fewer larger projects. These efforts were reinforced by the conclusion of the departmental demobilization exercise that increased the Programs’ non-rotational positions from 26% in 2016 to 88% in 2022, with the number of staff in substantial positions doubling in the last 2 years.

The Programs worked towards improving program management practices.

Previous evaluations of the Programs noted significant weaknesses in management systems and processes. During the evaluation period, and most notably since 2020, the Programs made improvements in management practices that included a more consistent project file management, updates to the Standard Operating Procedures, preparation of monitoring visit reports, entry of Vote 1 projects into the departmental systems and senior management notification of upcoming investments. However, a review of the Programs’ administrative platforms identified remaining gaps with respect to the upkeep of historical budget and disbursement data; central tracking of project proposals, field monitoring reports, committee Records of Decision; and project closure delays. A review of the Programs’ performance measurement tools and interviews with partners demonstrated that reporting tools did not sufficiently capture the complexity of funded programming and external factors.

Gender equality ratings of funded Vote 10 grants and contributions projects reflected a greater emphasis placed on gender integration.

Figure 10:

Source: Chief Financial Officer data, July 2022, presented by value of disbursement.

Figure 11:

The Programs quickly and significantly grew their focus on, and support for, the inclusion of gender equality in the largely male-dominated security sector.

In response to the 2016 evaluation’s recommendation to address the “gender blindness” of capacity building and the subsequent implementation of the Feminist International Assistance Policy and the National Action Plan on Women Peace and Security, the Programs took steps to strengthen gender integration capacity.  Prior to the evaluation period, funded projects did not carry out gender analyses or seek gender-specific results. Since 2015-16, gender equality analysis has become part of the proposal form and a criterion for proposal assessment. According to interviewed Vote 10 implementing partners, the inclusion of gender equality considerations became an important component of their proposals, and Canada was one of few donors who supported gender work in security.

Overall, the Programs relied on implementing partners to define the depth of gender integration commitments. Interviews and case studies highlighted that partners pursued 3 options for advancing gender equality: advancing women’s participation in capacity building activities; mainstreaming a gender perspective in training curricula and materials; and supporting gender-specific interventions. The latter led to the development of several seminal knowledge products on gender and security: a landmark report on women in law enforcement in the ASEAN region by UNODC, UN Women and INTERPOL; a handbook containing strategies and good practices for strengthening gender equality in counterdrug law enforcement agencies by the Inter-American Drug Abuse Control Commission; and studies by the United Nations Office of Disarmament Affairs on gender and small arms controls. CTCBP currently funds a digital platform to support integration of gender and intersectional factors into counter-terrorism and the prevention of violent extremism by the United Nations Office of Counter-Terrorism.

The evidence of human rights integration in the Programs’ cycle was limited.

The Special Rapporteur on the promotion and protection of human rights and fundamental freedoms while countering terrorism noted in her 2021 report that capacity building in the security sector suffered from a persistent neglect in mainstreaming human rights, minimal oversight and a dearth of adequate monitoring and evaluation. A review of project documentation for the projects sampled for the case studies showed that partners’ human rights assessments commonly contained high-level references to organizational, United Nations and other international human rights norms, statues, declarations (62% of proposals) and focused on including human rights in capacity building as content information (49%), primarily in training curricula and materials (31%) but also through awareness building and advocacy (24%). Only 33% of proposals contained some assessment of the human rights situation in the country and/or human rights record of the beneficiary, and 16% outlined specific steps to be taken to ensure that the project did not harm or negatively impact human rights.

Findings: Results

The literature notes that connecting long-term results of short-term capacity building exercises in relation to improved security outcomes is difficult to assess. Because security contexts are so complex, and capacity building provides one small contribution to the overall landscape, causal relationships and specific attribution between built capacity and larger-scale behavioural or institutional change is not measurable with any accuracy. On the other hand, best practices and key principles of effective security-focused capacity building are well documented and included in the below diagram.

Figure 12:

Source: Gerspacher, Hanlon & Weiland (2017). Sustainable Capacity Building: Guidelines for Planning and Project Design Communities. United States Institute of Peace.

The Programs achieved good results at the output and immediate outcome levels in developing beneficiary security staff’s knowledge, awareness, skills and in establishing networks.

Case studies documented that capacity building needs among case study countries were great and varied. In the case of Jordan, Guatemala and Niger, partner countries relied significantly on donor support for training, capital investments and equipment. Sampled projects in case study countries showed that the provided capacity building measures aligned with expressed beneficiary priorities and broadly met their needs. Projects across 5 sampled case study countries were generally well delivered and all of them implemented planned activities fully or partially, with some implementation delays attributed to the COVID-19 pandemic. Interviewed beneficiaries assessed the support they received as high quality and relevant at a personal and institutional level. They noted the high quality of trainers and expressed interest in advanced training. Beneficiaries highly appreciated the infrastructure and equipment support offered, which provided significant visibility of Canadian investment.

The vast majority of reviewed projects (81%) provided full or partial evidence of improvements in beneficiary capabilities in terms of staff’s knowledge, skills, awareness and access to operational networks (Annex 4). In projects that provided equipment and infrastructure support, measures aligned with beneficiary needs and led to improvements in operational readiness and agility (for example, the procurement of a zodiac boat and equipment for Jordan’s maritime counter-terrorism unit and rehabilitation of a road along the northern border with Syria).

Projects that demonstrated achievement of capacity outcomes exhibited characteristics of effective capacity building documented in the literature.

The literature review identified the following broad categories of best practices in capacity building: sensitivity and alignment of capacity building measures with local needs, context and levels of existing capability; development of partnership relationships and commitment between donors and partner countries to ensure that interventions are not “supply driven” and have follow-through; and a combination of quality capacity delivery with incentives to support behaviour and broader institutional change.

Case study analysis similarly demonstrated that sampled projects in which increased beneficiary capacity was observed demonstrated positive factors linked to implementation quality and local buy-in, including high quality implementation (71% of projects with observed capacity), appropriate design for the context (67%), sufficient intensity of provided capacity building measures (54%), and participant (50%) and leadership buy-in (42%). These projects clearly identified capacity gaps among targeted beneficiaries and provided comprehensive capacity building support in response to those gaps, which included training, hands-on practice, mentoring and study visits, required equipment and other support, as relevant. The provision of equipment was accompanied by corresponding training for its use and capabilities and included extended maintenance options.

Regional projects were an important platform for networking and knowledge sharing among partner states, but required tailoring to each country’s capacity building needs to be effective.

To address the transnational nature of terrorism and crime, the Programs pursued regional programming, which acted as a vehicle to increase cooperation, coordination and interoperability. Case studies showed that, in some instances, partner countries were able to learn from others in their region and provide guidance in areas where they were stronger (for example, through the Lawyers without Borders Canada project, Guatemala established a specialized court for crimes of human trafficking, becoming a regional leader). In other cases, applying a regional approach was perceived as insufficient to meet the country’s specific needs. To address this, implementing partners balanced the scope of regional projects through phased implementation of regional programming with country-specific follow-up or by working towards flexible country solutions.

Another example of leveraging regional programming strength was deployment support for an RCMP position to the Jakarta Center for Law Enforcement Cooperation. The funded RCMP officer worked with Australia, New Zealand and the United Kingdom to deliver training courses, deepening operational relationships with the Five Eyes partners. ACCBP reported that, in cooperation with RCMP, the Centre trained 1,200 law enforcement officers. 

In some cases, there were changes in partner countries’ operational practices and readiness that added value to existing security operations.

The evaluation identified tangible examples of behaviour change outcomes with corresponding changes in institutional practices (for example, UNODC’s cybercrime capacity building led to the formalisation of the cybercrime unit as a stand-alone department within Guatemala’s national police, while the Container Control Programmes established inter-agency port control units in different countries). These changes were most notably observed in projects that built operational capability, particularly in border security, policing and inspection and when staff stayed in positions for extended periods and identified lack of capacity as a security risk. Case studies showed several examples of project contribution towards global security. Some projects were able to quantify outcomes with respect to reducing threats through seizures or arrests; however, most observed changes were changes in quality of institutional practices and readiness rather than active deployment of capability.

Most sampled projects (88%), where changes in behaviour and practice were observed, had evidence of all 3 determinants of behaviour change (capability, opportunity and motivation) present. Conversely, 70% of projects that had evidence in capacity outcomes but no evidence of changes in practice also lacked evidence on personal and institutional incentives and opportunities to apply learned skills and knowledge. This suggested that changes in capacity were insufficient to lead to changes in practice without broader incentives to improve performance.

The Programs’ achievement of outcomes specific to legal instruments, frameworks and policies depended on the enabling environments within partner states.

The Programs provided important support for implementing and updating legal instruments, frameworks and policies, frequently as part of comprehensive capacity building efforts. The Guatemala, Jamaica and Thailand studies documented that some progress was made in drafting legal and institutional frameworks, including those specific to terrorism, illicit drugs, cybersecurity, as well as gender mainstreaming. However, efforts to achieve targeted outcomes were significantly impacted by the broader enabling environment in a country. Strong political and institutional support by partner countries facilitated progress towards these outcomes, while political instability, lack of political will, slow pace of change and unsupportive organizational cultures impeded progress.

Case studies highlighted a need for more and smaller investments in policy studies, think-pieces and knowledge exchanges to frame a subsequent course of action with respect to legal, policy and institutional frameworks. They also showed that capacity building programming targeting this area requires greater tailoring to national and local context and evolution, and greater advocacy efforts with beneficiary countries. Securing mission collaboration was viewed as an important contributor to the achievement of these outcomes, increasing beneficiary country cooperation, with strong results demonstrated in the case of Jordan.

Projects aimed at strengthening community-level resilience did not achieve targeted results due to limited reach, low intensity in programming, and lack of institutionalization paths.

The Counter-Terrorism Capacity Building Program expanded its focus on community engagement in response to the need to address root causes and links between violent extremism/radicalization and terrorism. For those projects, the primary beneficiary shifted from state institutions to community organizations and members. The impact of reviewed projects in Jordan and Niger was largely contained to small groups of individuals involved in project activities, many of whom did not demonstrate ability to lead change in the community. In the absence of a clearly targeted high-risk group, the limited intensity and short-term nature of programming, and the absence of a plan to institutionalize change for broader community engagement, reviewed projects demonstrated little prospect for achieving results. This observation was consistent with the evaluation of the Middle East Strategy, which noted little impact of Global Affairs Canada’s campaigns on violent extremism, despite reaching a broad audience. A literature review and case studies pointed to the need to focus on reaching a critical mass of community beneficiaries with longer-term programming targeted to the local context to be both effective and sustainable. While such programming extends beyond the Programs’ focus and into the development assistance domain, it provides an opportunity for the Programs to inform geographic bureaus of existing threats and opportunities to build resilience towards terrorism and crime.

Some partners experienced difficulties integrating and reporting on gender outcomes.

In some contexts, implementing partners reported difficulties in meeting Programs’ expectations with respect to gender integration, as those were not applicable due to the operational nature of their project or the lack of an enabling environment within a beneficiary institution. This was seconded by other government departments that undertook capacity building specific to their operational priorities and expertise. A review of Vote 1 disbursements showed that the Programs began gender assessment of projects implemented by other government departments and agencies in 2018-19, with the majority (81%) assessed as no gender equality outcomes between 2018-19 and 2021-22. While most federal partners integrated gender considerations in project delivery and had female implementers, projects did not pursue gender outcomes.

Interviewed federal government departments and implementing partners recognized the importance of integrating gender equality considerations into project design and implementation; however, reporting on program outcomes was operationalized in a limited fashion through the number of women participants, over which many implementing partners had little influence.

Increasing women’s participation in the security sector required long-term commitment and was not always feasible given the political, cultural or operational context.

Security institutions are nearly universally male-dominated, with women participating more actively in clerical roles, those specific to child protection, and medical and rehabilitation services. Case studies showed that advancing women’s full integration and employment in security sectors was a slow process, requiring  advocacy and significant investments and was not always feasible given the political, cultural or operational context. On the other hand, incremental steps were viewed as important because strength comes with numbers. In Jordan, sustained capacity building and advocacy efforts by Government of Canada staff at the mission added significant contribution to the achievement of gender equality outcomes in policing and defense. CTCBP gendered support ranged from female accommodations buildings and procurement of female personal protective equipment to language training and mixed-gender operational training on tactical search, tactical medical training and firearms skills. Canada was one of the 5 donors to support the Public Security Directorate (PSD) and the Jordanian Armed Forces in the development of their first multi-year gender-mainstreaming strategies. The strategies set specific goals, priorities and measures to advance gender across the institutions. Overall, achieved results were reflective of the supportive enabling environment, particularly political and institutional commitments and buy-in, and synergies with other efforts by Canadian staff at the mission and by other donors.

Findings: Sustainability

Projects within the Programs’ portfolios demonstrated a wide range of sustainability, ranging from none to full institutionalization of new capabilities.

Sustainability of intervention was a criterion in project selection. It assessed beneficiary commitment, established relationships between the implementing partner and the beneficiary, and use of the train-the-trainer approach. A review of available Priority Review Committee records from earlier years when the Committee had scheduled meetings showed that the Programs were able to lean on the knowledge of staff at missions and in other federal departments to support their assessment.

Field visits conducted for the case studies documented a range of examples of project sustainability. Sustainability was evident in both how these projects were conceptualized and the operational context in which they were delivered. Most effective projects that demonstrated sustainability after years of implementation were rooted in strong needs assessment that identified specific capacity gaps and targeted groups that were aligned with the priorities of beneficiaries. These were followed by well-designed, comprehensive support to address gaps through training, mentoring, facilitated knowledge exchange, technical assistance, equipment donation with hands-on training and a maintenance plan, and capital investments, as applicable. Some projects in the Guatemala, Jamaica and Jordan case studies also included the establishment of professionalized teams; institutionalization of training, mentoring and new processes into operations; the creation of internal training colleges and documentation; formalization of inter-institutional cooperation; and communication channels.

External factors impacting on project sustainability included insufficient understanding of beneficiary needs; significant staff turnover within beneficiary institutions; selection of participants with limited agency to apply or spread change; insufficient plan to sustain expenditures through internal budget; lack of support from institutional and political leadership; and major political changes within the country.

The train-the-trainer approach was widely used and had some successes, though it did not equate with sustainable capacity building practice. Such projects focused on building trainer capacity, but rarely addressed the issues of opportunity, motivation and institutional support required for the trainers to share their knowledge within institutions. Case studies found some anecdotal evidence that not all elements of training were subsequently passed on to staff by the trainers, particularly those related to gender equality and human rights considerations, as interviewed trainees only recalled operational training elements and could not recall any information specific to gender equality or human rights. Interviewed beneficiary staff also expressed strong preference for certification and delivery of training by international experts. The evaluation found that one way of ensuring this sustainability is to make the appropriate connections with other longer-term security programming ongoing in-country, and/or Global Affairs Canada development program priorities and objectives.

Conclusions

ACCBP and CTCBP added credibility and visibility to Canadian international engagements, and contributed to strengthening global security and advancing gender integration.

The evaluation found that the Programs were an important and relevant horizontal initiative that demonstrated an ability to lead security-focused capacity and practice change in evaluated projects. Achieved outcomes were most evident when focused on immediate security response. The Programs increased their support to gender-responsive capacity building and the development of knowledge products to advance gender integration in the security sector as a matter of priority. In all cases, immediate and sometimes broader project outcomes opened doors for Canada’s presence abroad and provided opportunities to initiate and sustain relationships in sensitive contexts.

Focus on both immediate security and long-term developmental objectives of capacity building blurred the lines of the Programs’ role and responsibilities.

The Programs are situated in a unique and potentially powerful security and development nexus, in which the immediate outcomes of capacity building activities may kick-start or reinforce development and diplomacy goals with regards to institutional, organizational and societal change. During the evaluation period, the Program expanded across the continuum of security capacity building, with both an immediate security approach and a longer-term developmental approach in their focus, which provided a solution to some inflexibility inherent in their programming processes. At the same time, the Programs lacked the engagement in time, staffing capacity and relationships that were necessary for long-term change to take effect. The move towards larger projects with a long-term view prevented the Programs from fully benefitting from their flexible Terms and Conditions, which allowed the Programs to work with non-traditional partners and engage in smaller impactful and innovative work.

The pivot away from interdepartmental coordination towards the management of grants and contributions left the Programs without a clear sense of their mandate as a horizontal security initiative.

The Programs provided a venue and rationale for the Government of Canada to pull together federal security expertise to support international anti-crime and counter-terrorism capacity building. Their design was meant to be mutually reinforced with specialized agency expertise informing investments to respond to threats to Canada’s national and global security. However, opportunities to coordinate security expertise horizontally across federal departments and agencies through program governance were impacted by the Programs’ organizational instability, staffing turnover and limited policy capacity. Further, limitations of the Vote 1 funding framework were addressed with workaround solutions that reduced the attractiveness of programs funds for federal partners. What made ACCBP and CTCBP impactful in their earlier years was their ability to bring together expertise across the Government of Canada and implement security assistance in a quick and direct way, which has since weakened.

Recommendations

The original intent and mandate of the Programs as articulated in the Terms and Conditions to support government-wide anti-crime and counter-terrorism security assistance abroad continue to remain relevant. The Programs are uniquely positioned as a convener of security expertise across the Government of Canada to inform and engage the department of existing threats and opportunities to build global resilience towards terrorism and crime. However, the mandate and role of the Programs have been diluted over time and have left them without a clear purpose. Some activities have extended into the development assistance domain further away from their operating niche, without clear and direct links towards potential security threats for Canada nor the priorities of other Global Affairs Canada programming work. Administrative processes and uncertainty regarding the eligibility of expenses for other government departments, have dis-incentivized key federal partners from engaging in capacity-building work with the Programs and have led to less agile project implementation – both key components of effective and relevant response to emerging security threats and priorities to Canada. In response to these challenges, this evaluation recommends the following:

1. The Programs should clarify their role, purpose and mandate, and revise their organizational structure in order to meet the Government of Canada’s anti-crime and counter-terrorism capacity building security objectives.
2. The Programs should improve use of existing specialized security expertise found within the Programs’ governance structures when coordinating and guiding programming strategic and investment decisions in support of Canada’s security priorities.
3. The Programs should proactively engage and develop working relationships with the department’s geographic divisions and missions where there are sizeable investments to ensure alignment with departmental priorities at the country and regional levels.
4. The Programs should review the existing funding framework and apply new streamlined instruments and processes for Vote 1 federal government partners, and incorporate faster and more flexible processes for administering Vote 10 grants and contributions.

Considerations

Greater policy – programming linkages: Situated in one Bureau since 2020, both teams previously had limited ability to inform each other's work and priorities, with gaps in policy development and coordination roles. The Bureau would benefit from establishing a framework of collaboration to link capacity building programming to policy priorities and to inform policy development with results from programming interventions, as well as bringing broader clarity around international engagement in global anti-crime and counter-terrorism fora.

Flexible performance measurement: There is a gap between the type and significance of achieved results found in project reports and corporate indicators and those identified during site visits. The majority of ACCBP and CTCBP projects were achieving more impactful capacity building and other outcomes than were documented, especially considering external factors affecting their implementation. Finding a more flexible approach to measuring program performance (for example, Outcome Harvesting used by the Peace and Stabilization Operations Program, or the Actor-based Change Framework) would allow to better capture the results and impact of capacity building on the ground.

Greater attention to and documentation of the integration of human rights considerations: The objectives of ACCBP and CTCBP are to deliver capacity building in a manner consistent with international and human rights obligations, norms and standards. Considering the nature of programming and its operation in fragile and conflict-affected states, as well as countries that have poor governance indicators, greater guidance to program staff is required to ensure consistent understanding and documentation of completed human rights assessments and risk mitigation strategies, which may include increased monitoring, restricted criteria for participation and others.

Field coherence for security programming at missions: There is a lack of coordination of security programming and activities on the ground and limited ability to provide feedback on engagements, priorities and local contexts to the Programs at Headquarters. Finding means to coordinate the Government of Canada security footprint in a country and/or region would help inform programming, build relationships and provide project oversight.

Common understanding of capacity building assistance: The department provides capacity building and technical assistance through a range of programs and mechanisms, all of which were used to support institutional development related to anti-crime and counter-terrorism. This includes ACCBP, CTCBP, Weapons Threat Reduction Program, Peace and Stabilization Operations Program, the Technical Assistance Partnership (TAP) and the bilaterally-focused project-based technical assistance mechanisms. This presents an opportunity to harmonize definitions and understanding of capacity building across the department, identify specific roles and complementarities, and develop common frameworks to measuring impact on the ground.

Annexes

Annex 1: ACCBP and CTCBP Ts&Cs (2020) and Terms of Reference for Programs’ Governance Structure (2021)

Annex 2: Actor-Based Change Framework

Figure 13:

The Actor-Based Change framework is a contribution analysis technique that aims to capture how change happens in a complex system (Koleros et al., 2020)^{Footnote 4}. By integrating concepts from complexity science and behavioural science, the framework helps identify which actors in the system are most closely associated with the outcome of interest and what conditions must be met to sustain a given behaviour or practice and thus contribute to a systemic change.

The Actor-Based Change framework applies 3 iterative steps when assessing the theory behind complex interventions:

• Development of an actor-based systems map: identify the actors in the system and understand how those actors behave with respect to one another and in response to their environment, by focusing on 3 essential determinants of behaviour (capability, opportunity and motivation).
• Development of a change agenda for the targeted groups: identify the conditions that need to shift over time to lead to sustainable changes in practices and relationships needed to address the issues targeted by the intervention.
• Articulation of causal impact pathways: actor-level changes, leading to organizational and systems change and other macro-level changes.

Applying an Actor-Based Change Framework to the Evaluation

The evaluation methodology included an application of the Actor-Based Change framework. The locus of the evaluation focused on whether desired behavioural change was evidenced among the Programs’ beneficiaries, what barriers to change existed in their environment, and how effective the activities and tools used to influence desired behaviour change were.

The framework helped conduct an assessment of the results and sustainability of program interventions, accounting for external factors that impacted on individual and organizational changes. The limitation of the approach in this evaluation was the team’s ability to contextualize achieved results within broader system-level changes at the country level due to many external factors beyond Canada’s capacity building assistance.

Annex 3: Donor Scan Summary

Lessons learned

The donor scan documented that there was no uniform approach to anti-crime and counter-terrorism capacity building assistance among donors, some of whom managed capacity building within broader peace and stabilization programming efforts. A summary of key lessons learned and good practices are provided below.

Integrated whole-of-government approach

Many donors (Denmark, France, Germany and the United Kingdom) pursued security capacity building in support of their broader security, defence, foreign affairs and/or development cooperation priorities by establishing a dedicated funding pool to coordinate assistance. Centralizing the management of these funds through interdepartmental steering committees helped guide and harmonize geographic and thematic priorities for investments, and promoted internal coherence.

France: France’s Directorate for Security and Defense Cooperation (DCSD) of the Ministry for Europe and Foreign Affairs is located in the General Directorate for Political Affairs. Given its mandate and expertise in the security-defense and security-development cooperation, the Directorate works closely with all ministries and public agencies working in those areas and is part of a large interministerial ecosystem (ministries of Armed Forces, Interior, Economy and Justice). It implements various training programs, including through 16 regional national schools across Africa, provides technical expertise and provides non-lethal equipment assistance. The directorate is composed of about 60 employees at Headquarters and 311 cooperation officers from 4 administrations (Armed Forces, Interior, Economy, Justice) assigned to 51 countries and international organizations to implement France’s structural cooperation.

Denmark: Denmark pursued an integrated, cross-government approach to peace and stabilisation by establishing the Peace and Stabilization Fund (PSF) to combine diplomatic, defence and development instruments funded by official development assistance (ODA) and non-ODA resources. The PSF is governed by the inter-ministerial steering committee (both strategic direction and funds approval), supported by a whole of government secretariat, anchored in the Ministry of Foreign Affairs, responsible for the overall management of the PSF. Overall responsibility for programmes and programming lies at headquarters, while day-to-day coordination and management is delegated to embassies, generally by a PSF Programme Manager/Stabilization Advisor.

Smaller Governance Structures: Many donors relied on smaller governance structures to manage funding pools used for anti-crime and counter-terrorism capacity building. This allowed them to use their governance structures as decision-making rather than consultative bodies, with a focus on joint selection of priorities and investment allocations and follow-up on lessons learned. Such governance structures primarily included foreign affairs and defence, but also home affairs, justice and police entities. They also allowed to bring other ministries with specialized expertise in the observer capacity.  

Defining a niche: Some donors defined capacity building programming niches for their investments. This was commonly characterized by priority geographic areas and, to a lesser extent, by a specific sector of intervention. While most of the countries had some level of global programming, their primary focus was linked to their national security interests (for example, Australia focused programming in the Indo-Pacific region, France focused on Africa, and Germany focused on the Middle East and Africa). Many donors made their programming priories public (the UK, Denmark, Germany, France).

A diplomatic tool: Several donors used capacity building programs as a tool for diplomatic influence. France, the United Kingdom and Denmark identified opportunities to use capacity building to build relationships, facilitate access, and garner support for their vision and priorities. For example, France deployed their staff in their embassies or in specialized schools to facilitate strategic use of capacity building assistance.

Field presence and alignment: Many donors worked towards creating regional field positions dedicated to security assistance and coordination (Netherlands, the UK, Denmark), and also identified other means to engage in the field, including: assigning project management and oversight to desk staff best placed to follow through on project implementation (for example, staff at mission as a preferred option or staff at Headquarters (Denmark); relying on external advisors contracted from program resources to provide technical assistance to partners and follow project implementation (UK, Denmark) and ensuring a more formal engagement of security ministry liaison officers posted at missions (Germany, France).

Annex 4: Case Study Project Results

Case studies included a sample of 45 projects implemented in Guatemala, Jamaica, Jordan, Niger and Thailand. The evaluation team constructed theories of change for each project using the Capability, Opportunity and Motivation model of Behaviour. The theory of change identified project-level capacity building outcomes targeted by the projects, along with assumptions regarding the motivational and opportunity factors required for each step in the theory of change to occur. The evaluation team gathered evidence through document reviews, direct observations of capacity and interviews with a range of stakeholders on the extent to which each step in the theory of change was achieved and what factors facilitated or inhibited its achievement. The graphic below present summary of this assessment in the 5 case study countries.

Figure 14:

Annex 5: Case Study Project Results, by Country

Overview outlining the evaluation results by overall result and by case study country: Guatemala, Jamaica, Jordan, Niger and Thailand.

Overall Result:

Activities Delivered:

71% - Full evidence that targeted change occurred

29% - Partial evidence that change occurred

Outputs Achieved:

65% - Full evidence that targeted change occurred

29% - Partial evidence that change occurred

6% - Evidence that change did not occur or no evidence (left blank)

Capacity Change:

54% - Full evidence that targeted change occurred

27% - Partial evidence that change occurred

19% - Evidence that change did not occur or no evidence (left blank)

Practice Change:

35% - Full evidence that targeted change occurred

27% - Partial evidence that change occurred

38% - Evidence that change did not occur or no evidence (left blank)

System Change:

25% - Full evidence that targeted change occurred

23% - Partial evidence that change occurred

52% - Evidence that change did not occur or no evidence (left blank)

Guatemala:

Activities Delivered:

78% - Full evidence that targeted change occurred

22% - Partial evidence that change occurred

Outputs Achieved:

78% - Full evidence that targeted change occurred

11% - Partial evidence that change occurred

11% - Evidence that change did not occur or no evidence (left blank)

Capacity Change:

78% - Full evidence that targeted change occurred

11% - Partial evidence that change occurred

11% - Evidence that change did not occur or no evidence (left blank)

Practice Change:

33% - Full evidence that targeted change occurred

44% - Partial evidence that change occurred

23% - Evidence that change did not occur or no evidence (left blank)

System Change:

33% - Full evidence that targeted change occurred

22% - Partial evidence that change occurred

45% - Evidence that change did not occur or no evidence (left blank)

Jamaica:

Activities Delivered:

70% - Full evidence that targeted change occurred

30% - Partial evidence that change occurred

Outputs Achieved:

40% - Full evidence that targeted change occurred

60% - Partial evidence that change occurred

Capacity Change:

50% - Full evidence that targeted change occurred

40% - Partial evidence that change occurred

10% - Evidence that change did not occur or no evidence (left blank)

Practice Change:

30% - Full evidence that targeted change occurred

30% - Partial evidence that change occurred

40% - Evidence that change did not occur or no evidence (left blank)

System Change:

20% - Full evidence that targeted change occurred

10% - Partial evidence that change occurred

70% - Evidence that change did not occur or no evidence (left blank)

Jordan:

Activities Delivered:

90% - Full evidence that targeted change occurred

10% - Partial evidence that change occurred

Outputs Achieved:

90% - Full evidence that targeted change occurred

10% - Evidence that change did not occur or no evidence (left blank)

Capacity Change:

60% - Full evidence that targeted change occurred

20% - Partial evidence that change occurred

20% - Evidence that change did not occur or no evidence (left blank)

Practice Change:

50% - Full evidence that targeted change occurred

10% - Partial evidence that change occurred

40% - Evidence that change did not occur or no evidence (left blank)

System Change:

40% - Full evidence that targeted change occurred

10% - Partial evidence that change occurred

50% - Evidence that change did not occur or no evidence (left blank)

Niger*:

Activities Delivered:

29% - Full evidence that targeted change occurred

71% - Partial evidence that change occurred

Outputs Achieved:

29% - Full evidence that targeted change occurred

57% - Partial evidence that change occurred

14% - Evidence that change did not occur or no evidence (left blank)

Capacity Change:

29% - Full evidence that targeted change occurred

43% - Partial evidence that change occurred

28% - Evidence that change did not occur or no evidence (left blank)

Practice Change:

29% - Full evidence that targeted change occurred

43% - Partial evidence that change occurred

28% - Evidence that change did not occur or no evidence (left blank)

System Change:

14% - Full evidence that targeted change occurred

57% - Partial evidence that change occurred

29% - Evidence that change did not occur or no evidence (left blank)

Thailand:

Activities Delivered:

75% - Full evidence that targeted change occurred

25% - Partial evidence that change occurred

Outputs Achieved:

75% - Full evidence that targeted change occurred

25% - Partial evidence that change occurred

Capacity Change:

50% - Full evidence that targeted change occurred

25% - Partial evidence that change occurred

25% - Evidence that change did not occur or no evidence (left blank)

Practice Change:

33% - Full evidence that targeted change occurred

17% - Partial evidence that change occurred

50% - Evidence that change did not occur or no evidence (left blank)

System Change:

17% - Full evidence that targeted change occurred

25% - Partial evidence that change occurred

58% - Evidence that change did not occur or no evidence (left blank)

* The Niger case study had a higher proportion of projects assessed as having partial evidence because the country’s security context limited the evaluators’ ability to visit project sites, interview different project stakeholder groups and triangulate evidence from various sources.