Formative Evaluation of the Anti-crime Capacity Building Program

October 2012

Abbreviations, Acronyms and Symbols

Acknowledgements

The Evaluation Division (ZIE) of the Office of Audit, Evaluation and Inspection would like to extend its appreciation to all employees of DFAIT, representatives of other government departments, international organizations and NGOs who agreed to participate in interviews for the formative evaluation of the Anti-Crime Capacity Building Program. ZIE also gratefully acknowledges the contributions of members of the Evaluation Advisory Committee who guided the evaluation throughout all its phases.

Executive Summary

Context and Program

The Anti-Crime Capacity Building Program (ACCBP) was established in 2009 to improve Canadian security by preventing and responding to threats from transnational organized crime. The Program focuses its activities in the Americas due to the region's strategic importance to Canada, the mounting threat to national and regional stability posed by its increasing crime rates, and the fact that criminal activities in the Americas are linked to crime and security in Canada.

The overall mandate of the ACCBP is to enhance the capacity of key beneficiary states to prevent and respond to threats posed by transnational criminal activity in the Americas. In this context, the ACCBP endeavours to address national and regional security concerns through the implementation of projects in the following thematic areas: Illicit Drugs, Corruption, Human Trafficking and Migrant Smuggling, Money Laundering and Proceeds of Crime, Security System Reform, and Crime Prevention.

Evaluation Objectives

This formative evaluation of the Anti-Crime Capacity Building Program, Foreign Affairs and International Trade Canada (DFAIT), was conducted as part of a Treasury Board of Canada requirement to report on the Program's progress toward the achievement of its expected outcomes. This evaluation was conducted by DFAIT's Evaluation Division (ZIE) from October 2011 to June 2012.

The overall objective of the evaluation was to review and assess the relevance and performance of the ACCBP. In this vein, the evaluation examined how the ACCBP addresses a continued need and aligns with Government and Departmental priorities; the degree to which management structures and systems have been put in place to support the achievement of the ACCBP's expected outcomes; and the extent to which individual projects under the ACCBP are contributing to the objectives of the Program in an efficient manner. The evaluation also examined how the ACCBP fits into the larger context of security programming in the Department and the Government of Canada and sought to identify best practices and lessons learned.

Key Findings and Conclusions

Relevance

Crime in the Americas poses serious security threats to people in the region, as well as to Canadians. It also hinders economic prosperity, prevents the development of democratic institutions and a full respect for human rights.

In this context, the ACCBP addresses an important need and plays a unique role in fulfilling Canada's commitments in the Americas. The ACCBP focuses its programming efforts to target countries where the threats posed by transnational crime are greatest while also considering the potential impact on the security of Canadians and Canadian interests, both domestically and abroad.

The ACCBP is fully aligned with Canada's priorities in the Americas and supports its international commitment to fight crime while promoting democracy and human rights. The Program also provides a tool that deepens bilateral relations, promotes greater stability in the region, and encourages trade. Moreover, the ACCBP is a unique Program within the Government of Canada, as it is the only program that is mandated and resourced to undertake anti-crime capacity building work abroad.

Performance

Overall, the ACCBP is a well-managed Program and has shown excellent progress toward the achievement of short-term outcomes. It has also established sound management processes and best practices that support the achievement of longer-term objectives.

The ACCBP has contributed to increasing Canada's visibility in the Americas and, with the ongoing collaboration and engagement of missions and the geographic bureaus, continues to highlight Canada's expertise and leadership on anti-crime issues.

Evaluation findings indicate that more consistent performance tracking at the Program level and more coherent and integrated information management could help the Program report more accurately on its successes, use performance information to make decisions, identify lessons learned and best practices, and plan more strategically over the long-term.

Efficiency and Economy

Early indications suggest that DFAIT is reaping a good return on its investment through the ACCBP. An effort has been made to ensure that limited resources are being maximized through the scrutiny of project costs by the Project Review Secretariat Committee and Project Officers.

Since 2010, the ACCBP has also reached out to other donors, including the United States, to engage in joint programming, in an effort to better coordinate programming in the region and enable the ACCBP to be more strategic about the projects it chooses to fund.

The officers within the division work efficiently. The ACCBP Secretariat is staffed to implement $15M of programming annually; however, it has disbursed a considerably larger amount of program funds in the past fiscal year. While the Program has been able to meet these additional commitments without increasing staff levels or funding, the current pace of work is limiting the ability of Program staff to plan strategically and maintain the good management processes that have been put in place.

Overall, stakeholders recognize that the ACCBP is giving Canada a big "bang for the buck" in that the Program is managing to increase Canadian visibility and credibility in a priority region for the Government and achieve results with relatively modest resources.

Funding for OGD Projects

The demand of OGDs for ACCBP funds exceeds the amount of funding currently allocated toward projects implemented by other federal government organizations.

The ACCBP provides funding to OGDs to implement projects. This funding is transferred as "Vote 1 programming" funds (also referred to as "OGD Programming Funds"), as Vote 10 funding (Grants and Contributions) cannot be transferred between federal government departments and Agencies. While this Vote 1 Programming allocation cannot be modified at the present time given that changes would require approval from the Treasury Board of Canada, OGD partners have expressed the view that the current allocation amounts prevent them from fully participating in the Program.

While additional funding would provide benefits for Canada and allow OGDs to participate more extensively in the ACCBP, the decision to reallocate money toward OGD projects needs to be carefully considered by the Program. Should additional funding be re-allocated through a Treasury Board submission in the future, monies allocated to OGDs should be commensurate not only with the increased demand to implement projects, but also with the capacity of OGDs to deliver programming in an efficient manner.

Recommendations

Recommendation #1:
During transitions in management of the Program, the ACCBP should continue to implement and refine the management practices and processes it has established.

Since its establishment in 2009, ACCBP has developed and implemented a number of sound management processes and practices to support the achievement of longer-term objectives in an efficient and effective manner. These practices include but are not limited to:

• Integrating the Priority Review Exercise into the annual management practices of the ACCBP and making the priority setting process multidimensional, with emphasis placed on beneficiary state needs, threats to Canadians and Canadian interests, severity and growth of security problems, and the extent to which Canada can add value through security programming.
• Increasing the efficiency of the decision-making process with regard to project selection by reviewing the roles and responsibilities of the ACCBP Secretariat.
• Incorporating subject-matter experts into the Program's governance structure.
• Moving from annual to multi-year funding for projects and initiatives to increase their impact and efficiency.
• Establishing Memoranda of Understanding with beneficiary States to highlight Canada's contribution through the ACCBP and to articulate the roles and responsibilities of Canada and the beneficiary state.

While most of these management practices have been well received by ACCBP staff and DFAIT Senior Management, they need to be consistently applied and further refined as needed by future Program Managers and staff, especially during periods of transition and changes in the management of the program.

Recommendation #2:
The ACCBP should establish standard operating practices with other Departmental security programs to ensure coordination and complementarity and to avoid duplication.

The ACCBP Secretariat collaborates closely with the Geographic Branches and Missions in the Americas. It also shares the same governance structure with the CTCBP. The Program does not, however, possess standard practices for consultation and exchange of information and best practices with other Departmental security programs such as START and the GPP. Increased collaboration could contribute to the coherence, effectiveness and efficiency in the planning and implementation of security related projects and initiatives, especially as other security programs seek to implement projects in the Americas.

In order to facilitate better information sharing, priority setting and programming coordination with other DFAIT security programs, especially at the country and regional level, the ACCBP should explore establishing more standardized cooperation practices.

Recommendation #3:
As it matures, the Program should focus on improving performance measurement and information management.

There are several means by which the ACCBP could improve the manner in which is measures its performance, including:

• Continuing to work towards the development of a more comprehensive information management system.
• Incorporating a Program-level analysis of results and lessons learned as part of its regular reporting process.
• Modifying its Results-Based Management and reporting templates to include more detailed guidance and instructions on the types of performance information it would like to collect from implementing organizations.
• Creating an applicant guide for recipients to detail the Program's expectations for the quality of performance information as well as expectations for performance and financial reporting and use of performance information in project management.
• Requesting additional clarification from implementing organizations on the performance of individual projects when the quality of performance information is lacking in reports and encouraging recipients to focus on tracking outcomes and successes.

Recommendation #4:
If an opportunity arises in the future, the ACCBP should re-examine the current funding allocation between Vote 1 Programming (Operating Budget) and Vote10 (Grants and Contributions). Funding in priority regions should be allocated based on the increased demand for the Program, as well as the capacity of OGDs to deliver programming in an efficient and effective manner.

Projects implemented by multilateral organizations do not provide the same degree of visibility for Canada as bilateral projects implemented through OGDs. In addition, projects delivered by OGDs highlight the skills and expertise available within the Government of Canada and encourage further bilateral partnerships and cooperation. Increased funding for OGDs in Vote 1 would facilitate a more active involvement of OGDs and enable the implementation of, longer-term bilateral initiatives that maximize the potential impact within the recipient country.

If the opportunity arises to increase and re-allocate funding for the Program, the ACCBP should increase the amount of funding available to OGDs in a manner commensurate with both their implementation capacity and their demand for ACCBP funds. The need for reallocation could be assessed through regular tracking of disbursement trends and consultations with OGD stakeholders.

1.0 Introduction

1.1 Background and Context

Security in the Americas is a serious problem. The situation has continued to deteriorate as gangs, drug traffickers, and other criminal organizations expand their operations, contributing to escalating levels of crime and violence that alarm citizens and threaten to overwhelm governments. The homicide rates in several Central American countries are among the highest in the world and citizens identify public insecurity as the top problem facing their countries.^{Footnote 1}

Underlying social conditions and weakness in governance inhibit efforts to improve security. Persistent poverty, inequality, and unemployment leave large proportions of the population susceptible to crime. At the same time, underfunded security/police forces and the failure to fully implement post-conflict institutional reforms have rendered the security sector weak and susceptible to corruption.

Several countries in the region also have a history of civil wars, armed conflicts or dictatorships. This legacy of conflict and authoritarian rule has stalled the development of democratic institutions and respect for the rule of law in many countries. Armed conflicts have also resulted in the creation of networks to support illicit trafficking of firearms in the region that have since been converted into transnational criminal networks used for drug trafficking and the smuggling of other illicit goods such as people and weapons.^{Footnote 2}

Drug trafficking, organized crime, and related violence and corruption are major challenges that confront many governments around the world and threaten Canadian security interests. Preventing and countering crime, drug trafficking, foreign corruption and money laundering not only protects Canadian communities and businesses, but also reinforces the rule of law on a global scale.

Through the implementation of the ACCBP, Canada, alongside other donor states and international institutions has made efforts to improve the situation by developing the capacity of countries in the region to be able to address these threats on their own.

The Anti-Crime Capacity Building Program (ACCBP) was established in 2009 to improve Canadian security by preventing and responding to threats from transnational organized crime. The Program was created to address national, regional, and international security threats posed by criminal activities such as drug and firearms trafficking, money laundering, corruption, human smuggling and urban gang violence. While these threats are present around the globe, the ACCBP focuses on the Americas due to its strategic importance to Canada, and the fact that security concerns in the Americas are linked to crime and security in Canada.

The ACCBP aims to strengthen the capacity of key beneficiary states in partnership with Canadian Government agencies, other donor countries, multilateral organizations and private sector partners in the following specific areas: 1) Illicit Drugs; 2) Corruption; 3) Human Trafficking and Migrant Smuggling; 4) Money Laundering and Proceeds of Crime; 5) Security System Reform; 6) and Crime Prevention.

The ACCBP is one of the two largest complementary and mutually supportive international security assistance programs under one management structure that aim to help develop security capacity abroad. The other main capacity building program, the Counter-Terrorism Capacity Building Program (CTCBP), was established in 2005 and its objective is to provide beneficiary states with assistance to enhance their ability to prevent and respond to terrorist activity.

The Program is managed by the Department of Foreign Affairs and International Trade (DFAIT). Given its focus on security, the ACCBP initiates partnerships with OGDs that possess domestic security mandates. In doing so, the ACCBP extends the reach of several Departments and Agencies and leverages their expertise in promoting Canadian security interests internationally.

1.2 Programme Objectives

The overarching mandate of the ACCBP is to enhance the capacity of key beneficiary states to prevent and respond to threats posed by transnational criminal activity in the Americas. In this context, the ACCBP endeavours to address national and regional security concerns, thereby contributing to improving Canada's national security.

The ACCBP expects to achieve the following short term-outcomes:

• Personnel in beneficiary States and organizations are more knowledgeable and skilled in anti-crime issues;
• Improved prioritization and coordination on anti-crime issues;
• New or improved anti-crime legal instruments, controls and frameworks; and
• Additional or improved anti-crime tools, equipment, networks and physical infrastructure are available and in use.

The ACCBP's expected intermediate outcomes are:

• Strengthened incident prevention, mitigation and operational readiness;
• Increased compliance with international anti-crime commitments; and
• Strengthened infrastructure to support anti-crime coordination and responseactivities.

The ACCBP's expected long-term outcomes are:

• Increased capacity of beneficiary states and government entities to prevent and respond to threats posed by criminal activities in a manner consistent with international norms and standards related to human rights;
• Reduction in threat to Canadians and Canadian interests, at home and abroad, from international criminal activities in identified target areas and according to Canadian priorities; and
• Increased capacity of States to address security issues on their own through the facilitation of regional coordination mechanisms.

The ACCBP achieves these objectives by implementing projects through three different delivery mechanisms:

• Bilateral programming: for the provision of training and expertise through other governmental Departments and Agencies (OGDs) including the RCMP, DND, and Public Safety, plus the provision of equipment through the Canadian Commercial Corporation (CCC);
• Multilateral programming: implemented primarily through international organizations to which Canada is a member, these projects often have a large number of beneficiaries; such institutions include the UN Office on Drugs and Crime (UNODC), the Organization of American States (OAS), Interpol, and the International Organization for Migration (IOM); and
• Partnerships: with key allies (e.g., US, UK, France) and other donors, as well as through reputable and well-positioned private organizations.

1.3 Logic Model

A logic model has been developed for both the ACCBP and other Programs managed by the Capacity Building Programs Division, as these Programs share a common management structure and aim to achieve similar results.

Activities

• Experts Placements
• Training Program Development
• Training Division
• Legal Support
• Provision of Tools, Technology, and Equipment
• Physical Infrastructure Support

Outputs

• Experts Placed
• Training Curriculums
• Training Sessions
• Legal Instruments
• Tools, Technology, and Equipment
• Physical Infrastructure Components

Short-term Outcomes

• Personnel in beneficiary States and organizations more knowledgeable and skilled in anti-crime and counter-terrorism policies, procedures and enforcement
• Improved coordination on anti-crime and counter-terrorism Issues
• New/improved anti-crime and counter-terrorism legal instruments, controls and frameworks
• Additional or improved anti-crime and counter-terrorism tools, equipment, networks and physical infrastructure are available and in use

Intermediate Outcomes

• Incident prevention, mitigation and operational readiness and responsiveness is strengthened
• Increased compliance with international anti-crime and counter-terrorism commitments
• Infrastructure to support anti-crime and counter-terrorism coordination and response activities are strengthened

Impact-Level Outcomes

• Capacity of beneficiary States and government entities to prevent and respond to criminal and terrorist activity is increased
• Capacity of international organizations responsible for supporting States’ anti-crime and counter-terrorism related efforts is increased
• Reduced risk, incidence and impact of international criminal and terrorism activity globally

1.4 Governance

The ACCBP shares common governance and management structures with the Counter Terrorism Capacity Building Program (CTCBP). While the Minister of Foreign Affairs is ultimately accountable for the Program, the governance structure consists of the following entities:

• The Interdepartmental Steering Committee: This entity, which is comprised of ACCBP personnel and 18 OGDs who partner with the Program, is responsible for developing priorities, providing the strategic direction of the Program, aligning it to all-of-government approach to security priorities, discussing emerging threats and trends and making recommendations. This is accomplished primarily through an annual priority review exercise.
• Project Review Committee: This consultative committee, which includes ACCBP staff and OGD partners, is charged with reviewing project concepts/proposals and providing input on projects within their area of expertise. In the case where a Project Review Committee member is also a project implementing body, it is also responsible for monitoring and overseeing project implementation, as well as reporting on performance.
• Capacity Building Programs Division: This DFAIT division houses the ACCBP Secretariat, which is responsible for the administrative and day-to-day management and reporting related to the ACCBP.

1.5 Project Approvals

Projects are approved and amended according to the following authorities:

• The Minister of Foreign Affairs and the Deputy Minister of Foreign Affairs are authorized to approve projects with a value up to $10M.
• The Assistant Deputy Minister of Foreign Affairs responsible for anti-crime and counter-terrorism programming is authorized to approve and amend projects with a value up to $5M.
• The Director General responsible for anti-crime and counter-terrorism programming can approve and amend projects with a value up to $3,000,000.

Where umbrella agreements are in use, approval authority is based on the value of each individual project rather that the cumulative amount of all combined projects covered under the overarching instrument.

Effective June 13, 2011, the Minister of International Affairs and the Minister of State for the Americas must provide approval for the initiation of every project. This procedural change is an additional requirement that must be satisfied prior to the formal signing approval of a project in accordance with the delegated authorities defined above.

1.6 Program Resources

In June 2008, the Government of Canada approved funding for the ACCBP. It was determined that $10M would be allocated for fiscal year 2009-10 and $15M would be provided on an on-going basis thereafter. The maximum amount payable per project in one year is $5 million. In the first three fiscal years of the Program, which is the time frame covered by the evaluation, the ACCBP disbursed approximately $42M.

The funding is broken down as follows:

The ACCBP's budgetary arrangement is unusual because Vote 1 (Operating Expenditures) includes both the costs related to the administration of the Program, as well as $700k in funding for projects implemented by OGDs. The remainder of the annual funding ($13M) is allocated to project-specific Grants and Contributions and is provided to other organizations such as NGOs or multilateral institutions. The overall Operating budget for the ACCBP Secretariat accounts for 2.5% of the total yearly budget.

1.7 ACCBP Secretariat Profile

The ACCBP Secretariat is housed under DFAIT's Capacity Building Programs Division (IGC) in the International Security Branch (IFM). The division is headed by a Director, who is also responsible for the Counter-Terrorism Capacity Building Program (CTCBP) and its regional envelopes, the ACCBP Human Smuggling Envelope and the newly-funded CISCA initiative, the Afghanistan Counter Narcotics Program (ACNP), the Annual Voluntary Contributions (AVCs) to the UNODC and OAS-CICAD, and who reports to the Director General of Non-Proliferation and Security Threat Reduction Bureau (IGD).

The ACCBP Secretariat is comprised of a Chief of Program (Deputy Director level), four Senior Program Managers working on core programming, one Senior Project Officer who is responsible for the Human Smuggling Envelope, and a Project Assistant and Finance Manager. Additionally, an Administrative Assistant is shared with the CTCBP.

The Secretariat collaborates and coordinates with 18 OGDs who are involved in the governance structure of the Program. It also maintains ties with a myriad of organizations that implement ACCBP-funded projects, including, but not limited to multilateral organizations such as the UNODC, the OAS, and Interpol, as well as NGOs, private partners and other governments. Additionally, the Secretariat regularly engages with Canada's key allies - the United States and the United Kingdom - to maximize donor coordination and further the dialogue on how the three countries can continue to work together to help fight crime in the Americas.

2.0 Evaluation Scope & Objectives

The overall objective of this formative evaluation was to review and assess the relevance and performance of the ACCBP. Through systematic data and information collection and analysis, the evaluation examined how the ACCBP addresses a continued need and aligns with Government and Departmental priorities; the degree to which management structures and systems have been put in place to support the achievement of the ACCBP's expected outcomes; and the extent to which individual projects under the ACCBP are contributing to the objectives of the Program in an efficient manner. The evaluation also sought to identify best practices and lessons learned. Since the Program was only established in late 2009, the Program's performance was examined to the extent possible based on the available information.

While this evaluation took into consideration the extent to which the ACCBP addresses the needs of beneficiary states and DFAIT, and is aligned with federal government priorities and with DFAIT's strategic outcomes, it also examined how the Program fits into the larger context of security programming in the Department and the Government of Canada. In this vein, the evaluation focussed on the following themes:

• Appropriateness of, and balance between, project types, thematic area and geographic coverage.
• Complementarity between the ACCBP and other DFAIT and Government of Canada security programs.
• Processes and systems in place to ensure the achievement and sustainability of results.

3.0 Key Considerations

The ACCBP is delivered in several potentially volatile countries with high crime rates, corruption and political instability where programming needs and opportunities can change suddenly. In the case of the ACCBP, activities and outcomes may be disrupted by any number of factors outside the control of the Program such as a sudden increase in violence, political instability, lack of political will, or corruption.

One of the ACCBP's objectives is to prevent and respond to security threats, both in Canada and in beneficiary countries. However, measuring the extent to which security threats have been prevented is inherently challenging and presents difficulties to attributing results to the Program. This issue is further complicated by the fact that baseline data are unreliable in many countries in the Americas because the illicit activities targeted by the Program often go unreported as a result of less developed security systems and the lack of trust in public institutions.^{Footnote 3}

Additionally, given the strategic importance of the Americas and the commitment to combat crime, the ACCBP has become an increasingly important tool used by the Government of Canada to advance Canada's foreign policy priorities and demonstrate sustained engagement in the region. Consequently, political drivers have played a role in the allocation of funding for certain projects.

4.0 Evaluation Complexity & Strategic Linkages

The ACCBP is one of the two largest complementary and mutually supportive international security assistance programs under the same management structure that aim to build capacity abroad. The second Program, the Counter-Terrorism Capacity Building Program (CTCBP), was established in 2005 with the objective of providing assistance to beneficiary states that will enable them to prevent and respond to terrorist activity.

The ACCBP also complements several other DFAIT Programs, namely: the Afghanistan Counter Narcotics Program, which focuses on decreasing the production of illicit drugs in Afghanistan; the Global Peace and Security Fund (GPSF), which provide support for conflict prevention and peace building and respond to international crises, particularly in failing and fragile states. Additionally, the ACCBP supports the international wing of the National Anti-Drug Strategy, which is coordinated by Justice Canada, through both its core programming and by managing the implementation of the AVC to OAS-CICAD.

In March 2012, the Prime Minister announced the establishment of the Canadian Initiative for Security in Central America (CISCA), which will focus on: police training; border security; enhancing regional dialogue; strengthening justice and security institutions; promoting human rights; supporting conflict resolution and reconciliation processes; and preventing and intervening in cases of violence against vulnerable groups, such as women and youth. Projects funded by CISCA will be implemented through the ACCBP and the GPSF.

Given its focus on the Americas, the ACCBP is also a key element under the Americas Strategy's security pillar. Specifically, anti-crime initiatives aim to strengthen conditions to achieve justice, prosperity, security and peace for the peoples of the Americas through protecting and promoting respect for human rights and fundamental freedoms; preventing and combating violence, crime, terrorism and corruption; fighting the global drug problem and related crimes; and encouraging broader civic participation of all citizens of the inter-American community.

Another long-standing aspect of Canada's international efforts to counter transnational criminal activity has been to partner with the United Nations Office on Drugs and Crime (UNODC), the Organization of American States (OAS), and the Inter-American Drug Abuse Control Commission (CICAD), which are dedicated to counter-narcotics and other anti-crime initiatives. DFAIT has contributed to these organizations since 1999 in order to enable them to deliver their mandates. While both the UNODC and OAS-CICAD are partners of the ACCBP and deliver projects that fall under the Program, both organizations continue to receive additional contributions from DFAIT through Annual Voluntary Contributions (AVCs) under separate Treasury Board authorities.

5.0 Evaluation Approach & Methodology

The Evaluation took place between October 2011 and June 2012. It was managed and conducted by DFAIT's Evaluation Division (ZIE). An in-house approach was chosen due to the sensitive nature of several of the ACCBP's projects. The Evaluation team consisted of a Senior Evaluation Manager, an Evaluation Manager and an Evaluation Officer.

The conduct of the evaluation was overseen by an Evaluation Advisory Committee (EAC) comprised of personnel from the ACCBP Secretariat; a representative from DFAIT's Program Services Division (IXS); representatives from Public Safety Canada, Finance Canada and Transport Canada; and ZIE. The role of the EAC was to guide the evaluation process and validate its findings and conclusions.

5.1 Evaluation Matrix

An evaluation matrix was designed as a framework for the evaluation to help guide and facilitate the analysis toward findings and recommendations. The matrix outlines the key questions to be answered by the evaluation, the indicators that were used to assess each issue, as well as their sources of information.

5.2 Data Collection Methods

Document Review

A variety of literature related to anti-crime capacity building and programming was reviewed to help determine the relevance and performance of the ACCBP. More specifically, international studies on the effectiveness of anti-crime programs and reports pertaining to crime in the Americas provided evidence about the continued need for the ACCBP and the continuing relevance of the assumptions that led to the establishment of the Program. Additionally, ACCBP program documents, other Government of Canada program documents, and DFAIT corporate documents were reviewed to assess the extent to which the Program is aligned with departmental priorities and strategic outcomes, and the degree to which it complements other DFAIT and OGD security programs.

Project File Review

A thorough review of a sample of 21 project files was also undertaken as to provide the evaluation team with a detailed analysis of the range of projects funded under the Program. Subsequently, 3 projects were excluded from the analysis due to missing information.

A purposeful sampling methodology was used to ensure that the different types of projects were adequately represented. In designing the sample, projects were grouped by theme, total cost (materiality), type of implementing body and implementation status. Projects from every year of the Program's implementation (2009-2011) were selected. Once the projects were categorized, the following steps were taken to determine the sample:

1. A proportionate number of projects under each theme were chosen. For example, there were a total of 45 projects that fell under the Security Sector Reform theme, while only 4 were implemented under Money Laundering and Proceeds of Crime. Consequently, the sample included a larger number of Security Sector Reform projects than Money Laundering projects.
2. Within each theme, projects with varying materiality were chosen, including those with the highest and lowest dollar values.
3. The type of implementing body was then considered in order to ensure that all categories of implementation instruments were represented, including UN Agencies, organizations within the Government of Canada, OAS bodies, NGOs, international institutions other than the UN and the OAS and private companies.
4. The status of the project was also taken into account to ensure a proportional combination of projects that have been completed as well as those that are still in progress.

The project sample was prepared in consultation with the ACCBP Secretariat to ensure that all project documentation would be available and complete.

The file review aided in the assessment of whether the ACCBP is achieving its expected results in a cost-effective manner and whether the proper processes and structures are in place to support the achievement of results.

The review was structured such that the evaluation could determine the extent to which individual projects were contributing to the relevance and performance and of the Program. The analysis was conducted based on specific indicators under each category. All evidence included within each project file was then reviewed and assessed against those indicators. Reported project results were compared to the commitments made in each proposal.

Documents that were examined include:

• Project proposals
• Interim reports
• Final reports
• Financial documents/budgets
• Project-related email correspondence
• Relevant descriptive and analytical reports
• Previous audits and evaluations related to the ACCBP Program
• Relevant planning documents
• Communications plans and tools

Key Informant Interviews

Interviews were conducted with 39 key stakeholders from the following groups:

• DFAIT employees responsible for the management and delivery of the ACCBP;
• DFAIT employees who work with the ACCBP;
• Representatives from OGDs who partner with DFAIT in the delivery of the ACCBP;
• Representatives from recipient organizations charged with the implementation of the ACCBP projects abroad;
• Representatives of other coordinating donor countries.

These interviews helped the evaluation team gain a better understanding of the ACCBP in terms of its design and delivery and obtain stakeholders' perspective about the relevance and performance of the Program. Interviews also provided important information that facilitated the analysis of the efficiency, effectiveness of the Program and identify key issues, challenges, best practices and lessons learned.

Country Comparative Analysis

Available documentation on similar Anti-Crime Programs implemented by the United States such as the Central American Regional Security Initiative, the Caribbean Basin Security Initiative and the Merida Initiative were reviewed to assess the effectiveness and efficiency of DFAIT's ACCBP relative to similar programs implemented by other donors. This provided important insights of the initiatives that Canada's closest partner is undertaking in order to combat crime and build capacity in the Americas.

6.0 Limitations to the Methodology

Some of the projects under the ACCBP are classified for national security reasons. As a result, the inclusion of these projects in the evaluation was limited to high-level summaries rather than a detailed account of results achieved. In order to mitigate this lack of information, the evaluation team ensured that a sufficient number of unclassified projects were reviewed to derive conclusions regarding program performance.

The evaluation was also not able to undertake a fulsome country comparative analysis due to the sensitivity inherent in security sector programming and the consequent lack of publicly available information. However, public documents pertaining to similar programs in other jurisdictions, particularly the United States, were reviewed to determine how like-minded countries have structured and implemented similar anti-crime programs. Interviews with officials from the U.S. Department of State also provided valuable information that contributed to the Evaluation team's assessment of the ACCBP's performance and efficiency in relation to the programs implemented by our closest ally.

7.0 Evaluation Findings

The following findings are based on the triangulation of data from relevant literature, project files and key stakeholder interviews, and highlight the perceptions of the majority of ACCBP stakeholders. Some issues raised under the Relevance section are directly applicable to the Program's performance, as many evaluation findings speak to both of these evaluation issues to varying degrees.

7.1 Relevance Issue 1: Continued Need for the Program

In order to assess the relevance of the ACCBP, the Evaluation team relied on several different sources of information, including: academic literature on transnational crime in the Americas; corporate documents of the ACCBP; reports and studies conducted by international institutions and other countries; speeches made by Government of Canada Officials; documents and press releases articulating Government of Canada and Departmental priorities; interviews with 39 key stakeholders, and a review of 23 project files.

Finding 1:
The ACCBP targets its programming to the needs of the beneficiary states while ensuring that the degree of threat and risk to Canada and Canadians are considered when selecting priorities and projects.

The review of Program documents and the available literature determined that transnational crime originating in the Americas contributes to criminality in Canada. For example, The Royal Canadian Mounted Police reports that all cocaine that comes to Canada originates from South America, with transit points through Central America and the Caribbean. Moreover, gangs that originate in the Americas, such as Mara Salvatruchas, are extending their networks across the world, including in Canada.^{Footnote 4}

Criminality in many countries of the Americas also impacts Canadian tourists, as Canadian visitors to the region fall victim to petty and serious crimes, including homicide, every year.^{Footnote 5} For the vast majority of countries in the Americas, including popular tourist destinations such as Mexico, Jamaica, the Dominican Republic and Costa Rica, DFAIT advises Canadian tourists to exercise a "high degree of caution" due to elevated crime rates.^{Footnote 6} Additionally, crime is deterring foreign investment in South and Central America, which has reduced worker productivity, prevented the development of supply chains and increased operating costs for companies.^{Footnote 7}

While the needs and threats are broad and exist throughout the region, the ACCBP determines where these needs are the greatest through its Annual Priority Review exercise (Priority Review). The Priority Review articulates the priorities for the ACCBP following an extensive consultation with OGDs and a review of multiple threat assessments. The potential impact on threats and security risks to Canadians is another important factor that is considered in the selection of projects and priorities.

Sixty percent of the projects from the evaluation's sample address issues that possess a direct link to Canadian security. For example, targeted training programs for police who deal with the security of tourists are being implemented in countries that receive a high volume of visitors from abroad. The remainder of projects have a more indirect link to Canadian security in that they are located in countries that are primary sources or transit points of criminal threats to Canada or the region.

7.2 Relevance Issue 2: Alignment with Government Priorities

Finding 2:
The ACCBP is aligned with the priorities of the Government of Canada and DFAIT.

Since 2006, the Government of Canada has pledged to protect the security of citizens and "crack down on gun, gang and drug violence."^{Footnote 8} Canada's National Security Strategy also focuses on protecting the nation and its citizens, ensuring Canada is not a base for threats to allies, and contributing to international security.

During a visit to the region in July 2007, the Prime Minister announced the Americas as a Canadian foreign policy priority, and, as a result, a whole-of-government Strategy of Engagement in the Americas was launched in 2008. The appointment the same year of a Minister of State of Foreign Affairs with special responsibility for the Americas further underscored Canada's commitment.^{Footnote 9} The Americas continues to be a foreign policy priority, as articulated by the last two DFAIT Reports on Plans and Priorities.

Moreover, the Minister of Foreign Affairs has committed to contribute toward effective global governance and international security and stability by:

• Enabling the whole-of-government effort in Central America and Mexico to address security threats and institutional deficiencies using capacity-building expertise to support government and non-government institutions;
• Collaborating with the United States, the European Union, Mexico and Latin American partners to enhance security and build capacity in Central America; and,^{Footnote 10}
• Advancing cooperation and capacity building, in particular with Mexico, Central America and the Caribbean, including Haiti, to address security and the rule of law.^{Footnote 11}

The ACCBP directly aims to achieve these objectives by addressing threats to Canada in narcotics source and transit states, protecting Canadian travellers and decreasing transnational criminal activity by working collaboratively with international allies and partners.

The ACCBP also supports the increase of economic opportunities for Canadian businesses, which is a top priority for both the Government of Canada and DFAIT, as stated in the most recent Speech from the Throne and in the 2012-13 priorities of the Minister of Foreign Affairs. By reducing the impact of crime in the Americas, businesses will become more likely to invest in those countries and opportunities for Canadian investors may be expanded.

The ACCBP is consistent with DFAIT's mandate as set out in the Department of Foreign Affairs and International Trade Act, which is to manage Canada's diplomatic and consular relations and to encourage the country's international trade. Specifically, the ACCBP is a tool to help deepen bilateral relations with countries in the Americas, promote greater stability in the region, and encourage trade.

7.3 Relevance Issue 3: Consistency with Federal Roles & Responsibilities

Finding 3:
The management and implementation of the ACCBP is an appropriate role for the Federal Government and for DFAIT.

One of the primary responsibilities of the Federal Government is to ensure the safety and security of Canadians. Given that one of the ACCBP's primary long-term objectives is to prevent crime in Canada, the Program is directly aligned with this key role of the Federal Government.

The ACCBP also supports many of the Federal Government's commitments in the Americas, including:

• promoting economic opportunity;
• working with partners to improve the region's security and strengthen its institutions; and
• building a stable foundation for Canada's engagement by deepening Canada's relationships with state and non-state actors.

The ACCBP is also well-aligned with DFAIT's mandate, which includes:

• Conducting all diplomatic and consular relations on behalf of Canada;
• Fostering the expansion of Canada's international trade and commerce;
• Managing Canada's diplomatic and consular missions; and
• Administering the foreign service of Canada.

The Minister may also develop and carry out programs related to the promotion of Canada's interests abroad including fostering the expansion of Canada's international trade and commerce and the provision of assistance for developing countries.^{Footnote 12}

As part of this mandate and as stated in the Departmental Program Activity Architecture (PAA), DFAIT has been assigned responsibility for the development and coordination of international security policy and programs. The management of the ACCBP also requires coordination with other donor states and international organizations, which necessitates the use of DFAIT's network abroad. Additionally, while DFAIT's responsibilities do not relate specifically to the security of Canadians, departments that fall under the Public Safety Portfolio are mandated to focus on domestic security issues and are not resourced to undertake capacity building work abroad.

The ACCBP is also a tool that is used by Canadian embassies abroad to leverage other Canadian interests in the region. If the program were not administered by DFAIT, the benefits related to a strengthened Canadian engagement in the region and a deepening of bilateral relations would likely not be realized.

7.4 Performance Issue 4: Achievement of Expected Outcomes

The assessment of the ACCBP's performance was primarily based on the interviews conducted with key stakeholders and the review of 23 project files.

The interviews provided valuable insights into stakeholders' perceptions of the ACCBP's performance, as well as of how its design and delivery contribute to maximizing the Program's impact. They also helped identify the major challenges facing the Program. The file review was critical to the analysis of the Program's performance, as it provided evidence about the degree to which individual projects are achieving their expected outcomes and allowed the evaluation team to derive conclusions about the performance of the Program as a whole.

A review of a variety of Program documents was used as a secondary source of information which facilitated the analysis on whether processes and structures were in place to support the achievement of results in the longer term.

Finding 4:
The ACCBP has made good progress toward the achievement of its short-term outcomes.

The ACCBP's expected short term outcomes include:

1. Personnel in beneficiary States and organizations are more knowledgeable and skilled in addressing anti-crime issues.
2. Improved prioritization and coordination on anti-crime issues.
3. New or improved anti-crime legal instruments, controls and frameworks.
4. Additional or improved anti-crime tools, equipment, networks and physical infrastructure are available and in use.

Based on interviews, program documents and project file reviews, evidence indicates that good progress has been made toward the achievement of the following three short-term outcomes:

1. Personnel in beneficiary States and organizations are more knowledgeable and skilled in anti-crime issues:

Out of the total number of projects analyzed in the sample, 62% included a component related to the training of officials working on crime issues, which has increased the knowledge and skills in anti-crime policies, procedures and enforcement. According to the results reported in the narrative reports, ACCBP-funded projects have also helped build relationships and have enhanced inter-agency cooperation and coordination. This is particularly true for approved multiyear projects that have a regional focus.

2. Improved prioritization and coordination on anti-crime issues:

It was reported that all the needs assessments undertaken as part of ACCBP-funded projects have allowed for implementing partners to identify programming priorities and better allocate limited programming resources.

3. New or improved anti-crime legal instruments, controls and frameworks:

The installation of new tools and equipment are also often an important component of many ACCBP projects. Accordingly, the physical infrastructure related to anti-crime work has been improved and is now being used by beneficiary States.

The extent to which the ACCBP has contributed to the improvement of existing policies, legislation, regulations and controls is less clear. This may be due to the fact that the majority of projects do not identify the enhancement of legislative and regulatory frameworks as an expected outcome and, therefore, do not track their progress in this regard.

Finding 5:
The ACCBP has contributed to increasing Canada's visibility and leadership on anti-crime and security issues.

Based on interviews with DFAIT staff at missions and in the geographic bureaus, one of the key benefits of the ACCBP is that it has increased Canada's visibility in the Americas. Beneficiary states have provided positive feedback about projects and have highlighted Canada's contribution to security during international conferences. Local media in beneficiary states report regularly on what Canada is doing in the region and local stakeholders are increasingly turning to Canada for guidance and expertise on security-related issues.

Multilateral institutions such as the UNODC, the OAS and Interpol actively seek the partnership of the ACCBP in the implementation of projects, as Canada has been established as a reliable and credible affiliate. The U.S. State Department also works closely with the ACCBP and acknowledges the quality of Canadian expertise in combatting international crime.

Evidence obtained through interviews with key stakeholders, and particularly with staff at missions, indicates that OGD-implemented projects have provided the highest degree of visibility for Canada. However, the ACCBP Secretariat is also making concrete efforts to improve the visibility of Canada when implementing projects through multilateral delivery channels.

The Program has been able to take advantage of Prime Ministerial and Ministerial travel to the Americas to both announce new programming and highlight successes of previous activities and efforts. Moreover, IGC is putting in place Memorandum of Understanding (MOUs) with beneficiary states that highlight each ACCBP project implemented in that country regardless of whether the project is delivered bilaterally through an OGD or multilaterally through an international organisation. The ACCBP has also developed a "Project Summary" which provides project information in a concise document and is distributed to stakeholders upon project approval. These steps, combined with a continued focus on the ACCBP's communications strategy, will facilitate Canada's ability to highlight Canada's activities through both bilateral and multilateral delivery mechanisms.

Finding 6:
The ACCBP complements the work done by other areas within DFAIT. However, complementarity could be enhanced further by the establishment of standard operating practices among different programming areas.

The Program is complementary to the work done by several other areas within DFAIT. There is a high level of cooperation and coordination between the ACCBP Secretariat, the Geographic bureaus and missions in the Americas. More specifically, these organizations collaborate closely through formal and informal meetings and discussions to ensure that priorities are set, projects are targeted to where the need is the greatest and that the projects enhance the visibility of Canada and the awareness of ACCBP projects. Since the amalgamation of the ACCBP and the CTCBP under one single Director, activities and processes have also been well-coordinated and harmonized between the two Programs.

The ACCBP Secretariat works in a more limited capacity with other departmental programs such the Stabilization and Reconstruction Task Force (START) and the Global Partnership Program (GPP). While there are ad hoc meetings to discuss certain projects with other security programs within DFAIT, no standard mechanism has yet been established to ensure coherence between the different Programs. Feedback provided during key informant interviews suggested that a more standardized mechanism to discuss possible complementarity between each of these Programs would be beneficial and enhance the effectiveness and efficiency, particularly in light of the fact that both START and GPP are now programming in the Americas. Furthermore, additional clarification of the programming focus of START and the ACCBP would be beneficial to avoid duplication, as each Program's thematic priorities are broad and may overlap in some instances.

Finding 7:
Coordination and complementarity between the ACCBP and the work done by OGDs is strong and contributes to the effective delivery of the Program.

The ACCBP Secretariat determines Program priorities and selects projects in partnership with 18 OGDs. While many of the ACCBP's partners (e.g., Public Safety Canada, the RCMP, and CSIS) have specific responsibilities that are directly related to the security of Canadians, their core mandates focus on domestic security. As a result, DFAIT is the only department that has both the mandate and the financial and human resources to undertake anti-crime capacity building work abroad. By partnering with DFAIT in the delivery of the ACCBP, OGDs are able to extend the domestic reach of their mandates and contribute to improving security both domestically and abroad. Furthermore, this partnership helps ensure that the strategic direction and projects of the ACCBP do not duplicate work being done by other organizations in the federal government.

The clarity of roles and responsibilities of all partners involved in the ACCBP also contributes to the effective delivery and coordination of the Program with OGDs. The Terms of Reference for the Interdepartmental Steering Committee and the Project Review Committee clearly articulate the roles and responsibilities of members of the governance structure and partners agree that these roles are appropriate.

The inclusion of OGDs within the ACCBP's Governance Structure has contributed to a whole-of-government approach to anti-crime capacity building and has expanded the expertise of the Program. However, several OGDs are involved in the selection of projects and also submit project proposals, which often make it difficult for those organizations to remain objective in their analysis and selection of projects. The decision to transform the Project Review Committee into a consultative rather than an approval body has mitigated this issue to some extent, as have the ongoing efforts made by the ACCBP Secretariat and OGDs to collaborate closely to achieve Program objectives.

Finding 8:
The demand for funding to implement projects by OGDs exceeds the funds available.

Under current Treasury Board authorities, $700k is allocated under Vote 1 Programming (Operating Budget) to implement OGD projects while $12.6M is provided for projects that are implemented by other organizations using Vote 10 (Grants and Contributions) on an annual basis. OGD partners have noted that this split appears to be unbalanced and prevents them from participating in the Program to the fullest extent possible.

At the time of the decision to allocate ACCBP funding, it was largely unknown which organizations would be best placed to implement projects. Accordingly, the Program made the decision to execute the majority of its projects through international organizations in its first fiscal year given their expertise undertaking work related to anti-crime issues. However, as the Program has matured, the demand from OGDs to implement projects has grown, as has their capacity to deliver projects abroad.

While multilateral organizations and NGOs have demonstrated their ability to execute projects effectively and efficiently, the implementation of projects through those organizations does not provide the same degree of visibility for Canada as those managed by OGDs. An increased presence abroad for OGDs also highlights the skills and knowledge of those departments, and may thus contribute further to Canada's image as a leader and desirable partner on anti-crime issues.

Additionally, given that OGD-funded projects are sometimes constrained in scope by the allocation limits within Vote 1, several different stakeholders indicated that additional funding for OGD projects would facilitate a more strategic delivery of the ACCBP and allow the OGDs to build upon previous work done in the region. It should be noted, however, that OGDs have historically returned approximately 20% of their funding to the ACCBP at the end of every fiscal year, which is higher than the rate of returns from other implementing organizations. While financial data indicate that returns are becoming lower from certain federal departments, the benefits to Canadian visibility need to be weighed against the ability of OGDs to implement projects on time and on budget.

Finding 9:
The selection of projects is well aligned to Program priorities and based on relevant and appropriate set of criteria.

The process by which projects are selected is robust and includes a broad consultation with subject matter experts from 18 OGDs. The first step in the selection of projects is the determination of Program priorities through the annual Priority Review exercise. Priorities are selected based on the existing threats, the location of these threats, and the potential to address them effectively through capacity-building programming.

Individual project proposals are then selected based on whether they fall within thematic and geographic priorities, as well as the capacity of the implementing organization to manage and implement the project. Consideration is also given to the likelihood that beneficiary states, including local authorities, will support and commit to the implementation of the project. Most of the project files reviewed contained evidence of a commitment on the part of beneficiary states to facilitate the implementation of ACCBP projects, suggesting that this criterion has been applied effectively.

The ACCBP is now moving toward a more strategic approach to help develop regional capacities, maximize impact with few resources, and counteract displacement effects of security system reform and crime prevention activities. For example, during the past fiscal year, emphasis was placed on projects that change the behaviour of managers to ensure maximum impact and reach (e.g., "train the trainer" programs).

The criteria above have been applied consistently in the selection of projects, with the exception of projects that are implemented as a result of emerging foreign policy priorities. In these cases, political drivers determine which projects will be funded rather than the usual project selection processes.

Analysis of the ACCBP project universe suggests that the approved projects are well aligned with Program priorities. In principle, 60% of the funding is allocated to top tier priorities while 30% and 10% are allocated to the lower two categories of priorities. This is to ensure that there is a good distribution of projects between thematic and geographic areas while maintaining a focus on priorities. In 2011-12, 61% of funding was allocated to top priority projects; 25% to the second category of priorities 2; 2.5% to the remainder of priorities and 11.5 % to emerging foreign policy priorities.

Finding 10:
Overall, the ACCBP is a well-managed Program with established management processes and practices that contribute to the achievement of longer-term outcomes.

The ACCBP was established in 2009 and was originally managed under the Chief of Program at the Deputy Director level, who reported to the Director General. Given that funding was only provided by Treasury Board in December 2009, the ACCBP's immediate focus was on ensuring the disbursement of $10M in funding for projects over the first 3 months. Consequently, there was very little time to establish sound management processes and project section criteria or to set priorities.

Since the Program's first year and aided by creation of the Capacity Building Programs Division in 2010, many refinements have been made to the ACCBP's management processes and practices to support the achievement of longer-term outcomes, including:

• The processes for setting the ACCBP's annual strategic priorities and selecting projects are robust and have been formalized through the establishment of the annual Priority Review exercise.
• Project Officers are engaged with implementing organizations during the life cycle of a project and regularly request clarifications and progress updates.
• The Governance Structure provides a forum of subject matter experts to help guide the selection of priorities and projects to maximize ACCBP resources.
• The ACCBP Secretariat meets on an informal basis with a myriad of internal and external partners to identify synergies, ensure complementarity where possible and avoid duplication.
• The implementation of multi-year initiatives and a greater focus in programming allows Canada to build on existing initiatives and contribute to more sustainable results.

Finding 11:
Additional emphasis on Performance Measurement would enable more accurate tracking of the Program's achievements and would allow the Program to plan more strategically.

Project-level Performance Measurement

Currently, all implementing organizations are required to report on the expected performance of projects during the proposal stage, as well as on their actual performance through the use of standardized templates in the ACCBP's narrative reports. The review of ACCBP project files demonstrated that while the majority of recipients fill out these templates properly, the quality of performance reporting submitted by implementing organizations is varied and is not always well aligned with the commitments identified in the proposals.

Furthermore, half of the recipients track performance exclusively at the output level, even in the case of multi-year projects where outcomes should be the focus. Where outcomes are properly identified, the selected indicators sometimes rely on the delivery of outputs.

It is not always clear how implementers are using performance information to improve the design and delivery of projects or to support decision-making. There is an opportunity for the ACCBP to encourage recipients, particularly those who submit proposals for more than one phase of an initiative, to use available performance information to inform decision-making and improve the efficiency and effectiveness of their projects.

Performance information at the project level is the foundation upon which the ACCBP can assess its longer-term performance at the Program level. It is therefore important that the performance information received from recipients at this level is of high quality.

Program-level Performance Measurement

A Performance Measurement Strategy for the ACCBP has been developed but data have not been collected systematically by the Program, making it difficult to determine which types of projects and implementing organizations have been the most successful.

Furthermore, the Project Review Committee currently focuses on the prospective selection and approval of projects rather than on the results of previous or multi-year projects. As a consequence, information on which types of projects have been most successful in achieving their objectives is not included in the planning and selection of new projects. This diminishes the ability of the ACCBP to take into account lessons learned from previous projects.

7.5 Performance Issue 5: Demonstration of Efficiency & Economy

In its appraisal of efficiency and economy, the Evaluation team relied mainly on information from ACCBP project documents, interviews, and literature about similar initiatives in other countries, particularly the United States. These sources provided the richest information about the extent to which the ACCBP compares to other similar Programs in different jurisdictions, the degree to which outputs are being maximized in relation to inputs, and the extent to which the ACCBP is delivering good value-for-money.

Finding 12:
Early indications are that the ACCBP is providing good value-for-money.

Early indications are that DFAIT is reaping a good return on its investment in the ACCBP. Overall, the Program is on track to achieve its expected results and is perceived as being a valuable tool for enhancing diplomatic relations with countries in the Americas.

An effort has been made to ensure that limited resources are being used optimally through the scrutiny of project costs by the Project Review Committee and Project Officers. The ACCBP also has strong financial management. Weekly one-on-one meetings between project managers and the Financial Manager allow for the Program to be well run and for managers to be aware of how project budgets are managed. Available correspondence suggests that ACCBP staff consistently plays a challenge function with regard to recipients' financial management in the implementation of projects.

Efficiency of the officers in the division is high, particularly considering the fact that the ACCBP Secretariat is staffed to implement $15M of programming and considerably more funding has been disbursed during this past fiscal year. The overall operating budget for the Secretariat accounts for 2.5% of the total yearly budget, which is low to similar grants and contributions Programs.

Stakeholders perceive that the ACCBP is giving Canada a big "bang for the buck" in that very modest resources are increasing Canadian visibility and credibility in a priority region for the Government, as well as achieving results at the project level.

Finding 13:
Information Management is a challenge for the ACCBP and leads to inefficiencies.

While the ACCBP has created several different documents to capture the information related to the Program, the lack of a more comprehensive information management system that captures tombstone, financial and performance information has presented efficiency challenges for the Program.

The ACCBP has developed a Master Project List that captures the tombstone information for every project. This list has been generally well-maintained and reasonably kept up-to-date but does not compensate for the lack of an effective database. The Program has also developed a simple and effective financial management system that allows for detailed accounting of how much funding is allocated and disbursed to each project. However, this spreadsheet cannot capture financial information by country or region.

The requirement to input similar information into several different documents is time-consuming for project officers. Maintaining multiple sources of information leading to the development of data quality issues that limit the ability of the Program to report accurately on its performance and use performance information strategically.

Retrieving and reconciling project information from several different sources is tedious and extremely time consuming work that limits the program's capacity to manage projects effectively. This issue is particularly relevant considering the volume of requests from Ministers' offices for briefing material and project-specific information, as well as increasing demands from other parts of the Department.

Finding 14:
Efforts are being made to better coordinate the ACCBP with the anti-crime initiatives of other countries, particularly the United States.

Since 2010, the ACCBP has reached out to other donors, and in particular, to the United States in an effort to better coordinate programming in the region and enable the ACCBP to be more strategic about the projects it chooses to fund.

In support of this effort, the ACCBP Secretariat and the U.S. State Department speak regularly about programming in the Americas and representatives at Canadian and American missions often work together to ensure that efforts are coordinated and not duplicated. Furthermore, Canada and the U.S. now ensure that all new equipment delivered through their projects is interoperable and certain projects are already co-funded. Both countries are actively looking for more opportunities to partner in the future.

The deepening partnership with the United States is a natural progression from the close bilateral relationship and the fact that the U.S. has implemented anti-crime initiatives targeting Central America that are similar to, but not duplicative of the ACCBP. Two notable differences between the programming in Canada and the U.S. include the vast amounts of funding and financial flexibility of American programs.

Canada also relies on international organizations such as the OAS and the UNODC to provide valuable information about what other donors are doing and help decrease the duplication of efforts. Due to the vastness of their operations, both organizations are able to provide information about the broader programming context, facilitating better donor coordination. Where possible, DFAIT missions and the Regional Office on Peace and Security centre also provide intelligence about what other donors are doing.

Finding 15:
As a result of an increased profile in the ACCBP by political officials, the current pace of work is challenging and may diminish the ability for staff to manage the Program strategically in the longer term.

The ACCBP staff has contributed to the delivery of several important departmental commitments in the Americas and has supported the Government's engagement strategy in the region. Overall, multiple lines of evidence suggest that the Program benefits target countries who receive assistance and promotes Canada's interests abroad.

While these successes are notable, the ACCBP Secretariat has recently experienced some work-load challenges in the implementation of the Program. Specifically, the Division's human resources have been stretched over the past year and a half as a result of the increased level of attention devoted to the Americas by political officials. The increase in the profile of the Program has increased the amount and intensity of information exchanges, particularly with the Office of the Minister of State of Foreign Affairs and the Americas. The rise in high-level bilateral visits and the increased role in contributing to the organization of international and regional conferences have placed significant strain on ACCBP staff. Moreover, the increased demand for the Program on the part of potential beneficiaries and by the Government of Canada has resulted in the ACCBP Secretariat disbursing a larger amount of funding than originally anticipated in the past fiscal year.

While the Program has been able to meet these increased commitments without increased staff levels or funding, the current pace of work is limiting the ability of Program staff to plan strategically and maintain a focus on the good management processes that have been put in place, including timely project monitoring practices.

8.0 ACCBP Best Practices

The ACCBP has undergone several changes since its establishment in 2009 and is maturing as a Program. Lessons have been learned and best practices established that have contributed to the improvement of the Program. These include the following:

1. Annual Priority Review Exercise

The formalization of the Priority Review into the annual management processes of the ACCBP in FY 10/11 has been one of the most important changes made by the Program since its establishment in 2009. Its purpose is to set Program priorities based on concrete evidence and identified priorities in order to maximize the ACCBP's impact.

The conclusions derived from the Priority Review are developed in consultation with the ACCBP's partner OGDs and are based on the analysis of threat assessments, academic literature, and meetings with subject matter experts. The priority selection process is sensitive to established priorities such as the Americas Strategy, as well as the presence of threats to Canadians and Canadian interests at home and abroad. The process of selecting priorities is multidimensional, with emphasis placed on beneficiary state needs, severity and growth of security problems, and the extent to which Canada can add value through security programming.

Under normal circumstances, the Priority Review should be conducted based on the following timelines:

• July: Determination and analysis of threats based on a document review and working-level meetings with partners such as Public Safety Canada, the RCMP, CBSA, CSIS, and PCO.
• August: Drafting of the Report.
• September: Circulation of the Report to DFAIT and OGD partners at the Director-level; phone calls by Director to the OGDs to discuss the priorities.
• October: Initial meeting with all OGD partners at the Director-level, who are allowed an additional week to provide comments in writing.
• October: Circulation of updated Report to DFAIT and OGD partners at the Director General-level.
• November: Concluding meeting with all OGD partners at the Director General-level, who are allocated an additional wee to provide final comments in writing.
• November: Report finalized and circulated to partners.
• December: Priority Review shared with the close allies, including U.S. and U.K. embassies.

2. Continuous Outreach and Engagement

The ACCBP is intended to bring a whole-of-government perspective to anti-crime capacity building and is delivered in a volatile environment with numerous other donors. As a result, the deepening of relationships with OGD partners and other donors and international institutions has been important as the Program matures.

On a strategic level, closer relationships with OGDs have allowed the ACCBP to integrate better quality threat analysis into the determination of priorities and have brought more subject matter expertise into the Program's decision-making processes. The ACCBP receives continuous feedback from partners regarding both priorities and projects, which enhances its ability to be focused on its programming and address countries where security needs are the most pressing. The contribution of OGDs is also important in ensuring that the ACCBP is well-aligned with Government of Canada priorities and that it is adding value to the efforts of other departments while avoiding potential duplication of efforts.

From a practical perspective, the strengthening of relationships with OGDs has resulted in smoother Program operations and has facilitated better cooperation between the ACCBP and its partners.

Enhanced collaboration with other donors and international organizations has also been beneficial. The sharing of Canadian priorities with the United States, the United Kingdom, Australia, the OAS and the UNODC has created a better coordination of programming efforts in the Americas and has helped ensure that equipment that is provided to beneficiary States is interoperable. Over the long-term, this collaboration is intended to increase the value-added of security sector initiatives through improved donor coordination and enhance the sustainability of the results achieved.

3. Criteria for the Selection of Projects

The ACCBP has recently established guidelines that help the Program propose projects for selections that maximize the impact of ACCBP funding. The criteria are set out in the Priority Review and encourage decision-makers to give priority to projects that build upon previous endeavours and that combine both training and equipment components.

These criteria were established to assist the ACCBP in providing strategic and targeted programming on behalf of Canada as well as to ensure sustainable results that will contribute to long-term gains in regional security and security sector capacity throughout the Americas. These criteria should be updated continuously as the Program progresses to reflect new lessons learned and emerging best practices.

4. Memoranda of Understanding (MOUs) with Beneficiary States

The ACCBP has begun to establish Memoranda of Understanding with beneficiary States to highlight Canada's contribution through the ACCBP and to place quasi-legal obligations on the beneficiary. The continued implementation of MOUs is important, as they have resulted in several important benefits to the Program, including:

• Increased operational efficiencies;
• The solidification of a beneficiary commitment to facilitate the implementation of a project;
• A means for Canada to properly track the equipment it provides to beneficiary;
• A mechanism that requires recipient States to report on results achieved;
• A means to control communications and announcements related to capacity building activities;
• Increased visibility for and recognition of Canada's capacity building projects in the beneficiary state by the beneficiary states.

While these MOUs are labour intensive, the recent development of a standardized template and the clarification of the purpose and scope of these documents with DFAIT's Legal Bureau have reduced the work associated with implementing these arrangements. The challenges that remain in the implementation of MOUs include tailoring each MOU to the specific recipient state and the intensity of some negotiations.

As MOUs cannot be put in place with all beneficiary states at the same time, prioritization is given to projects involving a significant amount of equipment, respond to an urgent need, or when a high-level visit is planned in a beneficiary state.

9.0 Conclusions of the Evaluation

This Evaluation has provided an opportunity to review the work of the Anti-Crime Capacity Building Program and the structures and processes it has put in place since its establishment in September 2009. Overall, the Program continues to be relevant and the initial indications regarding performance suggest that results are being achieved and will continue to be attained over the longer term. With some minor adjustments, it is expected that the ACCBP will be a valuable tool for the Department in the future, particularly if the Americas remains a priority region for the Government of Canada.

Conclusion 1:
The ACCBP addresses an important need in the Americas and plays a unique role in fulfilling Canada's commitments in the region.

Crime in the Americas continues to pose serious threats to the security of people in the region, as well as around the world, including Canadians. It also hinders economic prosperity and prevents the development of democratic institutions, and discourages the development of a full respect for human rights and the rule of law.

While security threats are multi-dimensional and exist in all countries in the region, the ACCBP focuses its programming efforts to target where the threats are the greatest through a thorough review of available evidence. The security risk posed directly to Canada and Canadians travelling abroad is also taken into account when priorities and projects are selected.

The ACCBP is fully aligned with Canada's priorities in the Americas, supports its international commitments to fight crime and terrorism, promoting democracy and human rights. The Program also provides a tool that deepens bilateral relations, promotes greater stability in the region, and encourages trade. Moreover, the ACCBP is unique within the Government of Canada in that it is the only Program that is mandated and resourced to undertake anti-crime capacity building work abroad.

Conclusion 2:
Overall, the ACCBP is a well-managed Program that is on track to achieve its expected results.

The ACCBP shows excellent progress toward the achievement of short-term outcomes and has established sound management processes and best practices that support the achievement of longer-term objectives.

Efforts to implement projects with training and equipment components, as well as a continued emphasis on selecting Program priorities and projects based on evidence and a clear set of criteria are critical for the ongoing operations of the Program. Additionally, a continued emphasis on implementing projects with a regional focus that build on previous work done by Canada and other donors will support the sustainability of project results and contribute to the achievement of the Program's longer-term objectives of building capacity and improving the security situation in the Americas.

The deepening of relationships with Government of Canada and external partners in order to improve collaboration and complementarity and avoid duplication has also enabled more effective and efficient program operations.

The ACCBP has also contributed to increasing Canada's visibility in the region and, with the ongoing collaboration and engagement of missions and the geographic bureaus, should continue to highlight Canada's expertise and leadership on anti-crime issues.

Conclusion 3:
Increased emphasis on performance measurement and information management would allow the Program to better track and report on the results it is achieving and program more strategically in the longer term.

The ACCBP has put in place structures to encourage implementing organizations to report on results, however, the use of performance information continues to be varied and generally focuses on outputs rather than outcomes. As a consequence, the Program has difficulties analyzing and demonstrating the full extent to which it is achieving results.

An increased awareness and emphasis on ensuring that the ACCBP is receiving meaningful performance data from implementing organizations, including the tracking of outputs and outcomes and the use of appropriate indicators, would help to guide project management and identify lessons learned and best practices that will support increased effectiveness of anti-crime programming.

It is also a challenge for the Program to report on results in an efficient manner because of the lack of a comprehensive mechanism to track and analyse performance data such as a database. More consistent performance tracking at the Program level and a more coherent and integrated information management system could help the Program to report more accurately report on its successes, use performance information to make decisions, identify lessons learned and best practices, and program more strategically over the long-term.

Conclusion 4:
The demand for ACCBP funding from OGDs has increased since the Program's establishment in 2009.

The current amount of funding that is allocated toward projects implemented by OGDs is lower than the demand for ACCBP funds from these federal organizations. While the allocation in Vote 1 Programming (Operating Budget) cannot currently be modified given that changes would require approval from the Treasury Board of Canada, OGD partners have expressed the view that current allocation amounts prevent them from fully participating in the Program.

The evidence indicates that although the funding of ACCBP projects through multilateral delivery mechanisms has resulted in the achievement of objectives, these projects do not provide the same degree of visibility for Canada as those projects managed through bilateral delivery mechanisms such as OGDs, the Canadian Commercial Corporation, and other Canadian NGOs. Additional programming through OGDs also highlights the skills and expertise that can be contributed by the Government of Canada to beneficiary states and would allow OGDs to plan their projects more strategically.

While additional funding to OGDs would provide benefits for Canada, the decision to reallocate money toward OGD projects needs to be carefully considered by the Program, particularly in light of the fact that historical trends indicate that OGDs return more funding at the end of the fiscal year than other implementing partners. While funding has been returned at progressively lower levels, indicating improved project management on the part of OGDs, additional monies allocated to OGDs should be commensurate not only with the increased demand for the Program, but also with the capacity of OGDs to deliver programming in an efficient manner compared to other implementing organizations.

10. Recommendations

Recommendation #1:
During transitions in management of the Program, the ACCBP should continue to implement and refine the management practices and processes it has established.

Since its establishment in 2009, ACCBP has developed and implemented a number of sound management processes and practices to support the achievement of longer-term objectives in an efficient and effective manner. These practices include but are not limited to:

• Integrating the Priority Review into the annual management practices of the ACCBP, and making the priority setting process multidimensional, with emphasis placed on beneficiary state needs, severity and growth of security problems, and the extent to which Canada can add value through security programming.
• Increasing the efficiency of the project selection process by reviewing the roles and responsibilities of the ACCBP Secretariat.
• Incorporating subject-matter experts into the Program's governance structure.
• Moving from annual to multi-year funding for projects and initiatives to increase their impact and efficiency.
• Establishing Memoranda of Understanding with beneficiary States to highlight Canada's contribution through the ACCBP and to articulate the roles and responsibilities of Canada and the beneficiary state.

While most of these management practices have been well received by ACCBP staff and DFAIT Senior Management, they need to be consistently applied and further refined as needed by future Program Managers and staff, especially during periods of transition and changes in the management of the program.

Recommendation #2:
The ACCBP should establish standard operating practices with other Departmental security programs to ensure coordination and complementarity and to avoid duplication.

The ACCBP Secretariat collaborates closely with the Geographic Branches and Missions in the Americas. It also shares the same governance structure with the CTCBP. The Program does not, however, possess standard practices for consultation and exchange of information and best practices with other Departmental security programs such as START and the GPP. Increased collaboration could contribute to the coherence, effectiveness and efficiency in the planning and implementation of security related projects and initiatives, especially as other Departmental security programs seek to implement projects in the Americas.

In order to facilitate better information sharing, priority setting and programming coordination with other DFAIT security programs, especially at the country and regional level, the ACCBP should establish more standardized practices.

Recommendation #3:
As it matures, the Program should focus on improving performance measurement and information management.

There are several means by which the ACCBP could improve the manner in which is measures its performance, including:

• Continuing to work toward the development of a more comprehensive information management system.
• Incorporating a Program-level analysis of results and lessons learned as part of its regular reporting process.
• Modifying its Results-Based Management and reporting templates to include more detailed guidance and instructions on the types of performance information it would like to collect from implementing organizations.
• Creating an applicant guide for recipients to detail the Program's expectations for the quality of performance information as well as expectations for performance reporting and use of performance information in project management.
• Requesting additional clarification from implementing organizations on the performance of individual projects when the quality of performance information is lacking in reports and encouraging recipients to focus on tracking outcomes.

Recommendation #4:
If an opportunity arises in the future, the ACCBP should re-examine the current funding allocation between Vote 1 Programming (Operating Budget) and Vote 10 (Grants and Contributions). Funding in priority regions should be allocated based on the increased demand for the Program, as well as the capacity of OGDs to deliver programming in an efficient and effective manner.

Projects implemented by multilateral organizations do not provide the same degree of visibility for Canada as bilateral projects implemented through OGDs. In addition, projects delivered by OGDs highlight the skills and expertise available within the Government of Canada and encourage further bilateral partnerships and cooperation. Increased funding for OGDs in Vote 1 would facilitate a more active involvement of OGDs and enable the implementation of, longer-term bilateral initiatives that maximize the potential impact within the recipient country.

If the opportunity arises to increase and re-allocate funding for the Program, the ACCBP should increase the amount of funding available to OGDs in a manner commensurate with both their implementation capacity and their demand for ACCBP funds. The need for reallocation could be assessed through regular tracking of disbursement trends and consultations with OGD stakeholders.

11.0 Management Response and Action Plan

Recommendation 1

During transitions in management of the Program, the ACCBP should continue to implement and refine the management practices and processes it has established.

Associated Findings:

Management Response & Action Plan: Agreed. The ACCBP will continue to implement and streamline the management processes it has established. The ACCBP recognizes that in order to continue as a successful program, this process needs to be dynamic and responsive.
Responsibility Centre: IGC
Time Frame: On-going

Recommendation 2

The ACCBP should establish standard operating practices with other Departmental security programs to ensure coordination and complementarity and to avoid duplication.

Associated Findings:

Management Response & Action Plan: Agreed. The ACCBP will continue to pursue systematic collaboration with other Departmental security programs. This could include, for instance, regular meetings between management teams of sister programs (e.g., GPP and GPSF), mutual consultation on PIAs, and joint needs assessment missions, where feasible.
Responsibility Centre: IGC
Time Frame: On-going

Recommendation 3

As it matures, the Program should focus on improving performance measurement and information management.

There are several means by which the ACCBP could improve the manner in which is measures its performance, including:

• Continuing to work towards the development of a more comprehensive information management system.
• Incorporating a Program-level analysis of results and lessons learned as part of its regular reporting process.
• Modifying its Results-Based Management and reporting templates to include more detailed guidance and instructions on the types of performance information it would like to collect from implementing organizations.
• Creating a recipient guide for implementing partners to detail the Program's expectations for the quality of performance information as well as expectations for performance reporting and use of performance information in project management.
• Requesting additional clarification from implementing organizations on the performance of individual projects when the quality of performance information is lacking in reports and encouraging recipients to focus on tracking outcomes.

Associated Findings: 11, 13

Management Response & Action Plan: Agreed. The lack of critical tools, such as a comprehensive information management system or programming database, constitutes an important management gap not only for the ACCBP, but for all DFAIT programs; indeed, this has been noted in the draft report of the Audit of Management of Transfer Payment Programs. IGC, working with IGA and IRD, has been pro-actively working on a "short-term" database solution that would see the Capacity Building Programs and the Global Partnership Program adapting START's project database to their respective needs. A more robust, medium-term solution that offers a more comprehensive set of information management solutions - including capturing performance measurement at the program level - is beyond the sole discretion of IGC and depends on the broader IM priorities of DFAIT and Service Canada (e.g., including the MapAmericas initiative requested by Minister of State of Foreign Affairs (Americas and Consular Affairs) which would look to capture the same information, but is specific only to the Americas.)
Responsibility Centre: IGC, DFAIT Information Management Division and Service Canada
Time Frame: Short-term database solution in place by early FY 13/14. Medium-term solution ongoing, and beyond control of IGC.

Management Response & Action Plan: Agreed. The Annual Priority Review Process currently includes a basic analysis of results and lessons learned. This section could be expanded to undertake a more robust analysis, within the limitations on gathering performance management information imposed by the lack of information management tools. It can also be reflected as appropriate in other forms of regular, ongoing reporting.
Responsibility Centre: IGC
Time Frame: On-going annual

Management Response & Action Plan: Agreed. IGC recently modified its Results-Based Management and reporting templates to ensure that the ACCBP received regular reporting on the performance and results of projects. Simplified and more clear guidance and instructions (e.g., a checklist) would be a useful addition.
Responsibility Centre: IGC
Time Frame: Templates updated by end of FY 12/13

Management Response & Action Plan: Agreed. Similar to the previous sub-recommendation, the creation of a recipient guidance for implementing partners (e.g., a checklist) would be a useful addition.
Responsibility Centre: IGC
Time Frame: Checklist created by end of FY 12/13

Management Response & Action Plan: Agreed. The ACCBP will continue to carefully review reporting information received, and regularly encourages recipients to focus on tracking outcomes.
Responsibility Centre: IGC
Time Frame: On-going

Recommendation 4

If an opportunity arises in the future, the ACCBP should re-examine the current funding allocation between Vote 1 Programming (Operating Budget) and Vote 10 (Grants and Contributions). Funding in priority regions should be allocated based on the increased demand for the Program and the capacity of OGDs to deliver programming in an efficient and effective manner.

Associated Findings: 8

Management Response & Action Plan: Agreed. Should an opportunity arise in the future, ACCBP will examine the funding allocation to ensure continuing efficient and effective program delivery adjusted for emerging circumstances. Any decision to increase the Vote 1 programming allocation to allow for increased program delivery by OGDs would need to take into consideration (1) project management challenges faced by some OGDs, including staff reductions; (2) the prospect of unexpected returned funds to the ACCBP, with the associated risk of a lapse; and (3) consistent implementation of MOUs between OGDs and DFAIT.
Responsibility Centre: IGC, in consultation with SWP
Time Frame: When increase in reference levels is sought.