Evaluation of the Global Partnership Program - Final Report

Global Affairs Canada
Inspector General Office
Evaluation Division

May 2016

Abbreviations, Acronyms and Symbols

Acknowledgements

The Evaluation Division (ZIE), Office of the Inspector General (ZID) of the Department of Foreign Affairs, Trade and Development Canada (DFATD), would like to thank the staff and management of the Global Partnership Program (GPP) for their cooperation and the members of the Evaluation Advisory Committee for their guidance and advice. Special thanks to all of the representatives of GPP partner organizations who agreed to be interviewed for the evaluation.

Executive Summary

This formative evaluation of the Global Partnership Program (GPP) was conducted by the Evaluation Division (ZIE), Office of the Inspector General (ZID), of the Department of Foreign Affairs, Trade and Development Canada (DFATD) as part of the departmental Five-Year Evaluation Plan. Consistent with the Treasury Board Policy on Evaluation, the purpose of this evaluation was to assess the relevance and performance of Phase IV of GPP, that is, since the 2011 decision to broaden its geographic mandate and focus on new programming priorities. The target audiences for this evaluation are the Government of Canada (GoC), DFATD’s Senior Management, program managers and the Canadian public.

Background

The Global Partnership (GP) against the Spread of Weapons and Materials of Mass Destruction (WMD) was launched in 2002 at the G8 Summit in Kananaskis, Canada. This partnership was established in order to address the risk of non-state and state actors of proliferation concern acquiring WMD and related materials. The Government of Canada (GoC) initially committed $1 billion over a period of 10 years for projects primarily in Russia and countries of the Former Soviet Union.

At the Deauville Summit in France in 2011, GP member states declared that the priorities identified at the 2002 Kananaskis Summit regarding Russia had largely been met and recommended that the GP mandate be extended beyond 2012 with an emphasis on global expansion. In 2012, GPP received approval for $367 million in new funding over five-years (2013-2014 to 2017-2018) for programming in Asia, the Americas, the Middle East, and Africa to address the new priority themes while winding down its commitments to chemical weapons destruction and nuclear and radiological security in the Former Soviet Union. Moving forward, the GP confirmed the following four new priority areas:

• Nuclear and radiological security
• Biological security
• Support for the implementation of UNSCR 1540 and scientist engagement (e.g. countering the proliferation of WMD knowledge).

Evaluation Scope and Objectives

The overall goal of this evaluation was to provide DFATD’s senior management with an evidence-based assessment of GPP’s activities and projects under its global expansion mandate and to provide program managers with information and feedback on how GPP implementation could be improved. The evaluation examined GPP from the inception of the spending authorities for its worldwide mandate in February 2011 to January 2015. The primary focus was on programming developed since the latest renewal of funding authorities for the Program that went into effect in April 2013. The evaluation also followed up on the recommendations of the 2012 Summative Evaluation of GPP.

Evaluation Approach and Methodology

The evaluation employed a combination of qualitative and quantitative methods and multiple lines of inquiry. Qualitative methods included 69 stakeholder interviews, observation through site visits to eight countries, a document review and a project file review to respond to all of the evaluation issues. These methods were complemented by quantitative methods to assess administrative data and other program information.

Key Findings

The evaluation found that there is a continuing need for GPP to address significant ongoing threats from WMD proliferation in the areas of nuclear, radiological, chemical and biological security. Evidence showed that the Program is aligned with DFATD’s Strategic Outcomes, the priorities and objectives of the federal government and is consistent with the federal role.

The evaluation found that since 2011 the GPP has largely supported short-term projects focused on the delivery of outputs such as equipment and training, and that results reporting focused on outputs rather than their contribution towards the achievement of outcomes. This said, the evaluation found some evidence that each of the GPP sub-programs are contributing to immediate outcomes. Results were most evident where GPP made focused interventions and/or was working with international organizations. For example, GPP’s support for the work of the IAEA contributed to that organization’s inspections of Iran’s nuclear facilities. GPP’s collaboration with the OPCW contributed to the destruction of chemical weapons stockpiles in Syria and Libya.

In Jordan, GPP’s support for multiple projects across its three sub-programs contributed to that country’s ability to detect WMD and respond to CBRN incidents that might arise from instability in Iraq or Syria. The evaluation found that such focused investments were also the most likely to have contributed to results for Canada in the form of Canadian leadership, influence and visibility. Where applicable, only limited evidence was found of projects that included approaches to ensure longer-term sustainability of results.

GPP completed the first step in its strategic planning process of identifying a broad list of countries/regions for engagement. GPP should review and expand on its annual priority setting process by developing strategic plans at the program and sub-program level to ensure that the Program is addressing the thematic and geographic areas of greatest and most imminent threats to Canada. Once developed, the strategic plans should be used to inform the development of project proposals.

GPP’s ability to plan strategically was undermined by the departmental project approval process in effect during the evaluation period. That process also dramatically reduced the ability of the program to disburse its approved funding, adversely impacting the efficiency and economy of program delivery.

GPP developed SOPs and the evaluation found that it is fulfilling some of the basic elements of project management in relation to financial and narrative reporting. The evaluation found, however, little documented evidence that GPP is systematically conducting needs and risk assessments or collecting and integrating lessons and best practices into its work. A lack of standardized reporting tools also hindered project management and reporting. The evaluation found no evidence of duplication between GPP and DFATD’s other security programs or programs working at cross purposes, however, there is scope for increased coordination between GPP and other security programs, other DFATD programs, and OGDs.

GPP has made progress in information management by adopting the Security Sector Program Management Tool (SSPMT), however further work is needed to integrate modules for risk, results reporting and lessons. GPP also needs to integrate its project documentation into the database and develop a registry for project files as record keeping varied from project to project.

GPP’s performance measurement was weak. It was not using its Performance Measurement Strategy to identify the outcomes its projects were expected to achieve and there was little reporting on results.

It is recommended that:

Recommendation #1: GPP update its priority setting process and develop strategic plans to shape programming activities.

Recommendation #2: GPP update its SOPs to include guidance for needs assessments, risk management, reporting, and integrating lessons learned into future programming initiatives. GPP should ensure that the SOPs are being consistently followed.

Recommendation #3: GPP work with other security and development programs, as well as geographic divisions and missions, to develop mechanisms that will allow for increased integration, coordination and synergy.

Recommendation #4: GPP management update and integrate its Performance Measurement Strategy into its planning, implementation and reporting to ensure that programming is focused on the achievement of expected outcomes.

1.0 Introduction

The Evaluation Division (ZIE) at the Department of Foreign Affairs, Trade and Development Canada (DFATD) is housed within the Office of the Inspector General (ZID) and is mandated by Treasury Board (TB) through its 2009 Policy on Evaluation to conduct evaluations of all departmental direct program spending, including grants and contributions programs. The Evaluation Division reports to the Departmental Evaluation Committee, which is chaired by the Deputy Minister of Foreign Affairs, the Deputy Minister of Trade and the Senior Associate Deputy Minister of Foreign Affairs.

The Evaluation of the Global Partnership Program (GPP) is part of the approved departmental Five-Year Evaluation Plan. Consistent with the Policy on Evaluation, the purpose of this evaluation was to assess the relevance and performance of GPP since 2011 when the Program received spending authorities for a world-wide geographic mandate and focus on the new programming priorities. The target audiences for this evaluation are the Government of Canada (GoC), DFATD’s Senior Management, program managers and the Canadian public.

1.1 Background and Context

The Global Partnership (GP) against the Spread of Weapons and Materials of Mass Destruction was launched in 2002 at the G8 Summit in Kananaskis, Canada. This partnership was established in order to address the risk of non-state and state actors acquiring WMD and related materials. The Kananaskis Summit resulted in international cooperation on non-proliferation aimed at destroying and securing Weapons of Mass Destruction (WMD) and preventing terrorist organizations and state actors from acquiring such materials. To this end, the GP committed over $20 billion over ten years to support cooperation and programming.

At the 2007 G8 Summit in Heiligendamm, Germany, it was remarked that the GP had to evolve to respond to new and emerging security threats. At the same summit, G8 leaders encouraged GP member states to support the implementation of United Nations Security Council Resolution (UNSCR) 1540. UNSCR 1540 places binding obligations on all UN state members to adopt legislation and domestic controls to prevent the proliferation of nuclear, chemical and biological weapons and their means of delivery.^{Footnote 1}

Subsequently, at the 2008 G8 Summit in Toyako, Japan, GP member states agreed to expand the reach of GP programming by adding new recipient and donor states on a case-by-case basis to help address WMD proliferation risks worldwide.^{Footnote 2} In 2010, at the Muskoka Summit in Canada, G8 leaders again encouraged the GP to expand its membership as a means of facilitating global programming.^{Footnote 3} At the Deauville Summit in France in 2011, GP member states declared that the priorities identified at the 2002 Kananaskis Summit regarding Russia had largely been met and recommended that the mandate be extended beyond 2012 with renewed emphasis on global expansion.^{Footnote 4} Moving forward, the GP confirmed the four new priority areas, which were first identified at the Muskoka Summit:

• Nuclear and radiological security
• Biological security
• Support for the implementation of UNSCR 1540 and scientist engagement (e.g. countering the proliferation of WMD knowledge).

1.2 Canada’s Global Partnership Program

Following the establishment of the GP in 2002, the GoC committed $1 billion over a period of 10 years for projects primarily in Russia and countries of the Former Soviet Union. Canada’s Global Partnership Program was established in 2003 and funding was allocated for Phase I and II of the Program (2003/04 to 2007/08) to five priority areas: 1) nuclear and radiological security; 2) chemical weapons destruction; 3) redirection of former weapons scientists; 4) nuclear-powered submarine dismantlement; and 5) biological non-proliferation.

GPP evolved through a number of phases marked by separate TB submissions, including submissions for specific priority areas (e.g. Chemical Weapons Destruction, Nuclear Submarine Dismantlement). In June 2009, GPP received policy authorities to expand its programming globally. In February 2011, the Program pursued and received spending authorities from TB for a notional allocation of up to $42 million in remaining GPP funding for threat reduction programming in countries and regions outside the Former Soviet Union in keeping with the decision made by G8 leaders at the 2010 Muskoka Summit.

In 2012, GPP received approval for $367 million in new funding over five-years (2013-2014 to 2017-2018) for programming in Asia, the Americas, the Middle East, and Africa to address the priority themes identified at Muskoka, while winding down its commitments to chemical weapons destruction and nuclear-powered submarine dismantlement in the Former Soviet Union.

1.3 Resources

GPP received a budget of $73.4 million per year in the period 2013-2014 to 2017-2018. Close to $67 million of this funding was allocated for assistance in the form of grants, contributions and operating funds used for programming via other government departments (OGDs). The remaining $6.4 million was allocated for operating costs, such as personnel, travel, audit, and evaluation.

1.4 Program Mandate and Objectives

The overarching mandate of GPP is to reduce the threat posed by weapons and materials of mass destruction and related expertise to Canadians and the international community by implementing projects in cooperation with G8 and other nations.

More specifically, GPP aims to achieve the following:

Expected immediate outcomes:

• Security measures for storing/transporting WMD related materials are improved
• Prevention, detection and response capabilities are improved
• Capacity to destroy/reduce/convert materials is increased
• Networks and partnerships are developed
• Awareness by technical experts of threats and opportunities is increased
• Legal instruments, legislation, controls, guidelines and frameworks are improved

Expected intermediate outcomes:

• Safety and security of materials in storage and during transportation is enhanced
• Supply of materials is reduced
• Prevention and detection capabilities are strengthened
• Compliance with and integrity of international WMD commitments is increased

Expected ultimate outcomes include:

• Access by state and non-state actors of proliferation concern to WMD-related materials, technology and expertise is reduced
• Readiness of national and global networks to respond to incidents of attempted or actual WMD proliferation is strengthened
• Reduce the threat posed by weapons and materials of mass destruction to Canadians and international community

GPP achieves these objectives by implementing projects through four different delivery mechanisms:

• Multilateral Delivery: GPP contributes to the WMD threat reduction activities of relevant international and/or regional organizations such as the International Atomic Energy Agency (IAEA), International Criminal Police Organization (INTERPOL), and the World Health Organization (WHO).
• Bilateral Donor Delivery: In cases when Canada’s GP partner countries have established strong project implementation and management frameworks, Canada can contribute financial resources to WMD threat reduction activities through these existing mechanisms.
• Bilateral Recipient Delivery: GPP can use bilateral instruments (memoranda of understanding, private international contracts, treaties if necessary) to establish comprehensive project implementation frameworks for work in a specific country.
• Other Government Department Delivery: Other federal government departments and agencies such as the Royal Canadian Mounted Police (RCMP), Department of National Defence (DND), the Public Health Agency of Canada (PHAC), Canadian Nuclear Safety Commission (CNSC) and the Canadian Commercial Corporation (CCC) are involved in delivering training, equipment and expert support activities.

1.5 Key Program Activities

GPP is a grants and contributions (Gs & Cs) program that provides transfer payments to implementing organizations to undertake activities under three sub-programs: 1) Biological Security; 2) Nuclear and Radiological Security; and 3) Support for the implementation of UNSCR 1540 and scientist engagement (e.g. countering the proliferation of WMD knowledge). As of January 2015, GPP had 103 projects approved under its global mandate at different stages of implementation: completed, in-progress or in- development. This evaluation reviewed only the 78 projects completed and in-progress. Projects in-development were not included in the review as they had not begun implementation. Table 2 provides a breakdown of projects by sub-program.

Source: GPP Master Project List as of January 7, 2015

Shaded cells reflect projects included in the scope of the evaluation.

Nuclear and Radiological Security

The primary focus of the Nuclear and Radioactive Security sub-program is to secure nuclear and other radioactive materials and technologies stored at vulnerable civilian and military facilities. Nuclear and Radiological Security programming also helps target countries to secure and dispose of excess nuclear material and to reduce WMD trafficking. Specifically, the sub-program works in the following areas:

• physical protection and secure transport of nuclear materials,
• prevention of illicit nuclear trafficking;
• radiological security; and
• material management; verification and compliance.

Biological Security

Biological Security programming aims to strengthen the ability of states to prevent and respond to bioterrorism attacks. The sub-program focuses on:

• securing dangerous materials;
• strengthening prevention, preparedness and response capabilities;
• strengthening disease detection for deliberate biological attacks;
• reinforcing biological non-proliferation principles, practices and instruments, and
• supporting bioethics and responsible conduct.

United Nations Security Council Resolution (UNSCR) 1540 / Chemical Weapons Destruction / WMD Knowledge Proliferation

UNSCR 1540

GPP supports countries’ capacities to implement the legally-binding obligations of UNSCR 1540 to adopt legislation that prevents the proliferation of nuclear, chemical and biological weapons, and their means of delivery, as well as to establish appropriate domestic controls preventing the illicit trafficking of related materials. GPP responds to countries requesting assistance to implement UNSCR 1540 provisions through:

• CBRN training and provision of related equipment;
• legislative and regulatory assistance; and
• support for the enhancement of export controls and border security systems.

Chemical Weapons Destruction

While Chemical Weapons Destruction was not anticipated to be a major thematic priority at the time of GPP’s renewal, provision was made for this activity. Subsequent events in Libya and Syria required a continued focus on chemical weapons and a rebalancing of priorities.

Countering WMD Knowledge Proliferation (Scientist Engagement)

While the shift of Canada’s programming from the Former Soviet Union to other parts of the world has led to a significant decline in “scientist engagement/redirection” activities, GPP has the mandate to address the human dimension of WMD proliferation. During the evaluation period, scientist engagement was incorporated into a number of GPP projects, including some projects under the biological security sub-program that had a training component with a focus on security culture and responsible conduct.

1.6 Program Management and Governance

The Global Partnership Program Division (IGA) is part of the Non-Proliferation and Security Threat Reduction Bureau (IGD), which is headed by a Director General, reporting to the Assistant Deputy Minister of the International Security Branch (IFM). The Director of GPP is responsible for the overall management of the Division and is supported by three Deputy Directors. Two of them manage the main portfolios (Nuclear and Radiological Security, Biological Security, and Implementation of UNSCR 1540). The Policy Section is also managed by a Deputy Director, who, along with two additional full-time equivalents (FTEs), provides policy support and coordination for Canada’s engagement with the Nuclear Security Summit process and the Global Partnership.

The Non-Proliferation and Threat Reduction Bureau also has a small Corporate Affairs team, which serves the bureau as a whole. It is formally housed within the Capacity Building Programs Division (IGC) but the budget for its staff rests within GPP Division and GPP envelope.

As of November 2014, IGA had 26 (FTE) staff members.^{Footnote 6} It was also supported by a Senior Legal Counsel from the Department of Justice housed in the Non-Proliferation and Security Threat Reduction Bureau (IGD), however, at the time of this report, the Legal Counsel was centrally located within a General Counsel Division (JUS) and no longer within IGD.

2.0 Evaluation Objectives and Scope

2.1 Evaluation Objectives

The overall goal of this evaluation was to provide DFATD’s senior management with an evidence-based assessment of GPP’s activities and projects under its global expansion mandate and to provide program managers with information and feedback on how GPP implementation could be improved. In accordance with the TB Policy on Evaluation, the evaluation assessed the relevance and performance of GPP and its sub-programs. Given this was a formative evaluation, it paid particular attention to program design and delivery including the management structure, operating procedures, systems, delivery mechanisms, and the extent to which they support successful program delivery, and the achievement of results. Based on the evaluation findings, recommendations identify areas for improvement.

The specific objectives of the evaluation were:

1) To determine whether GPP continues to address a demonstrable need and is an appropriate and effective mechanism for implementing Canada’s international commitments.

2) To determine whether GPP is consistent with DFATD’s priorities and objectives, as well as, where relevant, the priorities and objectives of partners and beneficiaries;

3) To determine whether GPP and its sub-programs is likely to achieve its declared outcomes;

4) To determine whether the current governance structure and resource allocation practices are optimal or whether there might be more effective or economical ways to achieve the same results;

5) To examine synergies between GPP and the two capacity building programs – the Anti-Crime Capacity Building Program (ACCBP) and the Counter-Terrorism Capacity Building Program (CTCBP) - within the Non-Proliferation and Security Threat Reduction Bureau, as well as with the Global Peace and Security Fund (GPSF) within the International Security Branch, and identify complementarities and potential duplications;

6) To derive lessons learned and best practices which could be applied to future project or program management practices;

7) To review the link between policy and programming and the extent to which policy priorities have been reflected in the design of the Program.

2.1.1 Evaluation Scope and Focus

The evaluation examined GPP from the inception of the spending authorities for its worldwide mandate in February 2011 to January 2015 with a focus on programming developed since the latest renewal of funding authorities for the Program in April 2013. The evaluation also followed up on the recommendations of the 2012 Summative Evaluation of GPP.

3.0 Key Considerations and Strategic Linkages

3.1 Geographic Expansion of Program

Since February 2011, GPP has expanded its programming activities from the Former Soviet Union to address changing threats from WMD. This required an evolution from a small number of high-value projects in collaboration with other GP countries to a large number of smaller projects in countries where GPP has not had any previous programming experience and where the Program can be working with beneficiaries who may not always be fully committed to or supportive of WMD security objectives.

3.2 Departmental Requirements for Project Implementation Approval

While developing and implementing projects under its new world-wide mandate, GPP had to adjust to new departmental procedures for the initiation of new projects. Since 2011, a new Project Initiation Authorization (PIA) process was put in place which precluded the conduct of detailed project discussions or negotiations of the legal instruments (e.g. Implementing Arrangements, Grant or Contribution Agreements) with implementing partners. As a result, the Program had to adopt a more flexible approach to project implementation. This was a key consideration in the assessment of the Program’s performance.

4.0 Evaluation Approach & Methodology

The following section outlines the evaluation methodology for assessing the five key evaluation issues and related evaluation questions. The five evaluation issues include:

• Continued need for the program
• Alignment with government priorities
• Consistency with Federal roles and responsibilities
• Achievement of expected outcomes
• Demonstration of efficiency and economy.

The evaluation was guided by an Evaluation Advisory Committee (EAC) comprised of GPP program staff, staff from other related DFATD divisions and OGD representatives. The EAC reviewed the Evaluation Work Plan, participated in a presentation to validate the evaluation preliminary findings and reviewed the draft evaluation report.

4.1 Approach and Methodology

To ensure a systematic evidence-based process for data collection and analysis, the evaluation employed a combination of qualitative and quantitative methods and multiple lines of inquiry. The available data from various sources was triangulated to identify trends, similarities and points of convergence in order to develop the evaluation findings.

4.2 Evaluation Matrix

An evaluation matrix was developed as part of the Evaluation Work Plan to act as a framework for the evaluation process. It was used to guide data collection and facilitate analysis and the development of findings and recommendations. The matrix outlined the key questions to be answered by the evaluation as well as the related performance indicators, data sources and data collection techniques.

4.3 Data Collection Methods

4.3.1 Primary Data Collection

Key Informant Interviews:

A total of 69 key informants were interviewed for the evaluation in Ottawa and during field visits. In Ottawa, evaluators interviewed 23 key informants including GPP staff (n=7), other DFATD staff at headquarters (n=9), representatives of OGDs (n=6) and a consultant to DFATD (n=1). The interviews assisted evaluators in analysing the performance of GPP and helped to identify key issues and areas for improvement.

Field Visits

The evaluators conducted site visits to observe project implementation and meet with implementing organizations, project partners and beneficiaries in eight countries (Austria, Colombia, Indonesia, Jordan, Malaysia, Singapore, Thailand, and Trinidad and Tobago). Forty-six key informants were interviewed during these field visits including mission staff working for DFATD and OGDs (n=17), and representatives of recipients, implementing partners, allied countries and other local experts (n=29). Evaluators also attended a GP Working Group meeting in Germany. The field visits focused largely on relevance and program effectiveness.

4.3.1 Secondary Data Collection

Document Review

Evaluators reviewed pertinent documents and literature to assess the relevance and performance of GPP. These included DFATD corporate documents, descriptive and analytical reports, previous program evaluations, progress reports, and relevant planning, policy and priority review documents. Evaluators also reviewed international studies pertaining to the evolving threat from WMD to assess the continued need for non-proliferation programming.

Project File Review

Evaluators conducted a review of 19 project files^{Footnote 7} to assess the extent to which GPP was implementing its Standard Operating Procedures and demonstrating sound project management. The file sample was selected based on the following criteria:

• distribution among program areas and across geographic regions;
• value of projects;
• delivery mechanisms and;
• status (completed).

5.0 Limitations to Methodology

While the methodological approaches used were adequate to report on the relevance and performance of GPP, there were limitations inherent in the evaluation methodology. These limitations and the respective mitigating measures adopted are briefly described below:

Sequencing of the Evaluation Planning Process

While the detailed Work Plan was initially planned for discussion in September 2014 with data collection and analysis to begin in October, this sequencing could not be followed due to a number of reasons. For efficiency purposes, the Evaluation Division had to combine field visits for GPP, where possible, with those planned for a parallel evaluation of the Capacity Building programs. While the discussion of the Evaluation Work Plan could not be scheduled before the visits, the evaluation team requested and received much of the needed information for GPP sites in the countries planned for the Capacity Building Programs. While IGA indicated that combined visits might have limited the extent to which field evidence could be gathered for GPP, the evaluation team had the opportunity to assess the extent to which GPP and the two capacity building programs, all housed in the Non-Proliferation and Security Threat Reduction Bureau, have consulted and coordinated their activities in the same theaters of operation.

Limited Reporting

There was little reporting by the Program on progress toward expected outcomes at both the project and program level. This made it difficult to assess the overall results of the Program. Evaluators were able to gather information on project results through interviews and observation during field visits, but only for a limited number of projects.

International Comparison

Short timeframes and the lack of directly comparable programs did not allow for a comparative analysis of Canada’s performance with that of other GP member countries in support of international WMD non-proliferation efforts. However, the evaluation team was able to gather strong evidence on program relevance from GP documents and assessments of WMD threats from the North Atlantic Treaty Organization (NATO), United States (US) Government agencies and international organizations. Further, key informant interviews provided some evidence on Canada’s relative position among GP members.

6.0 Evaluation Findings

The following findings are based on the triangulation of evidence from relevant literature, project documents, project file reviews, key informant interviews and observation during site visits.

6.1 Relevance Issue 1: Continued Need for the Program

Finding # 1: There is a continued need for Canada’s Global Partnership Program as it addresses significant and ongoing threats from WMD proliferation.

Recent events such as the repeated use of chemical weapons in Syria^{Footnote 8}, Iran’s nuclear program, the international impact of the spread of highly infectious diseases and reported attempts by terrorist groups to obtain WMD materials, highlight the continued need for Canada’s Global Partnership Program and global cooperation to combat these and other worldwide WMD proliferation threats.

There is evidence that the threat from WMD is evolving. According to a 2014 NATO report, terrorists are increasingly combining chemical, biological and radiological materials to conventional improvised explosive devices (IEDs) to create new versions better known as “dirty bombs” and many governments are not prepared to respond to this threat.^{Footnote 9} GPP staff and external experts told evaluators that nuclear, radiological and biological materials continue to be vulnerable to terrorist acquisition.

Evidence from interviews and site visits indicate that current GPP programming responds to priority areas identified by the GP, namely: nuclear and radiological security, biological security and implementation of UNSCR 1540.^{Footnote 10} Program documents and interviews with representatives of implementing organizations and beneficiary countries showed there is continued demand for assistance from countries with CBRN weapons or stockpiles of WMD materials which do not have the financial capacity or expertise to secure them. GPP staff told evaluators that there has been an increased number of requests for assistance as countries are becoming more comfortable with disclosing their CBRN-related vulnerabilities and needs.

The following sections examine the specific needs for GPP programming under its three sub-programs.

Nuclear and Radiological Security

Stakeholders identified a number of countries of nuclear and radiological proliferation concern, such as Iran, North Korea, Pakistan, India and South Africa. Although more than two decades have passed since the end of the Cold War, the existence of thousands of nuclear weapons continues to pose a serious threat to global security. Some countries continue to have active nuclear weapons programs or continue to be involved in regional conflicts, which make the possibility of using nuclear weapons a major concern. Remaining dual-use nuclear capabilities increase the threat of some countries becoming possible exporters of nuclear technology and know-how, and/or becoming a target for state and non-state actors seeking to procure nuclear materials.^{Footnote 11} Under its global mandate, GPP responded in two of these regions by supporting projects related to the physical protection of nuclear materials and IAEA verification and compliance activities for nuclear non-proliferation.

According to the most recent data available in the IAEA Incident and Trafficking Database (ITDB), between 1993 and 2013, there were 2,477 confirmed incidents related to unauthorized possession and related criminal activities, unauthorized activities and events, and theft or loss of nuclear materials or radioactive sources globally. ^{Footnote 12} More recently, the Center for Non-proliferation Studies (CNS) and the Nuclear Threat Initiative (NTI) reported that globally, there was an increase in the number of times where nuclear and radiological materials were lost, stolen or out of regulatory control between 2013 (155 incidents) and 2014 (170 incidents). The information contained in both the IAEA ITDB and CNS/NTI report indicate that there continue to be available unsecured high-risk nuclear and radiological materials in virtually every country; that materials are particularly vulnerable during transport; and effective border control measures can help to detect illicit trafficking.

There are a number of areas around the globe that are major transit points for illicit materials. Under its global expansion mandate, the GPP has had approximately 16 approved projects that deal with the physical protection of nuclear materials and preventing illicit nuclear trafficking. Working in collaboration with international organizations including the IAEA, INTERPOL, and the United National Office on Drugs and Crime (UNODC), as well as with allied countries including the US Department of Energy (DoE), GPP is supporting projects that will help to enhance the physical protection at vulnerable nuclear facilities, and enhance the capacity of countries to detect illicitly trafficked nuclear and radiological at cargo ports and points of entry in Asia, the Middle East, Latin America and the Caribbean, and Africa. In Asia for example, GPP is enhancing the physical protection security at a nuclear reactor in the Philippines, and is also working to address the threat of illicit trafficking of nuclear and radiological materials by creating intelligence sharing networks among law enforcement agencies throughout South East Asia. In Jordan, GPP is implementing projects aimed at helping the Energy and Minerals Regulatory Commission (EMRC) to detect radiological materials through the provision of radiation portals at three priority border crossings and air/sea ports.

The US Department of Energy (DoE) has identified the security threat from vulnerable stockpiles of civilian materials such as highly enriched uranium (HEU) and radiological materials that are often stored unprotected in hospitals and universities. According to the US DoE “A Radiological Dispersal Device, deployed with amounts of material found in normal use, could result in radioactive contamination that could require relocation, prohibit the use of that area pending cleanup, and cause economic impacts in the billions of dollars.” ^{Footnote 13}

Canada is currently one of the largest suppliers of radioactive sources used in hospitals and industrial applications. Under Phase IV of GPP, three projects have been approved aimed at enhancing the security of high activity, disused radioactive sealed sources, including those of Canadian origin in South and Central America. GPP has responded to requests for assistance from Costa Rica and Colombia to secure and remove disused radiological sources. GPP programming in this area is designed to prevent the theft of and unauthorised access to high-risk radioactive materials, including sealed sources, while at the same time reducing the risk of such materials causing harm or resulting in an accident that could negatively impact Canada's reputation.

Given that Canada is a leading exporter of radioactive sources, GPP is also supporting the delivery of training courses on best practices for securing nuclear sources to further ensure international security. Under Phase IV of GPP, ten projects have focussed on training, or contained a training element, in Asia, the Middle East, Latin American and the Caribbean, Africa as well as the development of a certification program for nuclear managers that will have a global reach. These projects are developed and delivered in partnership with leading Canadian and global nuclear security organizations including the Canadian Nuclear Safety Commission (CNSC), Bruce Power, IAEA, and the World Institute for Nuclear Security (WINS),

Overall, evidence showed that projects under the Nuclear and Radiological Security sub-program of GPP were, in many cases, responding directly to threats identified by the IAEA, INTERPOL or the US Department of Energy, and that projects fall within the priority areas identified by the Global Partnership.

Biological Security

In an increasingly interconnected world with greater movement of people, animals and goods between borders, the risk of exposure to public health risks, including infectious diseases, has also increased.^{Footnote 14} In recent years, there has been growing recognition among national policy makers of the serious impacts that health crises can have on national economic and security interests. The Global Health Security Agenda (GHSA), launched in 2014 by the US and the World Health Organization (WHO), lists today’s global health security risks as follows:

• the emergence and spread of new microbes;
• globalization of travel and trade; rise of drug resistance;
• the acceleration of biological science capabilities;
• the risk that these capabilities may cause the inadvertent or intentional release of pathogens; and
• continued concerns about terrorist acquisition, development, and use of biological agents.^{Footnote 15}

According to a WHO bulletin titled Health and Security in Foreign Policy, “research has demonstrated that acute and chronic changes in health status have direct and indirect impacts on security, and that epidemics may lead to destabilization, political unrest, civil disorder or long-term deterioration of the economic viability of a country or region.”^{Footnote 16} Recent examples of the economic impacts of health crises include the 2001 foot and mouth disease outbreak in the United Kingdom that was estimated to have cost more than $16 billion;^{Footnote 17} and the 2002-2003 outbreak of Severe Acute Respiratory Syndrome (SARS) in Canada that led to 44 deaths^{Footnote 18} and resulted in an estimated $1.5 billion in economic losses to the Canadian economy.^{Footnote 19} Most recently, the Ebola epidemic in West Africa was cited during interviews and the literature review as an example of the need for improved global capacity to prevent, detect, and respond to biological threats.^{Footnote 20}

Internal and external stakeholders interviewed by the evaluation team, as well as information collected at the Global Partnership meeting held in Berlin on November 4, 2014, reiterated the fact that irrespective of whether a health crisis is naturally occurring or intentionally caused (such as biological warfare or bioterrorism), the potential impacts and global efforts required to reduce the risks and address the threats are the same. Under Canada’s leadership in 2010, the GP agreed on five major deliverables for GP assistance in the field of biosecurity:

• Secure and account for materials that represent biological proliferation risks.
• Develop and maintain appropriate and effective measures to prevent, prepare for, and respond to the deliberate misuse of biological agents.
• Strengthen national and global networks to rapidly identify, confirm, and respond to biological attacks.
• Reinforce and strengthen biological non-proliferation principles, practices, and instruments.
• Reduce proliferation risks through the advancement and promotion of safe and responsible conduct in the biological sciences.

Through GPP, Canada as a member of the GP BSWG and GHSA Steering Group, is implementing a variety of projects to address the above-mentioned goals in countries and regions that are considered to be the most vulnerable due to gaps in the implementation of WHO International Health Regulations^{Footnote 21}, as well as in areas where there is a high prevalence of infectious diseases, including Africa and South-East Asia.^{Footnote 22}

GPP has partnered with and is supporting a wide range of implementing partners and recipients under a variety of implementing arrangements to achieve these deliverables, which have been endorsed by the GP’s Biological Security Sub Working Group (BSWG). Some examples of projects that GPP is implementing to strengthen global biological security under Phase IV include the provision of personal protective equipment to WHO in West Africa to support international Ebola response efforts and the provision of biohazard sample collection, diagnostic and transport kits to 32 countries in the Caribbean and Central America to increase the capacity for biosecurity and disease surveillance. Through the provision of modular or mobile biological laboratories in the Caribbean and Middle East, the Program aims to strengthen disease detection and biosecurity capacities. GPP is also working to improve the security and accountability for materials that represent biological proliferation risks by providing physical security upgrades at vulnerable veterinary biological laboratories that are working with dangerous pathogens in Africa. In Southeast Asia, the GPP is supporting a number of projects to mitigate biosecurity threats from infectious disease pandemics through funding that contributes to laboratory improvements, regional disease surveillance networks and pan-ASEAN coordination and cooperation.

Facilitating the Implementation of UNSCR 1540 and Chemical Weapons Destruction

Under this sub-program, GPP is working to reduce the threat posed by terrorist acquisition of WMD and related materials by increasing countries’ capacity to implement their obligations to prevent the proliferation of nuclear, chemical and biological weapons. This sub-program responds to Canada’s invitation to assist other signatory countries with the implementation of this UN Security Council Resolution 1540 within the priority areas identified by the GP including: chemical, biological, radiological and nuclear (CBRN) training and equipment, legislative language assistance, and support for export controls and border security related to WMD.

Overall GPP had 22 projects related to these priorities at the time of the evaluation. Partnering with the RCMP and DND, the majority of these projects have involved the provision of CBRN training to expert personnel in beneficiary countries (Latin America/Caribbean, Asia, Middle East) aimed at increasing their capacity to detect and respond to WMDs or related materials. In Colombia, GPP programming responded directly to a request by the Colombian National Police to the 1540 Committee for training and equipment to enable them to detect and identify CBRN materials. The Program has also provided funding for five regional workshops, meetings and/or seminars on topics such as improving WMD interdiction capabilities, sharing effective practices on implementing UNSCR 1540, and building the capacity of states to adopt domestic legislation to facilitate the ratification of international legal instruments related to nuclear security. In addition to Colombia, the GPP has also provided CBRN equipment contributions to Jordan to build on the CBRN training delivered, and is providing additional equipment to the Jordan Armed Forces (JAF) Crisis Management Centre to operate in the event of a national crisis including potential CBRN incidents. The GPP also has single projects in the areas of legislative assistance, placement of a coordinator to support implementation of UNSCR 1540, and a project with UNODC to prevent the illicit trafficking of WMDs and related materials in Asia.

Chemical weapons destruction was not expected to be a major thematic priority when GPP was renewed. However, the declaration of further chemical weapons stored in Libya in 2011 after the overthrow of the Qadhafi regime and the verification of chemical weapons use in Syria in 2013 highlighted the need for further work in this area. GPP has provided considerable support to efforts led by the OPCW to destroy these weapons. This case supports the need for GPP to have some degree of funding flexibility to respond to unexpected and emerging priorities.

6.2 Relevance Issue 2: Alignment with DFATD and Government Priorities

Finding #2: The objectives of GPP and its sub-programs are aligned with and supportive of the priorities and objectives of the federal government and DFATD’s Strategic Outcomes.

Through its suite of programming under its global mandate since 2011, GPP demonstrates Canada’s continued commitment to the GP made at the 2002 Kananaskis G8 Summit to prevent ‘terrorists, or those that harbour them, from acquiring or developing nuclear, chemical, radiological and biological weapons; missiles; and related materials, equipment and technology’.^{Footnote 23} GPP supports the renewed mandate of the GP at the G8 Summit in Deauville in May 2011 to address: nuclear and radiological security, biological security, countering WMD knowledge proliferation, and implementation of UNSCR 1540.^{Footnote 24}

GPP further supports Canada’s commitments to international treaties and disarmament initiatives, such as:

• the Nuclear Non-Proliferation Treaty;
• the Biological and Toxins Weapons Convention;
• the Chemical Weapons Convention; and
• UNSCR 1540.

GPP aligns with DFATD’s Strategic Outcome 3, International Assistance and Poverty Alleviation: poverty is reduced, and security and democracy are increased for those living in countries where Canada engages. GPP contributes to increased international security by enhancing the capacity of foreign governments, civil society and international organizations to manage international security challenges such as transnational crime, terrorism and WMD.^{Footnote 25}

GPP’s programming also supports Canada’s commercial foreign policy priorities. For example, its programming in Colombia, Brazil, Mexico, Costa Rica, and Peru is aligned with Canada’s Strategy for Engagement in the America’s^{Footnote 26} and its programming with ASEAN support Canada’s commitments to the ASEAN-Canada Plan of Action 2010-2015.

GPP also helps support domestic security strategies such as Canada's Chemical, Biological, Radiological and Nuclear, and Explosives (CBRN) Resilience Strategy, which seeks to “protect Canada and Canadians by taking all possible measures to prevent, mitigate and respond effectively to a potential CBRN incident”.^{Footnote 27} The Strategy names DFATD as the lead for the international coordination of Canada’s CBRN response.

6.3 Relevance Issue 3: Consistency with Federal Roles & Responsibilities

Finding #3: The delivery of GPP conforms to the responsibilities of the federal government, the mandate of DFATD and its federal delivery partners.

Ensuring the safety and security of all Canadians and developing and implementing Canada’s foreign policy are among the primary roles of the federal government. GPP is closely aligned with DFATD’s international mandate which includes enhancing security for Canada and Canadians at home and abroad. In order to deliver some of its programming, GPP relies on the technical expertise of OGD representatives from PHAC, CNSC, RCMP, and DND, and evidence collected from interviews and documents found that GPP programming is well-aligned with the mandates of these departments.

GPP occupies a niche area in the security program spectrum as the only GoC program that addresses CBRN security, countering WMD knowledge proliferation, and the implementation of UNSCR 1540. The evaluation found no evidence of overlaps between GPP and the other DFATD security programs such as the Anti-Crime Capacity-Building Program (ACCBP), the Counter-Terrorism Capacity Building Program (CTCBP) and the Global Peace and Security Fund (GPSF). Both GPP and ACCBP support container control projects with the United Nations Office on Drugs and Crime (UNODC), but these projects have separate and distinct mandates. Similarly, GPP and CTCBP have in the past supported CBRN response training, however, such training was handed over to and is currently supported solely through GPP.

Overall, the delivery of GPP is an appropriate role for DFATD because its management requires negotiation and coordination with the governments of both donor and recipient states, and multilateral organizations. Furthermore, GPP helps to fulfil Canada’s international commitments, promotes adherence to international law, and leverages DFATD’s network of missions to shape the international agenda in accordance with Canada’s priorities and values.

6.4 Performance Issue 4: Achievement of Expected Outcomes

Finding #4: There is some evidence that GPP is making progress toward expected outcomes. However, a full assessment was not possible due to a lack of performance reporting.

Given that this is a formative evaluation, the evaluators expected to find that GPP was achieving its immediate outcomes and was making progress toward its intermediate outcomes (listed on page 3). While the evaluation did find some evidence of progress toward expected outcomes, GPP’s ability to achieve and report on results is hampered by a lack of focus on outcomes, both in its programming and performance measurement. (See Section 6.5.5 Performance Measurement.) Evidence from interviews, site visits and project files showed that GPP is largely focused on the short-term delivery of outputs such as equipment and training with more than half of projects completed in a year or less.^{Footnote 28} The sections below provide examples of the results achieved by key initiatives in each of GPP’s sub-programs as a full assessment was not possible due to the lack of performance reporting.

6.4.1 Nuclear and Radiological Security

Finding #5: There was evidence of progress toward the immediate outcomes of improved security measures for storage and transport of WMD-related materials and the development of networks and partnerships, and the intermediate outcome of increased compliance to international WMD commitments.

In early 2015, the NRS sub-program had 16 projects completed or in progress with a value of more than $46 million. These projects spanned across 10 countries and regions. The bulk of this funding, $30 million, went to meet a 2011 commitment made at the G8 Summit in Deauville to support remediation of the Chernobyl Nuclear Power Plant in Ukraine. Canada’s contributions, along with those of numerous other countries are supporting a new safe confinement for the damaged nuclear reactor and an Interim Spent Nuclear Fuel Storage Complex. The two multi-million dollar construction projects are scheduled for completion in 2017.

During the evaluation period, GPP supported six projects worth more than $12 million through the IAEA, and it is these projects that appear to have achieved the most notable results. GPP provided $1 million to support IAEA compliance and verification monitoring in Iran. Such monitoring is an important component of the Joint Comprehensive Plan of Action on the Iranian nuclear program that was reached in July 2015 between Iran and the members of the UN Security Council. Thus GPP appears to have contributed to progress toward its intermediate outcome of increased compliance with international WMD commitments.

Through the IAEA, GPP is also supporting the safe destruction of disused radiological sources of Canadian origin^{Footnote 29} including projects in Costa Rica and Colombia. As a result of these projects, 45 disused radiological sources from Costa Rica were transported to Germany for disposal. In Colombia, a similar GPP-supported project identified 12 radioactive sources of Canadian origin that need to be destroyed, five more than originally identified by Colombian officials. According to interviews, the project led Colombia to ratify the International Convention on the Physical Protection of Nuclear Material. It also led to other international partnerships in dealing with the import-export of nuclear material for medical and research purposes including an agreement with the Canadian Nuclear Safety Commission for future cooperation. This provides evidence of progress toward GPP’s immediate outcomes of improved security measures for storage and transport of WMD-related materials and the development of networks and partnerships.

6.4.2 Biological Security

Finding #6: There was some evidence of progress toward the immediate outcomes of improved detection capabilities and the development of networks and partnerships.

At the time of the evaluation, GPP had 36 projects completed or in progress worth more than $33 million under its Biological Security sub-program. These projects spanned across 10 countries and regions. More than two-thirds of the project funding was for initiatives related to laboratory upgrades or the provision of equipment, such as personal protective equipment, diagnostic kits and specialized laboratory gear.

Among the projects supporting laboratory upgrades it was GPP’s work with the Caribbean Public Health Agency (CARPHA) in Trinidad and Tobago (T & T) that showed the strongest evidence of progress towards expected results. With this $ 2.5 million project, GPP supplied CARPHA a fully-equipped modular Biosafety Level 3 (BSL3)^{Footnote 30} laboratory. Based on documents reviewed by the evaluation team, as well as information obtained through interviews, the laboratory is reported to have enhanced regional capacities for fast detection, diagnosis and response to disease outbreaks, thus demonstrating progress toward the GPP’s immediate outcome of improved detection capabilities. Such improved disease detection capabilities are important to Canada given the high volume of Canadian travellers to the region.

In Nigeria and Ghana, the Program also funded two projects worth nearly $7 million to upgrade veterinary laboratories that handle dangerous pathogens of terrorist concern, such as plague and anthrax. Along with facilities and equipment, the projects provided biosafety training for the veterinary service personnel who use these new labs. According to GPP program staff, these laboratory upgrades were to create secure facilities to contain and reduce the holdings of dangerous pathogens. Project files contained detailed information that demonstrated that GPP is designing and delivering high quality facilities but there was little reporting available on project outputs or outcomes.

GPP has further contributed more than $9 million worth of biosecurity-related equipment, including $5 million worth of equipment to the WHO in 2014 for use in the response to the Ebola outbreak in West Africa. GPP also provided $300,000 to PHAC to support the deployment of two mobile laboratories and expert diagnostic teams to Sierra Leone. Some DFATD staff saw GPP’s rapid response to the Ebola outbreak as evidence of the Program’s ability to respond quickly to support efforts to contain and reduce the risks associated with this dangerous pathogen.

In the ASEAN region, GPP committed $6.4 million in 2014 to four projects to mitigate biological threats from infectious diseases. These projects aim to improve the ability of laboratories to detect and diagnose dangerous pathogens as well as establish networks that would allow countries in the region to work together to combat biosecurity threats. While it is too early to expect these projects to have achieved significant results, evidence from interviews showed there has been some progress toward the immediate outcome of developing networks and partnerships as several projects have established inter-agency collaborations to respond to biosecurity threats.

6.4.3 UNSCR 1540 and Chemical Weapons Destruction

Finding #7: There was some evidence of progress toward the immediate outcomes of improved WMD detection capabilities and development of networks and partnerships related to its support for UNSCR 1540.

In January 2015, GPP had 22 projects completed or in progress with a value of $14.5 million related to the implementation of UNSCR 1540. These projects were being implemented in a dozen countries and/or regions. More than half of these projects related to training or other capacity building initiatives designed to assist recipient countries to respond to CBRN attacks or prevent the transshipment of CBRN materials. Through site visits, document review and interviews, the evaluators found that there was evidence of progress toward immediate outcomes related to GPP’s support for CBRN training and equipment in Colombia, Malaysia and Jordan, and the UNODC Container Control Program.

In Jordan, GPP has committed about $24 million worth of equipment, infrastructure, training and related support to 18 projects to assist Jordan to respond to potential CBRN incidents arising from Syria and Iraq. These projects span across all three of GPP’s sub-programs. While many of the projects were still underway, the most significant results appeared to have been through CBRN training under the UNSCR 1540 sub-program. This training, along with equipment provided under the Biological Security sub-program, contributed to the immediate outcomes of improved WMD detection capabilities and development of networks and partnerships.

Through GPP assistance, training on how to deal with CBRN incidents was provided to the Jordanian Armed Forces, medical and police organizations according to the same model so that all organizations involved in responding to CBRN incidents would have shared knowledge. According to interviews, this joint training increased inter-agency coordination and improved the ability to respond to CBRN threats. The military personnel trained and equipment provided through GPP assistance have been deployed at Jordan’s borders to check for WMD materials enter the country.

Since 2012, GPP has provided about $2 million for RCMP training in responding to CBRN incidents. As of January 2015, seven training courses had been provided to police and other first responders in Colombia, Brazil, Malaysia, Jordan, Mexico and Indonesia and well over a hundred first responders have been trained. The evaluation found some evidence that this training has resulted in progress toward the immediate outcome of improved WMD detection capabilities in Malaysia and Colombia, where the RCMP has provided more than one training program. In Malaysia, while trainees have not been involved in responding to terrorist incidents following the CBRN training delivered by the RCMP, evaluators were told the training has helped police and other first responders to deal with a number of chemical spills and other accidents. Malaysian officials have also set up their own in-class CBRN training courses following the train-the-trainer course from the RCMP. Similarly, the Colombian National Police has reached out to the RCMP for further assistance to run its own training program.

Further, in South East Asia, there is evidence a $3 million UNODC project also contributed to improved WMD detection capabilities. The project was working to prevent the use of shipping containers for illicit trafficking of CBRN materials. According to UNODC officials, training and support for customs officials has led to changes in container screening practices. For example, in Indonesia customs officers were beginning to selectively (and not randomly) screen vessels based on criteria that identify high risk containers.

Finding #8: GPP support for chemical weapons destruction in Syria and Libya contributed to progress toward the Program’s ultimate outcome of reducing access to WMD materials.

GPP supported four projects with a value of $23 million that contributed to chemical weapons destruction in Libya and Syria. Several interviewees saw these contributions as the Program’s most significant achievements to date under its global mandate. By reducing the international supply of chemical weapons these projects appeared to be contributing to GPP’s ultimate outcome of reducing access to WMD materials. Canada contributed $2 million to the OPCW in 2013 that was instrumental in verifying the alleged use of chemical weapons by Syria. In 2014, Canada provided $15 million to the OPCW and the US Department of Defense to support the destruction of priority chemicals. The OPCW reported that as of February 2015, 98% of chemical weapons declared by Syria had been verified as destroyed including the blister agent sulfur mustard and a precursor chemical for the deadly nerve agent Sarin.^{Footnote 31}

In 2012, Canada contributed $6 million to the OPCW to support chemical weapons destruction in Libya. In May 2014, the OPCW verified that Libya’s known stockpile of useable chemical weapons had been destroyed and reaffirmed Libya’s plan to complete destruction of its stockpiles of precursor chemicals by the end of 2016.^{Footnote 32} However, instability within Libya hampered efforts to obtain and destroy these remaining stockpiles. DFATD staff told evaluators that in the absence of GPP it would have been difficult for Canada to contribute to these efforts to destroy chemical weapons.

6.4.4 Overall Assessment of Results

Finding #9: Focused GPP investments in some organizations and countries have contributed to Canadian leadership, influence and visibility.

Evidence showed that GPP has contributed to broader results in the cases where significant investments were made in multiple projects in a single country or international organization.

For example, GPP’s contributions to the IAEA have made Canada the organization’s third largest donor. According to interviews, these contributions, along with Canada’s reputation in nuclear security, have increased Canada’s visibility and role in the organization. IAEA’s officials have visited Canada on a number of occasions to brief Canadian officials, which in turn, has allowed DFATD to provide more complete information to the Government on the current situation in some countries where IAEA is implementing projects, such as Iran and Iraq. Canada’s leadership role in the IAEA has also provided it with increased influence in important decisions within the organization.

In the case of Jordan, evidence from interviews and site visits showed that the significant level of Canadian assistance has helped Canada gain credibility with partners and allies. According to GPP staff, the Program has recently received a request for assistance from another government in the region following referrals from Jordanian officials. Many of those interviewed agreed that GPP’s contributions in Jordan have led to improved relationships and increased awareness about the region and the regional context. Some interviewees suggested that the CBRN training provided by DND has been especially effective in building relationships between the Canadian military and the Jordanian Armed Forces as DND staff have been stationed in Jordan to work side-by-side with their Jordanian counterparts.

OGDs involved in GPP programming also noted the benefits for their work in Canada from their participation in the Program. For example, RCMP representatives said that relationships developed through the training programs can assist with domestic crime investigations. GPP programming also allows Canada to stay abreast of what is happening internationally in nuclear security.

Finding #10: The potential for sustainability of results was greatest among projects delivered through international partners and lowest among one-time training projects that were delivered bilaterally and were not tied to longer-term project activities.

In examining the effectiveness of GPP’s programming, evaluators looked at whether programming results were likely to be sustained after project funding ends^{Footnote 33}. It was found that projects delivered through international partners were more likely to generate sustainable results.

For example, the UNODC container control project put in place a number of measures to promote the sustainability of its work in building the capacity of port officials to detect WMD. Following training programs, UNODC instructors provide ongoing mentorship to assist trainees in applying the acquired knowledge and skills. UNODC introduced a train-the-trainer second phase to its program and encouraged those trained to become certified by the World Custom’s Organization. UNODC was also introducing a multi-agency approach to extend training to the chain of agencies involved in detecting and seizing illicit materials, such as customs officials and police, as well as prosecutors.

Further, projects under the NRS sub-program are often components of the IAEA Integrated Nuclear Support Plans which are negotiated between the IAEA and the participating country. According to IAEA officials, this means that GPP-funded activities are conducted in concert with other capacity building activities, such as the development of guidance and training, and peer review advisory services to put in place the necessary infrastructure for sustaining an effective nuclear safety regime.

Projects delivered through bilateral partners also demonstrated sustainability. For example, a review of project documents showed that biological laboratories implemented under the biological security sub-program factored sustainability elements into the project designs. Two projects included in the project sample, a biological lab in Trinidad and Tobago and another in Nigeria, showed that while GPP had assumed responsibility for the initial maintenance period, both recipients were found to have developed plans and budgets to support the longer-term maintenance and operations of the facilities after the contracts with GPP expired. GPP identified training requirements and worked with the recipients to identify skilled individuals to support the on-going operation and maintenance of the facilities and equipment. Staff were provided with necessary training and maintenance tools, which included sending maintenance staff from Nigeria to a training workshop at the National Microbiology Laboratory (NML) in Winnipeg. Canada’s Biosafety Standards and Regulations were used as the basis for the design of the facilities and detailed environmental assessments were completed. Sustainability, cost effectiveness and environmental considerations were also factored into the lab designs. For example, labs were designed to address local environmental conditions such as unreliable power supply, and designs were simplistic and based on technologies requiring lower maintenance so that the facilities and equipment can be maintained using local knowledge, thereby increasing cost effectiveness.

These projects also indicated some unique challenges related to sustainability. It was noted, for example, that, activities related to building or upgrading biosecurity laboratories need to be tied to the country’s strategic priorities, not only to ensure that adequate resources are made available for operations and maintenance, but also to monitor and enforce continued compliance with professional and safety standards, be it through the recipient government or through an international association. In Canada, laboratories dealing with human and animal pathogens are regularly inspected and certified by the PHAC Centre for Biosecurity. There was some evidence that similar oversight might be established for the Caribbean Public Health Agency laboratory that GPP supported in T&T. The Minister of Health of T&T has indicated that his department was in the process of developing a Medical Laboratory Regulation and Registration Policy.

However, several DFATD staff and implementing partners also raised doubts about the sustainability of one-time training courses or single workshops that were not tied into longer-term project activities. For example, in several countries, such as Mexico and Brazil, the RCMP delivered single CBRN training sessions for first responders. According to interviewees, several training sessions and the buy-in of senior level champions would be needed to develop a coordinated response to CBRN terrorist incidents. According to the international literature, to be sustainable, such capacity building initiatives require interventions beyond training at the individual level to address the organizational aspects of capacity such as champions, incentives, and organizational structures. An OECD report on capacity development recommends that service providers need to ensure that short-term activities, such as training courses, contribute to long-term learning and change strategies.^{Footnote 34}

6.5 Performance Issue 5: Demonstration of Efficiency & Economy

In the assessment of GPP’s efficiency and economy, the evaluation team relied on evidence from project documents, the project file review and key informant interviews.

6.5.1 Priority Setting and Strategic Planning

Finding #11: The Strategic Planning Process that was developed in the context of the 2013-2018 Renewal Mandate was not followed.

The approved Treasury Board Submission which renewed the GPP stipulated that the GPP will conduct an annual priority-setting exercise to ensure consistency with Departmental and Branch strategic objectives and foreign policy priorities. Through this exercise, the Submission states that the GPP will set criteria to drive the selection of thematic and geographic priorities including emerging threats. While these may be innovated, the criteria largely reflect the following high level requirements:

• intelligence gathering and assessment of the locus, nature and severity of the WMD threats;
• risks and considerations with associated political, economic or security conditions in a given country or region, or with a proposed activity;
• receptiveness, absorptive capacity and reliability of the direct recipient or an Implementing Body to use the funds for which they are intended; and
• alignment with Canadian foreign policy priorities, of which security is an integral part.

As part of the program renewal and in keeping with a recommendation from the 2012 GPP evaluation, GPP developed a Strategic Programming Framework process. The first component of this process was to identify a list of priority countries/regions for potential programming by sub-program by assessing the level of threat, sustainability, risk and alignment with DFATD security programming and Canadian foreign policy priorities. The second component was the development of a Strategic Directions Document (SDD) to be informed by consultations across the department and with relevant OGDs. Once complete, the SDD would further refine the proposed list of five to ten priority countries/regions for GPP work for the upcoming fiscal year, and outline notional financial allocations. The SDD was also meant to form the basis for the development of Priority Country/Region Plans based on which specific project proposals would be developed.

In 2012-2013, GPP first outlined a list of 25 priority countries. However, this list was not developed according to the above-prescribed process. For example, no SDD was developed, and no methodology was proposed to inform the assessment process.

In 2014-2015, GPP replaced the Strategic Programming Framework process with the Methodology for Priority Review Framework that was used to assess and rank 28 countries with a known or suspected CBRN vulnerability. This Framework was based on the high level requirements outlined in the Treasury Board Submission. Four key factors were assessed: WMD threat; programming environment (stability, absorptive capacity); sustainability (regional impact; capacity to sustain programming); and coherence/alignment with Government of Canada and DFATD priorities. Following extensive consultations with DFATD geographic desks and policy divisions, GPP issued a final ranking of these countries in October 2014. Unlike the original Strategic Programming Framework process, there was no attempt of identifying or limiting the number of priority countries based on budget allocations or capacity of the Program to manage and monitor projects in multiple countries. Rather, the ranked list of countries was viewed as a flexible tool to inform GPP sub-program planning, including consultations with other donors (largely in the context of the Global Partnership); international organizations; current and potential beneficiary states and Government of Canada Departments and Agencies, with a view to identifying appropriate niches for Canadian assistance, prioritising higher ranking countries as possible.

The Priority Review Framework methodology resulted in descriptions of the four key factors (WMD threat; programming environment; sustainability; and coherence/alignment with GoC and DFATD priorities) for each priority country/region. Some DFATD interviewees found the Priority Review Framework exercise cumbersome and time-consuming, particularly the five page matrix for scoring countries and regions, and felt that the final, ranking list of 28 priority countries came too late to influence the project proposals for the 2014-2015 fiscal year. Since GPP has a five year mandate and projects tend to be multi-year initiatives, the list was still considered to be relevant for future programming.

Unlike the original Strategic Programming Framework process, the Priority Review Framework methodology did not produce priority country/region plans to inform the development of project proposals, or to address the range of issues based on their importance in the strategic planning process such as specific programming priorities; resource implications including human resources, financial allocations, and monitoring and performance measurement plans; or results-based management frameworks and  logic models for the interventions in each priority country/region. Lack of strategic plans at the country and/or region level hindered GPP’s ability to plan, monitor and measure progress and report on the achievement of results against an approved strategy.

6.5.2 Project Approval Processes

Finding #12: Project approval processes hindered GPP’s ability to plan strategically and achieve results, and weakened the Program’s credibility with international partners.

In August 2011, a new project initiation authorization (PIA) process was initiated within the Department for projects under Vote 10 (Gs&Cs), which hampered strategic planning and caused major delays in project implementation due to the lack of project approvals. Under this process, DFATD managers were unable to predict if, or when, project proposals would be able to move forward for implementation. For example, out of the 160 project proposals developed by GPP between August 2011 and February 2015, just over one third of these proposals were approved for implementation. (See Table 3) Also, while all projects proposals addressed real WMD threats, the approval process did not provide opportunities for prioritization, meaning that the approved projects did not always address the greatest or most imminent threats.

A review of GPP spending for the last two years shows the Program spent less than two-thirds of its original budget allocations. (See 6.5.7 Resource Utilization.) DFATD staff told evaluators that they developed proposals valued at one and half times of their budgets in an effort to have a sufficient number of projects approved. Program staff indicated that the development of such a large number of project proposals put a heavy burden on them and reduced the time available for planning and project management. Several GPP staff members told evaluators that this process also had a demoralizing impact on employees.

The approval process not only hampered GPP’s overall ability to plan strategically, implement projects and ultimately to achieve results, but also led to a loss of credibility with international partners. Given that program staff could not predict if they would be able to proceed with a particular project, they were unable to make funding commitments at international forums or to commit to projects with like-minded countries. Interviewees from OGDs and international organizations indicated that GPP failed to involve them early on in the project planning process, which impacted the efficiency and effectiveness of project implementation. GPP staff explained that they were unable to engage with partners on needs assessments and project proposals as they were unsure projects would be approved.

According to some DFATD staff, the approval process often required “reverse engineering” whereby local needs were assessed and a detailed project plan developed only after a general concept was approved. This was the case with more than $6 million worth of projects in the ASEAN region. Some project partners and mission staff said that advance consultations with DFATD missions and beneficiaries would have led to a better match between proposed projects and the priorities of ASEAN-member countries

Finding #13: There is room for improved consultation and coordination with other DFATD programs, missions abroad and OGDs.

Both GPP’s Strategic Programming Framework and its Standard Operating Procedures (SOPs) identified the need for ongoing coordination with DFATD security programs, other DFATD programs and OGDs. While the evaluation found there has been some progress in improving coordination across DFATD’s security programs, there is an opportunity for greater coordination.

In an effort to improve coordination among DFATD’s security programs, the International Security Branch (IFM) established a Branch Policy and Programming Coherence Secretariat within the Stabilization and Reconstruction Task Force (START) bureau. Its mandate was to ensure coordination among all IFM Branch security programs (GPP, START, CTCBP and ACCBP), as well as coordination with the development and humanitarian sections. The Coherence Secretariat created special task forces, such as the Ukraine Task Force, to deal with some issues of high importance to the Department. The Ukraine Task Force was reported to have helped to facilitate communications and improve the coordination between DFATD’s security programs by bringing stakeholders together and by helping to define roles and responsibilities. GPP actively participated on the Ukraine Task Force and demonstrated good coordination with other DFATD security programs by working closely with ACCBP and START on a joint Container Control Project with UNODC aimed at countering illicit trafficking in Ukraine. A joint Corporate Affairs Team was also set up to support GPP and DFATD’s two capacity building programs (CTCBP and ACCBP). While both the Coherence Secretariat and the Corporate Affairs Team helped to increase coordination, DFATD staff also told evaluators that there was some confusion about the roles and responsibilities of these bodies and GPP’s own policy unit.

GPP held Project Initiation Approval (PIA) review meetings on as required basis, normally every month. In addition to GPP program staff, invitations were also sent to representatives of relevant policy, thematic, and geographic divisions, as well as to missions, and other DFATD security programs. PIA documents were circulated for comments prior to the meetings. GPP also participated in coordination efforts initiated by the other security programs. For example, the Capacity Building Programs Division invited program officers from GPP and GPSF to attend its 2015-2016 Priority Review Committee Meeting and GPSF invited representatives of the security programs, development programs and geographic divisions to regional-focused meetings to discuss its proposed programming.

However, evidence from interviews indicated that there is further room for GPP to increase the level of consultation and coordination with the other DFATD security and development programs, as well as with OGDs, in order to exchange information on implementing organizations, to conduct joint needs assessments where appropriate, and to share lessons and best practices. Some staff at missions expressed concern that neither GPP, nor some of the other security programs, were using existing coordination tools such as the on-line mission management tool “Strategia” to ensure field staff were informed of GPP’s activities. As well, some representatives of OGDs expressed a desire for increased consultation and coordination during project planning and implementation to ensure their staff are available to engage in projects when required and to ensure a whole of government approach to GPP programming. Coordination is especially important where there are geographic or thematic overlaps, such as under the biological security sub-program to avoid overlaps and duplication of effort with health security, development and public health initiatives.

6.5.3 Project Management

Finding #14: GPP was fulfilling some of the basic elements of its Standard Operating Procedures. However, it lacked a formal project review process, failed to conduct needs and risk assessments and lacked standardized reporting tools.

In keeping with another of the recommendations of the 2012 GPP Evaluation, the Program developed SOPs that provided a basic overview of the project management process broken down in three stages:

1. project proposal development and approval;
2. project implementation and monitoring; and
3. project evaluation and close-out.

Under stage 1 of the SOPs, project proposals were to be developed based on the Priority Country/Region Plans and a series of evaluation criteria. These proposals were to be subject to a formal review at monthly project review meetings attended by representatives of the sub-programs and other relevant DFATD divisions. Although there was no evidence that project proposals were developed based on Priority Country/Region Plans or according to evaluation criteria, project review meetings were held on an as required basis, normally every month.

The project file review and interviews showed that GPP did fulfil many of the aspects stage 2, project implementation and monitoring outlined in the SOPs. For example, it developed Implementing Agreements in consultation with stakeholders, obtained appropriate approvals for projects, and ensured that the financial and narrative reporting required by those Implementing Agreements was received. For all projects reviewed, financial reports from implementing bodies and recipients were received on time and according to the terms of the Implementing Agreements. Further, spending for the reporting period was explained along with any variances. For most projects, there was also evidence of clear and timely narrative reporting on project activities, as required by the Implementing Agreements. However, as discussed in Section 6.5.5, there was no evidence of reporting on outcomes. In stage 3 project evaluation and close out, the project file review found no evidence of project close-out reports that were required under the SOPs to evaluate project achievements.

The project file review also revealed significant deficiencies in the areas of needs and risk assessment, and a lack project reporting tools. Needs assessments are crucial to ensure that projects respond to well-defined needs and that proposed activities address those needs and have the potential to achieve expected results. Evaluators found that less than half of the projects reviewed contained evidence of preliminary needs assessments. As discussed, the inability of GPP staff to consult with project partners in advance of project approvals may, in some cases, explain this finding.

GPP programming under its worldwide mandate faced a number of risks including project implementation in fragile states, recipient capacity to deliver on projects and the implementation of a large number of smaller projects. The 2012 Summative Evaluation of GPP recommended that the Program establish a risk framework and risk mitigation strategies and GPP management committed to have such a framework in place by September 2014. While the Program was in the process of developing a risk module for the recently adopted electronic database (from GPSF), there was no evidence that a risk framework was being used at the time of the evaluation. Only 1 out of 19 project files reviewed for the evaluation had evidence of a thorough risk assessment, and there was no evidence of risk mitigation strategies. Given GPP’s security mandate, some Canadian government officials expressed concern about the Program’s lack of risk assessment and mitigation strategies.

Evidence from the file review and interviews showed that the Program also lacked standardized templates for needs assessments and financial, narrative and project close out reports. The use of standardized reporting tools by GPP staff and implementing organizations is essential if the Program is to collect the information needed from implementing organizations to effectively monitor projects and ensure reporting aligns with the Program’s Performance Measurement Strategy.

6.5.4 Information Management

Finding # 15: While GPP has made some progress with the implementation of an electronic database, more work is needed to improve information management processes.

The 2012 Evaluation of the GPP highlighted the need for an integrated records management system for financial and program data. At that time, GPP managers signalled their intent to adopt the project management database developed by the GPSF by September 2013.

Some project management staff saw the GPSF database as a good tool that would, in time, make their work easier. From a demonstration made to the evaluation team, it appears that the GPSF database contains the elements necessary for effective project management, including components for risk assessments, results reporting and lessons. For GPP, considerable work is still needed to bring uniformity and consistency to its information management. The database is yet to be integrated with project documentation and there is no overall registry for project files. Currently, record keeping varies from project to project and sub-program to sub-program. According to some DFATD staff, this lack of uniform processes means that it takes a long time for new staff to become effective at their jobs and presents a risk to project and program continuity when staff leave.

GPP uses the departmental Information Management System (IMS) database to manage payments. GPP program staff said that there were plans to create a bridge that would integrate the project information in the GPSF electronic database with the financial information in the IMS database.

One factor complicating progress on information management is the current amalgamation process between the former-Canadian International Development Agency (CIDA) and Department of Foreign Affairs and International Trade into one department, DFATD. Part of that integration is a move to harmonize processes across the new department. Such harmonization would respond to the concerns expressed by staff at several DFATD missions abroad about the lack of coordination among the Department’s security and development programs, as well as the lack of communication with missions on the part of GPP staff. Mission staff said there is a need for a database that captures all of DFATD’s security work that can be shared with missions abroad.

6.5.5 Performance Measurement

Finding #16: GPP was not focused on or reporting on results.

GPP developed a Performance Measurement Strategy in 2013, as part of the TB requirements for program renewal; however, as discussed earlier, it was not being used. In the project files reviewed for the evaluation, evaluators found that only 10% of the project proposals or Implementation Agreements linked project activities and outputs to expected outcomes. Several program staff told the evaluators that while they monitor the ongoing delivery of project activities, the Program does not track results once those activities have been completed. For example, for several projects designed to provide monitoring equipment to detect illicit trafficking of radiological materials at border posts, there were detailed plans to design, purchase and ensure the proper operation of the equipment. However, there were no plans to monitor and report on the results achieved by the provision of this equipment.

GPP’s Implementation Agreements did not contain a requirement for implementing organizations to report against GPP’s expected outcomes. So it was not surprising that there was no results-based reporting matched to the indicators in the GPP Performance Measurement Strategy in the files reviewed. Further, GPP had little reporting available that summarized the overall results of the Program and its different sub-programs. The one program-level report provided to evaluators was a presentation prepared for Central Agencies to report on the 2013-2014 fiscal year that focused largely on program outputs.

Program staff told evaluators that the preventative nature of GPP’s work makes it difficult to measure whether the Program has achieved its ultimate outcome of reducing the threat posed by WMD. However, the GPP Performance Measurement Strategy includes a logic model that provides a rational progression from activities to outcomes at the immediate and intermediate levels. The Strategy also provides indicators for measuring these short and medium-term outcomes that will contribute to the Program’s ultimate outcomes over the long term. Several of those interviewed expressed concern over the lack of reporting on outcomes and its implication for demonstrating results. Further, a survey of GPP staff found that less than a third of Program personnel had participated in results-based management training.

Finding #17: GPP lacked a process for collecting and integrating lessons and best practices into future programming.

The element of on-going learning was absent from GPP’s SOPs and evaluators found no evidence that GPP was capturing lessons and best practices and integrating them into future programming. The file review showed that in about half of the completed projects reviewed, implementing organizations shared lessons from their programming experience that could inform GPP’s future efforts. The GPSF database being adopted by GPP contained a component for capturing lessons, but GPP had yet to make use of it.

6.5.6 Human Resources

Finding #18: GPP has an experienced staff with considerable subject matter expertise. However, Program management is concerned about its ability to retain that expertise.

At the time of the evaluation, GPP had approximately 26 FTEs^{Footnote 35}, which appeared sufficient for current programming levels. Many of these staff, especially those involved in program management and delivery have been with GPP for a number of years and have gained considerable knowledge about their subject areas, a fact cited by several of those interviewed as one of the Program’s strengths. Interviewees praised GPP Program staff for their technical expertise and the clarity of their statements of work related to equipment procurement.

Some concerns were raised by GPP management about the ability to retain experienced employees due to two main factors: 1) the lack of parity that existed between Program Administration (PM) staff in GPP compared to other DFATD security programs; and 2) the introduction of a departmental mobility pool for PM which increases the risk of persons with specialized knowledge being easily replaced upon leaving the Program. The first issue was, however, recently resolved through the reclassification of PM-04 level positions in GPP to the PM-05 level, which may also lower the probability of staff at level leaving the Program.

6.5.7 Resource Utilization

Finding # 19: GPP Vote 1 and Vote 10 disbursements over the past two years have been significantly lower than original allocations.

According to GPP program staff, the $73 million annual budget allocation over five years was adequate for achieving the Program’s expected results. However, as shown in Table 5, the Program spent only $46 million in 2013-2014 and $42 million in 2014-2015, 63.1% and 57.7% respectively of the Program’s budget.

More specifically, in 2013-14 GPP spent just 62% of its allocation for Gs&Cs and in 2014-15 it spent just 55% of its Gs&Cs allocation. The percentage of spent funds over allocated funds for Vote 1 Programming fared even worse, spending only 44.6% of its budget in 2013-2014 and 25.1% in 2014-2015.

As discussed earlier in this report, underspending in the Gs&Cs category was largely attributed to the project initiation authorization process in effect during the evaluation period. The dramatic decline in spending on Vote 1 programming (44.6% of its allocation in 2013-2014 and 25.1% of its allocation for 2014-2015) was reportedly due to the Program not receiving full allocations at the start of each fiscal year.

The cost of administering the program as a percentage of overall program spending was originally projected at 8.6%, which is not unreasonable for such a complex and ambitious program. However, due to the significant underspending noted above, the costs of administering the program in relation to actual program spending increased significantly to 12.4% in 2013-2014. In 2014-2015, this ratio increased to 17.9% of program spending.

7.0 Conclusions of the Evaluation

The evaluation found that there is a continuing need for GPP to address significant ongoing threats from WMD proliferation in the areas of nuclear, radiological, chemical and biological security. Evidence showed that the Program is aligned with DFATD’s Strategic Outcomes, the priorities and objectives of the federal government and is consistent with the federal role.

GPP made some progress toward its expected outcomes; however a full assessment of Program results was not possible due to the lack of reporting on outcomes at the project and program level. The evaluation found that the GPP largely supports short-term projects that are focused on the delivery of outputs such as equipment and training, rather than the contribution of these outputs towards the achievement of outcomes.

Notwithstanding the foregoing, the evaluation found some evidence that each of the GPP sub-programs are contributing to immediate outcomes. Results were strongest where GPP made focused interventions and/or was working with international organizations. For example, GPP’s support for the work of the IAEA contributed to that organization’s inspections of Iran’s nuclear facilities. GPP’s collaboration with the OPCW contributed to the destruction of chemical weapons stockpiles in Syria and Libya. In Jordan, GPP’s support for multiple projects across its three sub-programs contributed to that country’s ability to detect WMD and respond to CBRN incidents that might arise from instability in Iraq or Syria. The evaluation found that such focused investments were also the most likely to have contributed to results for Canada in the form of Canadian leadership, influence and visibility. Only limited evidence was found of projects that included approaches to ensure longer-term sustainability of results.

GPP completed the first step in its strategic planning process of identifying a broad list of countries/regions for engagement. However it did not develop the Strategic Directions Document that was to narrow down the list to five to 10 priority countries/regions or the Priority Country/Region Plans that were to guide project selection. This resulted in GPP programming that was broadly and thinly spread across 25 different countries.

GPP’s ability to plan strategically was undermined by the departmental project approval process in effect during the evaluation period. That process also dramatically reduced the ability of the program to disburse its approved funding, adversely impacting the efficiency and economy of program delivery.

GPP developed SOPs and the evaluation found that it is fulfilling some of the basic elements of project management in relation to financial and narrative reporting. The evaluation found, however, little documented evidence that GPP is systematically conducting needs and risk assessments or collecting and integrating lessons and best practices into its work. A lack of standardized reporting tools also hindered project management and reporting. The evaluation found no evidence of duplication between GPP and DFATD’s other security programs or programs working at cross purposes, however, there is scope for increased coordination between GPP and other security programs, other DFATD programs, and OGDs.

GPP has made some progress in information management by adopting the GPSF database, however further work is needed to integrate modules for risk, results reporting and lessons. GPP also needs to integrate its project documentation into the database and develop a registry for project files as record keeping varied from project to project.

GPP’s performance measurement was weak. It was not using its Performance Measurement Strategy to identify the outcomes its projects were expected to achieve and there was little reporting on results.

Finally, GPP has experienced staff with considerable subject matter expertise. Program management expressed concerns related to the retention of this expertise given the staff mobility processes in the department.

8.0 Recommendations

It is recommended that:

Recommendation #1: GPP update its priority setting process and develop strategic plans to shape programming activities.

GPP should review and expand on its annual priority setting process by developing strategic plans at the program and sub-program level to ensure that the Program is addressing the thematic and geographic areas of greatest and most imminent threats to Canada. Based on that, priority countries and regions where these threats emanate from should be identified. These plans should address the range of issues of importance in the strategic planning process including: identifying specific programming priorities; notional financial allocations at the sub-program level; and monitoring and performance measurement plans. Once developed, the strategic plans should be used to inform the development of project proposals. Flowing from these strategic plans, GPP should implement a formal project selection process whereby potential projects are reviewed against strategic plans and other established criteria, including input from relevant DFATD divisions and OGDs. Effort should also be invested in producing results-based management indicators including a logic model for each project to measure project performance. Human resources needs should also be taken into consideration in the overall program and sub-program planning process.

Recommendation #2: GPP update its SOPs to include guidance for needs assessments, risk management, reporting, and integrating lessons learned into future programming initiatives. GPP should ensure that the SOPs are being consistently followed.

GPP should update its SOPs to include guidance for completing needs assessments, risk management strategies, reporting on results and lessons-learned. GPP should move forward with its plan to adopt the GPSF risk assessment module which will allow the Program to assess risk both at the program planning and implementation stages. GPP should also develop a checklist to ensure that all elements of its SOPs are followed and documented. In updating these SOPs, GPP may consider reviewing the SOPs of other security programs and adopting some of their good practices.

A standard set of reporting tools also needs to be developed for use across all GPP projects including templates for needs assessment reports, project monitoring reports, financial and narrative^{Footnote 36} reports by implementing organizations, and project close out reports, including lessons learned. Project documentation needs to be linked to the project database and a project file convention needs to be developed to organize project documentation according to the key components of the SOPs.

Recommendation #3: GPP work with other security and development programs, as well as geographic divisions and missions, to develop mechanisms that will allow for increased integration, coordination and synergy.

GPP should, through the development and implementation of SOPs, templates and best practices, ensure that thorough consultations and coordination is undertaken with relevant geographics, thematics, OGDs, and other DFATD security and development programs throughout the project life-cycle, including at the needs identification and assessment stages. Consultations undertaken should also be reflected in the project approval documents. Such mechanisms will assist in generating programming efforts that are complementary, ensure that information is shared about implementing partners, and provide opportunities to explore joint needs assessments and monitoring missions, and share lessons and best practices. GPP should also examine the opportunities for enhanced consultation with and involvement of OGD subject matter experts in its programming areas.

Recommendation #4: GPP management update and integrate its Performance Measurement Strategy into its planning, implementation and reporting to ensure that programming is focused on the achievement of expected outcomes.

GPP management should ensure that all of its program staff are trained in results-based management, ideally in sessions specifically related to security programming. A component of that training could be used to review and update the Program’s Performance Measurement Strategy as a way to begin to develop ownership for the strategy and a results culture within the Program. At the project level, performance indicators linked to the Performance Measurement Strategy need to be integrated in all Implementation Agreements to ensure that implementing organizations are working toward and reporting against GPP’s expected immediate and intermediate outcomes. Further, results-reporting from implementing organizations also needs to be documented in the project database. GPP needs to report annually on the achievement of expected results at the project, sub-program and program level.

9.0 Management Response and Action plan

Recommendation 1

GPP update its priority setting process and develop strategic plans to shape programming activities.

GPP should review and expand on its annual priority setting process by developing strategic plans at the program and sub-program level to ensure that the Program is addressing the thematic and geographic areas of greatest and most imminent threats to Canada. Based on that, priority countries and regions where these threats emanate from should be identified. These plans should address the range of issues of importance in the strategic planning process including: identifying specific programming priorities; notional financial allocations at the sub-program level; and monitoring and performance measurement plans. Once developed, the strategic plans should be used to inform the development of project proposals. Flowing from these strategic plans, GPP should implement a formal project selection process whereby potential projects are reviewed against strategic plans and other established criteria, including input from relevant DFATD divisions and OGDs. Effort should also be invested in producing results-based management indicators including a logic model for each project to measure project performance. Human resources needs should also be taken into consideration in the overall program and sub-program planning process.

Management Response & Action Plan (Agreed):

Priority Setting and Strategic Plans: As part of its regular priority setting exercise, IGA will identify priority countries and regions and prepare strategic plans at the program and sub-program level. These plans will be used to inform and guide the development of project proposals, recognising that unanticipated threats are likely to emerge that will require programming attention.

Responsibility Centre: IGA Director and Policy Deputy Director
Time Frame: June 2016

Management Response & Action Plan (Agreed):

Project Selection Process: IGA introduced a new Project Approval Document (PAD) and Standard Operating Procedures (SOPs) on December 2, 2015. The PAD and SOPs establish common selection criteria for all GPP projects, including consultations with relevant stakeholders, alignment with priorities and strategic plans, project monitoring, performance measurement and sustainability. The PAD serves as the basis for project selection, with final project approval sought from the appropriate management level in accordance with the sub-delegation of financial authorities.

Responsibility Centre: IGA Director and Policy Deputy Director
Time Frame: April 2016

Management Response & Action Plan (Agreed):

Program Logic Model: IGA is committed to developing Program and Project -level Logic Models (LM) and Performance Measurement Frameworks (PMF).  Initial drafts have been produced, and the project-level elements are identified in the PAD template. Following consultations with RBM experts, IGA project officers and Deputy Directors, final products are expected to be completed by April 2016. The LMs and PMFs will align with the corporate LM and PMF as well as the IFM branch-level Performance Measurement Strategy that is currently being drafted. It is anticipated that in most cases, the project level LM/PMF will be integrated into Implementing Agreements, and would be saved to the database.

Responsibility Centre: IGA Director and Policy Deputy Director
Time Frame: April 2016

Management Response & Action Plan (Agreed):

Human Resources: IGA will continue to review its Human Resources requirements on an on-going basis, taking into consideration human resource needs of sub-programs.

Responsibility Centre: IGA Director and Deputy Directors
Time Frame: Implementation ongoing

Management Response & Action Plan (Agreed):

Notional financial allocations: As part of the strategic planning process, IGA agrees to assign notional financial allocations to GPP sub-programs at the beginning of each fiscal year, and to review and adjust throughout the year, taking into consideration reference levels, TB Sub financial tables and GPP priorities.

Responsibility Centre: IGA Director and Deputy Directors
Time Frame: Implementation April 2016 and ongoing

Recommendation 2

GPP update its SOPs to include guidance for needs assessments, risk management, reporting, and integrating lessons learned into future programming initiatives. GPP should ensure that the SOPs are being consistently followed.

GPP should update its SOPs to include guidance for completing needs assessments, risk management strategies, reporting on results and lessons-learned. GPP should move forward with its plan to adopt an adapted version of the Security Sector Program Management Tool (SSPMT database) risk assessment module developed for the GPSF which will allow the Program to assess risk both at the program planning and implementation stages. GPP should also develop a checklist to ensure that all elements of its SOPs are followed and documented. In updating these SOPs, GPP may consider reviewing the SOPs of other security programs and adopting some of their good practices.

A standard set of reporting tools also needs to be developed for use across all GPP projects including templates for needs assessment reports, project monitoring reports, financial and narrative^{Footnote 37} reports by implementing organizations, and project close out reports, including lessons learned. Project documentation needs to be linked to the project database and a project file convention needs to be developed to organize project documentation according to the key components of the SOPs.

Management Response & Action Plan:

SOPs and Templates: IGA is in the process of updating its SOPs to include guidance on needs assessments, risk management, reporting on results and lessons learned. IGA is also drafting project level reporting tools, including financial, narrative, final, and project monitoring reporting templates, drawing on best practices from other Global Affairs’ security programs.

Responsibility Centre: IGA Director and Deputy Directors, Policy and Corporate
Time Frame: September 2016

Management Response & Action Plan:

Risk Assessment: IGA has now integrated a Risk Assessment Module in the Security Sector Program Management Tool (SSPMT database). Risk assessments are completed at both project planning and implementation stages, and are reviewed and validated by the Deputy Director. A copy of the initial risk assessment is incorporated into the PAD.

Responsibility Centre: IGA Director and Deputy Directors
Time Frame: Completed and implementation ongoing

Management Response & Action Plan:

Naming and Storage Conventions: As of December 2015, project officers have the technical tools to migrate project documentation into the database and documentation follows a standardised naming convention. The SOPs and project checklist will be updated to reflect this advancement, and project officers will receive training. All project titles entered into the database follow a standard file naming convention. On the I:drive, IGA is in the process of implementing SII – Information Services Enterprise Level Folder Structures initiative, which aims to standardise information management across Global Affairs Canada. As part of this initiative, IGA is standardising project naming and documentation storage conventions.

Responsibility Centre: IGA Director and Policy Deputy Director
Time Frame: June 2016

Recommendation 3

GPP work with other security and development programs, as well as geographic divisions and missions, to develop mechanisms that will allow for increased integration, coordination and synergy.

GPP should, through the development and implementation of SOPs, templates and best practices, ensure that thorough consultations and coordination is undertaken with relevant geographics, thematic areas, OGDs, and other DFATD security and development programs throughout the project life-cycle, including at the needs identification and assessment stages. Consultations undertaken should also be reflected in the project approval documents. Such mechanisms will assist in generating programming efforts that are complementary, ensure that information is shared about implementing partners, and provide opportunities to explore joint needs assessments and monitoring missions, and share lessons and best practices. GPP should also examine the opportunities for enhanced consultation with and involvement of OGD subject matter experts in its programming areas.

Management Response & Action Plan (Agreed):

Project level consultation, collaboration, communication: IGA agrees that consultation, collaboration and communication are essential at the project level, and will take steps to further enhance standards and guidance for project managers, most notably in the tracking of project-level consultations to ensure consistency of approach. Guidance to this effect has already been integrated into the Project Approval Document and related SOPs. Beyond this, IGA will examine best practices in other security programs related to other stages in the project life-cycle. In addition, IGA is currently examining how to increase OGD partner participation in GPP projects (e.g. conduct of needs assessments, delivery of training, project monitoring and evaluation) and is exploring the possibility of convening a GPP Advisory Group, modelled after the START Advisory Group.

Responsibility Centre: IGA Director and Deputy Directors
Time Frame: September 2016

Management Response & Action Plan (Agreed):

Coherence with Security and Development Programs: IGA Program, Policy and Corporate leads will continue to strongly support and actively contribute to the work of the IFM Security Coherence Unit, along with other coherence mechanisms as they arise, including: the International Assistance Envelope (IAE) Management Director-General led coherence reviews and mechanisms, such as country strategy reviews and the new IAE policy and funding framework that was called for in MINE’s mandate letter; the NGM-initiated Program Mechanism Review; IGA Directors’ regular participation at Business Operations Committee and IFM Directors Forum; and IGD’s participation in DG and ADM Programming Committees.

Responsibility Centre: IGA Director and Deputy Directors
Time Frame: Implementation ongoing

Recommendation 4

GPP management update and integrate its Performance Measurement Strategy into its planning, implementation and reporting to ensure that programming is focused on the achievement of expected outcomes.

GPP management should ensure that all of its program staff are trained in results-based management, ideally in sessions specifically related to security programming. A component of that training could be used to review and update the Program’s Performance Measurement Strategy as a way to begin to develop ownership for the strategy and a results culture within the Program. At the project level, performance indicators linked to the Performance Measurement Strategy need to be integrated in all Implementation Agreements to ensure that implementing organizations are working toward and reporting against GPP’s expected immediate and intermediate outcomes. Further, results-reporting from implementing organizations also needs to be documented in the project database. GPP needs to report annually on the achievement of expected results at the project, sub-program and program level.

Management Response & Action Plan (Agreed):

RBM Training: IGA accepts this recommendation, and has been working towards organising training sessions specifically related to the security programming environment, ideally in conjunction with the introduction of project level LM/PMFs. Eight out of eleven GPP project officers have participated in formal RBM training.

Responsibility Centre: IGA Director and Deputy Director, Policy
Time Frame: September 2016

Management Response & Action Plan (Agreed):

Performance Measurement (PM) Strategy: IGA agrees and priority will be given to ensuring that the PM Strategy, along with Results-Based Management best practices from other security programs and expertise of IGA staff, are translated into the SOPs and templates, including project level LMs/PMFs. Over the next year, IGA will develop a Program Renewal Strategy to propose a funding renewal of the GPP beyond its current mandate. In that context, relevant documents will be updated to include performance management considerations.

Responsibility Centre: IGA Director and Policy Deputy Director
Time Frame: March 2017

Management Response & Action Plan (Agreed):

LM/PMF: IGA agrees with the recommendation and is contributing to the development of an IFM LM & PMF.  IGA is contributing to the development of an IFM logic model and performance measurement framework (LM & PMF). In parallel, IGA is drafting Program and Project level LM/PMFs that will align with the IFM products. It is anticipated that the project level LM/PMF will be integrated in most cases into Implementing Agreements, and would be saved to the database. The LM and PMF will help support IGA in strengthening program and project level reporting on the achievement of outcomes, and not simply outputs.

Responsibility Centre: IGA Director and Policy Deputy Director
Time Frame: April 2016

Management Response & Action Plan (Agreed):

Program-level Reporting: IGA agrees that program level reporting should increasingly focus on the achievement of expected outcomes. IGA contributes to Global Affairs Canada’s report to Parliament on departmental activities and programming results in the annual Report on Plans & Priorities, the annual Departmental Performance Report and the annual Summary Report on the International Assistance Envelope. IGA will also ensure that its programming remains fully aligned with Government of Canada priorities and objectives of the Programs, which are ultimately reflected in the Department's Program Alignment Architecture and is built into Program SOPs and the PAD. IGA will continue to provide an annual verbal update and deck to Central Agencies on GPP spending and results. IGA will standardise project-level reporting practices, as detailed above. Together, these inputs satisfy the project, sub-program and program level reporting requirements laid out in the Program’s approved Treasury Board Submission and meet the recommendations presented by the GPP Formative Evaluation.

Responsibility Centre: IGA Director
Time Frame: Implementation ongoing

Annex 1: List of Evaluation Findings

Finding 1: There is a continued need for Canada’s Global Partnership Program as it addresses significant and ongoing threats from WMD proliferation.

Finding 2: The objectives of GPP and its sub-programs are aligned with and supportive of the priorities and objectives of the federal government and GAC’s Strategic Outcomes.

Finding 3: The delivery of GPP conforms to the responsibilities of the federal government, the mandate of GAC and its federal delivery partners.

Finding 4: There is some evidence that GPP is making progress toward expected outcomes. However, a full assessment was not possible due to a lack of performance reporting.

Finding 5: There was evidence of progress toward the immediate outcomes of improved security measures for storage and transport of WMD-related materials and the development of networks and partnerships, and the intermediate outcome of increased compliance to international WMD commitments.

Finding 6: There was some evidence of progress toward the immediate outcomes of improved detection capabilities and the development of networks and partnerships.

Finding 7: There was some evidence of progress toward the immediate outcomes of improved WMD detection capabilities and development of networks and partnerships related to its support for UNSCR 1540.

Finding 8: GPP support for chemical weapons destruction in Syria and Libya contributed to progress toward the Program’s ultimate outcome of reducing access to WMD materials.

Finding 9: Focused GPP investments in some organizations and countries have contributed to Canadian leadership, influence and visibility.

Finding 10: The potential for sustainability of results was greatest among projects delivered through international partners and lowest among one-time training projects that were delivered bilaterally and were not tied to longer-term project activities.

Finding 11: The Strategic Planning Process that was developed in the context of the 2013-2018 Renewal Mandate was not followed.

Finding 12: Project approval processes hindered GPP’s ability to plan strategically and achieve results, and weakened the Program’s credibility with international partners.

Finding 13: There is room for improved consultation and coordination with other GAC programs, missions abroad and OGDs.

Finding 14: GPP was fulfilling some of the basic elements of its Standard Operating Procedures. However, it lacked a formal project review process, failed to conduct needs and risk assessments and lacked standardized reporting tools.

Finding 15: While GPP has made some progress with the implementation of an electronic database, more work is needed to improve information management processes.

Finding 16: GPP was not focused on or reporting on results.

Finding 17: GPP lacked a process for collecting and integrating lessons and best practices into future programming.

Finding 18: GPP has an experienced staff with considerable subject matter expertise. However, Program management is concerned about its ability to retain that expertise.

Finding 19: GPP Vote 1 and Vote 10 disbursements over the past two years have been significantly lower than original allocations.