Evaluation of the North American Platform Program (NAPP) Partnership

Final Report
Prepared by the Diplomacy, Trade and Corporate Evaluation Division (PRE)
Global Affairs Canada
June 26, 2017

Table of Contents

• Acronyms and Abbreviations
• Executive Summary
• Program Background
• Program Resources
• Evaluation Scope and Methodology
• Limitations
• Findings
• Recommendations and Management Responses

Acronyms and Abbreviations

AAFC

Agri-Food Canada - Agriculture and Agri-Food Canada

ACOA

Atlantic Canada Opportunities Agency

BDC

Business Development Bank of Canada

CBSA

Canada Border Services Agency

CED-Q

Canadian Economic Development for Quebec

CFSI

Canada Foreign Service Institute

CH

Canadian Heritage

COOL

Country of Origin Labelling

CU

Coordination Unit

DND

National Defence

ECCC

Environment and Climate Change Canada

EDC

Export Development Canada

EOF

Economic Outcome Facilitated

ERI

Enhanced Representation Initiative

FPDS

Foreign Policy and Diplomacy Service

FY

Fiscal Year

GAC

Global Affairs Canada

HC

Health Canada

HC

Health Canada

HQ

Headquarters

IBD

International Business Development

INAC

Indigenous and Northern Affairs Canada

IRCC

Immigration, Refugees and Citizenship Canada

ISED

Innovation, Science and Economic Development Canada

LES

Locally Engaged Staff

NNB

North America Advocacy Division

NNC

North America Commercial Programs Division

NRC

National Research Council

NRCan

Natural Resources Canada

OGD

Other Government Department

OP

Opportunity Pursued

PRD

Evaluation and Results Bureau

PRE

Diplomacy, Trade and Corporate Evaluation Division (PRE)

PS

Public Safety

SWC

Status of Women Canada

TB

Treasury Board

TC

Transport Canada

TCS

Trade Commissioner Service

TD

Temporary Duty

TRIO2

Client Relationship Management (CRM) system used by the Trade Commissioner Service (TCS)

WD

Western Economic Diversification

Executive Summary

The NAPP is a well-designed and well-managed program with effective governance structures that support the achievement of results through sound project planning and implementation.

The NAPP is achieving its objectives of fostering an integrated approach, advancing Canadian interests, and promoting international business development in North America:

• Collaboration between missions and NAPP Partners is strong and partnerships are taking place regularly which has led to a more unified and coordinated approach to advocacy and trade initiatives. However the benefits from the integrated approach are confined to only NAPP Partners preventing a greater integrated approach from taking place.
• The NAPP is contributing to the advancement of Canadian interests in North America. Advocacy initiatives using NAPP funding are reaching and engaging their target audiences leading to increased awareness of Canadian positions, increased support for Canadian interests, and tangible changes in foreign policies that are aligned with Canadian interests. However NAPP advocacy priorities are not well aligned with the priorities for Canada in Mexico limiting the advancement of Canadian interests in Mexico.
• NAPP is contributing to the promotion of international business development in the North American market. The Key Performance Indicators (KPIs) for North America demonstrate successes are being achieved in part with NAPP funding. In addition, many Trade Commissioner Service (TCS) officers at missions indicated that the NAPP is their most important source of program funding and indicated the NAPP has enabled many successes.

The Temporary Duty (TD) and LES Professional Development programs are deeply valued by all stakeholder groups fostering understanding and collaboration, enhancing knowledge, skills, and networks, and increasing the government’s capacity to deliver projects in North America.

Performance measurement systems exist for two of the NAPP’s objectives but not for the objective of fostering an integrated approach, leaving gaps in knowledge of the totality of results achieved.

Program Background

About the NAPP

The NAPP Partnership

The North American Platform Program (NAPP) is horizontal initiative of the Government of Canada that provides a coordinated and integrated approach to advancing Canadian interests and promoting international commerce in the United States and Mexico. The NAPP, originally the Enhanced Representation Initiative (ERI), was established in 2003 for a five-year term. The NAPP has since been renewed three times (2008-2013, 2013-2016, and 2016-2019). There are six Government of Canada departments and agency Partners (including GAC) who make up the NAPP partnership: Two Partners support advocacy programming: Agriculture and Agri-Food Canada (AAFC) and Atlantic Canada Opportunities Agency (ACOA), and three Partners support International Business Development (IBD) programming: Canada Economic Development for Quebec Regions (CED-Q), Export Development Canada (EDC), and Innovation, Science and Economic Development Canada (ISED).

NAPP Governance: Currently, the NAPP Partnership is governed by a committee structure that includes a Director General Committee and a Working Group (WG), both of which are supported by the NAPP Coordination Unit (CU) within GAC. It is also supported by other divisions in GAC, as well as a network of 19 North American Missions.

Program Resources

Text version

• GAC - 64%
• AAFC - 8%
• ACOA - 8%
• CED-Q - 8%
• ISED - 8%
• EDC - 3%

The total budget for the NAPP is $4.2 million annually (2015/16). GAC contributes $2.7 million to the NAPP Partnership while Partner departments and agencies have committed to spend at least $350,000 towards achieving NAPP priorities and objectives.

Core funds are disbursed between advocacy and IBD program streams. In FY2015/16, $1.3 million was spent on IBD initiatives while $1.275 million was spent on advocacy initiatives.

A portion of the Partner contributions are also transferred to GAC for programming, for the optional TD and Training Program, and for partnership administration. The remaining portion is retained by each Partner to pursue their own initiatives and projects.

Program funds are provided to missions from GAC Headquarters through direct budget transfers through a fund (N702) created in 2014. Before 2014, NAPP allocations were made available to missions through a series of Fund Reservations (RS) held at Headquarters in Canadian dollars.

GAC contributes the largest portion of funds to the NAPP by far. (2015/16)

Sources: NAPP Annual Report, 2015/16 and interviews with program staff.

Evaluation Scope and Methodology

Evaluation Scope

Evaluation Questions

The evaluation focused on the following questions:

1. To what extent has the NAPP contributed to:
   1. Fostering a coordinated and integrated approach,
   2. Advancing Canadian interests through advocacy efforts, and
   3. Promoting International Business Development?
2.  Is the TD and Training Program achieving its objectives? 
3.  To what extent are the recent changes to the NAPP driving program improvement?
4.  How well is the current governance structure supporting the achievement of results?
5.  How does the NAPP track its performance? To what degree is performance information enabling decision-making?

Methodology

This evaluation employed a mixed-method design using both quantitative and qualitative methods to evaluate the Program between 2013-2016, the most recent agreement for the NAPP Partnership. Interviews with a variety of program stakeholders (e.g., FPDS and TCS officers, NAPP Partners, TD and training participants and program staff) were triangulated with program documents, performance data on advocacy initiatives from Strategia (GAC’s Planning and Reporting tool), and performance data on trade initiatives from TRIO 2 (the Client Relation Management System for the Trade Commissioner Service) to answer the evaluation questions

Limitations

One of the primary purposes of the NAPP is to enable the TCS, Foreign Policy and Diplomacy Service (FPDS) and partners to carry out their work more effectively. NAPP programming is implemented along with numerous complementary activities (e.g., other funding programs, partner organizations, etc) in order to achieve results. As a result, NAPP’s performance and successes cannot be directly attributed to the NAPP. GAC’s performance monitoring systems are set up to capture overall results rather than results of the various programs that contributed to those results. This created methodological and analytical challenges as summarized below. Given these limitations, the evaluation focused on assessing the contribution of the NAPP to its objectives of promoting international business development, advancing Canadian interests through advocacy, and fostering a coordinated and integrated approach through triangulation of data collection methods and data sources.

Limitations

• The NAPP is one of multiple programs supporting the objectives of the TCS. Separating out the achievements of the NAPP from those of the TCS as a whole is a challenging endeavor given that the Key Performance Indicators (KPIs) are tracked by initiative rather than funding program (TRIO). KPIs can be tracked back to their funding source in Strategia, however KPIs in Strategia are not validated by the performance measurement unit (BTB) and therefore are not reliable.
• Advocacy and trade successes achieved in part or in whole with NAPP funding cannot be directly attributed to the Program. Multiple external factors in addition to the initiatives that missions implement with NAPP funding have contributed to the achievement of successes making it difficult to differentiate between the contribution of the NAPP and the contribution of external factors to any given success. In addition, information from advocacy KPIs in Strategia was incomplete and entered inconsistently resulting in issues of reliability.
• Information regarding the results of the initiatives carried out by Partners with their retained funding was not available for analysis by the evaluation team.
• Some performance measurement data was only available for part of the reference period (2015/16) rather than the full reference period. 

Mitigating Strategies

• Information from TRIO 2 relating to KPIs for all of North America was triangulated with interview data from TCS officers, program staff, and program documents to determine the NAPP’s contribution to promoting international business development.
• Information from Strategia relating to the advocacy KPIs was triangulated with interview data from FPDS officers, program staff and program documents to determine the NAPP’s contribution to advancing Canada's interests in the U.S. and Mexico. 

Findings

Finding 1: The NAPP is fostering a coordinated and integrated approach among the NAPP Partners through coordination and collaboration, leading to better quality initiatives and services for clients.

Evaluation question 1a: To what extent has the NAPP contributed to fostering a coordinated and integrated approach?

Partnerships between missions and NAPP Partners are taking place regularly.

All North American missions are collaborating with NAPP Partners on both advocacy and IBD. Each mission reported working with a NAPP Partner at least once over the past three years.

For IBD, formal partnerships through the pooled funding mechanism are taking place between missions and NAPP Partners across the U.S. and in Mexico (see map on right for 2015/16). ISED and EDC are the most active IBD Partners. At least 10 missions reported partnering with each of them within the past three years.

In total, missions and NAPP Partners collaborated on 39 initiatives through the NAPP pooled IBD process in 2015/16

Text version

ISED partnered with 11 missions on 20 different missions. EDC partnered with 8 missions on 17 initiatives. CED-Q partnered with 1 mission on 2 separate initiatives.

For advocacy, collaborations are also taking place though not as often as on the IBD side, as there are only two NAPP advocacy Partners. AAFC is the most active advocacy Partner, with 9 missions reporting partnering with them within the past three years. No missions interviewed reported working with the second Partner, ACOA.

Note: Map displays formal partnerships through IBD pooled funding mechanism in 2015/16 only. Additional informal partnerships may have occurred outside the pooled funding mechanism,.

Sources: NAPP Coordinating Unit, IBD pooled projects, 2015/16, interviews with FPDS and TCS officers.

Text version

• A map of the United States and Mexico which displays the formal partnerships through IBD pooled funding mechanism in 2015/16.
• Lines connect ISED with Canadian missions in SEATL, SFRAN, PALTO, LNGS, SNDG, DENVR, MNPLS, WSHDC, CNGNY and BOSTN.
• Lines connect EDC with Canadian missions in SFRAN, PALTO, MXICO, DALAS, DTROT, WSHDC and BOSTN.
• Lines connect CED-Q with the Canadian mission in BOSTN.

The NAPP has encouraged a collaborative, unified approach leading to better quality initiatives and services for clients.

Partnerships between missions, TD participants, and NAPP Partners have led to:

• A more unified and coordinated approach to advocacy and trade initiatives
• Increased access to subject matter expertise for officers at missions
• Increased networks and client contacts for both missions and NAPP Partners
• More accurate messaging conveyed to target stakeholders, and
• Better quality support to clients

Staff at North America missions, NAPP Partners, and TD participants indicated the NAPP fosters activities that promote a coordinated and integrated approach.

Missions indicated they had stronger strategic coordination of initiatives, increased access to subject matter expertise and the opportunity to widen their networks and client contacts as a result of the NAPP.

NAPP Partners reported that one of the greatest values of the NAPP is enhanced access to and collaboration with North American missions which has led to a host of benefits including: larger networks, the ability to influence both IBD and advocacy strategies, appropriate messaging being conveyed, improved business intelligence and better support to key clients.

Both GAC and NAPP Partner employees who participated in the TD Program reported an increased understanding between GAC headquarters, missions, and other departments and were able to more effectively collaborate with those missions when they returned to their permanent positions.

Sources: interviews with FPDS and TCS officers at missions, NAPP Partners, and TD participants.

Finding 2. There are opportunities to increase collaboration within the NAPP through partnerships with additional organizations.

Evaluation question 1a: To what extent has the NAPP contributed to fostering a coordinated and integrated approach?

Given that there are only five Partners in the NAPP, a fraction of the number of OGDs with strategic and/or operational interests in North America, there are opportunities to increase collaboration within the NAPP through partnerships with additional organizations.

OGDs that are not NAPP Partners have little incentive to provide support to missions or coordinate activities with them. Missions have reported that collaboration with OGDs outside the NAPP is more challenging and have encountered difficulties in engaging OGDs to provide critical information and support in order to successfully implement their initiatives.

In order to truly foster a coordinated and integrated approach, collaboration is required from all OGDs who have strategic interests and priorities in advocacy and business development in North America.

In addition, many missions in North America indicated their interest in collaborating with OGDs who are not NAPP Partners yet have mandates that are very well aligned with NAPP priorities (e.g., the advocacy priority of “Energy/Environment” aligns well with Natural Resources Canada and  Environment and Climate Change Canada).

Text version

A radial diagram with GAC at the center showing the connection with the five NAPP partners: ISED, CED-Q, EDC, AAFC, ACOA. Other government departments (OGDs) are placed farther from the center with a dotted line to demonstrate that missions have encountered difficulty collaborating with OGDs outside the NAPP.

The following departments were mentioned by FPDS officers as great additions to the NAPP Partnership. NRCan and ECCC were the most often mentioned departments (5 missions each).

Text version

• NRCAN and ECCC were mentioned by 5 missions.
• INAC was mentioned by 2 missions.
• The remainder were mentioned by 1 mission each: SWC, IRCC, TC, CBSA, CH, PS, DND, NRC, HC.

The following departments were mentioned by TCS officers as great additions to the NAPP Partnership. ACOA, AAFC, and NRC were the most often mentioned departments.

Text version

• ACOA, AAFC and NRC were mentioned by 3 missions each.
• BDC and NRCAN were mentioned by 2 missions each.
• WD, HC and DND were mentioned by 1 mission each.

Sources: interviews with FPDS and TCS officers at missions, and program management

Finding 3: NAPP is contributing to the advancement of Canadian interests in North America by reaching and engaging target audiences, raising awareness and support, and through the achievement of policy changes.

Evaluation question 1b: To what extent has the NAPP contributed to advancing Canadian interests through advocacy efforts?

Text version

• NAPP funds: 50%
• GENad, OPS, Other: 50%

The NAPP represents a significant amount of advocacy funding available to missions in North America.

NAPP funds make up half of all advocacy funding for initiatives in North America, or $1.275 million (2015/16).

Numerous advocacy initiatives are being implemented with NAPP funding. Out of the almost 500 advocacy initiatives carried out in FY2015/16, over half (54%, 259) were funded in part or in whole by NAPP.

Text version

• WSHDC: 33
• CNGNY: 27
• MNPLS: 27
• BOSTN: 23
• DALAS: 22
• DTROT: 20
• SEATL: 20
• DENVR: 17
• ATNTA: 16
• CHCGO: 15
• LNGLS: 14
• MIAMI: 14
• SFRAN: 9
• MXICO: 2
• Total advocacy initiatives implemented with NAPP funding in FY 2015/16: 259

Performance data collected through Strategia indicates NAPP-funded initiatives are reaching and engaging their target audiences leading to increased awareness and willingness to support Canadian positions.

Initiatives are engaging target audiences. Over half of all initiatives resulted in strong engagement from target audiences while another 6% of initiatives succeeded in the target audience fully supporting Canada’s position in 2015/16 (n=252).

Text version

• 6% of initiatives had full engagement from the target audience.
• 55% of initiatives had strong engagement from target audiences.
• 34% had partial engagement from target audiences.
• 2% of initiatives had minor engagement from target audiences.
• 3% of initiatives had no engagement from target audiences.

The majority of initiatives are reaching their target audiences. Two-thirds reached their target audiences in 2014/15 (n=252).

Text version

• Fully Reached: 65%
• Partially Reached: 32%
• Not Reached: 3%

Sources: Strategia 2015/16, NAPP annual report, and North America Advocacy report 2015/16.
Note: Graphs  are based on data from Strategia. Strategia data from FY2015/16 is incomplete and has not been entered  by FPDS officers at missions consistently. As a result some indicators are not represented here due to reliability issues. (e.g., intelligence gathered)

Awareness and support for Canadian interests has increased  and changes in U.S. policies  have been achieved.

Awareness of Canada’s positions has increased.

The majority of missions in the U.S. and Mexico reported that awareness of Canada’s positions has increased in their respective regions since 2013. Some missions indicated that an equally important goal they are achieving is the maintenance of awareness. This is especially true in regions where target audiences include political staff where there is a high degree of turnover.

Willingness by audience to support Canadian interests has increased.

Missions have indicated there has been an increase in support for Canadian interests. FPDS officers reported their key stakeholders are not only well aware of their positions but are also advocating for Canada’s interests. For example, one mission reported the conversation in the region changed from whether there was a need for hydroelectricity, to how much hydroelectricity is needed.

Tangible changes in U.S. policies are being achieved.

Missions reported that they were able to change or prevent key U.S. policies that challenge Canadian interests through the use of NAPP funds. Key examples mentioned were the repeal of the Country of Origin Labelling (COOL) provisions in December 2015 that restricted imports of Canadian meat products; the legislation of the Act Relative to Energy Diversity in Massachusetts (August 2016) authorizing the import and procurement of large-scale long-term Canadian hydropower; and convincing the U.S. Environmental Protection Agency for the definition of “clean energy” in the Clean Power Plan’s to remain broad enough to include Canadian Hydro.

Sources: Interviews with FPDS officer at missions.

Finding 4. NAPP advocacy priorities are not well aligned with the priorities for Canada in Mexico creating challenges for the advancement of Canadian interests in Mexico.

Evaluation question 1b: To what extent has the NAPP contributed to advancing Canadian interests through advocacy efforts?

The NAPP advocacy priorities of competitiveness, security, and energy/environment are well aligned with the advocacy priorities for Canada in the U.S. but not in Mexico. The context in which FPDS officers operate in Mexico is very different from that of FPDS officers in the U.S.. Mexico faces many challenges related to governance which continue to hamper its socio-economic development.

The most significant and pressing advocacy initiatives in Mexico have to do with addressing these governance challenges through initiatives that encourage better governance, improved rule of law, respect for human rights and increased transparency. These initiatives are challenging to fit into the existing NAPP priorities and are therefore not often funded through the NAPP. This is significant, given that the NAPP is Mexico’s largest source of funding for advocacy initiatives and sometimes only source of funding (in FY2015/16, NAPP was the only source of funds for advocacy initiatives in Mexico).

The misalignment in priorities and resulting lack of funding for pressing advocacy initiatives in Mexico, creates challenges in advancing Canadian interests in Mexico.

In 2015/16, the NAPP was Mexico’s only source of advocacy funding and received  less than each mission in the U.S. except for MIAMI (Strategia 2015/16).

Sources: Interviews with FPDS officer at missions, Strategia 2015/16, and program documents.

Finding 5: The NAPP is promoting International Business Development by providing a significant source of program funding to TCS officers leading to Opportunities Pursued and Economic Outcomes Facilitated.

Evaluation question 1c: To what extent has the NAPP  contributed to promoting international business development?

NAPP provides a significant amount of funding available to the TCS at North American missions. NAPP funds make up 40% (average between 2013-2016) of total funding for IBD initiatives in North America (or $1.1 million) and has contributed to over 63% of all IBD initiatives in North America (FY2015/16, base and pool-funded initiatives).

Staff at missions and NAPP Partners both indicated the NAPP has enabled many successes through the provision of funding and many TCS officers noted that the NAPP is their largest and most important source of program funding . The KPIs below demonstrate the successes achieved in North America over the past three years, in part with NAPP funding.

Some highlights of successes include:

• Game Developers Conference (GDC) in Palo Alto, California: PALTO, in collaboration with EDC and CED-Q, organized a program of networking activities at the annual GDC to enhance the conference experience for participating Canadian firms. At the conference the TCS introduced Double Stallion Games, an independent studio producing its own games and providing services to other developers, to the Atlanta-based Cartoon Network, a major producer of games and TV shows for kids. Follow-up negotiations between the two companies led to a partnership where the two developed OK K.O.! Lakewood Plaza Turbo!,  which launched a flagship new cartoon series in February 2016.
• BIO International Convention in San Diego: SNDGO, ISED, and EDC collaborated to provide funds to a Canadian biotechnology company, Neurodyn, for mentoring services from a pharma scout to improve the investment pitch, executive summary, and strategic approach to investors and technology partners. In addition, the company received upgraded conference passes to the BIO International Convention in San Diego in 2014, allowing the company to fully participate in the partnering program. These services helped the company complete a $1 million funding round from existing shareholders and a range of investors in both Canada and the U.S..    
• Canadian Technology Accelerator (CTA) in Boston: Since its inception in 2013, the CTA has hosted over 70 Canadian companies for four-month residencies at the Cambridge Innovation Centre. Participant companies were small to mid-sized Canadian companies working in life sciences, information and communication technologies, robotics and security technologies, and emerging food-product sectors. Following their residencies, participants reported $55 million in raised capital and a $23 million increase in revenue.

Sources: Interviews with TCS officers , ICCB-Q4 TCS Performance Report and YTD (2013/14, 2014/15, 2015/16), the NAPP Annual Report 2015/16, and IBD funding for 3 years.

Finding 6: The Temporary Duty Program is deeply valued by all stakeholder groups. It fosters understanding and collaboration, enhances knowledge, skills, and networks, and increases the TCS’ and FPDS’ capacity to deliver projects in North America.

Evaluation Question 2: Is the TD and training Program achieving its objectives?

TD participants, NAPP Partners, and FPDS and TCS officers at missions all deeply value the TD Program. The TD Program is the only program of its kind in the department and many employees are taking advantage of the opportunities it provides. Over the past three years (2013-2016), 56 employees have participated in the NAPP TD Program with the majority going to SFRAN, LNGLS, DALAS, and MNPLS.

The majority of (65%) TD participants are GAC employees (2013-2016).

Through the TD Program, many outcomes have been achieved (in order mentioned most often to least from missions):

• The TD Program’s greatest value is its ability to foster an increased understanding of the work undertaken by missions. The majority of TD participants indicated they had a better understanding of functions at post, increased awareness of the preparation required for high level visits, and enhanced knowledge of the activities GAC undertakes to support business opportunities for Canadian companies abroad.
• The TD Program has fostered collaboration and a coordinated and integrated approach. The Program enabled greater collaboration between headquarters and missions, and OGDs and missions through more efficient coordination of projects, greater communication, and easier file-sharing, among others.
• Through the TD Program, participants were able to broaden their contacts and networks within their respective fields. 
• Through the TD, participants increased their skillset and gained a valuable professional development experience.
• Finally, the TD Program provided additional human resources to missions enabling them to carry-out higher quality initiatives than they would have otherwise been able to. 

Sources: interviews with TD and training participants, interviews with FPDS and TCS officers at missions, and program documents

Finding 7: The LES Professional Development and Training programs are achieving its objectives and is valued by missions and NAPP Partners.

Evaluation Question 2: Is the TD and training Program achieving its objectives?

The Locally Engaged Staff (LES) Professional Development Program

The LES Professional Development Program provides opportunities for LES to enhance collaboration with NAPP Partners in Ottawa and develop a more robust understanding of how Global Affairs Canada Headquarters (HQ) works.

In 2015/16, 12 LES travelled  to Ottawa to meet and engage with their Canadian counterparts. They participated in meetings, events and various formal and informal training activities as part of the LES Development Program.

FPDS and TCS officers at missions indicated that training opportunities through NAPP were valuable and appreciate the Program for the opportunities it provides locally engaged staff (LES) to:

• Learn and understand how HQ works,
• Meet with OGDs and understand how OGDs work,
• Meet Canadian companies and understand their capacities

Training Program

The Training Program provides NAPP Partners with access to training opportunities provided mostly through GAC’s Canada Foreign Service Institute (CFSI).

NAPP Partners found the trainings they attended to be of high quality and gained knowledge on the U.S. economy and Canada-U.S. relations among other areas.

In 2015/16, 20 employees of three NAPP Partners participated in courses offered by CFSI or in specialized training offered by GAC. The courses offered were:

• Social Entrepreneurship: New and Emerging Approaches for International Commerce, Policy and Development (CFSI)
• Introduction to Canada-U.S. Relations (CFSI)
• Canadian Economy Primer (CFSI)
• Training on the Global Economy and Global Economic Rebalancing (CFSI)
• Advocacy (specialized course)
• International Business Development (specialized course)
• Pre Posting Training (specialized course)

More CED-Q employees participated than employees from other Partners.

Sources: the NAPP Annual Report 2015/16, NAPP Training Courses 2015/15, interviews with TD and training participants, FPDS and TCS officers at missions, and NAPP Partners.

Finding 8: The NAPP is a well-designed and well-managed program with effective governance structures that support achievement of results through sound project planning and implementation.

Evaluation Question 3 and 4: To what extent is the governance structure and management of the NAPP supporting the achievement of results?

Program stakeholders are satisfied with the governance structure of the Program and with the leadership of GAC. Overall, Program stakeholders are satisfied with the governance structure of the NAPP and expressed the view that it enables the achievement of results. Specifically, the Director General’s committee was deemed to be an effective body to share information, set priorities and discuss proposals. This function is complemented by the role of the interdepartmental working group, which is well-attended by NAPP Partners. There was also consensus that GAC’s leadership of the Program was strong and responsive to the needs of the Partners and missions. Guidance and feedback from the NAPP CU, NNC and NNB was noted as timely, thoughtful and of good quality.

NAPP’s design allows the TCS and FPDS to implement initiatives that are considered to be high impact.

NAPP funding is provided on both a competitive and non-competitive basis. While base funding, which is allocated on an annual basis, provides a predictable and consistent source of funding to implement initiatives, the pooled funding mechanism encourages missions and Partners to design highly effective projects and leverage each other's resources in their execution.  However, it was noted that the workload associated with developing a proposal for pooled funding is high compared to the amount of money disbursed for the implementation of projects.

NAPP was also consistently mentioned as one of the most flexible sources of funding available to the North American network. Stakeholders stated that priorities and requirements are clear, but there is flexibility to make adjustments to proposed initiatives throughout the year. The ability of missions and Partners to include hospitality and labour costs in projects was also mentioned as a significant advantage of the NAPP as it enables staff to attract their target audience to an event more easily without supplementing from operational budgets.

The management of the NAPP is enabling sound project planning and implementation.

In 2015, the Program created a discreet Fund  for  its programming, which transfers money directly to missions in USD on an annual basis. This decision has improved the ability of missions to plan and implement initiatives within budget, as they no longer need to manage currency fluctuations. Moreover, the Program’s planning and funding allocation cycle enables them to disburse funding on the first day of each fiscal year (April 1st), which allows missions and Partners to move forward with their projects on a timely basis.

Finding 9: Performance measurement systems exist for two of the NAPP’s objectives but not for the objective of fostering an integrated approach or for measuring the contributory effect of Partner’s use of retained funding towards the effectiveness of the NAPP, leaving gaps in knowledge of the totality of results achieved.

Evaluation Question 5: How does the NAPP track its performance? To what degree is performance information enabling decision-making?

The NAPP CU develops a NAPP report each year highlighting success stories from missions and NAPP Partners as well as metrics related to outputs. However, the performance of NAPP activities is not centrally managed by the NAPP CU. Rather, the performance of the objectives of promoting international commerce and advocating for Canadian interests is managed by  the North America Commercial Programs Division (NNC) and the North American Advocacy Division (NNB) who also track the performance of initiatives from other funding sources. This allows a broad picture of results to emerge for North America rather than for a single funding program. 

However, there exists no performance tracking or measurement system for the third objective of the NAPP, fostering a coordinated and integrated approach. Furthermore, NAPP Partner retained funds are also are not being systematically tracked by the Program, but for a narrative of how the funds were used in the NAPP annual report.

These gaps in performance measurement present lost opportunities for program improvement and marketing of achievements. Opportunities exist for increased efforts to develop, refine, and use performance information to support decisions on program improvement.

Objective 1: International Business Development

NNC tracks the performance of all IBD initiatives in North America through TRIO 2, the Client Relation Management System.  NNC regularly takes stock of initiatives in order to inform decision-making. Results are looked at from a mission perspective, rather than by individual funding source. This approach of not tracking performance by funding source is fitting for IBD initiatives given that:

• The TCS has a robust and well-established system that tracks the same IBD objectives that the NAPP aims to achieve.
• This approach allows a broad picture of results to emerge for a region (e.g., North America) rather than for a single funding program.

Objective 2: Advocating for Canadian interests

Results from advocacy initiatives for North America as a whole, are tracked by NNB in Strategia, GAC’s Planning and Reporting Tool. Strategia contains qualitative narratives of results achieved and quantitative indicators developed by staff at headquarters and the FPDS network. Similar, to IBD initiatives, this approach is fitting for the same reasons: it allows for a broad picture of results to emerge for a region, rather than a funding program.

However, although significant efforts have been made in recent years to develop clear and measurable indicators for advocacy activities, the indicators are not being used by NNB to inform or support decisions on which initiatives to fund. In addition challenges remain capturing the actual results of advocacy initiatives in Strategia for a the following reasons:

• Information in Strategia is incomplete
• FPDS officers are not entering data into Strategia consistently.
• Advocacy indicators are collecting mostly output data rather than outcome or results (e.g., # of initiatives gathering intelligence, # of initiatives engaging audiences, etc).

Objective 3: Fostering a coordinated and integrated approach

Unlike the NAPP’s first two objectives, the performance of the third objective of fostering a coordinated and integrated approach, is not systematically tracked or monitored by the NAPP CU nor any other divisions. As such, there is no data to support decisions about this objective or take stock of what has been achieved to date.

Partner retained funding

NAPP Partners also commit retained funds to the NAPP.

While the NAPP CU collects narratives of how each Partner spent their retained funds for the annual NAPP report, there is no method of systematically tracking activities carried out with these funds, let alone the outcomes they have achieved. Given that retained funds make up a significant amount of the program’s budget, $800,000, or 20%, a large gap in knowledge exists in terms of what the Partners have achieved with the funding.

Conclusion

The NAPP is achieving its objectives of fostering a coordinated and integrated approach in North America, advancing Canadian interests, and promoting international business development. The Program’s strengths are its management structure and design as well as the Temporary Duty and Training Programs which are deeply valued by all stakeholder groups and further enable the NAPP in the achievement of its goals.

Challenges for the Program exist in regards to the limited number of Partners, the misalignment of NAPP priorities with the priorities for Canada in Mexico, and the lack of performance measurement systems for the objective of fostering coordinated and integrated approach.

Recommendations and Management Responses

This section outlines the evaluation’s recommendations and the Program management’s response and actions.

Recommendations

The Management Response and Action Plan (MRAP) for the North American Platform Program (NAPP) Partnership’s Evaluation

Recommendation 1

The NAPP should develop a concrete strategy for seeking additional Partners to join the NAPP.

(Supported by Finding #2)

Recommendation 2

The NAPP should clearly define which Canadian advocacy priorities in Mexico it supports.

(Supported by Finding #4)

Recommendation 3

The Program should develop a performance measurement approach for the Program’s objective of fostering a coordinated and integrated approach, as well as for measuring the contributory effect of Partner’s use of retained funding towards the effectiveness of the NAPP.

(Supported by Finding #9)

Recommendation 4

The NAPP, in conjunction with other divisions that oversee advocacy activities, should continue developing and refining advocacy indicators in order to drive program decision-making.

(Supported by Finding #9)