Evaluation of the North American Platform Program (NAPP) Partnership

September 2012

Abbreviations, Acronyms and Symbols

Acknowledgements

The Evaluation Division (ZIE), Office of Audit, Evaluation and Inspection (ZBD/ZID), of the Department of Foreign Affairs and International Trade (DFAIT), would like to the thank the members of the Evaluation Advisory Committee (EAC) who guided the evaluation throughout all its phases, and all those individuals who shared their thoughts on the relevance and performance of the North American Platform Program (NAPP) Partnership. ZIE also wishes to acknowledge the contribution of TNS Global for their part in the design, administration, and analysis of data generated from the mission survey implemented in support of this evaluation and the contributions of Ms Gabrielle Biron-Hudon to the production of this report.

Executive Summary

Introduction

This evaluation of the North American Partnership Program (NAPP) Partnership was conducted by the Evaluation Division (ZIE), Office of Audit, Evaluation and Inspection (ZBD/ZID), of the Department of Foreign Affairs and International Trade (DFAIT), in compliance with a requirement in the NAPP Partnership Governance Framework Agreement (GFA) of 2008 to report on the relevance and performance of the Partnership prior to the end of Fiscal Year (FY) 2012-2013. The target audience for this evaluation is DFAIT's senior management, the NAPP Executive Coordinator, the Assistant Deputy Minister, North America, DFAIT, and the NAPP Partners.

Background

The NAPP Partnership finds its roots in the Enhanced Representation Initiative (ERI) which was established in 2003 with a mandate to engage other government departments (OGDs) in a whole-of-government effort to stimulate investment in Canada, increase trade with the United States (US), promote science and technology (S&T) cooperation and development, reduce trade barriers and generally advocate key Canadian interests and issues deemed vital to a strong Canada-US relationship. With the sunset of ERI in 2008, and in recognition of the continued need for a cohesive whole-of-government approach to promoting Canada's policy and business development interests in the US and Mexico, the NAPP Partnership was formed with three objectives: 1) promote international commerce; 2) advance Canadian interests in North America, and; 3) foster a whole-of-government approach.

Following preliminary outreach with existing and prospective partner departments and agencies, activities to form the new Partnership concluded in 2008 with the signing of the GFA between DFAIT, Agriculture and Agri-Food Canada (AAFC), the National Research Council (NRC), the Atlantic Canada Opportunities Agency (ACOA), Canada Economic Development for Quebec Regions (CED-Q), and Western Economic Diversification Canada (WED). Under the GFA the Partners contribute funding only for programming and coordination. DFAIT contributes $4 million annually; AAFC contributes $1 million: NRC contributes $500,000; and the regional development agencies (ACOA, WED, CED-Q) contribute $750,000.

Program Description

The NAPP Partnership has three core programming components: the business development program, the advocacy program, and the human resource (HR) program. Within the business development program there are two sub-programs or funding sources: International Business Development (IBD) and the NAPP Technology Partnering Initiative (TPI). The former focuses on three overlapping and inter-rated areas (trade, investment and S&T, the latter focuses on increasing the participation of Canadian firms with counterparts in the US and Mexico for the purpose of technology commercialization and investment. Within the advocacy program there are two inter-related components: political advocacy, which includes raising awareness and correcting misconceptions with the view to promoting Canadian interests, and commercial advocacy, which encompasses the promotion of free and open trade, maintain market access, and defending against protectionist measures. The HR program supports business development and advocacy programming through the provision of training to Partners relating to the foregoing as well as funding for the management of Temporary Duty (TD) assignments and long-term assignments to missions.

Governance

The governance structure of the NAPP Partnership is comprised of several inter-departmental committees; an Assistant Deputy Ministers' Committee (ADMC), a Directors' General Committee (DGC), and two subcommittees, including the Standing Committee on Advocacy and Business Development (ABDC) and the Standing Committee on Human Resources (HRC). Under the GFA, facilitation of the Partnership governance mechanism was to be provided by DFAIT through a "Virtual Secretariat" comprised of an Executive Coordinator, an Administrative Assistant, and functional specialists drawn from divisions within DFAIT's North American Branch. In addition to supporting the aforementioned governance bodies, the Virtual Secretariat provides a range of support services including the review of business plans and project proposals, financial management, including transfers to missions and Partners, fund utilization and reallocation, and oversight with respect compliance with Government of Canada (GoC) policies and procedures.

Evaluation Scope and Objectives

Although Partnership programming is delivered through the mission network under the NAPP, the evaluation focussed exclusively on the relevance and performance of the former. The temporal scope of the evaluation covers FY 2008-2009 to 2010-2011. The overall objective of the evaluation was to assess the relevance and performance of the current systems and practices in place to achieve expected outcomes of the NAPP Partnership. The specific objectives of the evaluation were:

• To determine whether the NAPP Partnership in its design and intent continues to be an appropriate and effective mechanism for promoting Canada's advocacy and business development interests, notably trade, investment and S&T in the US and Mexico;
• To determine the extent to which the NAPP Partnership is consistent with and supportive of GoC and Departmental priorities and objectives, as well as the priorities and objectives of Partner departments/agencies;
• To determine the extent to which the NAPP Partnership and its projects contribute to the realization of declared outcomes, namely to promote Canada's participation in international commerce, advance Canadian interests in North America, and foster a whole-of-government approach to the GoC's advocacy and business development objectives in the US and Mexico;
• To determine whether the current systems of governance and resource allocation are optimal or whether there are alternative, more coordinated, effective or economical ways to achieve the same results or better;
• To assess the effectiveness of the virtual secretariat in meeting the needs of the Partnership and to assess the organization, provision, quality and quantity of required services, and accessibility of such by Partners;
• To determine whether the composition of the Partnership in terms of OGD participation is appropriate to give effect to Canada's objectives for North America; and
• To derive lessons learned which could be applied to the strategic focus of Canada's commercial advocacy and business development relations with the US and Mexico.

Evaluation Approach and Methodology

The three programming components of the NAPP Partnership are distinguished by different objectives, priorities, results articulations and modalities of implementation. Accordingly the evaluators calibrated their approach and methodology to these characteristics. For results reporting purposes, the evaluators adopted a statistical sampling approach to the examination of projects implemented during the study period. This resulted in the selection of 170 projects, accounting for over 20% of disbursements, out of approximately 969 business development projects implemented during the study period and the selection of 132 projects, again accounting for over 20% of disbursements, out of the 664 advocacy projects implemented during the study period.

The project samples for both business development and advocacy provided, for statistical inference purposes, a confidence level of 90%, with a margin of error of 10% for the former and 6.4% for the latter. For business development, the evaluators relied heavily on the performance data recorded in mission Commercial/Economic Program (CEP) and Partner reports, supplemented by data retrieved from DFAIT's Client Relationship Management System (TRIO) which records the outputs of activities (contacts, leads, outcalls, service requests) performed by Trade Commissioner Services at missions. For advocacy, the evaluators relied on performance data gleaned from narrative reports pertaining to NAPP Partnership projects.

This aggregate performance data for business development and advocacy was supplemented by information obtained from 6 project case studies - 3 for business development (Greenbuild, Canadian Technology Accelerator, and the International Boston Seafood Show) and 3 for advocacy (All Politics is Local, Rising State Leaders, and the Government of Canada Continental Analysis Research Tool).

For the HR program, the evaluators relied on performance data gleaned from documents (e.g., TD reports), interviews with HR program participants, and results from a Mission Survey, wherein Trade Commissioner Service (TCS) and Foreign Policy and Diplomatic Service (FPDS) managers where asked to comment on the contributions of TDs and long-term assignments to mission business plans, among other things.

Further in support of the evaluation, the evaluators conducted interviews with 93 individuals from the following stakeholders: DFAIT, including representatives from the Regional Offices (ROS), WED, ACOA, CED-Q, including representatives from their respective regional offices, AAFC and NRC. In addition to the foregoing, the evaluators conducted interviews with representatives from 6 non-Partner departments and agencies (Heritage Canada, Industry Canada, Environment Canada, Natural Resources Canada, the Department of National Defense, and the Federal Economic Development Agency of Southern Ontario) as well as representatives from Industry and Trade Associations from across the country.

Key Findings

Relevance

• While Canada's commercial relations with the US and Mexico are mature, Canadian small and medium size enterprises remain challenged in finding market information, contacts and access to capital to enhance exports to these markets, thereby affirming the continued relevance of the kind of support provided to small and medium size enterprises through the NAPP Partnership. Further, access to these markets continues to be encumbered by foreign government policies and regulations that are unfavourable to Canadian exporters, again affirming the relevance of NAPP Partnership efforts in advocacy.
• As a horizontal initiative, the NAPP Partnership marshals the resources (human and financial) of its members, which is inclusive of not only other government departments, but regional development agencies as well, thereby aligning its activities with the interests of the country as a whole, which is a unique feature of the Partnership, and bringing greater coherence to Canada's commercial engagement with the US and Mexico. This said the Partnership could benefit from an expanded membership, both among federal government departments and regional development agencies.
• While the three programming components of the NAPP Partnership (business development, advocacy and human resources) are regarded as appropriate and complementary, few Partners, apart from DFAIT, have been significantly engaged on the advocacy file, which has emerged an area of concern given that a third of NAPP Partnership resources are allocated in support of advocacy programming. That existing Partner interest in the NAPP Partnership primarily relates to business development provides further evidence of need to diversify its membership if allocations to advocacy are to be sustained.
• The pooled funding model, which is a fundamental design feature of the NAPP Partnership, is generally regarded favourably by the Partners for reason that: Grants and Contribution resources can be converted to Operational and Maintenance resources; financial resources once pooled become fenced, thereby facilitating planning; and, unspent funds can be returned to the pool for reallocation. This said some stakeholders remarked that the Partnership programming could benefit from greater access to and use of Grants and Contribution resources.
• Other benefits to partnership cited by Partners include access to the missions network and their resources (human and financial), greater collaboration and cooperation among Partners, missions, and regional offices, and the leveraging of resources among the same for optimum results, to name a few. Notwithstanding these benefits, the NAPP Partnership is under some strain owing to the diminution in the quantum of resources available for international business development and advocacy and recognition that Canadian commercial interests and opportunities lie outside North America which likewise require attention.
• Though this evaluation concludes that the NAPP Partnership in design remains relevant as an instrument to advance Canada's commercial interests in North America, the model may not be easily replicated in other international markets owing to factors specific to Canada's commercial relationship with the US and Mexico, namely the existence of a plurality of sectors of interest to Canadian businesses, evidence of a pan-Canadian stake in the market, and the existence of extensive mission network to support programming.

Achievement of Expected Outcomes

• During the study period, the NAPP Partnership had neither a common performance measurement framework to report on results or systems in place to generate results data that could be attributable to the NAPP Partnership. Notwithstanding this deficit, the evaluators were able to make some inferences on the performance of the NAPP Partnership based on a review of the documentary records, including CEP and Partner reports, TRIO data, and the narrative reports pertaining to advocacy projects.
• In the domain of business development, the evaluation found that NAPP Partnership resources had been instrumental in facilitating the participation of thousands of Canadian businesses from a wide range of sectors and regions of the country in events which allowed the same to show case their respective capacities with business counterparts in the US and Mexico. Meetings facilitated by the NAPP Partnership between Canadian businesses and foreign buyers generated, in excess of program disbursements, millions of dollars in confirmed and potential sales, as well as commitments by the same to explore opportunities for cooperation.
• While advocacy outcomes directly attributable to the NAPP Partnership are difficult, given the nature of advocacy, the review of the records confirmed that advocacy programming over the study period had reached thousands of influencers and decision makers and had some impact on the same in terms of shaping attitudes that were aligned with Canadian commercial interests in the US and Mexico.
• Partners conveyed a high degree of satisfaction with the human resource program. Participants in the training program remarked on the valuable theoretical and practical knowledge received as well as contacts needed to conduct their activities both in Canada and abroad. Partners and participants in the Temporary Duty subprogram also remarked on the benefits of same in terms of gaining better insights into the operations of missions and the markets relating thereto, as well as obtaining hands on experience in business development and advocacy programming. Among the three service offerings made available to Partners through the human resource program (training, temporary duty assignments, and long-term assignments), satisfaction with the long-term assignment subprogram was, however, qualified.

Efficiency and Economy

• While structures and procedures have been created to facilitate Partner engagement in decision making, arguably governance of the Partnership could have benefited from greater involvement of different centres of expertise, including the regional offices, within Partner departments and agencies. Further, and notwithstanding Partner participation in all the governance bodies established under the NAPP Partnership, Partners conveyed the view that decision making, both at the strategic and operational levels, has not been wholly consensus based. This observation was most pronounced with respect to advocacy programming, but also cited, albeit to a lesser extent, in reference to business development and human resource programming.
• Strategic guidance offered by the NAPP Partnership governance bodies was found to be weak, providing sub-optimal direction as to the relative importance of different priorities and how programming should be designed in support thereof. While Partners and stakeholders conveyed qualified satisfaction with the existing strategic priorities and guidance relating thereto, it is surmised that this satisfaction is likely borne from the very generality of these strategic priorities which leaves them open to interpretation.
• Roles, responsibilities and accountabilities of the governance bodies under the NAPP Partnership have been fluid and never formally documented, which has generated some uncertainties with respect to the same. Though the service offerings of the Virtual Secretariat were documented, challenges in delivering on several of these services created uncertainty as to what the Virtual Secretariat was actually tasked with performing. Additionally, the Virtual Secretariat did generate confused accountabilities within DFAIT by making managers accountable for personnel over whom they had no direct authority and conversely personnel accountable to managers not directly responsible for the NAPP Partnership.
• Partners and missions reported that the current planning systems work well, though Partners conveyed the view that access to the CEP Plans would facilitate better business development program planning. Coordination at the program level was similarly reported to be good, owing to the system of peer review for business development proposals, though less so for advocacy, owing to the fact that peer review on project proposals is only engaged if the project is over $50 thousand, which is rare. Further, there appears to be little interface between the groups charged with administering the business development and advocacy programs, both at DFAIT headquarters and at missions.
• Financial comptrollership over NAPP Partnership resources was found to be good, evidenced by the successful reallocation of surplus resources and the record of modest year end lapses. While Partners conveyed satisfaction with the quality and timeliness of financial reporting from the Virtual Secretariat, the dissemination among the Partners of program level information was found to be weak, owing to the absence of a functional and robust communications service/program, the outcome of insufficient resources being allocated to this function and the failure of the Partners to consistently provide the Virtual Secretariat with program data in a prescribed format.
• While the technical infrastructure is in place to support sound information management, the realization thereof has been encumbered by data being housed in different databases that is not easily accessible to those who need to know and in a format to support effective decision making - outcomes that in part can be attributed to the absence of Standard Operating Procedures (SOPs).
• Though the evaluation makes note of several improvements to instruments used to track performance, the NAPP Partnership was for the duration of the study period bereft of a common performance measurement framework as well as systems to collect and collate performance data, thereby handicapping the Program's ability to demonstrate results.
• Horizontal initiatives involving a plurality of federal government departments and agencies always engender governance challenges that often translate into higher administrative costs. While administrative overhead for the NAPP Partnership was high for much of the study period, administrative charges have declined and are now proportional to the overall value of the Program. Additional efficiencies may be gained from streamlining and consolidating different funding sources relating to business development and providing training in the regions.
• Real efficiencies, however, are to be found in declared benefits of partnership i.e., the leveraging of resources (financial and human) of Partners, cooperation and collaboration between the same, and institutional capacity building - benefits that would be lost were the Partnership not to exist. Further, when the return on investments is considered, in terms of increased exports, the NAPP Partnership is good value for money.

Conclusion and Recommendations

This evaluation found that the concept of partnership within the North American context has merit. Additionally, this evaluation found the NAPP Partnership in design, including the pooled funding model, works and contributes to the achievement of results, evidenced by the increase in exports to the US and Mexico, progress in addressing several advocacy issues of import to Canada, and increased capacity of the Partners through their participation in the HR program. Further, the evaluation found that, despite challenges in the administration of the Program, most notably in the area of performance monitoring and results reporting, the Partnership is reasonably well managed and at a cost not out of proportion with the value of the Program.

Based on the foregoing observations and analysis, the evaluation recommends the following to improve the relevance and performance of the Partnership.

Recommendation 1:
That consultation with Partners and prospective partners commence as soon as possible to renew a North American Platform Program Partnership.

Recommendation 2:
That a renewed NAPP Partnership establish clear roles, responsibilities and accountabilities of the Partners and governance bodies in order to enhance transparency, accountability and performance.

Recommendation 3:
That the Partnership establish a Secretariat to meet service requirements in support of a fully effective Partnership in consideration of costs and resource requirements.

Recommendation 4:
That the Partnership and Partners commit to improving the existing performance measurement systems to ensure that the collection, consolidation and analysis of NAPP related performance data is used to support decision making by management.

1.0 Introduction

The Evaluation Division (ZIE) at the Department of Foreign Affairs and International Trade (DFAIT) is housed within the Office of Audit, Evaluation and Inspection (ZID) and is mandated by the Treasury Board Secretariat (TBS) through its 2009 Evaluation Policy to conduct evaluations of all Direct Program Spending of the Department (including Grants & Contributions). ZIE reports Quarterly to the Departmental Evaluation Committee (DEC) which is chaired by the Deputy Ministers (DMs). The North American Platform Program (NAPP) Partnership was identified in the ZIE evaluation plan as a candidate for evaluation for fiscal year (FY) 2011/12 on the basis of the Partnership's roots in the Global Commerce Strategy (GCS) and requirement, reflected in the NAPP Partnership Governance Framework Agreement (GFA), for an evaluation to be undertaken prior to the end of FY 2012-2013. The target audience for the evaluation is DFAIT's senior management, the NAPP Executive Coordinator, the Assistant Deputy Minister (ADM) North America (GNM) and the NAPP Partners.

2.0 Background and Context

2.1 Program Context

Canada's international trade and investment interests have historically and remain today primarily tied to the North America market. In the 2002 Speech from the Throne, the Government of Canada (GoC) alluded to the need for a more sophisticated approach to increasing Canadian representation in the United States (US) to improve advocacy of Canadian interests in the country. In this context, DFAIT established the Enhanced Representation Initiative (ERI).The primary objective of the ERI was to address the GoC's priorities such as: stimulating investment in Canada, increasing trade with the US, promoting science and technology (S&T) cooperation and development, reducing trade barriers and generally advocating key Canadian interests and issues that were considered to be vital to a strong Canada-US relationship (borders, security, agriculture, trade, energy and the environment). The ERI targeted this objective through broader representation throughout the US, particularly in the West, South and Southwest.

To this end, ERI was provided a budget of $118.2 million over five years, beginning in FY 2003/04 to fund both increased representation and programming. ERI Partners provided approximately 50%, the balance being provided through the fiscal framework. Contributions generally increased over the life of the ERI, peaking in 2005-2006 and 2006-2007 at $31 million. Program funding was directly correlated to this, reaching close to $10 million in 2006-2007. Although total contributions were reduced to $28 million in 2007-2008, funding was supplemented by a re-profile of and a carry-forward of funding from the previous years which resulted in a not to be repeated year high of $9.6 million (2007-2008).

Although the 2007 GCS provided $20 million for maintaining the mission network following the sun-setting of ERI in 2008, there remained concern for a potential loss of established political relations and access to its largest sources of Foreign Direct Investment (FDI), venture capital, as well as the world's largest repository of knowledge-intensive industries. This concern was exacerbated as other countries with similar ambitions to Canada were enhancing their presence in the US at that time. The GCS therefore included a commitment by DFAIT to form a horizontal initiative building on the ERI and focussing on continued advocacy and business development activities. The NAPP Partnership was consequently established in recognition of the following factors:

• The growing integration of the North American economic space
• The breadth of Canadian interests in the US and Mexico, and the importance of those interests to Canadian economic competitiveness and prosperity
• Ongoing Canadian vulnerability to US domestic politics and actions
• The requirement for Canada to develop proactive advocacy stances tailored to win the support of Americans who play key roles in influencing regional and national decisions affecting Canada
• The need for Canada to address regional diversity and interests both in Canada and in North America (particularly the US) through its advocacy and business development activities
• The necessity of adopting an integrated approach to advocacy and business development
• The desire of the GoC to coordinate policy development and program delivery pertaining to the US and Mexico
• The proven benefits of working in partnership, as shown by the success of the ERI

2.2 Partnership Strategic Objectives and Priorities

Pursuant to the NAPP Partnership Governance Framework Agreement (GFA) (August 2008) and elaborated on in the NAPP Results-based Management and Accountability Framework (RMAF) (January 2009) Strategic Objectives of the Partnership are to achieve synergies through collaboration and resource-sharing among the Partners in order to:

Strategic Objective #1: Promote International Commerce

Increase opportunities for business through enhanced business development activities, including trade and investment promotion and the advancement of Canada's Science and Technology (S&T) and innovation agenda in the US and Mexico. More specifically, NAPP Strategic Priorities for 2008/2009 were as follows:

• Exploit value chains
• Promote Gateways
• Create access to research, technology and investment for Canadian firms and institutions
• Provide integrated International Business Development (IBD) support to Priority sectors.

Priority sectors identified in 2008-2009 were:

• Aerospace and defense
• Agriculture and food
• Bio-industries and life sciences
• Environmental technologies
• Information and communications technologies
• Other, including:
  • Mining and energy
  • Building products
  • Cultural Industries
  • Automotive
  • Advanced manufacturing and materials
  • Oceans/Marine

The priority sectors are reviewed and updated annually by the Partners.

Strategic Objective #2: Advance Canadian interests in North America

Effectively advance understanding of Canadian capabilities and perspectives in the US and Mexico and to seek changes in those countries' legislative/regulatory initiatives to protect and advance Canadian interests in various aspects of the bilateral relationships, including economic competiveness and market access. Priority areas under this objective include:

• Decrease the "thickening of the border" (costs, barriers to people, goods and services), while promoting security investments
• Correct misperceptions (e.g., energy from Canadian oil sands, border security, impact from Canadian aquaculture, etc.)
• Increase market access (e.g., Country of Origin Labeling, International Traffic in Arms Regulations, Buy American provisions, etc.)
• Create awareness of Canada, its fiscal strengths, its reliability as a security partner, its economic capabilities, and the integrated nature of Canada-US markets (e.g., auto sector, energy, Gateways and Corridors)

Strategic Objective #3: Foster a whole-of-government approach

Better coordinate policy discussions and strategic direction in the Partnership (and within its Partners) and other government departments (OGDs), leading to more effective advocacy and business development programming in the US and Mexico within the context of the GoC's strategic priorities. More specifically,

• Improve communication and coordination among the Partners
• Improve communication and collaboration with other government departments
• Ensure the North American representational network meets NAPP and OGD needs
• Create understanding of Canada-US, Canada-Mexico and trilateral issues and the use of advocacy.^{Footnote 1}

These Strategic Objectives and Priorities were intended to provide direction to the NAPP Partners and are reviewed and updated annually by the same. These Strategic Objectives and Priorities, however, have remained relatively consistent throughout the study period.

2.3 NAPP Outcomes

According to the NAPP RMAF (2009), the NAPP Partnership was intended to achieve the following Short-Term and Medium-Term Outcomes:

Advocacy

Short-term:

• Increased engagement and coordination of Canadian stakeholders (departments, agencies, provincial territorial governments, institutions and businesses) in advocacy initiatives.
• Raised awareness by Americans of Canada and its importance to the US

Medium-term:

• Canadian interests not adversely affected by US legislation/policy/regulations/industry codes
• Improved settlement of Canada-US issues.

Business Development

Short-term:

• Increased Canadian capacity/diversity to export
• Raised awareness in the US of investment opportunities (DFI, Venture capital, technology commercialization) in Canada
• Raised awareness in the US of Canadian technological capabilities evidenced by increased Canada-US technology partnerships

Medium-term:

• Increased export development evidenced by an increase in the number and type of Canadian firms exporting to the US.
• Increased investment development evidenced by the value of inbound foreign direct investment (FDI) from the US to Canada
• Increased commercialization of S&T evidenced by the number of collaborations (partnerships, joint ventures and strategic alliances)

These short-term and medium-term outcomes are expected to contribute to: an "improved flow of people, goods and services across the Canadian-US border and, as a consequence, improve the prosperity and security of Canadians."

It is noted that the NAPP RMAF was prepared by DFAIT to fulfill a central agency requirement respecting the approval of the $20 million allocated to DFAIT annually through the GCS. The RMAF reflects the activities of the NAPP Partnership and expands on these to meet RMAF results reporting requirements. Notwithstanding that the GFA also called for the development of an RMAF, the Partners neither participated in its development nor officially endorsed the same.

2.4 NAPP Overview

2.4.1 NAPP Partners

The NAPP Partnership is a well-established horizontal initiative of GoC departments and agencies that provides a coordinated and integrated approach for promoting Canada's advocacy and business development interests, including trade, investment, and S&T, in the US and Mexico. The NAPP Partnership is led by DFAIT and currently includes five other Partners

• Agriculture and Agri-Food Canada (AAFC)
• Atlantic Canada Opportunities Agency (ACOA)
• Canada Economic Development for Quebec Regions (CEDQ)
• National Research Council (NRC)
• Western Economic Diversification (WED).^{Footnote 2}

As stipulated in the NAPP Partnership GFA, other departments and agencies of the GoC that have advocacy and business development interests in the US may join the NAPP Partnership at any time. The responsibilities and privileges of future Partners will be the same as current Partners and will include contribution of funding for common activities, alignment of their organization's US and Mexico advocacy and business development programming with Partnership strategic priorities, and cooperation with each other and with Canada's posts in the US and Mexico to achieve NAPP Partnership strategic and operational objectives. The Partnership achieves its objectives through the Canadian representational network of missions in the US and Mexico, the network of Regional Offices (ROs) in Canada, and NAPP Partner offices across Canada.^{Footnote 3} As per the GFA, the Partnership is considered to be in effect from April 1, 2008 for a five year term ending March 31, 2013.

2.4.2 Program Description

Under the NAPP, the network of missions in the US established through the ERI has been maintained.^{Footnote 4} It was argued that the NAPP would enable these missions to continue providing services that assist in attaining success in North America by making Canada a partner of choice for international business, negotiating improved access to global markets, capital, technologies and talent; and connecting Canadian businesses with expanding global opportunities. There was no change to the number of offices or staff in the US during the transition from the ERI to the NAPP. However, DFAIT, with the addition of $20 million from the GCS, now covers the entirety of physical infrastructure and staff costs in the ERI network.

The NAPP Partnership provides strategic focus, collaboration and accountability to Canada's relations with the US and Mexico. Although the focus of NAPP programming is primarily on the US, programming in Mexico is also eligible for NAPP Partnership funding. Indeed, the NAPP Partnership began earmarking resources for bilateral initiatives for Mexico in FY 2008/09.The NAPP Partnership supports continental alignment of regulations, standards and policy approaches. Most Partners have specific goals for Mexico, and the NAPP Partnership now plays a role in integrating those organizational goals into a coherent whole-of-government approach.

Through collaboration and the harmonization of their advocacy and business development initiatives and activities, the Partners:

• Participate in strategy setting and program planning for enhanced advocacy and business development efforts in the US and Mexico
• Extend Canada's reach and enhance its profile in the US and Mexico
• Enhance effectiveness of joint initiatives that benefit from resource sharing, collaboration and synergies
• Establish and strengthen ongoing working relationships with specialists and experts at the Canadian posts in the US and Mexico
• Provide direct access to and sharing of information gathered by or generated at Canadian posts in the US and Mexico

The NAPP Partnership has three core programming functions: advocacy, business development, and human resources (HR) programming which are described briefly below:

Advocacy

Advocacy is recognized as having two inter-related components - Political and Commercial Advocacy. Political advocacy includes raising awareness and correcting misconceptions with a view to promoting Canadian interests and advancing the overall bilateral relationship. Commercial advocacy includes promoting free and open trade, maintaining market access and defending against protectionist measures.

Business Development

Business development within the NAPP Partnership focuses on three overlapping and inter-related areas: trade, investment and S&T. Activities in these areas serve to assist Canadian clients and stakeholders in realizing export opportunities to the US and Mexico, in acquiring investment funding from these countries, and in forming collaborations for research, technology commercialization, including the accessing of venture capital funding. Commercial advocacy is also carried out within the business development programming stream.

In addition and complementary to the resources targeted to business development are the resources made available through the NAPP Technology Partnering Initiative (TPI). The objective of the TPI is to advance the participation of Canadian firms with counterparts in the US or Mexico for the purpose of technology commercialization and Investment.

HR Programming

The NAPP Partnership HR program serves to support the advocacy and business development programming needs of the Partnership by providing capable staff to missions and enhancing the capabilities of staff engaged in these activities at mission in the US or in Partner organizations in Canada. The NAPP HR program provides staff and funding for the management of the Temporary Duty (TD) assignments (1 to 4 months) and the multi-year assignment process.

The TD sub-program provides candidates with an opportunity to experience first-hand the day-to-day operations of a mission in the US or Mexico and to jointly contribute to a project or event of mutual interest. Assignments (hereinafter, long-term assignments) are longer-term in duration (4 years) and are intended, in the first instance, to staff an operational position at post, and to provide employees with a more fulsome experience working at a mission.

The HR program also offers candidates training to allow the same to plan and conduct their activities in the US or Mexico in an effective manner. Under this service offering participants are provided with pre-posting training which includes professional development training aligned with the position the employee will fill. Participants are also entitled to receive training offered by the Canadian Foreign Service Institute (CFSI). Targeted development training is also provided to participants in specialized areas (e.g., Buy America/Stimulus Workshop, Agri-Innovation, Social Media). Finally, participants are eligible to attend and participate in the Annual FPDS/STC meeting. The HR program covers all eligible expenses related to training, including fees and, if applicable, travel and per diem costs.

2.5 NAPP Resources

Funds received specifically for the NAPP under the GCS ($20 million annually) are used for managing the facilities and maintaining the offices and positions established under the ERI. This is augmented, in respect to the Honorary Consuls Program, by a $1.2 million contribution from DFAIT - See Table 1: Funding for Mission Network

As provided in the GCS, programming funds of the NAPP Partnership is based on DFAIT's contribution, plus the contributions from other Partners through reallocation of their respective departmental/agency resources. The contributions of the Partners over the study period is summarized in Table 2: Program Funding 2008-09 to 2010-11.

Partner contributions are applied to advocacy, business development and HR programming and the NAPP Secretariat (hereinafter, the Virtual Secretariat). The specific allocation of funds is decided annually through the governance process and on approval of the Assistant Deputy Ministers' Committee (ADMC). Each of the Partners is allocated program funding, in a manner determined by the Director General Committee (DGC), on an annual basis to undertake a broad range of advocacy and business development projects that address the strategic objectives and priorities of the Partnership - See Section 2.6 NAPP Governance.

Throughout the FY, programming and related expenditures are monitored actively by Partners and the Office of the Executive Coordinator (GND) to ensure that funds are used in accordance with approved allocations and to identify potential funding surpluses and opportunities for increasing partner programming. Funding can also be re-allocated at the request of Partners however only upon the approval of the Executive Coordinator to ensure that the best overall interests of the Partnership are upheld. Planned and actual expenditures of NAPP Partnership financial resources, by activity, are presented in Table 3: NAPP Financial Summary 2008-09 to 2010-11.

2.6 NAPP Governance

The NAPP Partnership's responsibilities and privileges, ongoing governance, and program oversight are defined by the NAPP GFA, as signed by the Partners in 2008. Pursuant to the GFA, DFAIT is the lead Partner in the NAPP Partnership, responsible for managing the network of posts and Honorary Consuls in the US. DFAIT provides to the Partnership the services of the Platform in respect to the collaboration and delivery of advocacy and business development programming. DFAIT also reports on the initiative's plans and results through the Department's Report on Plans and Priorities (RPP) and Departmental Performance Report (DPR).

The NAPP is a partnership of equals founded on the values of trust and flexibility, operating by joint decision-making and coordination of Partnership resources. The interdepartmental governance of the NAPP Partnership allocated pooled funding strategically for advocacy/market access and business development initiatives. The governance structure is comprised of an Assistant Deputy Ministers' Committee (ADMC) and a Directors' General Committee (DGC), under which there are two subcommittees: the Standing Committee on Human Resources (HRC) and the Standing Committee on Advocacy and Business Development (ABDC).

Support for governance and Partnership programming is provided by DFAIT through the Executive Coordinator and a Virtual Secretariat - see 2.6.5 for details.

2.6.1 Assistant Deputy Minister Committee (ADMC)

The ADMC is responsible for ensuring that the NAPP Partnership meets the expectations of its Partners in advancing the advocacy and business development interests of the GoC and its provinces/territories. The ADMC is also charged with ensuring that Partner programming is aligned with NAPP Partnership Strategic Objectives and Priorities. More specifically, it has authority to:

• establish direction and review/revise annually, Strategic Priorities for the NAPP Partnership in consideration of the full range of policy issues with respect to Canada, the US and Mexico;
• allocate NAPP Partnership funding to reflect the importance of the Partnership's Strategic Objectives and Annual Priorities and to meet the needs of Partners;
• establish performance measurement and respond to external evaluation of the NAPP Partnership; and
• annually review the performance of the NAPP Partnership in respect to its Strategic objectives and Strategic Priorities in consideration of the NAPP Partnership's Performance Measurement Framework.

2.6.2 Directors' General Committee (DGC)

The DGC is the principal working committee for the NAPP Partnership, responsible for establishing the overall operational budget of the NAPP Partnership and the Secretariat. It has authority to approve major initiatives, programs and projects reviewed and proposed by each of the Standing Committees. Each member is expected to have responsibility for an advocacy or business development function related to the US and Mexico in their respective department/agency. The chair of the DGC is the Executive Coordinator of the NAPP in order to ensure that it is being managed in an unbiased manner and enables proper time commitment to the development and implementation of the NAPP Partnership. The DGC reports directly through its Chair to the ADMC.

More specifically, the DGC has authority to:

• determine the size and composition of the NAPP Partnership Secretariat and provide it with general direction;
• direct the Secretariat to perform functions related to the DGC specifically and the best interests of the NAPP Partnership in general
• provide recommendations to the ADMC on issues related to advocacy and business development and on issues related to NAPP Partnership governance; and
• consider for approval all advocacy projects greater than $100,000 and any project/proposal on the advice of its Standing Committees.

Furthermore, the DGC is responsible for all aspects of NAPP Partnership financial management, including:

• the establishment of the overall budget for the NAPP Partnership, both annually and on a multi-year basis;
• the determination and allocation of funds to NAPP Partnership activities (advocacy, business development, HR and communications) as well as the resulting allocations to Partners and missions;
• decision-making related to the allocation of funds from year to year, and
• decision making, in the absence of specific policy direction or direction from the ADMC, related to the utilization of surplus funds.

2.6.3 Standing Committee on Human Resources (HRC)

The HRC provides coordination and oversight to the implementation of human resources staffing plans and TD placements. It also assists in the identification of training needs, the establishment of training initiatives, the identification of required staffing incentives for the US and Mexico^{Footnote 5} and development of initiatives to address these, as well as assisting in the annual staff selection process for positions in the US.^{Footnote 6}

2.6.4 Standing Committee on Advocacy and Business Development (ABDC)

The ABDC provides advice on strategic direction and funding for advocacy, business development and technology/innovation. The ABDC is authorized to approve projects between $50,000 to $100,000. This committee also is responsible for advancing communication and collaboration within the Partnership and it governance.

2.6.5 NAPP Secretariat/Executive Coordinator

Facilitation of the Partnership governance mechanism is provided by DFAIT through an Executive Coordinator with an Administrative Assistant and designated functional specialists from DFAIT program and corporate service functions. The Executive Coordinator, Administrative Assistant and functional specialists operate as a virtual Secretariat to meet the needs of the Partnership. The Executive Coordinator chairs the DGC and facilitates governance of the Partnership, promotes the principles of the Partnership and ensures that programming is aligned with objectives. The Virtual Secretariat also performs the following functions:

• supports the Partnership and the functioning of the governance structure, including the organization of meetings, preparation and circulation of minutes and decisions of meetings, preparation of any required analysis, recommendations and information, and communication of information regarding programming and governance;
• ensures that the actions recommended by any Committee are addressed by appropriate authorities;
• supports the Partnership in the pursuit of its strategic objectives by:
  • Reviewing the Partnership's operational environment, by facilitating planning and visioning exercises and by developing a supportive funding allocation process
  • Working with Partners to structure activities to meet Partnership objectives
  • Monitoring and reporting Partners' activity expenditure
  • Developing a comprehensive performance measurement framework with the Partners for the Partnership and promoting performance measurement by individual Partners
  • Collecting Partners' performance measurement data, aggregating it and reporting on it
  • Performing analyses of Partnership operations and performance measures and providing recommendations as appropriate
• to ensure that the measures recommended by the committee are implemented by the relevant authorities;
• supports liaison between and among the Partners and their ROs, the US posts, and HQ bureau to ensure that the strategic objectives of the Partnership and its governance are being met;
• supports independent evaluations of the NAPP Partnership by ZIE and ensures that such evaluations are conducted to meet the needs of the Partnership and its governance; and
• plans and manages the Partnership budget and periodically reports on the governance and financial state of the Partnership.

3.0 Evaluation Scope & Objectives

3.1 Scope

The evaluation makes an important distinction between the Partnership and the Platform and Program dimensions of the NAPP as referenced in the GCS. An evaluation of the Platform and Program will be undertaken as part of an evaluation of the GCS, scheduled to take place in FY 2013/14. Accordingly, this evaluation focuses attention on the relevance and performance of the activities undertaken by the Partners with the $7.7 million annual budget in support of declared Partnership objectives. The evaluation does not specifically address results/programming relating to funding provided by the GCS for the NAPP under the Sustaining Canada's North American Advantage (the $20 million for the Platform and the $10 million for new programming with respect to Investment, Global Value Chains [GVC] and Market Access), except to the extent to which these impact on the relevance and performance of the NAPP Partnership. The evaluation covers the period between Fys 2008/09 to 2010/11.

3.2 Objectives

The overall objective of this evaluation is to assess the relevance and performance of the current systems and practices in place to achieve expected outcomes of the NAPP Partnership. The specific objectives of the evaluation are outlined below:

• To determine whether the NAPP Partnership in its design and intent continues to be an appropriate and effective mechanism for promoting Canada's advocacy and business development interests, notably trade, investment and S&T in the US and Mexico;
• To determine the extent to which the NAPP Partnership is consistent with and supportive of GoC and Departmental priorities and objectives, as well as the priorities
• To determine the extent to which the NAPP Partnership and its projects contribute to the realization of declared outcomes, namely to promote Canada's participation in international commerce, advance Canadian interests in North America, and foster a whole-of-government approach to the GoC's advocacy and business development objectives in the US and Mexico;
• To determine whether the current systems of governance and resource allocation are optimal or whether there are alternative, more coordinated, effective or economical ways to achieve the same results or better;
• To assess the effectiveness of the virtual secretariat in meeting the needs of the Partnership and to assess the organization, provision, quality and quantity of required services, and accessibility of such by Partners;
• To determine whether the composition of the Partnership in terms of OGD participation is appropriate to give effect to Canada's objectives for North America; and
• To derive lessons learned which could be applied to the strategic focus of Canada's commercial advocacy and business development relations with the US and Mexico.

4.0 Key Considerations

4.1 Economic Context

In evaluating the performance of trade and investment promotion program it is important to bear in mind the economic context within which the program is implemented, and in the case of the NAPP Partnership the study period happens to coincide with one of the most severe economic down turns in recent memory. Real Gross Domestic Product (GDP) in the US grew by 0.3% in 2008, compared with 1.9% in the preceding year. Mexico registered a 1.2 % increase in GDP over 2007. Despite slumping demand in the US for Canadian exports of goods and services, exports to the US in dollar terms increased by 5.2%. FDI both to and from the US recorded a modest increase in 2008. The value of Canadian merchandise exports to Mexico experienced a significant increase over 2007 (18%), though FDI, both to and from Mexico, remained flat - see Table 4: Canadian Trade and Investment - US and Mexico.

The rise in the value of exports to the US in 2008 was largely due to price increases in energy products. These gains were offset by a severe decline in exports of automotive products, textiles, forestry (wood, pulp, and paper) products, and transportation equipment. FDI, both to and from the US, in forestry, transportation, and consumer goods sectors all experienced declines in 2008, with US FDI in the forestry sector experiencing the most dramatic decline.

In 2009, real GDP in the US contracted by 3.5%, largely due to down turns in non-residential fixed investment, in inventory investment, and in consumer spending. This decline impacted heavily on Mexico, whose GDP fell by 6.5%. The value of Canadian exports of goods and services to the US declined by around 25%. Canadian merchandise exports to Mexico in 2009 declined by nearly 18% over 2008, while the Canadian FDI in Mexico registered a 1.2% decline at the end of 2009. Though US FDI in Canada experienced a modest increase in 2009, Canadian FDI to the US decreased.

Energy products led the downward movement, followed by the export of consumer goods, automotive products, machinery, and agriculture products (beef and pork products). The decline in Canadian FDI to the US was most pronounced in the energy, and service sectors, while US FDI in Canada declined in the energy and machinery sectors.

In 2010 real GDP in the US grew by 2.8% over 2009. Canadian exports of goods and services in turn grew by 8.2%, led by increases in the value of mineral fuels and oils, motor vehicles (first time in 5 years), and precious metals (gold) and stones. US FDI in Canada increased at the end of 2010 by 6.4%, while Canadian FDI in the US continued to decline. Canadian merchandise exports to Mexico also rebounded, registering a 4.3% increase in 2010 over the preceding year. FDI, both to and from Mexico, registered positive figures with outward investment increasing by 1.4% and inward investment by 13% at the end of 2010.

In addition to the foregoing, the evaluators also examined the performance of the Canadian economy in relation to several of the strategic priorities of the Partnership. Support for Global Value Chains (GVC) participation is a case in point. While there is evidence of increased Canadian participation in GVC over the last decade, Canada's level of participation remains well below that of many other Organization of Economic Cooperation and Development (OECD) countries.

Based on data generated by the Vertical Specialization Index (VSI), which serves as a proximate measure of the level GVC participation, Canada's level of GVC participation has declined during the study period.^{Footnote 7} The drivers behind this negative trend are not well understood, calling for more research on the subject, but it is speculated that the results may be due to the rapid appreciation of the Canadian dollar and the ascendency of the resource/extractive industries which typically have fewer intermediate inputs that can be imported. ^{Footnote 8}

Support to Information Communications Technology (ICT) is another priority sector. Though ICT firms are highly export oriented, Canadian ICT exports of goods declined by 7.9% between 2001 and 2010. The US accounts for roughly 64% of all Canadian ICT exports. Notwithstanding the increase in the export of ICT services, Canada has an overall trade deficit in the ICT sector (- $26.6 billion).^{Footnote 9} Venture capital, which is crucial to the ability of ICT companies to innovate and commercialize technologies, also registered a decline during the study period (- 25%).^{Footnote 10}

Data on the use of Canada's gateways and corridors, measured in terms of the volume of traffic, during the study period has also been mixed. For example, but for the port at Prince Rupert, British Columbia, all other major ports in Canada experienced an overall decline in container traffic between 2008 and 2009, though 2010 and the first half of 2011 saw an increase in traffic in the ports of Metro Vancouver and Halifax. Vessel traffic on the St. Lawrence Seaway also registered a decline (-17%) in conformity with a decade long trend.

Rail traffic, which is largely inter-provincial, experienced a decline in 2008, but increased in 2009, 2010 and the first half of 2011. Road traffic, which accounts for over 60% of cross-border trade with the US, declined during the study period in Eastern Canada, but registered a modest increase in traffic in Western Canada. Air traffic likewise experienced a decline in between 2008 and 2010 (-10%) in Eastern Canada, but registered a modest increase in Western Canada (2%), again reflecting the emerging economic strength of Western Canada in the Federation.^{Footnote 11}

4.2 Global Commerce Strategy (GCS)

As provided in section 3.1., the focus of this evaluation is on the relevance and performance of the NAPP Partnership, as distinct from the Platform and the Program. This said the NAPP in all its dimensions obtains its policy coverage and financial authority under the GCS. Consequently, the evaluation examined how the NAPP Partnership relates and contributes to DFAIT's wider efforts to improve Canada's competitiveness and support Canadian firms in their pursuit of opportunities in the North American market. In this respect, the findings of the mid-term evaluation of the GCS, as they pertain to the NAPP Partnership, were considered in assessing the relevance and performance of the NAPP Partnership.

4.3 Economic Action Plan 2012

The GoC is in the process of implementing an economic action plan wherein all federal departments/agencies are required to submit separate plans for reducing expenditures by 5% and 10% by reallocating funds, ending programs, or cutting positions. To the extent that decisions have been made prior to completion of this evaluation of the NAPP Partnership, their impact or potential impact on the NAPP Partnership and its members were examined.

4.4 Other Considerations

In the course of developing the Evaluation Work Plan for this evaluation, preliminary consultations with the Partners were conducted where representatives thereof were invited to identify areas of interest that they wished to see examined in the evaluation. These consultations produced the following list of areas for examination:

• The appropriateness of the NAPP Partnership funding authorities (terms and conditions), particularly as they relate to operations (Vote 1) and grants and contributions (Vote 10).
• The efficiency and effectiveness of the pooled funding model with the view to determining whether alternative models exists which would generate the same benefits to the Partners.
• The efficiency and effectiveness of the current governance structure in reference to priority setting and strategic guidance.
• The efficiency and effectiveness of the Virtual Secretariat in terms of the alignment of services provided with perceived needs.
• The alignment of advocacy and business development efforts with Partner priorities.
• The adequacy of systems in place to support performance monitoring and results reporting.
• The adaptability of the NAPP Partnership model to other markets.

The aforementioned issues were covered in the mission survey and in interviews with respondents and the outcomes are addressed in this evaluation report.

5.0 Strategic Linkages

The NAPP Partnership is strategically interlinked with the North American Advantage Initiative and the GCS. The North American Advantage Initiative is a strategic long-term economic plan designed to improve Canada's economic prosperity and the GCS is a strategic framework aimed at helping Canada compete by: Making Canada a partner of choice for international business; Negotiating improved access to global markets, capital, technologies and talent; Connecting Canadian business and expanding global opportunities. Accordingly, this evaluation of the NAPP Partnership assessed the relevance and performance of the NAPP Partnership in reference to these broader strategic objectives.

6.0 Evaluation Approach and Methodology

6.1 Introduction

The Evaluation took place between November 2011 and July 2012, and was conducted and managed by DFAIT's Evaluation Division. As remarked in section 4.4, preliminary interviews were conducted with senior staff from DFAIT, AAFC, NRC, CEDQ, WED, and ACOA to identify the issues of interest to the Partners (those subjects they would like to see examined in the evaluation), to identify sources of information (people, organizations, and documents) relevant to the evaluation, and to discuss different approaches to data collection, data analysis and reporting. The results of those deliberations, combined with the information received, formed the methodological approach used in the conduct of this evaluation.

6.2 Evaluation Matrix

An Evaluation Matrix was developed based on the evaluation issues and questions contained in the Work Plan as well as on input received from the NAPP Partners. The Evaluation Matrix sets out the evaluation issues, the questions relating thereto, associated performance indicators, data sources, and data collection techniques. This matrix serves as the main tool for designing the data collection instruments and reporting on the evaluation findings.

6.3 Approaches to Reporting on Results

The NAPP Partnership has three main programming components (advocacy, business development, and human resources), each of which call for an approach to performance analysis and reporting calibrated to their specific objectives, priorities, results articulations, and modalities of implementation. The following outlines the approach used for each of these components.

6.3.1 Business Development

Over the study period the NAPP Partnership supported, through the business development component, approximately 969 projects^{Footnote 12} valued at around $19.6 million. Given this large number of projects, the evaluators were challenged with identifying a sample size that is manageable, representative of the activities undertaken by the Partners, inclusive of the missions involved in program delivery, and statistically significant for inference purposes. These factors considered the evaluators determined that a sample size of 170 projects for all three years, representing over 20% of disbursements, would constitute the minimum number of projects for statistical inference purposes, providing a confidence level of 90% with a margin of error of 10%. The number of projects in the sample roughly corresponds to the contributions of the missions and the Partners to the total project population - See Table 5: Business Development Project Sample Overview.

Establishing the total population of projects administered under the business development component of the NAPP Partnership proved to be a challenge as the evaluators were not provided with a comprehensive list of business development projects. Some NAPP Partnership business development projects, for example, are administered exclusively by the Partners themselves and thus not reflected in the mission reports. Accordingly, the evaluators had to review the annual reports of missions and the Partners to come up with an approximation of total project population. This exercise was complicated by the fact that not all the Partners produce an annual report, thereby requiring the evaluators to rely on the most current business plans to determine the quantum of business development projects.

In addition to the foregoing, a single project frequently involves the contributions of several Partners and missions each which report their contributions as discrete projects, thereby inflating the total number of projects and magnifying the risk of double, triple, and quadruple counting. To address this problem, the evaluators had to review the contributions of the Partners to ascertain whether they pertained to one project. Where this was the case, the collective contributions of the Partners and missions were consolidated and counted as one project. Conversely, the evaluators found initiatives that were reported as projects, but which were actually a cluster of projects. In this case, the evaluators had to disaggregate the projects under the declared initiative. This exercise was rendered all the more difficult because the names and descriptions of the projects were variable. In sum, the total project population cited here will not correspond to the total number of projects reported by the Partners.

For results reporting purposes the evaluators first considered grouping the projects under priority sectors, since the Partners generally report against sectors. However, it was determined that the sectors were far too numerous and variable in articulation between the Partners to serve as a practical framework within which to group projects. Moreover, sectors in reports were often framed as "initiatives" which may include non-NAPP Partner projects, thereby rendering it difficult to attribute results to NAPP Partnership. This was particularly pronounced for FY 2010-11.

In light of the foregoing, the evaluators grouped projects in the sample under the four strategic priorities identified for business development, as these priorities remain the only strategic framework common to all Partners that are more or less consistent throughout the study period. Notwithstanding that all project proposals for NAPP partnership funding must demonstrate alignment with one or other of the declared strategic priorities, projects, as stated above, are not grouped under the strategic priorities, nor reporting done against them, but rather against the sectors.

This being the case, it was necessary for the evaluators to review the documents (proposals and or reports) of each project in the sample to ascertain which strategic priority the project is most closely aligned. The exercise was further complicated by the fact that most business development projects declare support for more than one strategic priority, thereby necessitating a judgement call as to which strategic priority the project is most closely aligned. ^{Footnote 13}

Although projects under the business development sample were grouped against the strategic priorities relating thereto for analysis purposes, the results reported herein are set against three objectives: 1) increase the number and variety of Canadian businesses exporting to the US and Mexico; 2) increase awareness of Canadian capacity; and 3) contribution to the achievement of outcomes.

6.3.2 Advocacy

Over the study period the NAPP Partnership has supported, through the advocacy component, approximately 664 projects valued at around $6.7 million. Given this large number of projects, and the limited resources available to conduct this evaluation, the evaluators were challenged with identifying a sample size that is manageable, representative of the activities supported under this component of the NAPP Partnership, inclusive of the missions involved in program delivery, and statistically significant for inference purposes.

These factors considered, a sample size of 132 projects for all three years, representing roughly 22.2% of total disbursements, were selected that constituted the minimum number of projects for statistical inference purposes, providing a confidence level of 90% with a margin of error of 6.5%. These projects have been grouped under each of the strategic priorities by year in a manner roughly proportional to the dollar value allocated to these strategic priorities - See Table 6: Advocacy Project Sample Overview.

For results reporting purposes, the evaluators had expected to rely on the outputs and outcomes documented in the DFAIT's Advocacy Report Card (ARC) and Performance Management Tracking Template (PMTT) systems, which were the primary instruments used during the study period to record performance.^{Footnote 14} The ARC, however, was configured to report on activities and outputs by mission, and not against projects or advocacy priorities, thereby rendering it nearly impossible to attribute information contained therein to the NAPP Partnership. The PMTT was designed to support attribution however it was only operable for about six months, thereby rendering it of limited value to the evaluation. As such, the evaluators had to review the narrative reports pertaining to projects to obtain performance data.

As with business development, although performance data was grouped against the advocacy strategic priorities for analysis purposes, the results reported herein are set against three objectives: 1) increase awareness of Canadian capacity; 2) expand Canada's network of influence, and; 3) contribution to the achievement of outcomes.

6.3.3 Human Resources

Over the study period, the HR component has disbursed approximately $1.1 million to support TDs and long-term assignments and related training. - See Table 7: HR Expenditures.

Training

Each year NAPP partners are invited to take part in professional development training and targeted training - see Table 8: Summary of NAPP-Funded Partner Participation.

During the study period, participation in the training subprogram increased significantly, from a total of 17 participants from Partner departments in FY 2008-2009, to 26 participants in 2009-2010 and reaching a high of 34 participants in 2010 -2011 - see Table 9: Partner Participation in NAPP Funded training.

It should be noted that in FY 2008-2009, the NAPP HR Program also contributed funding for the development of the Venture Capital and the Trade Advocacy courses which are external to the CFSI. It is noted that the increased participation also reflects increased course diversity. The numbers do not include DFAIT employees' participation.

The Program developed a Performance Measurement Framework related to training, assignments, TD assignments and other activities, however it was found to be process oriented, focusing on activities to be undertaken rather than the expected results. Consequently, to assess the benefits and results achieved related to both training and TD assignments, the evaluators relied on interviews with participants who took part in TD assignments and on the post TD assignments reports which highlighted the learning objectives of the TD, the duties performed while at post and the lessons learned from the experience.

Temporary Duty Assignments

Over the study period, the HR program supported 41 TDs - see Table 10: Number of TD assignments facilitated by missions for FY 2008-2009 to 2010-2011. Among the Partners, DFAIT accounts for roughly 36% of the total number of TDs to missions, followed by WED 21%, ACOA 17% and AAFC at 14% respectively, and NRC at 12%. The most frequent destinations have been San Francisco (8) followed by Washington (5) and Chicago 5 candidates.

TDs are requested to prepare and submit a TD Report which documents their activities and accomplishments within two weeks of the completion of their placement. These reports constituted the primary documentary source of evidence for results reporting purposes. For sampling purposes, the evaluators examined 16 of these TD reports. These TD took place at various locations, namely, Washington, San Francisco, Chicago, San Diego, Seattle, Los Angeles and Dallas. This documentary evidence was supplemented by 12 in-person and telephone interviews with candidates who took part in these TD assignments.

Long-Term Assignments

Additionally, the HR program has supported over the last three years the posting of 39 long-term assignments to missions in the US. Among the Partners, DFAIT accounts for roughly 76 % of the total number of assignments to missions followed by AAFC at 16% and ACOA at 8%. NRC, WED and CEDQ have had no assignments over the study period. The vast majority (31 or 80%) of assignments during the study period were to trade positions. Among the few assignments to political positions, all but one (AAFC) were from DFAIT. Of the 19 missions that have received long-term assignments, 7 missions (Chicago [4], Boston [4], Dallas [4], Minneapolis [4], San Francisco [3], Denver [3], and Atlanta [3]) account for over 64% of assignments - see Table 11: Assignments by Partner and Mission.

Given the duration of assignments (4 years), there is no requirement for persons assigned to produce a post- assignment report as do TDs. Reporting is done through regular channels. Performance data was obtained from interviews and the Mission Survey - see section 6.4.1 (b) re Mission Survey.

6.4 Data Collection Methods

6.4.1 Primary Data Collection

a) Key informant interviews

In the course of developing the Work Plan, the evaluators consulted with NAPP Partners to solicit their assistance in identifying organizations, administrative bodies and interlocutors whose knowledge of and experience with the NAPP Partnership could add value to this evaluation. As a result of this consultation, the evaluators undertook an extensive interview process that included in-person and telephone interviews with 93^{Footnote 15} internal and external stakeholders. The interviews were conducted using a structure interview guide that covered the issues of relevance, performance, governance and efficiency and economy.

Interviews were conducted with the following groups:

DFAIT Interlocutors at Headquarters

• Assistant Deputy Minister Americas (GNM)
• North America Program and Operations (GND and GWX)
• North America Commercial Programs (GNC)
• North America Partnership and Operations (GNB)
• North America Policy and Relation Division (GGA)
• Members of the ADM Committee , DGC, HRC and ABDC

DFAIT interlocutors at the Regional Offices

• Vancouver
• Calgary
• Toronto
• Halifax
• Montreal

Representatives from Partner Departments who have responsibility for advocacy, Business development, HR intergovernmental relations or program management function related to the US and Mexico

• Western Economic Development (WED)
• Agriculture and Agri-Food Canada (AAFC)
• Atlantic Canada Opportunities Agency (ACOA)
• Canada Economic Development for Quebec Regions (CED-Q)
• National Research Council (NRC)

Representatives from other Government Department (Non- Partners)

• Heritage Canada (HC)
• Industry Canada (IC)
• Federal Economic Development Agency of Southern Ontario (FEDASO)
• Environment Canada (EC)
• Natural Resources Canada (NRC)
• Department of National Defence (DND)

Representatives from Industry and Trade Associations

• Atlantic Canada
• Quebec
• Alberta
• Manitoba
• British Columbia and
• Saskatchewan

Candidates on Temporary Duty Assignments

• DFAIT
• ACOA
• WED
• CED-Q
• NRC
• AAFC (provided no names for interview)

b) Mission Online Survey

An online survey of missions in the US and Mexico was conducted by an independent contractor with a sample of 40 respondents. The survey was launched on May 22, 2012 and by June 14, a total of 35 survey questionnaire were completed.

• The purpose of the survey was to solicit information from the same on:
• The business development and advocacy activities carried out at mission;
• The value added by the partners to these activities;
• The quality of guidance and support received from HQ;
• The level of coordination between the Political, Foreign Policy and Diplomacy Service (FPDS) and Trade Commissioner Service (TCS) staff complements at missions; and
• The level of coordination between the missions

The sample was composed of those whose primary program responsibility at their mission was:

• International Business Development (IBD) and included 19 Program Managers
• International Business Development (IBD) and included 19 Program Managers

Information from the survey is integrated into this report.

6.4.2 Secondary Data Collection

Document Review

Documentary evidence formed the basis of secondary data collection and served to inform the evaluation's historical record as well as to provide concrete sources of verification as part of triangulation of findings. Documents that were reviewed include:

• Departmental Performance Report (DPRs) and Reports on Plan and Priorities (RPPs);
• Submissions to central agencies (GCS Memoranda to Cabinet [MC] and TBS Submissions);
• Governance Framework Agreement;
• RMAF;
• Business Plans (missions, Partners);
• Decision records of ADMC, DGC, ABDC and HRC;
• Minutes of meetings, memos and briefing notes;
• Descriptive, analytical and statistical reports (e.g., Canada's State of Trade: Trade and Investment Update reports);
• Partner Annual Reports;
• Project proposals;
• HR Project selection and approval documents ;
• Databases (Canadian Commercial Interest List [CCIL], Commercial/Economic Program [CEP] reports, Dashboard, Client Relationship Management System [TRIO] ARC, PMTT, Mission Activity Tracking Template[ MATT]);
• Project and mission level Performance reports;
• Reports of previous studies;
• Report of the NAPP partnership for the years (2008-2009 to 2010-2011)
• NAPP Extranet site content;
• Mission Inspection Reports; and
• NAPP Wiki.

In addition, the evaluators reviewed selected administrative data, including financial documentation detailing FY budgets and expenditures and funds allocation to determine the efficiency and economy with which resources have been utilized. The evaluators also reviewed information from databases reflecting Program activities and/or outcomes.

Case Studies

In addition to the aggregated performance data obtained through the review of project reports in the business development and advocacy samples, the evaluators selected 6 projects (three for advocacy and three for business development) as candidates for case studies. Case studies allowed the evaluators to collect more detailed relevance and performance data on an initiative and to deriving lessons learned therefrom. In selecting initiatives for case studies, the evaluators took into consideration the following factors:

• the dollar value of the initiative;
• the level of participation of different missions;
• the level of alignment with Partner priorities; and
• the longevity of the initiative i.e., multi-year.

These factors considered, combined with the advice received from the Partners the following list of case study initiatives were selected for Advocacy and Business Development:

Advocacy

• All Politics is Local (APIL)
• Government of Canada Continental Analysis Research Tool (GoCCART)
• Rising State Leaders (RSL)

Business Development

• GreenBuild
• International Boston Seafood Show (IBSS)
• Canadian Technology Accelerator (CTA)

7.0 Limitations to Methodology

While the approaches and methodologies used to report on the relevance and performance of the NAPP Partnership are scientifically rigorous, and ambitious given prevailing resource and time constraints, the approaches and methodologies nonetheless engender a number of limitations. These limitations are documented below along with measures to mitigate the risk to the evaluation results.

7.1 Business Development Sample

As described in Section 6.3.1, the absence of a consolidated repository of project information for the business development component of the NAPP Partnership required the evaluators to reconstruct an inventory of business development projects. In doing so, the evaluators had to make a number of judgement calls, often based on incomplete and or inconsistent information, in order to come up with a total project population from which a sample could be derived. As such, the evaluators acknowledge that the inventory of projects created herein, and the project sample derived therefrom, may likewise engender inaccuracies and inconsistencies.

Again, as remarked in Section 6.3.2, the evaluators relied heavily on the information contained in the CEP (DFAIT) and Partner reports for performance data. Regarding the CEP reports, results reporting is done against sectors/initiatives and not projects per say, which increases the risk of conflating the results of non-NAPP Partnership projects with NAPP Partnership projects. This risk was further exacerbated by the fact that, for FY 2008-2009 and 2009-2010, CEP reports only listed initiatives that were undertaken which provided no indication whether NAPP funding was in fact used.

To address this problem, the evaluators attempted to make use of performance data (local contacts, outcalls, leads, service requests) in the Department's Client Relationship Management System (TRIO). This involved identifying the date and mission where a NAPP Partnership project was executed and matching the same with the date performance data was entered in TRIO which provided the basis, albeit a fragile basis, to make an inference that the data in the latter likely related to the former.

7.2 Advocacy Sample

Over the study period the strategic priorities for the Advocacy component of the NAPP partnership have undergone some modifications. This said GNB, DFAIT, has grouped Advocacy projects implemented under the NAPP Partnership over the past three years under the current 5 strategic priorities, and the evaluators have relied on this taxonomy to organize its sample. While the evaluators have made best efforts to ensure that the dollar value of projects selected under the sample correspond roughly to the proportion of project spending per strategic priority for the total population, there are a few exceptions, notably in FY 2009-10 under the strategic priorities of "Increase Market Access" and "Create Awareness of Canada's Economic Footprint." In the former strategic priority the sample is under representative of the dollars committed to this priority for the year while the latter is over representative. This imbalance in the sample selection was mitigated in part through the selection of a case study under "Increase Market Access."

As remarked earlier, the evaluators had originally planned to rely on performance data in the ARC and PMTT, but found neither useful: the former because data in the system could not be attributed to the NAPP Partnership, and the latter because it was only in effect for six months. To address this problem, the evaluators requested access to the narrative reports of all advocacy projects supported by the NAPP Partnership for the study period and then identified the same pertaining to the sample through a word search. While this approach provided the evaluators with access to project data, the narrative reports were variable in detail and were, for the most part, bereft of quantitative data. This data was supplemented with data gleaned from the case studies.

7.3 Human Resource Programming

As remarked earlier, in the absence of a PMF to report on results, the evaluators constructed their own, focusing on the value of the training to participants, the value of the TD and long-term assignments programs to the same, and their respective contributions to mission objectives and the overall objectives of the program. Responses to these issues were obtained through interviews and supplemented with data obtained through the Mission Survey.

8.0 Evaluation Findings

8.1 Relevance

Issue 1: Continued need for the Program

Finding 1:
Despite the rising importance of new and emerging markets, the North American market remains and will continue to remain critical to Canada's economic health.

The US is Canada's largest trading partner, accounting for around 75% of Canada's total exports abroad. While Canadian exports, including those of Canadian Small and Medium Size Enterprises (SMEs), are increasingly shifting to new and emerging markets in Asia, Europe and Latin America, the US market is today and expected to remain for some time to come Canada's main export market, with Mexico likewise remaining among Canada's top 5 trading partners.^{Footnote 16}

Canadian commercial relations with the US are mature and robust. This said rapid changes in the US market, driven by the accelerated pace in the globalization of supply chains serves a warning that the US market is not one to be taken for granted. As remarked earlier, US domestic policies, such as Buy America, combined with the more active engagement of other countries in the US, challenge Canada's standing in the US and highlight its vulnerability. Responding to these challenges call for a proactive approach to commercial engagement to maintain and enhance market share.^{Footnote 17}

Finding 2:
Key to Canada's success will be the ability of small and medium size enterprises to penetrate the North American market.

SMEs constitute the most dynamic sector of the Canadian economy, accounting for 45% of GDP and 75% of net employment growth. Although the number of SMEs engaged in exports is small (around 9%), their contribution to exports is significant, accounting for over 40% in the total value of global exports - 25% for Small enterprises and 18% for Medium size enterprises.^{Footnote 18} Although SME exports to the US and Mexico account for roughly 37% and 38% respectively of total Canadian SME exports globally,^{Footnote 19} success in the US market is reported to be seen by many in the SME sector as a precursor to global expansion.

Finding 3:
NAPP Partnership support to both business development and advocacy are aligned with the needs of Canadian SME exporters.

Challenges in finding market information, the right contacts, financial resources (e.g., venture capital) have been cited as impediments to SME growth in the US. Addressing these challenges is well suited to the business development program.^{Footnote 20} Other impediments to growth include legislation and regulation barriers such as the Food and Safety Modernization Act, Buy-America, Country of Origin Labeling (COOL), cross border security measures, and negative or erroneous perceptions of Canada. These challenges are well suited to advocacy.

Consultations with Canadian industry and trade associations undertaken in support of this evaluation confirmed that the aforementioned challenges are material to their constituents and that government can play a role in mitigating their impacts. As such, government support to the SME sector is still required in the immediate future in order to preserve and strengthen market access and share. Studies indicate that it is Medium size enterprises, as distinct from Small enterprises, that are most likely to be "export ready" and thus best positioned to expand internationally.^{Footnote 21}

Relevance of Program Design

Finding 4:
The whole-of-government approach facilitated by the NAPP Partnership is relevant and key to Canada's success in preserving and enhancing market share.

The NAPP Partnership is designed to facilitate a "whole-of-government" approach to Canada's commercial engagement with the US and Mexico. The whole of government approach is predicated on the assumption that no single department in the Government of the Canada (GoC) possesses the requisite knowledge, expertise, and resources to fully represent and adequately advocate for the commercial interests of Canada in the US and Mexico. This approach is consistent with and supportive of GoC policy and the merits relating thereto now accepted as axiomatic.

Notwithstanding the foregoing, the level of Other Government Department (OGD) engagement under the NAPP Partnership is not considered optimal. All Partners interviewed in the course of this evaluation remarked that the Partnership could benefit from an expanded membership. Typical candidates cited by Partners included, in no particular order, Industry Canada, Natural Resources Canada (NRCan), the Departmental National Defense (DND), the Export Development Corporation (EDC), and the Federal Economic Development Agency of Southern Ontario (FEDASO).

STC and FPDS managers at missions were, through the Mission Survey, asked the same question. Again, the answer was the same - the NAPP Partnership could benefit from greater OGD participation. For STCs, OGDs identified included: Industry Canada, EDC, NRCan, Environment Canada, and FEDASO. For Advocacy, OGDs included: NRCan, Environment Canada, the Canadian Border Security Agency (CBSA), DND, and Public Safety.

Even though the GFA imposes no specific requirement on the Partnership to expand its ranks, the GCS, from which Partnership derives its authority, does make reference to the need for the Partnership to be inclusive of all relevant stakeholders to make good on its whole-of-government mandate, and DFAIT certainly adopted this as a goal. Although considerable efforts were made to reach out and persuade non-Partner OGDs - see Finding 5 - to join the Partnership, little progress has been made in expanding the same.

The most frequently cited obstacle to participation is the financial commitment involved. The second most frequently cited obstacle was the lack of alignment of NAPP Partnership priorities with those of non-Partner OGDs, though arguably were these OGDs to join the Partnership their priorities would, by virtue of membership, become Partnership priorities. Another disincentive to participation cited by Partners is the view that non-NAPP OGDs can partake in and profit from NAPP Partnership sponsored events without incurring the obligations associated with partnership - a practice referred to as "free riding".

With the financial commitment to Partnership being cited as the biggest obstacle to participation, thought was given to an alternative contribution formula wherein the rights and privileges of members would be calibrated to the size of their contribution, but this found little favor among the Partners. An alternative approach proposed would allow Partners and prospective partners to have a greater say in where their contributions go, be it to business development or advocacy or to HR programming. This approach may make it more attractive to prospective partners to join.

Finding 5:
Through its work with the regional development agencies, the NAPP Partnership supports a whole-of-Canada approach to commercial engagement, thereby benefiting all Canadians, however the omission of Ontario remains a gap that needs to be addressed.

The NAPP Partnership, through the regional development agencies, supports a "whole of Canada" approach to Canada's commercial engagement with the US and Mexico, which is a unique feature of the Partnership. This said a significant gap in the Partnership is the absence of Ontario, whose exports to the US accounts for over 44% of total exports and where 39% of exporting SMEs reside.^{Footnote 22} All Partners affirmed the importance of getting Ontario on board, an opinion shared by both STCs and FPDS managers at missions in the US and Mexico.

As remarked in the preceding Finding, the absence of official representation from Ontario in the Partnership, as with other OGDs, was not for reason of a lack of effort - FEDASO was approached on several occasions to persuade them to join the Partnership.^{Footnote 23} It must be recalled, however, that FEDASO was only launched in late 2009, and that when these overtures took place, the agency was in the early stages of establishing itself and its network. Consultation with representatives from FEDASO confirmed that institutional capacity challenges figured prominently in the decision to hold back from joining the Partnership at that time.^{Footnote 24}

Finding 6:
The three programming components of the NAPP Partnership are complementary and appropriate.

The three programming components of the NAPP Partnership (business development, advocacy, and HR) are considered appropriate and complementary. All the Partners represent clients in the business community who have an interest in accessing the North American market, and the activities supported by the NAPP Partnership are aligned with their respective needs. While not all the advocacy priorities are aligned with the priorities of the Partners (e.g., the Arctic), Partners recognize that advocacy is an important adjunct to business development and critical to advancing Canada's commercial objectives in the North American market.

The HR component of the NAPP Partnership supports business development and advocacy by offering specific training relating to both, as well as providing participants with first-hand experience in business development and advocacy programming, which strengthens the institutional capacity of the participating Partner. Though a sub-program that is highly valued by the Partners, of the three service offerings made available (training, TD assignments and long-term assignments), support for long-term assignments is qualified, largely because Partner personnel that succeed in securing a posting become in effect DFAIT employees - see Finding 25.

Finding 7:
The pooled funding model, despite some disadvantages, finds favor with the Partners and thus remains relevant to the Program.

As remarked in Finding 4, though the obligation to commit funds to the pool as a condition to join the Partnership has been cited as an impediment to greater OGD engagement, the pooled funding model is perceived by the Partners themselves to have merit. Advantages to this model include the following:

• Allows Partners to convert G&C funds into O&M funds, which is generally favored by the Partners for reasons that it is flexible and fast disbursing, an observation more or less echoed by STCs and FPDs at missions.
• Contributions once pooled become "fenced," thereby providing Partners, including missions, with a stable and predictable funding source which facilitates planning.
• Unspent funds are returned to the pool (retained within the Partnership) and, where feasible, reallocated to where the funds can best be used.

Though the ability of the pooled funding model to convert G&Cs resources into O&M resources was generally favored by the Partners, and therefore never raised as a concern prior to this evaluation, greater access to and use of G&C resources for programming purposes has emerged recently as an issue. ACOA, for example, conducts its business development activities almost exclusively through G&Cs, which is consistent with its funding authority. Despite limitations on the ability to transfer G&Cs to another Federal Partner, including Canadian missions abroad, this funding vehicle is reported to work well for ACOA.

Greater access to and use of G&C resources to support programming was also raised in the results of the Mission Survey. For example, though around 90% of STC respondents to the Mission Survey conveyed the view that O&M funding was appropriate for business development, around 50% of FPDs respondents remarked that access to G&Cs would be beneficial for advocacy programming, particularly with regards to the provision of funds to academic institutions and "think-tanks" to undertake work aligned with declared advocacy priorities

Notwithstanding the dissenting views on the merits of O&M vs G&Cs funding authority, there is a consensus among the Partners that the pooled funding model works and that absent this financial commitment the Partnership would likely fall apart and the benefits of partnership lost - see Finding 8.

Finding 8:
Despite the obligations and the challenges associated with Partnership, Partners see significant benefit to partnership.

Declared benefits associated with partnership are as follows:

• Access to mission network, including ROs and their resources.
• Access to information on the priorities and activities of other Partners which results in greater collaboration between Partners and between partners and ROS and missions.
• Leveraging of resources with Partners and non-partners.
• Supports, through the HR Program, institutional capacity building.
• Complements other funding sources.
• Supports innovation (e.g., pilot projects).

Further, the resources made available to missions from the NAPP Partnership constitute a significant source of funding, ranging from one to two thirds of funds available for business development and advocacy.^{Footnote 25} The loss of NAPP Partnership resources would have a significant negative impact on both the kind and quantity of activities undertaken at missions. Results from the Mission Survey further affirm the importance of NAPP Partnership resources to both business development and advocacy activities, with two thirds of both STC and FPDS respondents indicating that the availability of NAPP partnership resources were "critical," meaning they could not deliver on their respective mandates effectively.

Finding 9:
While none of the Partners declared a withdrawal from the North American market were the NAPP Partnership to cease to exist, all remarked that access to the mission network would be significantly diminished to the detriment of their clients.

While all Partners remarked that they would continue business development, and to a much lesser extent advocacy, activities in the US and Mexico were the NAPP Partnership not to exist, there was a consensus among the Partners that they would do less work and not nearly as effectively for the reasons cited in the preceding Finding. In particular, most Partners cited the potential loss of access to the mission network as a serious impediment to effective programming.

The results from the Mission Survey similarly confirmed that though both business development and advocacy activities would continue, they too would not do so nearly as effectively were the NAPP Partnership not to exist. The reasons include, in descending order of importance, the loss of access to financial resources to support programming, a reduction in the ability to leverage resources from other funding sources, a diminution in the ability to plan effectively, and a loss of access to human capital/knowledge possessed by the Partners and their regional offices.

Issue 2: Alignment with Government of Canada Priorities

Finding 10:
The NAPP Partnership priorities and objectives are consistent with and supportive of GoC priorities and objectives.

While greater importance has been accorded to expanding and diversifying commercial relations with new and emerging markets (e.g., BRICs^{Footnote 26}), strengthening Canada's commercial ties with the US and Mexico remains a GoC priority. Accordingly, the priorities of DFAIT, and the activities it supports through the NAPP Partnership, are aligned with GoC priorities. A review of the DPRs and RPPs of the Partners similarly confirms the importance of the North American market to the same, providing further evidence of alignment with their respective priorities with those of the GoC.

Finding 11:
There is an emerging and inherent tension between the exigencies of globalization and the limited geographic focus of the NAPP Partnership.

As important as the North American market is to Partners, Partners are also mindful of the opportunities that exist beyond North America and the need to engage with commercial partners wherever such partnership is advantageous. Indeed, the logic of global value chains (GVC), which is driving international economic integration, knows no boundaries, allowing for elements of the production process to be carried out wherever the necessary skills and materials are available at competitive costs. In a climate of scarce resources and competing priorities, the geographic focus of NAPP Partnership programming and the economic imperative to "think globally" is generating some tension in terms of the level of effort and financial resources to be dedicated to North America.

With Partners conveying interest in programming outside the North American market, questions have been raised about whether the NAPP Partnership model is one that could be applied to other markets? In responding to this question, it is important to keep in mind that the NAPP Partnership works because of a convergence of a number of factors (a plurality of sectors of interest to Canadian business, an alignment of Canadian capacity with commercial opportunities, evidence of a pan-Canadian stake in the market, evidence of a pan-governmental stake in the market, existence of an extensive mission network to support program delivery) which creates the requisite "critical mass" to support a formal partnership arrangement. While these factors exist within North America, thereby providing evidence of the continued relevance of the NAPP Partnership as a vehicle to advancing Canadian commercial interests in the same, they may not exist to the same extent in other markets. This does not preclude interdepartmental cooperation on business development and advocacy in the other markets, but it would likely be on a sectoral or project by project basis.

Finding 12:
While the importance of advocacy is acknowledged by Partners, activities undertaken in relation thereto are not fully aligned with the priorities of the Partners.

Notwithstanding that roughly a third of NAPP Partnership resources are allocated to support advocacy activities, and that Partners do have a say in shaping advocacy priorities - see Finding 27 - there has been little engagement of the Partners, with the exception of AAFC and WED, on the advocacy file. While the Partners, for the most part, conveyed a qualified comfort with this arrangement, there is clearly a disconnect between the programming priorities of the Partners, which to date has been largely business development focused, and the resources committed to advocacy, which has become by default an almost exclusive DFAIT concern - see Finding 30.

Finding 13:
Though priority business development sectors under the NAPP Partnership are more numerous than the priority business development sectors of DFAIT, NAPP Partnership business development activities are aligned with DFAIT's priority sectors, with the priority sectors of non-DFAIT Partners and of the missions. Advocacy allocations to declared priorities are fully aligned with DFAIT priorities.

DFAIT priority sectors find expression in the Sector Practice Groups. The first tier of priority sectors include: 1) Aeorspace, 2) Clean Tech, 3) ICT, 4) Infrastructure, and 5) Life Sciences. The second tier of priority sectors include: 1) Automotive manufacturing; 2) Defense and Security, 3) Forestry Innovation, 4) Ocean tech., and 5) Wine. Results from the Mission Survey, where STCs were invited to identify their top sectors of priority, produced the following list of sectors ranked in order of importance: 1) ICT, 2) Agriculture and Agri-food, 3) Aerospace and Defense, 4) Clean Energy, and 5) Bio-Industry and Life sciences.

NAPP Partnership programming resources touch on all of the aforementioned sectors. That certain sectors under the Partnership may assume greater importance or weight than may be accorded to the same by DFAIT should not be a surprise, given that the NAPP Partnership sector priorities are those of the Partners, not just DFAIT. Nor should it be a surprise that the sector priorities of missions would also differ slightly from those of the Partners, given their proximity to different markets. Indeed, missions remarked that one of the benefits of the NAPP Partnership is that it provides access to resources to promote Canadian commercial interests in a greater diversity of sectors which, though outside the strictures of the sector practices, nonetheless present real opportunities for Canadian business.

As remarked in Finding 12, despite opportunities to shape advocacy priorities, few Partners have engaged in advocacy programming, leaving programming decisions largely to DFAIT. This being the case, it is of no surprise that the results of Mission Survey revealed a strong alignment between the advocacy strategic priorities of the NAPP Partnership with those of the missions - NAPP Partnership expenditures under each of the strategic priorities match perfectly with the declared strategic priorities of the missions - because the mission priorities are in effect DFAIT priorities.

Finding 14:
NAPP Partnership funding to missions are not fully aligned with the regions of opportunity.

A comparison was made between NAPP Partnership business development expenditures at missions in the US and the Canadian Commercial Interest List (CCIL). The CCIL measures the commercial importance of a country or mission to Canada based on a combination of 6 broad factors: 1) trade, 2) market size, 3) FDI, 4) Service Demand, 5) R&D Spending, and 6) Growth Potential.

This comparison revealed that there is no relationship, proximate or otherwise, between the scores accorded to US missions under the CCIL model and NAPP Partnership funding to missions. For example, whereas the combined missions of Los Angeles and San Diego rank number 1 under the NAPP in terms of business development expenditures, under the CCIL model these two missions rank 4th among the US missions. While the CCIL model ranks Texas (Dallas and Houston) number 1 in terms of growth potential, Texas ranks 5th in terms of NAPP Partnership programming.^{Footnote 27}

The divergence in NAPP Partnership funding to missions with the identified regions of opportunity can partly be explained by reference to the dynamics of the Partnership, wherein Partners determine where their investments will be directed. This said the observation is noteworthy for two reasons: first, the inference that there may be untapped potential in these regions, and; second, the current funding formula for missions may be ill equipped to exploit these potentials - see Finding 30.

Issue 3: Consistency with Federal Roles and Responsibilities

Finding 15:
The activities undertaken by the Partners through the Partnership are consistent with their respective mandates and deemed an appropriate responsibility of the Federal Government.

While different levels of government and the private sector can and do play a role in the planning and delivery of business development and advocacy activities abroad - see Finding 16 - there is a consensus among stakeholders, both in the public and private sectors, that the Federal Government must take the lead in defining policy and in establishing the infrastructure/programs needed to give effect to policy.

As remarked earlier, all the current Partners possess either express or implied mandates to support international business development. Each Partner has their sector priorities, as well as unique subject matter expertise relating thereto, which is the underlying rationale for the whole-of-government approach to international commercial engagement. In short, the NAPP Partnership in reference to its mandate and design is wholly consistent with the roles and responsibilities of the Federal Government.

Finding 16:
Funding sources available to NAPP Partners grants the same flexibility to engage whatever organization may be best positioned to deliver on a given project or initiative.

All the Partners work in close collaboration with government (e.g., provincial governments) and non-government organizations (e.g., industry and trade associations) in program planning and delivery. These organizations can be supported or engaged through O&M contributions or G&Cs. All the Partners have access to G&C resources, albeit at different levels, to support programming. Though G&C resources may be the appropriate vehicle to engage some partners in project delivery, such as think-tanks to undertake research, O&M resources is generally considered appropriate to support the staging of events, such as trade fairs and workshops.

For Partners, whether one funding source is regarded more favorably than another depends on the type of programming contemplated, and of course the relative availability of these funding sources to support programming. Some Partners are O&M rich, while others are O&M poor. For the latter, as remarked in Finding 7, the pooled funding model is considered to have merit for reason that G&C resources can be converted to O&M resources, thereby addressing a deficit. The appropriate balance between access to O&M and G&C resources is something that can only be determined by the Partners themselves, however to the extent that all Partners do have, at least nominally, access to both funding sources, they are vested with a degree of flexibility to use one or the other where it is advantageous.

8.2 Achievement of Expected Outcomes

While a PMF was prepared by DFAIT for the North American Platform Program, which included elements pertinent to the Partnership, the PMF was never formally endorsed or implemented by the Partners and thus never populated with data. As a consequence the evaluators were obliged to rely on results data available in the systems in effect during the study period, which were not configured to generate performance data that can be easily attributed to the NAPP Partnership. Nor can the evaluators confirm the accuracy and reliability of the data. Accordingly, the numeric results data reported hereunder should be regarded with caution.

8.2.1 Business Development

Introduction

Of the 969 business development projects supported by the NAPP partnership over the study period, 170, distributed proportionately between the strategic priorities, were selected for examination. The following summarizes the outputs and outcomes gleaned from records. Although "Seek opportunities created by the US Stimulus Package" was included as a priority for 2010-2011, no performance data was found for the three projects in the sample. Consequently no attempt is made to report on results against this strategic priority.

Results

Finding 17:
The NAPP Partnership has facilitated the participation of thousands of Canadian businesses, industry and government representatives from across the country in activities and events that show-case Canadian capacity.

An examination of the records pertaining to the projects within the sample demonstrate that the NAPP Partnership has facilitated the participation of over 2,000 Canadian businesses, industry and government representatives (hereinafter clients) in events, including site visits, that show-case Canadian industry and infrastructure (gateways and corridors) capacity. These clients represent a wide array of sectors (agri-business and agri-food, transportation, aerospace and defense, clean technology, energy, life sciences and bio technology, automotive, oceans and marine, and ICT) from all across the country.

Clients, through their participation in NAPP Partnership funded events, were exposed to tens of thousands of their counterparts in the US, Mexico and many other countries. Events such as Greenbuild, which is held annually in different cities across the US, brings together thousands of participants (producers, distributors, architects and builders) from dozens of countries, Photonics West, which again brings thousands of industry representatives together, and the Fancy Food Show and the International Boston Seafood Show, which also bring together food suppliers, distributors and wholesalers in the thousands, provide an indication of the reach facilitated by the NAPP Partnership.

Further, innovative projects like the Canadian Technology Accelerator (CTA) program, which is designed to prepare Canadian ICT firms for access to the California high tech market, has also provided Canadian businesses with an opportunity to show case their capacities. This program has, over the study period, engaged around 264 Canadian companies in the "boot camp" sub-program, supported over 100 in "48 hours in the Valley" sub- program, and 80 companies in the business accelerator/incubator program.

Finding 18:
Canadian business participation in NAPP Partnership funded events has helped Canadian businesses forge ties with their business counterparts in the US and Mexico and generate interest in doing business with the same.

In addition to facilitating Canadian business participation in events, the NAPP Partnership, with the assistance of STCs at missions, has facilitated hundreds of match-making (face-to-face) meetings between Canadian participants and their US and Mexican counterparts. An examination of the records pertaining to the projects within the sample revealed that these match-making meetings generated over 4,700 contacts, 2,300 leads, and close to 400 hundred services (outcalls and service requests) delivered.

Examination of the records pertaining to the sample further revealed that participation in NAPP Partnership funded events generated 5 MOU/Agreements to support cooperation in such sectors as bioenergy (Doing business with DOE), transportation/gateways and corridors (CentrePort Missions), aeronautics (Aeromart - Montreal), and life sciences (FADE Research and Business Partnership Forum). Hundreds of US and Mexican businesses following meetings facilitated by the NAPP Partnership at events (e.g., Pacific Market Centre, Food Technology Exchange, Mexican Environmental Technology Trade Show, Photonics West, etc.) are on record conveying interest in exploring business opportunities with their Canadian counterparts.

Further, results from the recent Trade Commissioner Service (TCS) Online Client Survey, which was launched in April 2010 and has been repeated annually since, revealed that over 57% of Canadian exporters that received TCS assistance reported pursuing commercial agreements with their US and Mexican counterparts.^{Footnote 28} One can make an inference based on the foregoing that an equal percentage of participants in NAPP Partnership sponsored events are likely pursuing commercial agreements with their US counterparts.

Finding 19:
NAPP Partnership investments in business development appear to have generated significant returns.

The value of disbursements over the study period relating to the projects comprising the business development sample is around $2 million. According to the records reviewed, these investments generated around $3 million in confirmed sales (Incoming US Seafood Buyers Mission, International Builders Show, Irish Fest, Photonics West, PA LCB Wine Festival, Atlanta International Environmental Trade Mission), and around $42.8 million in expected and potential sales (Fancy Food Shows, GreenBuild and Photonics West). Additionally, the results from a survey of CTA program participants revealed that 50% reported increases in sales by a factor of 10 following their participation in the program.

Additionally, a study undertaken by DFAIT on the impact of TCS on Canadian exporter performance revealed that exporters that received TCS assistance export almost 18% more on average than comparable exporters that did not receive such assistance. This was reported to translate into $27 in export sales for every dollar spent on the TCS. Though these results are not specific to North America or to the evaluation sample, an inference can be made that assistance to Canadian exporters through the NAPP Partnership has likely yielded similar outcomes.^{Footnote 29}

In terms of generating FDI, the review of the records pertaining to the project sample found examples of US FDI in Canada which include a $23 million investment in a pharmaceutical laboratory in Laval Quebec (LA Region Outreach) and a reported $55 million in potential new investments in the manufacturing sector (Manufacturing GVC). The review of the project sample also showed unexpected results in respect to Canadian Direct Investment Abroad (CDIA), as several Canadian firms reported setting up offices in the US to take advantage of contract opportunities with the US Department of Defense (Mission to US Air Force Logistics Centre).

8.2.2 Advocacy

Introduction

Of the 664 advocacy projects supported by the NAPP Partnership over the study period, 132, distributed proportionately between the strategic priorities, were selected for examination. Again the following summarizes the outputs and outcomes gleaned from the records, which may likewise be multiplied by a factor of 5 to arrive at the proximate level of outputs and outcomes generated by the entire population.

As stated in section 2.6 of this report, the evaluators found little quantitative data in the records. Consequently, the results reported hereunder are largely qualitative in nature.

Results

Finding 20:
NAPP Partnership advocacy programming has contributed to an increased awareness of Canadian interests and capacities.

By targeting key groups including politicians, business and trade organizations, federal, state and municipal governments, academia, the energy industry and media, advocacy programming has contributed to an enhanced understanding of Canadian interests and capacities. During the study period, NAPP Partnership sponsored events included within the sample attracted over 120,000 participants. Examples noted include the "Offshore Technology Conference" held in Dallas, the "Canadian Snowbirds Air Show" event held in Denver, and the 31st annual Starz Denver Film Festival which attracted tens of thousands of attendees and high-level sponsors. Additionally, there was the "Aerospace Trade Study and Event" where the President and Chief Executive Officer of Campbell Soup Company delivered a keynote address to 180 guests at an Embassy dinner that included journalists, academics, and representatives of associations and companies.

In addition to the events cited above, NAPP advocacy programming during the study period sponsored over 65 promotion or media relations events that were aimed at raising awareness and/or highlighting Canadian capacities. These included things such as press conferences, interviews or similar media events. The use of print media (letters, brochures, maps) was also an important tool used to reach key actors and raise their awareness of Canada. For example, a letter signed by Atlanta HOM outlining the importance of the Canada-State relationship and including the individual State fact sheets was sent to State Representatives and Senators in the region. Many events that NAPP takes part in are covered by print media and this often presents an opportunity to increase awareness. Coverage by Business Week, Bloomberg, the Hill and MSNBC.com of the Aerospace Trade Study and Event which NAPP took part is just one example.

Furthermore, through speaking engagements and meetings with key decision makers and by the development of new contacts NAPP advocacy programming helped enhance Canadian interests in the US. In the sample, 76 speaking engagements were noted which were used to highlight Canadian capabilities. An example of this was at the World Food Prize event where the Minister of Agriculture and Agri-Food Canada conveyed messages on food security and trade to a high-level audience of government, business, NGO and academic experts in Des Moines Iowa. The review of the project sample showed that there were over 1,000 new contacts that were developed as a result of the various NAPP advocacy activities that were undertaken.

Among the initiatives that stand out in terms of raising awareness and highlighting Canada's important role in North America is the "All Politics is Local" (APIL) initiative.^{Footnote 30} It is an annual one-day economic advocacy blitz on Capitol Hill that involves the Canadian Embassy in Washington, D.C., Canadian Consuls-General from across the U.S. network, NAPP Partners, and business leaders. APIL brings together both political and business influencers from both countries and is aimed at raising awareness and highlighting the close economic relationship between Canada and the U.S. It demonstrates the benefits of the relationship for the Congressional districts of the Members of the Senate and House of Representatives that are visited during the event. The aim is to educate members of Congress to the importance of the U.S.-Canada economic relationship.

APIL adopts a unique theme each year. In 2009 APIL took place under the theme "Partners in Recovery: Rebuilding the North American Economy Together." The Deputy HOM at Washington, D.C and Canada's 13 Consuls General along with US business leaders conducted more than 80 meetings with elected members of Congress, underscoring the scope and depth of Canada-US trade and the millions of US jobs that depend on it (approximately 7 million at the time of the initiative).^{Footnote 31} In 2010, under the theme "U.S.-Canada: Creating Jobs Together" there were meetings with 87 Members of Congress to underscore the importance of U.S.-Canada trade. More than 30 Canadian and American company representatives took part. The event was capped by a reception and dinner with 170 guests hosted by the Ambassador.^{Footnote 32} In 2011 the objectives of APIL was to build support for the next round of Canada-U.S. procurement negotiations. The event was useful in identifying and building support amongst U.S. domestic stakeholders for a more comprehensive Canada-U.S. reciprocal procurement agreement.^{Footnote 33}

APIL is important in terms of creating a favourable image of Canada with Members of Congress. For many American legislators having a Consulate General accompanied by a company representative in meeting as well as having facts and data on job creations in their respective electoral districts significantly helped raise the importance bilateral trade with Canada has on jobs in their electoral districts or states. Overall, APIL remains a highly visible and effective mechanism to reach large numbers of US elected legislators, as well as to build relationships with key advocacy partners.^{Footnote 34}

Another tool that is crucial to NAPP advocacy programming is Government of Canada Continental Analysis and Research Tool (GoCCART). It is an interactive, password-protected, geographic information system that centralizes the necessary information to increase Canada's political and trade advocacy capabilities, and allows this information to be sorted and searched on any category and on any geographic area including political districts.^{Footnote 35} The posts are consumers of this tool, using it to promote Canadian interests across America and focus policy-makers' attention on Canadian priorities, and at the same time contributors to its banks of information.

The data serves as the foundation for a number of key events and products including "All Politics is Local," speeches, and state trade fact sheets, used to engage key players in Congress, members of the business community, and a variety of other policy and advocacy targets. GoCCART serves approximately 200 users across DFAIT and NAPP partners and stakeholders.^{Footnote 36} Some key highlights of results for the GoCCART tool during the years covered under this evaluation were:

• GoCCART was accessed thousands of times by hundreds of unique members throughout DFAIT and NAPP partners. Reports generated from GoCCART were used for internal activities and hand-outs to members of Congress and business leaders.
• Research reports and findings from GoCCART were regularly used for meetings with Congressional members and staffers to demonstrate the economic benefit that Canada brings to a Member's district or state.
• The research and reports generated by GoCCART and shared with members of Congress helped build and solidify positive attitudes within Congress about Canada and Canada-U.S. trade.^{Footnote 37}
• Products such as the QuickFacts Report and Canadian Footprint Report were used by the Ambassador and Consuls General for briefing notes and speeches, and were well received by elected officials and their staff.
• By utilizing GoCCART, WSHDC was able to build upon the CAPP list of U.S. suppliers to the Canadian oil sands by providing specific locations, line of business, and employee figures in support of the ongoing oils sands advocacy efforts.
• Over the course of FY 2010-11, GoCCART was accessed over 1,300 times (representing a 13% increase from FY 2009-10) by users at DFAIT and NAPP partner departments.^{Footnote 38}

Finding 21:
Advocacy programming has contributed to expanding Canada's network of influence in the US.

The review of the projects undertaken by the program over the study period shows that as a result of advocacy programming there were over 14,000 influencers and/or decision makers that were reached. These contacts were mostly through briefings, luncheons, roundtables or similar activities where influencers or decision makers were the main target audience. Examples include events such as "Canada Day in Trenton," "Health Care Round Table" initiative held in Denver and the "Nurture Bi-national Partnerships around an Energy Agenda" project in Buffalo. The access to key decision makers and influencers in these settings is crucial to the advancement of key Canadian security, economic, political, interests in the US.

Other notable events where key influencers were targeted include the "Talking Economics Speaker Series" project implemented by the Consulate General in New York which partnered with the Foreign Policy Association to attract 100 guests from the business, academic, think tank, media and government communities and "NPT Review Conference Outcomes Seminar" in Washington where over 100 decision makers were present. And as mentioned in the previous finding APIL is a key initiative undertaken by NAPP which contributes to raising awareness about Canada but it is also crucial in terms of expanding Canada's network of influence in the U.S.

One of unique advocacy project which helps to expand Canada's influence with U.S. policy makers is the Rising State Leaders (RSL) Program.^{Footnote 39} The Embassy of Canada in Washington, D.C. partners annually with the Council of State Governments (CSG) to deliver the RSL program, which selects promising and influential US state legislators to participate in an intensive working tour of Canada. Tours have involved briefings with GoC representatives and meetings with Members of Parliament and Federal Cabinet Ministers in Ottawa. ours have also involved meetings with elected officials from provincial and territorial governments, senior corporate executives, and other key persons touching on all key advocacy issues from across the country. The aim is to facilitate direct relationships between state legislators and Canadian political, business and cultural leaders, with the broader objective of developing participants' first-hand knowledge of the value and complexity of Canada-US relations.^{Footnote 40}

Finding 22:
While there are few outcomes that can be directly attributed to NAPP Partnership advocacy programming, an inference can be made that advocacy programming has contributed to the achievement of outcomes.

Though outside the study period, progress has been noted in the media on several advocacy files of importance to Canada e.g., the Key Stone Pipeline, Beyond the Border Plan, renewal of the Great lakes Water Quality Agreement, to name a few. While it is difficult to say how and to what extent NAPP Partnership programming contributed to these outcomes, the knowledge influencers and decision-makers gain from the advocacy program contributes greatly to an increased awareness of Canada's position. The benefits of participating in the program have been relayed often by individual participants of the program. The following provides anecdotal evidence of media coverage and several examples of feedback received from decision-makers following participation in NAPP Partnership events:

• After taking part in the 2009 program Assemblyman (D-NV), stated "The biggest lesson I'll take away from this experience is recognition of Canadian provinces as partners and allies."^{Footnote 41}
• Following her participation in the 2009 program the chair of the Council of State Governments (CSG) pledged to introduce the idea of a permanent Canada-U.S. panel at national and regional CSG meetings to discuss topics such as "Buy American" and other relevant issues.^{Footnote 42}
• The delegates who participated in the program in 2010 left with a substantial appreciation for the complexity of the bilateral relationship between Canada and the U.S. and the opportunities and challenges which exist. An immediate impact was an op-ed piece which appeared in the Manchester Journal.
• •Participants over the years, have become instrumental in leading regional gatherings of the Council of State Governments to pass resolutions demonstrating common cause with Canadian interests (e.g., on Buy American). The program alumni continue to serve as valuable partners to further Canadian advocacy objectives in the U.S.^{Footnote 43}

8.2.3 Human Resource Programming

Introduction

The HR programing serves to support the business development and advocacy programming needs by providing staff to missions, training the same, and funding for the management of the TD assignment and/or long-term assignment. Although a PMF in name was developed by the Program related to training, TD and long-term assignments, the framework essentially documents the activities to be undertaken rather than the expected results to be derived from those activities, and thus determined by the evaluators to be of limited utility for results reporting purposes.

To report on performance, the evaluators focused on assessing the value of training to participants, the value of the TD and long-term assignments programs to the same, and their respective contributions to mission objectives and the overall objectives of the program. Each of the components or subprograms of the HR program are examined discretely hereunder.

Training

Finding 23:
Despite some minor issues around Partner access to training, the training offered is highly valued by the Partners and appears to have contributed to enhanced capacity and performance.

Each year NAPP partners are invited to take part in professional development training and targeted training. During the study period, the demand for training increased significantly, from a total of 17 participants from Partner departments in FY 2008-2009, to 26 participants in 2009-2010 and reaching a high of 34 participants in 2010 -2011. Training expenditures also increased by 6.6% from the allocated budget in 2009-2010 and 51.2% from the allocated budget in 2010-2011. As a result, the Program had re-allocated funds to meet the demand for training.

Participants in the training program interviewed for this evaluation spoke highly of the relevance and quality of the training they received. In particular, participants remarked on the valuable theoretical and practical knowledge they received as well as the contacts needed to conduct their activities both in Canada and abroad.

STC and FPDS managers at missions were also, through the Mission Survey, asked to comment on the training program as it relates to TD and long-term assignments. The results revealed that most respondents (75 to 80%) agreed that TDs and those participants in long-term assignments were well prepared to assume their duties at post upon arrival, though they did remark that many upon arrival found it difficult to integrate into the DFAIT culture.

Notwithstanding the foregoing, several former TDs interviewed remarked that, owing to late confirmations of their postings, they were unable to profit from pre-posting training. Additionally, TD alumni remarked that late notices for training, combined with the fact that training is delivered in Ottawa, made it difficult to plan for attending training sessions. Another impediment to access to training cited was the language with which training is provided - see Finding 37.

Temporary Duty

Finding 24:
The TD program is highly valued by the Partners and is reported to have greatly contributed to the institutional capacity of the same and to enhanced performance. This said the TD program has not fully capitalized on its potential.

Temporary Duty (TD) Assignments are short-term assignments through which partner's employees are assigned to a post for terms not exceeding four (4) months. During the study period, a total of 41 candidates participated in TD assignments involving all the Partners, but for CEDQ.^{Footnote 44} Most of the candidates were from DFAIT which represents 34% of the total participants (14 candidates) followed by WED 21% (9 candidates), ACOA 17% (7 candidates), AAFC 16% (6 candidates) and NRC 12% (5 candidates). The actual expenditures for TD assignments during the study period have been slightly less than the actual budget with the exception of FY 2010-2011 where the actual expenditures exceeded the budget by $17,899.

Candidates returning from TD assignments listed a number of benefits including the following:

• Better understanding of the operations of posts and better understanding of the markets and the potential of further activities in the location where they were assigned;
• Better insight as to the challenges faced by Canadian companies when looking at expansion into the international market;
• Greater insight of the important role of continued and sustained advocacy to promote Canadian interests;
• Extended networking and contacts and development of relationship with key contacts;
• Significant experience in planning and organizing large and small events; and
• Strengthened working relationship with missions and DFAIT.

With respect to benefits to their departments, participants reported that the TD assignment provided them with the opportunity to showcase their region, to develop contacts in sectors that are of importance to their departments/agency and to share the knowledge gained with their colleagues. The missions, through the Mission Survey, remarked that they too gained a better understanding of the operations of the Partners and the opportunities available in the region where the Partner employee came from. Missions further benefited from an extra employee for the delivery of tasks that may not have been possible without the additional support.

Notwithstanding the popularity of the TD program, Partners remarked on the challenges associated with backfilling positions when the incumbents are at mission. The evaluators found that there is no cross sharing of personnel between Partner departments or between missions, ROs and Partners. Sharing of resources approach may assist in dealing with the backfill issue.

Beyond simply addressing the backfill issue, such interdepartmental exchange of personnel would be of mutual benefit to Partners. Were personnel from DFAIT, for example, to participate in TDs assignments to one or other of the regional development agencies, and personnel from the regional development agencies to participate in TD assignments at DFAIT, this would likely strengthen knowledge transfer, mutual understanding and cooperation between the same. Were personnel from missions to participate in TD assignments to one or other of the regional development agencies or ROs, this too would likely have the same outcome. That such an exchange of personnel among the Partners, missions and ROs has not occurred constitutes a missed opportunity

Long-Term Assignment

Finding 25:
While the long-term assignment program was expected to be of mutual benefit to Partners and missions, support for the program from the same is qualified.

Long-term assignments are typically of four (4) year duration. The NAPP Virtual Secretariat provides partners with advance notice of the deadline for application for Canada-based position in the US. Partner departments are responsible for reviewing potential application to confirm that they meet the general and specific criteria of the position. During the study period, a total of 223 candidates met the qualification.

Out of 223 candidates, 199 candidates were interviewed through the NAPP Selection Process. Out of the 199 candidates, only 52 candidates were successful in securing a posting in the US. From the 52 candidates that were selected, 36 were from DFAIT and 8 from partner department (AAFC, ACOA and WED) for a total of 44 candidates. Out of the remaining 8 candidates, 6 were extended and 2 were selected through an alternate process.

Notwithstanding that Partners were expected to benefit from the long-term assignment program in terms of having a durable contact at mission, and that missions were expected to benefit from the regional knowledge and subject matter expertise that Partner personnel would bring to their assignments, neither Partners nor missions conveyed the view that these expectations have been fulfilled.

In the first instance, the level of non-DFAIT Partner participation in the long-term assignment program has been small. The modest level of non-DFAIT personnel participation in the long-term assignment program is attributed variously to: challenges in receiving authorization from the donor Partner to have the candidate released for an extended period of time, and the reported difficulties that non-DFAIT Partner personnel have had in meeting DFAIT eligibility requirements, in particular official language competencies. Further, while non-DFAIT Partners recognize the professional development value to participants, non-DFAIT Partners saw successful candidates as lost to them since they become in effect become DFAIT employees.

The missions, through the Mission Survey, similarly conveyed qualified support for the long-term assignment program - roughly half of respondents declaring that the contributions of OGD personnel participating in long-term assignments as being only "somewhat significant." A common complaint cited by respondents who conveyed qualified satisfaction with the long-term assignment program was that candidates did not have the requisite training or experience to assume responsibilities at mission. Another complaint was the inability of candidates to adjust to DFAIT culture.

8.3 Efficiency and Economy

Introduction

Findings under the following section are grouped under two broad headings: 1) Governance, and 2) Program Administration.

8.3.1 Governance

Stakeholder Engagement

Finding 26:
While structures and procedures were established to facilitate Partner engagement, governance of the Partnership arguably could have benefited from greater engagement of centres of expertise housed within Partner departments.

While Partner departments are represented in all the NAPP partnership governance bodies, these departments contain within them a plurality of centres of expertise relevant to the mandate and objectives of the Partnership. Within AAFC, for example, there are different units dedicated to marketing (the AgriMarketing Program) and market access (the Market Access Secretariat). Within NRC, there are different units dedicated to research institutions (e.g., National Institute for Nanotechnology, Institute for Biological Sciences, etc.) and to the Industrial Research Assistance Program (IRAP), which provides grants and financial support to businesses looking to bring new and innovative technologies to market. A more fulsome engagement of these various centres of expertise would arguably benefit the Partnership.

DFAIT likewise houses a variety of centres of expertise which could add value to the Partnership. For example, the Trade Policy and Negotiations Branch (TMG), which includes the North America Trade Policy Bureau, possess knowledge and expertise highly relevant to the NAPP Partnership. Similarly the International Business Development, Investment and Innovation Branch (BFM), which is comprised of 7 bureaus,^{Footnote 45} also possess knowledge and expertise highly relevant to the NAPP Partnership. Arguably the Partnership could benefit from the input and counsel of these branches.

Among the bureaus within BFM is the Canada Bureau (BSD), which is charged, among other things, with managing the DFAIT's network of ROs across Canada. The ROs perform a very important support role to both Partners and missions, providing the same with market, industry and client information. The ROs also provide assistance and advice directly to clients within the Canadian business community relating to international trade and investment. Notwithstanding the foregoing, and the fact that the ROs are eligible for NAPP funding, the ROs have not received funding from the NAPP Partnership nor have they been accorded any representation in the NAPP governance structure. This neglect of the ROs is reported to have resulted in the missed opportunities and duplication of effort.

Finding 27:
Structures and procedures have been put in place to facilitate, in the spirit of the GFA, Partner participation in decision making, though there have been reported lapses in this regard.

In compliance with the principles of the GFA, governance structures and procedures have been established to facilitate Partners participation in decision making. Though Partners are represented in all the governance bodies formed under the NAPP Partnership, there is a perception among non-DFAIT Partners that decision making has not been wholly consensus based. Examples include the decision, undertaken unilaterally by DFAIT, to disband the NAPP Secretariat, formally a stand-alone body, and replace it with the Virtual Secretariat, as well as the decision, again undertaken unilaterally by DFAIT, first in 2008 and repeated in succeeding years, to abandon the automatic carry-forward of unspent funds and redirect the same to missions to make-up for funding shortages.

Non-DFAIT Partners further conveyed the view that priority setting over the years had increasingly come to be directed by DFAIT, a trend that was reported to be particularly pronounced with respect to advocacy. As stated in Finding 12, non-DFAIT Partner engagement on the advocacy file has been minimal, thereby relegating priority setting and programming to DFAIT. This outcome has likely more to do with the interests of the Partners, which have been largely business development focused, than on defects in the governance structure. This said, the practice of engaging the ABDC only when advocacy project proposals exceed $50 thousand, which is rare, has likely contributed to the perception of DFAIT programming without full consultation with the Partners. Additionally, the reversal of a DGC decision regarding strategic priorities in FY 2011-2012 at the behest of DFAIT also has likely contributed to the perception among non-DFAIT Partners that priority setting is DFAIT driven.

Non-DFAIT Partners also conveyed concern that HR programming was favorable to DFAIT. While the results of the Mission Survey suggested a certain preference for DFAIT personnel in the TD assignment processes, thus seeming to confirm the view among some non-DFAIT Partners that the selection process is favorable to DFAIT personnel, the actual number of TD assignments from DFAIT roughly corresponds to the fiscal framework established under the Partnership. Consequently, the perception of bias is just that - a perception without supporting evidence. This said 81% of the long-term assignments have been DFAIT personnel - see Finding 25.

Strategic Guidance

Finding 28:
While stakeholders are generally satisfied with the quality of strategic guidance provided by the NAPP Partnership governance bodies, there is room for improvement.

The ADMC was originally charged with priority setting, but this was delegated by default to the DGC when the ADMC ceased to convene regularly after 2008. Consideration was given at one time to assigning the strategic priorities of both business development and advocacy budgets, which would have provided some indication of the weight or importance relating to each, however this option was eschewed in favor of maintaining the strategic priorities as a programming framework with funding relating thereto to be determined by the projects identified by missions and Partners.

Priority setting within the context of the Partnership has from the outset been something of a challenge, owing to the need to accommodate the interests of all the Partners, resulting in what the Partners themselves have referred to as a list of broad objectives rather than actual strategic priorities. A review of the minutes of the DGC further provided evidence of disquiet with the strategic priorities, which were characterized as vague and overlapping. Enhancing Canada's participation in GVC is a case in point. Although enhancing Canada's participation in GVC is commonly interpreted to mean increasing Canadian exports of intermediate goods and services, GVC participation also involves the import of goods and services, as well as outsourcing of elements of the production process to other countries. Promotion of Canada's Gateway and Corridors, while a key GVC enabler, is arguably better suited as an advocacy priority than a business development priority, unless the objective is to foster FDI in Gateway and Corridor infrastructure.

The strategic priorities for Advocacy are similarly broad in construct and can likewise be read as variations on a similar theme. For example, "Decrease in the thickening of the border" and "Increase market access" more or less address the same issue, and are both pursued by the same activities and with the same audiences, unless market access deals specifically with domestic US regulations. New guidelines however have recently been developed which bring greater clarity to some of these issues.

Although both the Partners and missions are comfortable with the priorities, likely for reason that they are open to interpretation and can accommodate just about any activity, target group or sector, more specific guidance on what these priorities actually mean would benefit programming.

Roles, Responsibilities and Accountabilities

Finding 29:
Roles, responsibilities, and accountabilities within the NAPP Governance structures have not been formally documented, thereby contributing to ambiguities relating to the same.

The roles and responsibilities, and accountabilities of the various governance bodies have been fluid over the course of the study period. The delegation of the responsibility for priority setting from the ADMC to the DGC is a case in point. The roles, responsibilities, and accountabilities with respect to the other governance bodies (ABDC and HRC) have similarly been fluid, with the terms of reference (TORs) for these latter two committees having not been fully elaborated on or formally adopted. It is surmised that ambiguities around the precise roles and responsibilities of these governance bodies appear to have fostered some uncertainty as to what these bodies are actually accountable for and to whom - see Finding 27.

Though the service offerings of the Virtual Secretariat are documented, during the transition from the ERI model to the Virtual Secretariat model some service offerings, with the exception of financial management, were reported to have been neglected, thereby contributing to uncertainties as to what the Virtual Secretariat was actually responsible and accountable for - see Finding 37. Moreover, the Virtual Secretariat did generate confused accountabilities within DFAIT by grafting on NAPP Partnership responsibilities to the job descriptions of employees who are accountable to managers not directly responsible for the NAPP Partnership. Conversely, some managers found themselves responsible, and thus accountable for, work performed by personnel over whom they have no direct authority. Divided accountabilities under the Virtual Secretariat model has likely contributed to the perception among non-DFAIT Partners that personnel within the Virtual Secretariat largely work for DFAIT with the interests of the Partnership being subordinate to the interests and priorities of DFAIT.

A lack of clarity on roles, responsibilities and accountabilities was also noted with respect to the HR program, particularly the TD subprogram. TDs, while at mission, are to receive direction from and report to their immediate supervisors, be it a STC or a senior FPDS. This said there were reported instances where TDs undertook work and attended meetings without prior notification, and who never reported back to their supervisor, leaving the same with the impression that they were not there to advance the mission objectives, but rather to serve their home department. While these instances are likely rare, it provides further evidence of the need to establish very clear guidelines on the roles and responsibilities of TDs while on assignment - see Finding 33.

While some of the perceived ambiguities relating to roles, responsibilities and accountabilities are structural - e.g., the Virtual Secretariat model - most can be attributed to the absence of well documented TORs and processes, which one would normally find in a Standard Operating Procedures (SOPs) manual. This is not to suggest that guidelines and procedures do not exist, they do, but rather that they are not consolidated into a single document. SOPs, of course, are of limited utility if they are not shared with those who need to know, which highlights the importance of also having in place a robust communications service/program - see Finding 34.

8.3.2 Program Administration

Resource Allocation

Finding 30:
While Partners and missions convey a modest level of satisfaction with the current formula for resource allocation, some dissenting opinions were noted.

Though Partners convey satisfaction with current allocation formula to the same, which is based on the contributions of the Partners, the allocations assigned to the different programming components of the NAPP Partnership has emerged as an issue. As stated in Finding 12, although allocations to advocacy account for roughly a third of total programming dollars available, non-DFAIT Partner engagement on the advocacy file, but for the AAFC and to a lesser extent WED, has been minimal. Though Partners acknowledge the importance of advocacy, in the current climate of austerity many are questioning the quantum of resources allocated in support thereof. Partners have also conveyed concern over the allocation in support of the Virtual Secretariat itself, which accounted for around 10% of the NAPP Partnership budget during the study period, and which exceeds the generally accepted threshold of 6 to 7% overhead - see Finding 37.

While non-DFAIT Partners do not participate directly in decisions relating to the allocation of funding to missions, these funds do have a direct impact on programming. As remarked in Finding 7 and addressed in greater detail in Finding 32, funding to missions permit the same to identify projects for prospective Partner participation. The missions, therefore, perform a critical role in program planning and thus have a significant impact on where Partner resources will be directed. Though the Mission Survey revealed modest levels of satisfaction with the current formula used to allocate resources to missions - roughly half of STC and FPDS respondents indicated they were very satisfied with the current funding formula - some dissenting opinions were noted.

For example, funding to missions is currently based on mission capacity - i.e., the number of Full time Equivalents (FTEs) at mission. While it is not unreasonable to use the staff complement at a mission as a proximate measure of the commercial significance of the same to Canada, it is at best a blunt measure, and may not accurately reflect real commercial opportunities within the area, or potential for growth. As remarked in Finding 14, NAPP Partnership funding to missions bear no relationship to those regions and missions in the US deemed by DFAIT's CCIL model to offer the highest growth potential. Questions relating to the reliability of the CCIL model aside, the apparent disconnect between the geographic allocation of NAPP Partnership resources and the geographic regions of growth potential was identified by several respondents to the Mission Survey as a weakness in the current funding formula.

Respondents to the Mission Survey who conveyed some dissatisfaction with the current funding formula also remarked that funding to missions is not linked to performance - missions get the same funding regardless of what they do, which is said to foster complacency and discourages risk taking and innovation. It is difficult to comment on the merits of this contention, but it does allude to an interesting proposition - that base/fixed allocations should be supplemented with a competitive application process, as per the Client Service Fund (CSF) model, whereby missions would have to compete with other missions for additional funds to support projects.

Financial Management

Finding 31:
There is a general consensus among the Partners that financial management and reporting is good.

While the Virtual Secretariat was initially charged with managing program resources of the Partnership as a whole, over time the management of program resources was delegated, through the instrument of a Memorandum of Understanding (MOU), to each of the Partners, ostensibly to facilitate the speedy reallocation of funds among approved projects, with the former assuming more of an oversight and coordination role.

Notwithstanding the aforementioned MOU, which outlined financial reporting requirements, and the existence Financial Guidelines (2009), one Partner remarked that financial reporting requirements were not clear, thereby contributing to uncertainties as to the content and format of financial reporting. This said no other Partners conveyed dissatisfaction with the guidelines. In terms of financial reporting from the Virtual Secretariat, Partners conveyed a high degree of satisfaction with the accuracy and timeliness of financial reporting.

Partners further remarked that late notification from missions of unspent project funds have generated financial management challenges, particularly with respect to the reallocation thereof. While the return of unspent funds to the Virtual Secretariat late in the FY has contributed to lapses, the Virtual Secretariat has been very successful in putting these funds to productive use thereby minimizing the quantum of funds lapsed at year end. Over the study period, lapses have ranged from 1 to 4% of total commitments, which is quite acceptable - see Table 3: NAPP Financial Summary 2008-09 to 2010-11.

Project Planning, Selection and Approval

Finding 32:
While there is a general level of satisfaction with the current system of project planning, selection and approval, some dissenting opinions were noted.

As remarked earlier, the missions perform a key role in project planning, identifying projects for prospective Partner participation. Some challenges were noted by both Partners and missions with respect to the different planning cycles among the same, though both Partners and missions conveyed the view that the challenge is manageable. This said some Partners remarked that a lack of access to or unfamiliarity with the business plans (CEP) of missions constituted an impediment to effective planning.

Some ambiguity was reported on the eligibility criteria for NAPP Partnership funding by both Partners and missions to the point that clarification from the Virtual Secretariat was sought on what constitutes a "NAPP Partnership project" - ambiguities which are alleged to have contributed to project proposals being approved which were not projects at all, but rather operational expenses. The problem highlighted the need to update and bring greater clarity to the Eligible Expenditure Guidelines, which in 2010 had not been revised since the days of ERI.

Though the Eligible Expenditure Guidelines were subsequently updated, some Partners have conveyed the view that the current eligibility criteria and the current system of project proposal review are onerous and time consuming. Missions, through the Mission Survey, similarly conveyed some frustration with the length of time it takes to secure project approval, which can lead to implementation challenges in the latter half of the FY. In particular, Partner sign-off on business development project proposals was cited as an irritant, the complaint being: why should a Partner sign-off on a project advanced by another Partner for which the Partner has no involvement or interest?

While the process of subjecting business development projects to peer review may be cumbersome, the practice does have the benefit of helping to keep Partners informed of what other Partners are doing, which was cited by the Partners themselves as one of the virtues of partnership. Further, and notwithstanding complaints relating to the length of time it takes to secure project approval, the NAPP Partnership's record of disbursements, which has resulted in minimal lapses, would suggest that project implementation has not been unduly frustrated by the project selection and approval process in effect.

Finding 33:
While both Partners and missions cited problems with TD and long-term assignment planning, selection and approval processes, these problems have recently been addressed.

Although guidelines for the selection and approval of TDs and long-term assignments have been extant since the inception of the NAPP Partnership, these early guidelines were considered vague and incomplete. For example, though the TD Program Guidelines of 2009 speak to the roles and responsibilities of Partners sponsoring a TD at mission, the roles and responsibilities of the mission host, apart from the provision of administrative and logistic support, were almost entirely neglected. This omission likely contributed to the perception, confirmed in the results of the Mission Survey, that TDs were "imposed" on missions. Nor did the TD Program Guidelines then in effect document clear timelines for the submission of proposals, consultations, selection and approval. While missions were required to approve the objectives of the TD proposal, the HR Coordinator at DFAIT was granted full authority to approve TDs and related funding. The guidelines relating to long-term assignments were reported to be similarly vague and incomplete.

The TD Program Guidelines have since been updated, and now clearly affirm the role of the missions as an originator, not just a recipient, of TD proposals. Each phase of planning, selection, approval and deployment process is now well documented with clearly defined timelines (6 weeks). The candidate review and approval process has been completely overhauled, vesting the authority to review and approve proposals, formerly delegated to a single individual, to an HR Committee, composed of Partner representatives. The review and approval of long-term assignments is likewise now conducted by an Inter-Partner Selection Committee. Though some have remarked that this new structure is cumbersome and time consuming, owing to the multiple sign-offs, most acknowledge that it supports greater transparency and fairness.

Communications and Coordination

Finding 34:
Infrastructure in support of communications remains underdeveloped, and though coordination is reported to be good, there is still room for improvement.

Although an Extranet site was created to post and share information of common interest to Partners and other stakeholders, the site remained for some time underdeveloped and largely unpopulated with current and, from the Partners' point of view, useful data. Partners reported having rarely visited the site for the reasons stated above. The Virtual Secretariat also created a Wiki site to serve as an electronic platform to share information and ideas, but again the Wiki was found to be underdeveloped and underused and is, at any rate, unavailable to Partners.

The existence of a robust communications infrastructure and program is an important enabler, particularly for a horizontal program such as the NAPP Partnership, and it is surmised that at least part of the negative perceptions of the Partners on work and value of the Virtual Secretariat can be attributed to the absence of a robust communications infrastructure and program. Reasons for challenges in developing this capacity relate to inadequate resources assigned to the task and difficulties in securing access to program data from all the Partners in an intelligible format - see Finding 35.

Further, though structures (the governance committees) and procedures (project review and selection) are in place to support coordination, there is a general perception among the Partners and missions that coordination is not working as well as it should. Both personnel at HQ and at missions, for example, remarked that there was very little interface between the business development and advocacy programs. Though arguably commercial advocacy, which falls within the mandate of the business development program, and political advocacy, which falls within the mandate of the advocacy program, are two separate things, there is a need for the latter to be aligned with the former if the two are to be truly complementary and mutually supportive, which in turn suggests a need for greater communication and coordination of activities between these two programs.

As previously stated, though all Partner projects are vetted through the Virtual Secretariat, thereby providing the same with an opportunity to both identify potential areas for collaboration and to mitigate the risk of duplication, both Partners and missions remarked that there had been lapses in this regard, though without reference to specific cases. This perception is perhaps due to the lack of Partner access to CEP Plans - see Finding 32 - and the fact that advocacy projects under $50 thousand are not subject to peer review - see Finding 27 - which makes it incumbent on the business development program (GNC) and the advocacy program (GNB) to alert Partners of potential areas of both overlap and collaboration. Reference has been made to the apparent neglect of ROs - see Finding 26 - which is reputed to have led to some missed opportunities and ROs sponsoring projects that were, unbeknownst to them, NAPP Partnership projects - see section 9.0 Economic Action Plan 2012.

STCs and FPDS managers were asked in the Mission Survey about their level of satisfaction with respect to communications and coordination. With regards to facilitating communications between missions, Partners, ROs and other missions, most STC (56%) and FPDS (64%) respondents reported to be only "somewhat satisfied" with communications. Satisfaction with respect to the level of coordination between missions, Partners, ROs and other missions was as follows: 56% of STCs and 74% of FPDSs reporting to be "somewhat satisfied" with coordination.

Information Management

Finding 35:
Information management systems remain undeveloped and balkanized which has contributed to challenges in consolidating and sharing information of interest to Partners.

Under the NAPP Partnership, program information is dispersed not only among Partners and missions, but also between different divisions within DFAIT - in part a function of the Virtual Secretariat model - that has frustrated both access to and consolidation of this information. Moreover, the information available was not found to be configured in such a way as to support reporting. As remarked in section 6.0 of this report, the evaluators had to construct databases not only for the purpose of reporting on results, but also to obtain an understanding of how NAPP Partnership funds have been spent. Concurrent with the conduct of this evaluation, the Virtual Secretariat was in the process of finalizing its three year "Report on the NAPP Partnership." Consultations with Secretariat staff charged with preparing the said report similarly encountered challenges in collecting and collating project data (financial and descriptive) to populate the report with program information. This challenge was particularly acute with respect to business development projects.

Sound information management begins with clear direction as to what information is required and how it should be organized. With this in place, it is then incumbent on the producers of data to supply the same to those who need to know in the prescribed format. During the study period, the evaluators found little evidence of such direction or adoption of any standardized system of information management. Nor did the evaluators find evidence of consistent and timely reporting of data, which highlights an important principle of partnership - that information management is a shared responsibility. Unless Partners and their respective managers accept certain standardized information management and reporting protocols, and respect these, the Virtual Secretariat will remain challenged in producing the kind of data desired by the Partners.

Performance Measurement

Finding 36:
Over the study period the NAPP Partnership had no common Performance Measurement Framework in operation. Nor were performance measurement systems at DFAIT configured to populate such a framework with performance data.

As remarked earlier in this report, though a Performance Measurement Framework (PMF) was developed for the NAPP ($20 million), which included elements pertinent to the Partnership, the PMF was never adopted by the Partners, and thus never populated with data. In the absence of a common PMF, each Partner was left to collect performance data from their respective systems and report on the same against their respective PFMs, which were all different. Despite the absence of a common performance reporting framework, efforts were made during the study period to strengthen performance reporting. For example, the Virtual Secretariat proposed the adoption of a directive that would see formal reporting on all projects over $5 thousand. Partners were further encouraged to adopt the annual performance reporting format developed by WED, which was seen as something of a model for results reporting. Unfortunately, neither initiative garnered much support from the Partners. Reporting on projects over $5 thousand was sporadic, and there was no uptake on the proposal to adopt WED's annual performance report model.

Given the above, the evaluators were obliged to rely on the performance measurement systems in effect during the study period which, as remarked in section 6.0 of this report, were not designed to facilitate performance reporting on the NAPP Partnership. Within DFAIT, systems for reporting on business development performance (CEP reports) do not report against projects or even programs, but sectors wherein results are aggregated, rendering it extremely difficult to attribute the same to a NAPP Partnership investment. Nor is TRIO designed to attribute results to a project or program. The evaluators had to identify the mission and date of a NAPP Partnership sponsored event and then match this with the entry date of data in TRIO in order to make the fragile inference that the TRIO data likely related to the NAPP Partnership event.

Performance data on advocacy proved equally challenging and problematic. Data entered in ARC did not relate specifically to the NAPP, thereby rendering attribution nearly impossible. While the PMTT was designed to track activities related to NAPP Partnership advocacy programming, its life was short lived and yielded little information of utility to this evaluation. In response, the evaluators had to undertake a word search of narrative reports on initiatives in order to gather information on the performance of advocacy investments, which were largely bereft of quantitative data. This exercise produced mixed results. In short, the NAPP Partnership had neither a framework to measure nor a means to report on performance during the study period.

Notwithstanding the foregoing, the evaluators note several improvements to the performance measurement systems within DFAIT which, though instituted too late to yield data of use to this evaluation, stand to strengthen results reporting. For example, in the spring of 2011 DFAIT introduced new mandatory performance indicators, the Economic Outcomes Facilitated (EOF) and Opportunities Pursued (OP), for the trade services. Both EOF and OP data is to be recorded in TRIO. Though there would still be difficulties in attributing this data to the NAPP Partnership, the challenge could be addressed by applying a program code to the entries, which would allow for easy data recovery. The technical feasibility of this approach would have to be explored with the Performance Measurement and Strategic Trade Planning Division (PDC), which has carriage of TRIO.

Additionally, the advocacy program adopted in June 2011 the Mission Advocacy Activity Tracker (MAAT) to replace the ARC and PMTT. The MATT marks a significant improvement over its predecessors, capturing both qualitative and quantitative information on performance. Further, new guidelines and templates have been developed to strengthen performance and results reporting. The advocacy program has also committed to producing an annual Advocacy Year End Report for the US and Mexico using data gleaned from the MAAT, which is expected to be shared with Partners. Being able to demonstrate the results from advocacy activities may in future make it easier to convince existing and prospective partners of the merits of investing resources in advocacy.

Efficiency and Value for Money

Finding 37:
Though there is a common perception among the Partners that the Partnership is "governance heavy" and therefore costly to administer, the costs are not out of proportion with the value of the Program. Further, judging from the NAPP Partnership's contribution to increased Canadian exports, the NAPP Partnership is good value for money.

All Partners remarked that the existing governance structure with its varied committees and complex approval procedures is administratively heavy, with the inference being that it is inefficient. However, these structures and procedures were set up to support Partner participation in the decision making process in compliance with the spirit of the GFA. All horizontal initiatives that involve a plurality of government departments and agencies present unique administrative challenges, and the NAPP Partnership is no exception.

As remarked in Finding 30, allocations to the Virtual Secretariat over the study period amounted to roughly 10% of the total budget, which is high. The reasons therefore can in part be explained by the fact that resources committed to the Virtual Secretariat were used to cover the full salaries of certain DFAIT personnel, even though some DFAIT staff devoted only a portion of their time to supporting the NAPP Partnership, thereby inflating the real costs associated with program administration. This said the breakdown of allocations to personnel has been adjusted and is now reported to be around 6% of the total program budget which is wholly reasonable for a program of the NAPP Partnership's size.

Partners further remarked that the HR program was "governance heavy" and therefore costly to administer. As with other selection and approval processes associated with the NAPP Partnership, those relating to the selection and approval of candidates for training, TD and long term assignments are intended to be inclusive, transparent and fair. Respecting these principles, in particular the principle of inclusiveness in decision making, can impose additional administrative burdens. Conversely, any compromise of these principles in the interest of expediency and cost efficiency would expose the Partnership to charges of being non-inclusive, nontransparent, and unfair.

Although Partners undertake their own pre-screening of candidates for TD and long -term assignments, the number of candidates far exceed the number of positions available, resulting in large pools of qualified candidates whose credentials must be reviewed and their persons interviewed. Arguably, the process could be expedited without compromising the aforementioned principles were the Partners to apply more exacting criteria for pre-selection which would reduce the number of qualified candidates for second tier examination.

The cost associated with having to go to Ottawa to obtain training was cited by Partners as an issue. Costs could be reduced were courses to be made available on-line, but on-line training is a poor substitute for class room instruction. Moreover, the provision of alternative modes of delivery is beyond the control of the Partnership. Cost savings may be realized were the training to be provided in the regions.

In sum, both perceived and real inefficiencies relating to the administration of the NAPP Partnership must be weighed against the declared efficiencies generated from partnership: the ability to leverage the resources of other programs and Partners to maximize impact, institutional capacity building facilitated through the HR program, avoidance of duplication of effort through information sharing and collaboration, etc. These aspects of partnership, which are cited by the Partners as the benefits of partnership, constitute evidence of real efficiencies that would be lost were the Partnership not to exist. Further, when one considers the overall return on investment, as demonstrated by the increase in Canadian exports to the US and Mexico, the NAPP Partnership provides good value for money.

Finding 38:
The plurality of funding instruments and application processes associated therewith imposes an administrative burden on missions.

While funds made available to missions through the NAPP Partnership are almost invariably used in conjunction with other funding sources, which is cited as a benefit, each of these other funding sources have their own eligibility criteria and application processes which effectively increases the administrative burden on missions tasked with staging an event.

Within the NAPP Partnership business development program there are two funding sources (IBD and TPI) which have two separate application procedures. TPI has a distinct focus on technology commercialization and investment while IBD is more inclusive of other sectors. The review of the project sample found instances of projects funded by both IBD and TPI. Some STCs at mission have conveyed a preference to have these, and potentially other funding sources, consolidated to reduce the administrative burden of having to apply for funding through multiple channels.

9.0 Economic Action Plan 2012

As remarked in section 4 of this report, at the end of the study period federal government departments and agencies, including those in the NAPP Partnership, embarked on a process of cost cutting to contribute to deficit reduction. In the case of DFAIT, this translated into the closure of a number of missions in the US as well as the closure of several ROs within Canada, which reportedly will no longer be engaged in programming, thereby reducing the capacity of the mission network within North America, which will undoubtedly impact on the level and type of NAPP Partnership programming in future. Partners themselves, in the face of reduced budgets, are reviewing their priorities relative to their core mandates, and adjusting allocations in support of international business development and advocacy accordingly. For example, DFAIT in the spring of 2012 announced that its contribution to the Partnership would be reduced. In short, in the current climate of austerity, it would not be unreasonable to project that the NAPP Partnership will not have the same level of resources available to it as it has enjoyed in the past and that this will impact on the future of the Partnership.

10.0 Conclusion and Recommendations

10.1 Conclusion

This evaluation found that the concept of partnership within the North American context has merit. Additionally, this evaluation found the NAPP Partnership in design, including the pooled funding model, works and contributes to the achievement of results, evidenced by the increase in exports to the US and Mexico, progress in addressing several advocacy issues of import to Canada, and increased capacity of the Partners through their participation in the HR program.

With Partners conveying interest in programming outside the North American market, questions have been raised about whether the NAPP Partnership model is one that could be applied to other markets? In responding to this question, it is important to keep in mind that the NAPP Partnership works because of a convergence of a number of factors (a plurality of sectors of interest to Canadian business, an alignment of Canadian capacity with commercial opportunities, evidence of a pan-Canadian stake in the market, evidence of a pan-governmental stake in the market, existence of an extensive mission network to support program delivery) which creates the requisite "critical mass" to support a formal partnership arrangement. While these factors exist within North America, thereby providing evidence of the continued relevance of the NAPP Partnership as a vehicle to advancing Canadian commercial interests in the same, they may not exist to the same extent in other markets.

Despite the merits of the model, the NAPP Partnership has not realized its full potential. The Partnership has been unable to elicit the participation of other important OGDs which has resulted in gaps in regional representation and a membership structure wherein few existing members have a significant commitment to advocacy. Despite the existence of structures to support the horizontal governance and management of the Partnership, there are residual perceptions among non-DFAIT Partners that decision making has not been fully consensus based or transparent. Nor have these governance bodies provided the kind of strategic guidance needed for targeted programming.

While this evaluation concludes that the Virtual Secretariat does provide valuable service, and at a cost that is not out of proportion with the value of the program, its operations have not been wholly transparent, it has contributed to a fractured administrative structure with confused accountabilities, and has been challenged in realizing several key deliverables. For example, program level SOPs have not been prepared and information management systems remain underdeveloped. The Virtual Secretariat has struggled to maintain an effective communications service/program, and it has not produced an operable Performance Measurement Framework or developed systems in support thereof, which has left the NAPP Partnership handicapped in its ability to report on performance and results. On balance, however, the Partnership and its resources have been well managed.

10.2 Recommendations

Based on the foregoing observations and analysis, the evaluation recommends the following to improve the relevance and performance of the Partnership.

Recommendation 1:
That consultation with Partners and prospective partners commence as soon as possible to renew a North American Partnership Program.

While this evaluation concludes that the NAPP Partnership remains relevant, and therefore worthy of continued support, the evaluation also recognizes that its continued relevance will depend on a different kind of partnership, one that is more accommodating of the diverse interests and priorities of both existing and prospective partners. Such a partnership will invariably involve a different contribution and allocation formula.

Recommendation 2:
That a renewed NAPP Partnership establish clear roles, responsibilities and accountabilities of the Partners and governance bodies in order to enhance transparency, accountability and performance.

The renewal of the GFA presents an opportunity to revisit and clarify the roles, responsibilities and accountabilities of the partners and governance bodies, which as remarked in this report have been hitherto fluid and ambiguous. This will include, among other things, developing and formally adopting new terms of reference for the governance bodies, as well as new Standard Operating Procedures (SOPs).

Recommendation 3:
That the Partnership establish a Secretariat to meet service requirements in support of a fully effective Partnership in consideration of costs and resource requirements.

A renewal of the GFA will also present an opportunity to clarify the role, responsibility, and accountability of a Secretariat. While service requirements will be determined though consultation and negotiation with partners and stakeholders, it is envisaged that the Secretariat would, among other things, likely assume the lead in developing SOPs, in managing a communications service/program, and developing and instituting a Performance Measurement Framework.

Recommendation 4:
That the Partnership and Partners commit to improving the existing performance measurement systems to ensure that the collection, consolidation and analysis of NAPP related performance data is used to support decision making by management.

While this evaluation concludes that the NAPP Partnership does produce results aligned with the interests and priorities of the Partners and the GoC, the Partnership has not developed either a framework or systems in support thereof to demonstrate results, thereby casting doubt on the value of the program. Partners must agree on a common Performance Measurement Framework, and adhere to reporting requirements. Further, these efforts should take into account DFAIT performance measurement systems given the high degree of integration of NAPP business promotion programming with DFAIT trade operations. Consultation with data providers should be undertaken in the development of the Performance Measurement Framework to assure that the data is available and in a format that can allow for attribution.

11.0 Management Response and Action Plan

Recommendation 1:

That consultation with Partners and prospective partners commence as soon as possible to renew a North American Partnership Program.

While this evaluation concludes that the NAPP Partnership remains relevant, and therefore worthy of continued support, the evaluation also recognizes that its continued relevance will depend on a different kind of partnership, one that is more accommodating of the diverse interests and priorities of both existing and prospective partners. Such a partnership will invariably involve a different contribution and allocation formula.
Associated Findings: 4, 5, 7, 11 and 16

Management Response & Action Plan: The North America Branch has developed a proposed model for the renewal of the NAPP partnership which remedies a number of concerns that have been raised in the Summative Evaluation.

Furthermore, the branch has begun to engage and consult with existing NAPP partners on the proposed model at both the DG and ADM level. Over the next few weeks, the proposed model will continue to evolve to reflect ongoing discussions with current and prospective partners

A formal written proposal will be sent to existing and prospective new partners in the next few weeks with a view to establishing a renewed and hopefully expanded partnership by April 2013.
Responsibility Centre: GNB, GNC, GND
Time Frame: Immediate

Recommendation 2:

That a renewed NAPP Partnership establish clear roles, responsibilities and accountabilities of the Partners and governance bodies in order to enhance transparency, accountability and performance.

The renewal of the GFA presents an opportunity to revisit and clarify the roles, responsibilities and accountabilities of the partners and governance bodies, which as remarked in this report have been hitherto fluid and ambiguous. This will include, among other things, developing and formally adopting new terms of reference for the governance bodies, as well as new Standard Operating Procedures (SOPs).
Associated Findings: 26, 29, and 35

Management Response & Action Plan: Redefined roles, responsibilities and accountabilities of partners and governance bodies will be the outcome of negotiations with partners, the principles of which will find expression in a renewed GFA. With these principles established, the Bureau will prepare detailed Terms of Reference for each of the governance bodies proposed along with Standard Operating Procedures for all partners.

To ensure strategic engagement between partners, the North America Branch will organise timely strategic discussion on specific themes or sectors involving HOMs, senior officials, OGDs and as appropriate external experts.

This will also provide greater opportunities for NAPP partners to guide the programming in North America on an ongoing basis.

The renewed NAPP will also bring about a streamlined approval process for partner projects, assorted with clear accountability requirements.
Responsibility Centre: GND
Time Frame: Upon renewal of a partnership

Recommendation 3:

That the Partnership establish a Secretariat to meet service requirements in support of a fully effective Partnership in consideration of costs and resource requirements.

A renewal of the GFA will also present an opportunity to clarify the role, responsibility, and accountability of a Secretariat. While service requirements will be determined though consultation and negotiation with partners and stakeholders, it is envisaged that the Secretariat would, among other things, likely assume the lead in developing SOPs, in managing a communications service/program, and developing and instituting a Performance Measurement Framework.
Associated Findings: 29, 32, 35, and 36

Management Response & Action Plan: Included in the proposed model is the establishment of a low and fixed cost, dedicated secretariat comprising of a Senior Coordinator, Program Analyst, Financial Analyst and Partner Liaison officer.

The secretariat will be responsible for providing support to the governance structure including the NAPP Executive Coordinator, communicating and liaising with NAPP partners, managing the human resources program, financial allocation and reporting, performance measurement, and corporate reporting including annual reports. Further, the secretariat would aim to facilitate collaboration and increase synergies between partners' through enhanced transparency, networking and showcasing models of collaboration.

It is important to note that the roles and responsibilities ascribed to the proposed secretariat may evolve as engagement and consultations with existing and potential NAPP partners continue to ensure an adequate level of service.
Responsibility Centre: GND
Time Frame: December 2012 - August 2013

Recommendation 4:

That the Partnership and Partners commit to improving the existing performance measurement systems to ensure that the collection, consolidation and analysis of NAPP related performance data is used to support decision making by management.

While this evaluation concludes that the NAPP Partnership does produce results aligned with the interests and priorities of the Partners and the GoC, the Partnership has not developed either a framework or systems in support thereof to demonstrate results, thereby casting doubt on the value of the program. Partners must agree on a common Performance Measurement Framework, and adhere to reporting requirements. Further, these efforts should take into account DFAIT performance measurement systems given the high degree of integration of NAPP business promotion programming with DFAIT trade operations. Consultation with data providers should be undertaken in the development of the Performance Measurement Framework to assure that the data is available and in a format that can allow for attribution.
Associated Findings: 36

Management Response & Action Plan: While the evaluation covered the years 2008-2011, significant improvements have since been made to performance measurement. For instance, in 2011 the Mission Advocacy Activity Tracker (MAAT) was created to help us better track our advocacy efforts and results in North America. In addition, our international business development efforts in the U.S. are tracked by TRIO and Client Surveys, as it is for the rest of the world. These tools capture results for all programming funds, including the NAPP, which accounts for close to 75% of programming funds in North America. We have also developed a tool to track partner projects and how NAPP funds are disbursed and spent.

It is also important to note that the initial work on developing a performance measurement framework has already taken place and will need to be further refined in consultation with existing and prospective partners. Efforts to develop and implement a performance management system for a renewed partnership will build on the work that has already been done.

GND has recently identified an individual to assume the role that will be responsible for performance measurement of a renewed partnership and will commence in December 2012. The individual would be responsible for consulting with partners to finalize and implement a rigorous performance measurement system that would allow for the effective collection, consolidation and analysis of performance data that would ably support corporate reporting and decisions made by management.
Responsibility Centre: GND
Time Frame: December 2012 - December 2013