Evaluation of the Client Service Fund and the Integrative Trade Strategy Fund

Final Report
Prepared by the Diplomacy, Trade and Corporate Evaluation Division (PRE)
Strategic Policy Branch (PFM)
Global Affairs Canada
May 2017

Table of Contents

• Acknowledgements
• Executive Summary
• Acronyms and Abbreviations
• Program Background
• Program Resources
• Evaluation Scope
• Methodology
• Findings
• Conclusions
• Recommendations
• Management Response and Action Plan (MRAP)

Acknowledgements

PRE would like to thank the Evaluation Advisory Committee (EAC) and members of the following divisions and bureaus who provided their support and contributions to this evaluation: Sector Practices (BBD) and Trade Commissioner Service – Operations (BTD) bureaus; Geographic, Functional, Financial Management, Mission Support, and Trade and Economic Analysis divisions; and Senior Trade Commissioners (STCs) and Trade Commissioners (TCs) at missions abroad and Regional Offices (ROs) in Canada.

Executive Summary

The Trade Commissioner Service (TCS) relies on the CSF and ITSF to fund strategic initiatives, projects and services that help Canadian businesses to participate and expand into global markets. As a proportion of total trade budgets, the CSF and ITSF represent an increasingly important source of funding to Commercial-Economic sections due to a decline in funding from Other Government Departments and partners.

The ITSF serves as a highly targeted and strategic source of funding aligned with the Sector Analysis and Priorities for the TCS, while the CSF continues to be a discretionary source of funding that allows the TCS to respond to local, market-driven opportunities and client demands. Access to discretionary CSF and targeted ITSF funding enhances TCS capacity to serve Canadian businesses in their efforts to expand into global markets.

Trade initiatives funded in part through the CSF and ITSF support the advancement of Departmental international business development objectives by contributing to the expansion of Canadian business participation in global markets, increasing interactions between Canadian clients and global business partners, and generating new business opportunities and results for Canadian clients.

Strategic guidance provided by the Trade Sectors Bureau for the ITSF was found to be clear for all of the TCS’s priority sectors, however, greater clarity and consistency in the provision of operational guidance across all sectors is needed. Further, stakeholders at missions and headquarters expressed a need for greater access to hospitality to support CSF and ITSF- funded initiatives since hospitality is considered an essential cost of doing business.

The CSF and ITSF-funding mechanisms are well-managed and have effective governance structures that provide oversight and support the achievement of results through sound project planning and implementation. The financial monitoring and reallocation systems in place are highly effective and contribute to low lapse rates. However, uncertainty of the ITSF budget and timing of ITSF disbursements to posts was found to adversely impact on program delivery, and the transaction costs of administering the ITSF was reported to be high at missions, headquarters and Common Service Delivery Points.

Finally, the database review demonstrated a lack of interoperability between CSF and ITSF planning (Strategia) and reporting (TRIO2) systems which increases the administrative burden on posts, introduces a risk that data will be entered inaccurately, and limits effective performance measurement.

Acronyms and Abbreviations

B2B

Business-to-Business

BBD

Trade Sectors

BBH

Clean Technology, Infrastructure and Life Sciences

BBI

Multi-Sectors Practices

BBR

Aerospace, Automotive, Defence and ICT Practices

BFM

International Business Development and Innovation Branch (Chief Trade Commissioner)

BI

Business Intelligence

BIS

Investor Services

BTA

Trade Planning, Coordination and Responsible Business Practices Division

BTB

Systems and Analysis

BTD

Trade Commissioner Service - Operations

CCIL

Canadian Commercial Interests List

CDIA

Canadian Direct Investment Abroad

C-E

Commercial Economic (section)

CEP

Commercial Economic Plan

CFIA

Canadian Food Inspection Agency

CME

Canadian Manufacturers and Exporters

CO

Fund Commitment

CRM

Client Relationship Management

CSDP

Common Service Delivery Point

CSF

Client Service Fund

CSR

Corporate Social Responsibility

CTA

Canadian Technology Accelerator

EOF

Economic Outcome Facilitated

FPDS

Foreign Policy and Diplomacy Service

FY

Fiscal Year

GAC

Global Affairs Canada

GC

Government of Canada

GVC

Global Value Chains

HOM

Head of Mission

HQ

Headquarters

HRMS

Human Resource Management System

IBD

International Business Development

ICCB

International Commerce Coordination Board

ICT

Information and Communication Technologies

IMS

Integrated Management System

ITSF

Integrative Trade Strategy Fund

KPI

Key Performance Indicator

MNE

Multi-National Enterprise

MRAP

Management Response Action Plan

NAPP

North American Partnership Program

OGD

Other Government Department

OECD

Organisation for Economic Co-operation and evelopment

OP

Operational budget

OP

Opportunity Pursued (KPI)

OPI

Opportunity Identified

OPR

Opportunity Referred

PIF

Post Initiative Fund

Q1 (2,3,4)

First Quarter (second, third, fourth)

RO

Regional Office

ROI

Return on Investment

SME

Small and Medium-Sized Enterprise

SPA

Sector Practice Approach

STC

Senior Trade Commissioner

TC 

Trade Commissioner

TCA

Trade Commissioner Assistant

TCS

Trade Commissioner Service

Program Background

The CSF was launched in April 2002 as the Department’s principal fund for supporting proactive initiatives undertaken by the Trade Commissioner Service (TCS) to assist Canadian industry to compete globally and influence foreign decision-makers to select Canada as a source of goods and services.

On April 1, 2009, the Integrative Trade Strategy (ITS) was introduced whereby 50% of the total CSF allocation was devoted to funding strategic initiatives aligned with the Department’s priority markets and sectors through a competitive application process.

In 2013-2014, the Global Value Chains (GVC) Fund and the Canadian Direct Investment Abroad (CDIA) Fund were amalgamated under the ITS, establishing a single, larger fund known as the Integrative Trade Strategy Fund (ITSF).

The Corporate Social Responsibility (CSR) and Edu-Canada funds are allocated and tracked through the same processes as ITSF funds, but are administered separately from the ITSF funding envelope.

CSF and ITSF Governance Structure

Program Resources

The CSF began with annual funding levels of $8.7M in fiscal year (FY) 2003-2004, which was reduced to $6.3M in 2004-2005, and $5.9M per year from 2005-2006 to 2007-2008. Since 2008-2009, the yearly base level budget has been maintained at $4.9M, divided between the CSF and ITSF components. Due to departmental priorities and/or supplemental injections of funding, actual annual budgets have fluctuated from $4.76M in 2013-2014 to $5.88M in 2015-2016.

Geographic Disbursement of CSF and ITSF Funds

Each year, CSF and ITSF funds are disbursed to approximately 140 missions abroad and Regional Offices (ROs) in Canada.

Evaluation Scope

The focus of the evaluation is on the relevance, performance and efficiency of the CSF and ITSF, as well as stakeholder perceptions on the funds’ usefulness. The Corporate Social Responsibility (CSR) and EduCanada programs were not part of the evaluation. EduCanada will be subject to a separate evaluation (expected in 2018), and CSR was recently evaluated in 2014.

The evaluation reference period is from 2013-2014 to present, to reflect recent CSF and ITSF practices and processes.

The evaluation objectives were to assess the following:

1. The continued relevance of the CSF and ITSF to the TCS community in support of Canadian business clients.
2. The extent to which CSF and ITSF have contributed to departmental international business development objectives and results for Canadian businesses.
3. The efficiency and effectiveness of the governance structure, program practices and processes.

Methodology

The evaluation utilized a mixed method approach that consisted of quantitative and qualitative data collected using the methods and data sources described below to respond to the evaluation questions. Data was collected between October 2016 and February 2017.

Findings

Relevance

Finding 1: The CSF and ITSF represent an increasingly important source of program funding to the TCS.

As illustrated in Figure 1, the CSF and ITSF represent an increasing proportion of the total funds available to the TCS due to a decline in external funding from other government departments (OGDs) and other partners^{Footnote *}.

Figure 1. Mission Trade Budgets by Funding Type, 2013-2014 to 2015-2016

As illustrated in Figure 2, since 2013-2014, total mission trade budgets (including CSF, ITSF, operational budgets (OPs), other GAC programs, and funding from OGDs and other partners) have declined by $6M or 25%. With increasingly limited program funds, particularly for travel and hospitality, STC and TC survey respondents said that without the CSF and ITSF, their ability to organize events or travel to support Canadian client’s needs would be limited.

Figure 2. Total Mission Trade Budgets, 2013-2014 to 2015-2016

Survey respondents strongly agreed that the CSF and ITSF represent a significant and meaningful contribution to the available resources of the Commercial-Economic (C-E) section at their post:

• For the CSF, 96% of STC and TC respondents agreed or strongly agreed with this statement, and 84.6% agreed or strongly agreed for the ITSF.
• Highest levels of agreement were recorded in Europe (92%), Asia (92%), the Middle East (96%), South America and the Caribbean (100%). Lowest levels of agreement were recorded in Africa (71%)

Interview data: 100% of stakeholders interviewed agreed that the CSF and ITSF are vital sources of funding to the TCS.

Finding 2: Access to discretionary and targeted funding supports TCS capacity to serve Canadian businesses in their efforts to participate in global markets.

STCs, TCs, and stakeholders affirmed the advantages of each fund and the niche that they occupy:

The ITSF

• Targeted and strategic source of funding aligned with the Sector Analysis and Priorities for the TCS
• Consultations between the Sector Bureau (BBD) and Private Sector Advisory Groups, OGD partners, industry and other key stakeholders inform the Sector Practice Strategies and help the TCS to align proposals with Canadian capabilities and interests
• Competitive nature encourages more strategic and focused initiatives
• Costlier activities that may not otherwise be possible with limited C-E budgets, i.e. supporting medium to large-scale delegations to key, regional, multi-post events (e.g. Farnborough Air Show) which are crucial to building the TCS network and promoting Canadian capabilities
• Customized business-to-business (B2B) activities are encouraged to bring Canadian clients and local buyers into direct contact

The CSF

• Discretionary source of funding that allows the TCS to respond to local, market-driven opportunities and client demands
• Stable source of funds outside of the operational budget (OPs)
• Allows for cross-sectoral programming and emerging sectors, or sectors not covered by the ITSF (e.g. Agriculture / Agri-food)
• Complements and covers deficits from other sources of funding and foreign exchange deficiencies
• Provides STCs with the flexibility to respond when changes to initiatives occur and to emerging, mid-year demands

Achievement of expected outcomes

Finding 3: The CSF and ITSF support Government of Canada International Business Development (IBD) objectives.

CSF and ITSF contributions to the expansion of Canadian business participation in global markets:

The Report on Plans and Priorities reported that in 2015-2016 the TCS supported Canadian companies by conducting 1,220 trade initiatives funded through the CSF and ITSF (and EduCanada) which contributed to 963 commercial agreements concluded as compared to 934 commercial agreements concluded in 2014-2015.

Research conducted by the Organisation for Economic Co-operation and Development (OECD)^{Footnote 1} and Canadian Manufacturers & Exporters (CME)^{Footnote 2} indicate that the types of initiatives and activities funded by programs such as the CSF and ITSF are aligned with the needs of Canadian Small and Medium-Sized Enterprises (SMEs).

The CSF and ITSF were highly rated by STCs and TCs in terms of their effectiveness at contributing to the following objectives:

• Contribute to the expansion of Canadian business participation in global markets by funding strategic initiatives, projects and services within the scope of the TCS.
• Increase interaction and strategic alliances between Canadian clients and global business partners.
• Generate new business opportunities, leads and results for Canadian clients.
• Provide detailed market information and intelligence to specific Canadian clients.
• Support strategic initiatives / partnerships as defined in the commercial plans of C-E sections.

However, the CSF and ITSF received lower ratings in terms of their effectiveness at contributing to the following objectives, for the reasons provided below:

• Provide information and advisory resources to TCS clients wishing to pursue foreign investment.
  • May be due to respondents failing to recognize that CDIA is included under the ITSF.
• Expand Canadian business participation in, and awareness of, Multinational Enterprise (MNE) Global Value Chains (GVC).
  • GVC is no longer viewed as a discrete programming domain.
• Promote Canada as a competitive location and partner for investment and innovation.
• Funded under Invest in Canada program, Tier III, and Champions program rather than ITSF.

Overall, the CSF (83%) received a higher ranking compared to the ITSF (76%) in terms of its effectiveness at contributing to CSF and ITSF objectives.

Survey respondents and stakeholders interviewed at HQ identified the following types of initiatives/activities as the most effective at generating outcomes for clients: workshops/seminars featuring Canadian capabilities; matchmaking events; trade missions in market; participation in trade shows; market studies; and reverse trade missions to Canada.

For the CSF specifically, support for core TCS work, such as local travel to conduct outreach and gain contacts, market intelligence and hosting events with local contacts were also noted to be highly effective initiatives/activities.

Finding 4: The project sample demonstrates that the CSF and ITSF contribute to results for Canadian businesses.

Project sample description

The evaluation relied on performance data from a sample of 123 CSF and ITSF-funded initiatives at 8 missions (Beijing, Beirut, Bogota, Guadalajara, London, Nairobi, Tokyo, and Washington). Of this sample, 79 (65%) initiatives were traceable to KPIs in TRIO2.

Based on mission expenditure data in Strategia, the project sample represents 80% of the ITSF funds ($343.8K) and 53% of CSF funds ($213.3K) allocated to these missions in 2014-2015. The sample spans across different sectors including Aerospace, Automotive, Clean Technology, Information and Communication Technologies (ICT), Infrastructure and Life Sciences. Key events in the sample include: Farnborough Air Show, Mobile World Congress, and Game Developers Conference.

Other sources of funding comprised 60% ($880K) of the total funding for the project sample, with federal and provincial partners contributing the highest share (31%).

KPI data reported in this finding includes KPIs achieved up to January 2017.

As noted in the Methodology section, performance data must be regarded with caution for the following reasons:

• A representative sample was not possible, therefore, the results of the evaluation project sample are indicative of the sample only and do not represent the total population. Different results may be observed among a different sample of projects.
• Within the project sample, all KPIs may not be captured.
• The CSF and ITSF initiatives in the project sample were frequently jointly funded, or funded in conjunction with other sources of funding including OPs, other internal GAC funding, OGD, provincial, private industry & associations, and/or foreign governments, which complicated outcomes attribution.

KPIs generated by funding source:

The project sample was linked to a total of 1,537 KPIs including: 847 services delivered, 493 outcalls, 164 opportunities identified/referred (OPI & OPR), 22 opportunities pursued (OP) and 11 economic outcomes facilitated (EOF).

Key observations from the project sample based on average number of KPIs generated by funding source type (see Table 2):

• Cost effectiveness: On average, the CSF used alone or in conjunction with other funding sources, was more cost-effective in terms of the amount spent on the initiative vs. the number of KPIs generated ($666/KPI).
• Highest average number of KPIs: On average, the CSF used alone or in conjunction with other sources of funding generated a slightly higher average number of KPIs per initiative (20.6) including the highest average number of outcalls per initiative (8.2) and the highest number of opportunities identified/referred per initiative (2.5).
• Success KPIs: Both the CSF and ITSF (used alone or in conjunction with other sources of funding) resulted in approximately the same average number of successes (OPs and EOFs combined) per initiative (0.4-0.5/initiative).

Key observation on leverage (not shown in Table 2): Key events, funded primarily through the ITSF, leveraged the greatest amount of funding from external partners ($423.9K in total). Funds were primarily leveraged from federal or provincial partners (e.g. $250K from Atlantic Canada Opportunities Agency and Industry Canada to support the Farnborough Air Show and $93K from the Government of Alberta for the 2014 Expo Oil and Gas Congress & Exhibition).

Key observations from the project sample based on KPIs generated by type of activity/initiative (see Table 3):

• Outreach, events and core services to clients generated the highest average number of KPIs per initiative (24.7), but at the highest average cost per KPI ($1,289).
• On average, incoming and outgoing trade missions generated the highest average number of OPs (0.6) and EOFs (0.4) per initiative.
• Trade missions were the most cost effective in terms of the average amount spent per KPI generated ($341/KPI). This was followed by seminars / workshops featuring Canadian capabilities ($538/KPI).

Additional details on trade missions:

• While there were fewer outgoing trade missions (6) in the sample than incoming missions (9), outgoing trade missions generated on average 1.5 successes per initiative (9 OPs and EOFs) which was more than double the number of successes generated from incoming trade missions (4 OPs and EOFs).

Of the 33 successes (OPs and EOFs) linked to the project sample, 11 success stories reported in TRIO2 contained details on the actual or estimated value of the contract or agreement. The total combined actual/estimated value to clients was greater than $40M.

Client Satisfaction

Beyond the performance data examined above, client satisfaction is also an important indicator of performance. Based on the 144 online client feedback surveys linked to the project sample, the satisfaction rate with the overall quality of TCS service delivery was 86%.

Between the two funds, client satisfaction was 6% higher for CSF-funded initiatives than the ITSF. However, with regards to specific client satisfaction drivers (such as helping clients to overcome barriers to overseas business; improve client profile or credibility; improve client knowledge of the competitive environment in a foreign market; gain access to information or intelligence; connect with customers/partners or other contacts; avoid mistakes, delays or costs; and improve business development strategy), the ITSF scored higher than the CSF.

Efficiency and economy

Finding 5: The CSF and ITSF governance structure provides effective guidance and oversight over the use of the funds.

1. The Steering Committee is a representative and inclusive body that engages the right stakeholder groups in decision making at the appropriate level

Chaired by the Director of BTA, the steering committee includes Director-level representatives (or designated appointees) from the sector and geographic bureaus, the regional network, and functional divisions.

BTA reports annually on the performance of the CSF and ITSF to the ICCB - the ultimate trade governance body.

Geographic representatives interviewed indicated that the Steering Committee provides a forum that supports effective consultation on key CSF and ITSF decisions.

2. Roles and responsibilities are clear

All members contribute to key decisions relating to expense eligibility, how to allocate the CSF reserve, and CSF and ITSF reallocations.

The Geographics determine the distribution of CSF resources among the missions in their regions based on a number of factors and considerations including the Canadian Commercial Interests List.

BBD distributes ITSF funds to missions based on a competitive process supported by strategic guidance provided in part through Sector Practice Strategies and private sector advisory boards.

3. Steering Committee members provide good guidance and oversight in the delivery of CSF and ITSF resources

Steering Committee meetings are held 3 to 4 times per year timed to correspond with key milestones.

CSF and ITSF Guidelines and Sector Practices Strategies and Guidelines provide direction on program objectives, allocations and usage criteria.

Ongoing monitoring ensures the funds are used according to guidelines and expense eligibility criteria, and that unused funds are returned and repurposed.

Successful reallocation of returned funds contributes to high burn rates/low lapse rates.

Finding 6: There is general satisfaction with the eligible activities and expenses in the CSF and ITSF guidelines, however, stakeholders conveyed a need for greater access to hospitality.

77% of STC and TC survey respondents are either satisfied or neutral with the eligible activities and expenses in the CSF and ITSF Guidelines and 23% are dissatisfied.
 
Survey and interview feedback demonstrates dissatisfaction with the following ineligible expenses and activities in the CSF and ITSF Guidelines:

• Lack of access to hospitality funds under the CSF, and limited access under the ITSF
• Overtime expenses for ITSF events
• Limitations to French translation of market reports
• Client travel to market using CSF base funds

Hospitality

Stakeholders agreed that hospitality is an essential cost of doing business, particularly for networking and B2B events and in regions where providing hospitality is the cultural norm (e.g. China, India).

Access to OPs budget for hospitality at missions is not guaranteed or sufficient to support the CSF and ITSF for the following reasons:

• OPs budgets for hospitality are subject to ceilings
• competition for the OPs hospitality budget between C-E sections and other players (Head of Mission (HOM), Foreign Policy Diplomacy Service (FPDS)) for non-trade related expenses
• access to the OPs budget for hospitality requires HOM approval. Some HOMs view OPs hospitality funds solely for local networking and not for larger trade events/activities.

Similar limitations and ceilings on OGD and partner hospitality budgets limits partner’s ability to cost-share.

For these reasons, stakeholders conveyed the view that the CSF and ITSF warrant dedicated hospitality resources. Stakeholders proposed granting missions the discretion to use hospitality within specified limits and conditions as permitted under the following GAC trade programs:

• North American Partnership Program (NAPP) Base and Pooled funds: hospitality is permitted, but cannot be the only cost within a project.
• Export USA/Mexico: hospitality is permitted to a limit of 20% of the total cost of an initiative, and is limited to basic refreshments (i.e. no meals or alcohol).
• EduCanada: hospitality is eligible in support of education promotion activities, however, cost-sharing with mission budgets is strongly advised.

Overtime

Events can take place outside of regular working hours thereby requiring significant amounts of overtime to fully participate. Lack of eligibility for overtime expenses under the ITSF means that missions may have to reconsider participating in certain components of events if they involve overtime which may result in missed opportunities to serve clients.

Finding 7: Strategic guidance provided for the ITSF is clear, however, operational guidance was found to be less clear.

1. Demonstrated strategic and operational guidance provided by BBD

Analysis and sector priorities documents are provided to posts to coincide with the launch of the Strategia commercial planning process to help STCs prepare their ITSF applications. In 2016-2017, new guidance was provided on the creative industries/arts & culture sector, an e-commerce guide for consumer products, and a detailed sub-sector guide on the digital media/video-game industry.

The Trade Sectors Wiki contains tactical guidance, fact sheets and detailed background on all of the TCS’s priority sectors, as well as organizational charts and contact information for all sectors to encourage missions to proactively reach out to the sector practices for guidance on their ITSF proposals.

2. STC and TC perceptions on ITSF strategic guidance

93% of survey respondents are satisfied or neutral with the strategic guidance provided by BBD.

Satisfaction with the strategic guidance provided by BBD appears to have improved since 2014 when only 70% of STC and TC respondents to the 2014 Sector Practice Approach (SPA)^{Footnote 3} Survey agreed or were neutral that the Sector Practices provided timely and meaningful sector intelligence and information.

3. STC and TC perceptions on ITSF operational guidance

Despite noted improvements in project approval rates (from 59% in 2013-2014 to 67% in 2015-2016), survey and interview feedback from STCs and TCs indicates different levels of satisfaction with the operational guidance and advice provided by sector teams and officers in the following areas:

• lack of clear benchmarks and scoring criteria which contributes to a perception that decision making is arbitrary and subjective;
• levels of transparency and feedback provided, in particular for rejected proposals and partial funding, differs among sector officers;
  • 21% of STC and TC survey respondents do not agree that ITSF funding decisions are transparent
• different application of rules concerning hospitality and travel;
• inconsistent approaches among sector teams and officers concerning the repurposing of funds in response to changes to initiatives;
  • some sectors/officers are flexible when changes to initiatives occur (e.g. change of client or location), while other sectors/officers are less flexible and changes require that a post re-compete for funding.

These issues were previously flagged in the 2014 SPA survey.

Finding 8: High burn rates demonstrate efficient financial monitoring and reallocation systems.

1. High burn rates / low lapse rates

As shown in Figure 3, for the last 3 years, CSF and ITSF burn rates were between 95% and 98.8%.

Figure 3: CSF and ITSF Burn Rates, 2013-14 to 2015-16

2. Effective financial monitoring and reallocation systems

BTA and other colleagues at HQ continually monitor the status of CSF and ITSF initiatives, and conduct on-going follow-up with missions and ROs on funds utilization.

For CSF, missions are asked to move uncommitted funds to a hard commitment. For ITSF, BTA committed the funds to hard commitments for all approved initiatives to help reduce the administrative burden on missions, and to better track funding to results.

Two annual re-allocation phases take place to ensure that unused and returned funds are repurposed toward new initiatives.

Phase 1 in Q3 (October - December)
Phase 2 in Q4 (January – March)

3. Year-end accountability for all CSF and ITSF expenditures

BTA provides detailed, year-end accounts of all expenditures on the departmental Intranet site and presents annual highlights to the ICCB.

Finding 9: Lack of stable funding, and late release of funds to missions, impacts ITSF program delivery.

1. Timeliness of ITSF approvals and disbursements

Survey feedback demonstrates overall satisfaction (70% agree and 25% are neutral) with the timeliness of the ITSF adjudication process (survey question D2(iii)):

However, satisfaction with the timeliness of ITSF allocations to posts is lower (survey question D02(vi)):

2. Lack of stable ITSF budget

Each year, CSF and ITSF budgets are subject to reductions if other BFM branch pressures, such as salary deficits, arise. For example, in 2013-14 and 2014-15, initial CSF and ITSF budgets were reduced by 5% and 10%.

For the ITSF, this resulted in reduced allocations to posts below amounts notionally approved in March. Partial allocations is one of the main reasons cited by posts for cancelled ITSF initiatives and returned funds. 

In addition to reductions at the start of the FY in 2013-14 and 2014-15, in 2016-17, ITSF funds returned by missions were frozen from reallocation by BFM from April to October. Uncertainty regarding the Phase 1 reallocation process impeded planning at HQ and posts.

3. Timing of ITSF disbursements to posts

The CSF and ITSF budgets are generally communicated by BFM to TCS – Operations (BTD) and BBD in April. BTA reconciles the funding decisions on ITSF in February-March against sector/function budget allotments. Upon receipt of the budget confirmation, BTA prepares financial coding for each mission. CSF and ITSF allocations are usually sent to missions and ROs in May. Funds can be accessed early in the first quarter (Q1) upon request.

The timing of the release of ITSF funds to posts in late May impacts on ITSF program delivery in the following ways:

• Discourages STCs from submitting ITSF proposals for spring/Q1 events due to the need to fund ITSF events from other budgets which leads to complicated accounting problems. This may result in missed opportunities.
• Travel and event related fees cannot be paid for as far in advance to obtain the lowest prices.
• Foreign currency fluctuations can result in insufficient funds by the time funds are received at mission. 
• Uncertainties relating to actual funding impacts on posts’ ability to leverage partner funding, especially for spring/Q1 events. 

Finding 10: Organizational restructuring has led to administrative efficiencies, however, the cost of administering the ITSF remains high.

1. Organizational restructuring and adoption of financial management practices

Amalgamation of the GVC, CDIA, CSR and EduCanada funds with the ITSF has resulted in more streamlined financial, submission and reporting processes, reducing the administrative burden on missions and HQ.

The funding guidelines of the GVC and CDIA funds are now the same as the CSF and ITSF, reducing the number of guidelines and rules that officers have to follow.

The Funding Unit creates individual fund commitments (COs) for each ITSF initiative, a practice that has reduced the administrative burden on missions and that is highly appreciated by officers.

For the last two years, missions received 100% of their allocations at one time, as opposed to receiving it in two tranches.

2. Level of effort invested in the annual Strategia commercial planning process

Despite administrative efficiencies gained, interview and survey respondents indicated that the number of transactions required to manage the ITSF (e.g. proposal development and submissions, adjudication, cash management, cancellations, returned funds, and reallocations) contributes to high administrative costs for missions, Common Service Delivery Points (CSDPs), and HQ, which is not necessarily commensurate with the amount of funds received.

While the ITSF approval rate for projects submitted as part of the annual Strategia commercial planning process has increased from 59% in 2013-14 to 67% in 2015-16, when cancellations are factored in, the time and effort involved in this process results in projects being implemented only half the time.

Figure 4. Number of ITSF^{Footnote 4} proposals submitted, approved, cancelled and added, 2013-14 to 2015-16

Additionally, as noted in Finding 9, ITSF funds for approved projects are disbursed to posts in late May, requiring missions to “cash manage” to pay for ITSF events taking place in the first quarter using funds from other budgets. Cash management creates complicated accounting problems and high transaction costs because funds have to be reallocated to the appropriate budgets later in the fiscal year. Further, the replacement of in-house financial staff with CSDPs has placed the administrative burden for all financial transactions previously done by the accounting section on STCs, TCs and Trade Commissioner Assistants (TCAs).

3. ITSF cancellations and returned funds

Of the approved projects, roughly 20% are cancelled and funds returned for repurposing in the third and fourth quarters. The reasons cited for ITSF project cancellations and returned funds include:

• lack of participation for an event/trade show
• initiatives completed under budget
• partial funding below amounts initially requested and/or notionally approved which can render the funds unusable if the mission is unable to revise the scope of the initiative, or find other sources of funding to cover the ITSF deficit
• changes to the initiative

While the reallocation process has contributed to low lapse rates of ITSF funds, personnel at both HQ and missions have noted that the process of returning funds, resubmitting proposals, adjudication and re-disbursement entails significant transaction costs.

Further, roughly half of ITSF annual expenditures occur in the fourth quarter which imposes significant administrative burden on personnel at post and HQ. Stakeholders indicated the need to spread expenditures and activities more evenly throughout the year.

The CSF, unlike the ITSF, provides STCs with the flexibility and discretion to repurpose funds to respond to changes to initiatives or cancellations. For example, as illustrated in Figure 5, returns of CSF base funds over the last 3 years were 87% less ($332.7K) than the ITSF returns which totalled $2.07M.

Figure 5. CSF^{Footnote 5} and ITSF^{Footnote 5} Funds Returned by Missions, 2013-14 to 2015-16

STCs have noted that the lack of flexibility to repurpose ITSF funds to respond to changes to initiatives does not reflect the planning cycle since initiatives can change between February, when the Strategia commercial planning process takes place, and the actual execution of initiatives which can be up to a year later.

Finding 11: General agreement that the existing KPIs accurately capture the outcomes of CSF and ITSF funded activities and initiatives, however, there is room for improvement.

1. General satisfaction with KPIs

In response to the survey question “Existing performance indicators (KPIs) accurately capture the outcomes of CSF and ITSF activities/initiatives/projects”, 82% of survey respondents conveyed satisfaction with the KPIs (only 18% disagreed).

However, survey feedback and stakeholders interviewed at post and HQ identified the following concerns and challenges with regards to using the Trade Commissioner Service (TCS) KPIs, which were not specifically designed to measure the performance of CSF and ITSF initiatives:

• too much focus on quantity and lack of focus on quality
• KPIs don’t measure value, e.g. a $1M dollar economic outcome facilitated (EOF) is weighted the same as one worth $10K
• KPIs do not reflect the level of effort or resources invested
• KPIs incentivize/drive short-term transactional behaviours rather than long-term ones
• officers have little control over ultimate outcomes (OPs and EOFs)
• KPIs do not cover the full range of TCS tasks/work
• frustrations that KPI definitions, especially successes, are too rigid
• KPIs can be manipulated
• results/KPIs can take longer than the current one-year timeframe for reporting and performance measurement, e.g. requirement to close initiatives in Strategia by the end of the fiscal year, before some results, particularly successes, have materialized
• officers may not make the linkages between the KPIs and the initiative and funding source

Finding 12: Lack of interoperability between CSF and ITSF planning and reporting systems creates administrative burden and limits effective performance measurement.

1. System developments since the last evaluation have improved departmental capacity to monitor performance

Strategia, which replaced the Commercial Economic Plan (CEP), is a comprehensive, one-stop, mission planning tool. The TCS uses it to enter their commercial action plans and initiatives, and to report on expenditures, outcomes and KPIs.

TRIO2, originally intended to be a Client Relationship Management (CRM) tool, is now used to track the performance of the TCS based on KPIs. It is also used to record and approve successes (OPs and EOFs). The Case tab, introduced in 2014, has increased interoperability by linking results to key TCS events.

A Business Intelligence (BI) pilot project is currently under development to link trade data from Strategia, the Integrated Management System (IMS), Human Resource Management System (HRMS), and TRIO2. The objective is to better align KPIs and outcomes to trade initiatives, and to measure the return on investment based on the human and financial resources invested. The tool will also be used to inform future ITSF funding decisions. Since FY 2015-2016, COs are created by BTA for each approved ITSF funded initiative to support this project.

2. Despite the noted improvements, challenges still remain

Due to limitations of the Strategia tool, trade officers are required to manually enter CSF and ITSF expenditures (IMS) and performance data (TRIO2) into Strategia. This increases both the administrative burden on posts and introduces a risk that data will be entered inaccurately, evidenced by data discrepancies for both expenditure data and KPIs in Strategia.

Despite KPI data in Strategia, TRIO2 is considered the only authoritative source for validated KPI data.

Outside of the Case tab and BI pilot project, there is limited ability to generate reports that link initiatives and funding sources with KPIs in TRIO2, which limits the ability to compare performance at the initiative/activity and funding levels.

The sector bureau, however, has demonstrated the ability to track performance at a sector level to inform decision making

The sector bureau has conducted detailed TRIO2 data analysis to determine the scope and scale of TCS activity within the ICT and creative industries sectors. This was achieved by using TRIO2 performance data for the sector, including all funded and unfunded initiatives/activities.

Information presented in the sector analyses includes:

• Client profile, number of clients served, location of clients served, client organization type (e.g. SME), client data by sub-sectors, etc.
• KPI trends by markets covered, and successes by post and by region. However, KPIs are not disaggregated by funding source(s) or amounts.
• Types of initiatives/activities that lead to successes

In addition, on an annual basis, sector officers review Strategia and KPI performance data in TRIO2 to determine the results achieved from previously funded ITSF activities to inform funding decisions.

Best practice: WSHDC

WSHDC has developed their own performance measurement system using a standardized naming convention, e.g. ICT0 (ICT=sector and 0 = general outreach) that links all their commercial initiatives to KPIs in TRIO2.

KPI descriptions in TRIO2 reference the initiative name, making it simple to link Strategia data (initiatives, funding sources) to TRIO2 data (results/KPIs).

WHSDC plans to use the TRIO2 Case tab for all initiatives, not only key events, to track results and to inform future STC funding decisions.

Conclusions

Access to discretionary CSF and targeted ITSF funding has become increasingly important for the Commercial-Economic sections of the Department and enhances TCS capacity to support Canadian businesses in their efforts to participate and expand into global markets. Trade initiatives funded in part through the CSF and ITSF support the advancement of Departmental international business development objectives by contributing to the expansion of Canadian business participation in global markets.

Strategic guidance provided by the Trade Sectors Bureau for the ITSF was found to be clear for all of the TCS’s priority sectors, however, there is a need for greater clarity and consistency in the provision of operational guidance across all sectors. With respect to eligibility criteria, stakeholders at missions and HQ expressed a need for greater access to hospitality to support CSF and ITSF- funded initiatives.

Mechanisms are in place to support sound project planning and implementation, and the system of reallocation contributes to low lapse rates. However, uncertainties over the ITSF budget and timing of disbursements to posts adversely impact on program delivery. Further, the transaction costs of administering the ITSF were found to be high at missions, headquarters and CSDPs. Finally, lack of interoperability between planning (Strategia) and reporting (TRIO2) systems increases the administrative burden and limits the ability of the Program to monitor performance and report on results. 

Recommendations

Management Response and Action Plan (MRAP)

Recommendation 1

BTD and BBD review the guidelines and expense eligibility criteria which govern the use of funds for hospitality with the view to making funds more accessible to support CSF and ITSF funded initiatives. (linked to F6)

Recommendation 2

BBD adopt a more consistent approach within the branch and divisions in the provision of operational guidance to STCs. (linked to F7)

Recommendation 3

BTD and BBD explore options to increase the predictability of funding, release ITSF funds earlier, and reduce the administrative burden at missions and HQ. (linked to F9 and F10)

Recommendation 4

BTD and BBD, in consultation with the relevant responsibility centres, and aligned with the Departmental Data Strategy, explore options to improve data quality and reporting on CSF and ITSF initiatives and their results to help make informed, evidence-based decisions.  (linked to F12)