Evaluation of the Canadian Technology Accelerator – Final Evaluation Report

December 2015

Abbreviations, Acronyms and Symbols

Acknowledgements

The Evaluation Division (ZIE), of the Office of the Inspector General at the Department of Foreign Affairs, Trade and Development (DFATD), would like to extend its appreciation to the employees of DFATD, both at Headquarters and missions abroad, including the CTA Unit in the BBR Division, the Trade Sectors Bureau of DFATD/Global Affairs Canada (GAC). We would also like to express our gratitude to the many internal and external stakeholders who agreed to participate in the evaluation of the Canadian Technology Accelerator Initiative including Business Development Canada Capital (BDC Capital). We also gratefully acknowledge the support and advice provided by the members of the Evaluation Advisory Committee (EAC) in reviewing the approach, findings and recommendations of this evaluation.

Executive Summary

Introduction

In 2009, the Canadian Technology Accelerator (CTA) Initiative began as a pilot program at the Canadian Consulate General in San Francisco and Palo Alto. The Initiative provided Canadian Small Medium-Sized Enterprises (SMEs) with access to unique, in- market resources to assist them in their internationalization efforts. Developed and managed by the Canadian Trade Commissioner Service (TCS), the Initiative intended to accelerate international opportunities for high-growth, export ready Canadian SMEs in the Information and Communications Technology (ICT), Life Sciences and Sustainable Technologies sectors at critical growth stages. The Initiative provides the tools, partners and programming to help Canadian SMEs make connections with venture capitalists and strategic partners (including multinational enterprises (MNE) and expatriates) in global market technology hubs.

Since 2009, the CTA has shown both versatility and value in addressing the commercialization and export challenges faced by young Canadian companies attempting to access international markets. In 2013, the perceived success of the pilot resulted in the CTA Initiative receiving $5 million over 3 years from the Government of Canada (GoC). Global Affairs Canada (GAC) formerly Department of Foreign Affairs Trade and Development (DFATD) received central agency approval to align existing CTA pilot programming across the United States (U.S), to support new CTA pilot locations, and to initiate stronger coordination and reporting frameworks. More specifically, the Initiative expanded from its collection of pilot programs in California, Colorado and New York to locations across the U.S. and later in 2014, to additional locations overseas. Over three sequential stages the Initiative, in alignment with government priorities to expand support to SMEs at home and abroad, grew from a single pilot in one market in 2009 to 12 programmes in 8 markets in the U.S., United Kingdom (UK), France and India in Fiscal Year 2014/2015.

This evaluation was conducted in response to a requirement to review the Initiative at the end of its initial 2013 funding allocation. The evaluation was conducted by the Evaluation Division at GAC. The purpose of the evaluation was to assess the relevance and performance of the Initiative and identify lessons learned regarding the efficiency and economy of its delivery. The evaluation examines the CTA from its initial pilot phase in 2009 until the end of Fiscal Year (FY) 2014/2015. However, a greater emphasis in the evaluation was placed upon the two years following the CTA Initiative’s receipt of central agency funding which resulted in the creation of two positions at GAC Headquarters (HQ) to administer the Initiative. Only at that point was a formal governance structure for the Initiative formed and systematic tracking of outputs and outcomes undertaken. The period prior to 2013 is therefore considered primarily for historical context and to understand the evolution of the Initiative over time.

Key Findings

Relevance

With Canada’s SME growth profile in export markets remaining well below the Organization for Economic Cooperation and Development (OECD) average, the SME tech sector in Canada continues to require assistance when attempting to internationalize. The CTA actively engages Canadian SMEs looking to internationalize in the U.S., UK, Europe and India. The proactive efforts of the Initiative reduce barriers to internationalization by helping Canadian SMEs identify their market niche and refine their offering. As a result, CTA companies are achieving notable job increases domestically and internationally, increased revenue generation, and increased capital investment. CTA companies are, as a result, becoming competitors in international markets. Participating companies have noted that the CTA has been integral to their company success; at times propelling their businesses years ahead. Thus, the CTA has demonstrated itself to be a relevant instrument for advancing government priorities to increase Canada’s ability to compete in the global innovation economy. In doing so it is fulfilling a unique role within the GoC.

The relevance of the Initiative is also demonstrated by the continued demand from Canadian SMEs.  From the initial pilot in 2009 until FY 2014/2015, the CTA has grown from 6 companies participating in its first cohort to a high of 140 participating in FY 2014/2015. With approximately 298 applicants in FY 2014/2015 the Initiative has a 47% admissions rate. It is projected that approximately 75-100 participants will participate in FY 2016/17. The reduced number of participants is a reflection of the increased selectivity of the Initiative and a refinement of the Initiative’s offerings to more mature companies.

Despite the foregoing, one of the challenges the CTA has faced has been aligning the services of the Initiative with the expectations of the participating companies. As the CTA has grown, its service offerings and the companies they recruit have evolved in an effort to align the needs of Canadian SMEs with relevant and responsive programming. Participants and administrators agreed that when Initiative offerings and company demands or expectations are aligned, a CTA company was more successful in the Initiative, more likely to complete, and more likely to express satisfaction with the CTA. Thus, ensuring that the CTA is not only providing responsive activities and services but also recruiting the appropriate companies is necessary for success.

The Initiative is fully aligned with the priorities of the GoC. The CTA Initiative aligns with GoC ambitions to drive further improvements to the trade promotion and support service efforts of the Trade Commissioners Service (TCS). It does so by helping SMEs access global value chains and promote inward investment and innovation partnerships in markets where the TCS can have the greatest impact, thereby helping to grow Canada’s SME footprint and create new jobs for Canadians. It is furthermore aligned with GoC ambitions to grow the Canadian economy by investing in Canadian businesses and entrepreneurs.

Finally, the CTA is appropriately situated and is fulfilling a unique role under the TCS. There is no other government department with the same connections around the world to successfully run the CTA. While further efforts can be made to broaden the involvement of the TCS through the inclusion of Regional Officers (ROs) in company recruitment and the delivery of the Initiative, the efforts of the TCS are a necessary asset and integral component of the CTA. Notably, the Initiative has also proven to serve as a powerful client recruitment tool enabling the TCS to reach and service younger companies that might become TCS clients in the future. Thus, the CTA is also aligned with the broader ambitions of the TC service.

Performance – Achievement of Results

Overall, the CTA Initiative has had a positive impact on participating Canadian SMEs as well as the TCS. The Initiative has made significant strides in achieving its objectives and delivering on key intermediate and final outcomes. Yet, important considerations regarding resource allocations must be made to ensure the continued success of the Initiative.

Over the course of the CTA’s evolution from a pilot project in the U.S. to an Initiative in 8 international locations, its service offerings have also evolved as administrators learn and work to ensure that the Initiative delivers relevant and responsive services to participating and qualified companies in all locations. Notably, the access the CTA provides to mentors and the TCS network, as well as the in-market address and office space, are vital offerings of the Initiative that are highly valued by CTA participants. Through such tailored offerings and expertise the CTA has increased the capacity of Canadian SME’s to compete in the global economy.

While the strength of the CTA remains its adaptability, agility, and the willingness of staff to learn and draw upon their own resources, a renewed effort should be placed on the need to standardize some minimum service offerings to ensure the achievement of the CTA’s expected outcomes and the maintenance of the CTA brand. A cited challenge for the CTA is providing structured, consistent offerings and projecting a consistent brand across locations. Notably, this evaluation found several inconsistencies in Initiative activities and outputs across locations in areas such as orientation and coordinated events, mentorship and adaptive services by the TCS. This is a cause for concern for two key reasons. First, it has been documented in the evaluation to have produced confusion among participants, partners and stakeholders regarding the offerings of the CTA. Second, and as a result, despite success advancing a strong and valued CTA brand the various offerings and weak internal and external communications have produced confusion over what the CTA offers and thus the Initiative’s branding proposition.

In addition to programming, recruitment has also evolved since the CTA’s inception. Over time the Initiative has moved to taking on fewer and more qualified companies. This was a natural evolution that occurred as administrators identified the types of companies that fared better in the Initiative. Further impetus for refining the selection criteria to more mature and qualified companies came through the CTA’s expansion to the UK, France and then India. These CTAs were often viewed as sequential steps for Canadian companies at different stages of maturity, and with different market goals in mind.

Performance – Efficiency and Effectiveness

Tensions have also emerged in the delivery of the Initiative. TCS officers frequently cited the challenge of balancing TCS and CTA responsibilities. A contributing factor to this tension appears to be an imbalance in the allocation of full time employees relative to the number of applications and participants in some CTA locations. Furthermore, there appears to have been little clear guidance on how traditional TC activities should be balanced with the CTA Initiative. This has resulted in several administrative challenges at Post.

The transition of the CTA from several independent pilot projects in to a recognized and funded Initiative with a clear strategic direction, and governance structure that outlines the roles, responsibilities and accountabilities in support of declared objectives has proven a challenge for management. This is due in part to the fact that some CTA locations operated autonomously for several years prior to the Initiative securing its funding authority in 2013 while others were established only after 2013. As a result, in many locations a resistance to the involvement of Ottawa has emerged as well as a lack of understanding regarding the strategic goals and objectives of the CTA as a whole.

Challenges in instituting a clear governance structure contributed to poor internal and external communications with relevant partners, stakeholders and potential clients. As the Initiative has grown in size, this problem appears to have only increased. While the CTAs expansion has resulted in more market options for clients, weak communication among the various locations means that while each CTA has a clear understanding of their respective market challenges, they often fail to view how their CTA is situated within the larger Initiative. As a result in many cases they are unable to communicate to perspective clients and partners the value added of each location. With no clear communications structure, prospective participants are being referred on an ad-hoc rather than informed basis.

Similarly, the lack of clear governance structure has limited the Initiative’s ability to cultivate a consistent and strategic recruitment process. The current process of individual CTA’s recruiting companies independently with different application deadlines and methods of outreach to partners is unsustainable and confusing to both companies and internal and external CTA partners. This has been aggravated, as referenced above, by the lack of communication regarding the offerings of each location and the clients they recruit.

Further, the lack of strategic direction and defined roles and responsibilities for personnel within the Initiative has contributed to an inconsistent reporting of the data necessary to assess the Initiative’s relevance, performance and effectiveness as measured against its objectives and final outcomes. For example, inconsistencies in the coding of applicants and participants and their completion of the programme as well as poor reporting to headquarters on relevant outcome data such as company success has resulted in the Initiative underperforming in this regard and often duplicating data collection efforts. Mechanisms such as TRIO that should assist in the collection of data are regarded by administrators as insufficient and not reflective of the reality of the SME business cycle in any of the locations or industries in which the CTA operates. As a result, poor reporting on outcomes and a weak alignment of Key Performance Indicators have occurred. Thus, the overall ability of the evaluation to assess the specific results of the Initiative has been limited by the absence of clear objectives, expected results and performance indicators.

Recommendations

Based on the foregoing analysis, the following recommendations are presented for consideration.

Recommendation #1: To meet programmatic objectives, the CTA should continue to refine and clearly define the target client and the selection and recruitment process. To ensure the recruitment of qualified companies, the process should be managed through a clear, coordinated and centralized two-pronged approach that:

• Includes a targeted and qualified recommendation process through ROs, partners (BDC, NRC-IRAP), TCS and previously successful CTA companies;
• Includes an open, informed and transparent application process via social media and through an easily accessible CTA website that specifies company requirements and expectations.

Recommendation #2:  The CTA should clearly define minimum requirements and standards across all current and future locations/ missions to ensure the Initiative fulfills its programmatic objectives.

Recommendation #3: To ensure a consistent value offering the resources of the CTA should be better aligned in order to establish an optimal Trade Commissioner to company ratio.

Recommendation #4: The CTA should regularly and actively leverage the extended networks of the entire TCS and improve coordination and communication/ information sharing with the regional offices and non-CTA missions during the recruitment, program and post-program phases.

Recommendation #5: Communications efforts between CTAs, partners and the public should be improved via the development of a single web presence which outlines the offering at each CTA location and the objective of the CTA as a whole. The web presence and all communications tools should be streamlined and updated regularly.

Recommendation #6: Improved reporting and recording of CTA outputs and outcomes is necessary from Posts and HQ. Company performance and outcomes should be tracked for an additional follow up with companies 12 to 18 to 24 months after graduation. Harmonized standards of reporting across CTAs are also required.

1.0 Introduction

The evaluation of the Canadian Technology Accelerator Program (CTA) was conducted by the Evaluation Division (ZIE), Office of the Inspector General (ZID), Global Affairs Canada (GAC), formerly the Department of Foreign Affairs, Trade and Development (DFATD)/, in compliance with a Treasury Board (TB) requirement to report on the relevance and performance of the program. ZIE reports to the Departmental Evaluation Committee (DEC), which is co-chaired by the Deputy Minister of Foreign Affairs, the Deputy Minister of Trade and the Associated Deputy Minister of Foreign Affairs and convenes as required to approve evaluation reports.

The evaluation of the CTA Initiative responds to the requirement to review the Initiative before its three year mandate and financing from the GoC expires in 2016. The purpose of this evaluation is to assess the relevance and performance of the Initiative, as well as the effectiveness of its governance structure and delivery systems. The target audience for the evaluation is GAC’s senior management; the Assistant Deputy Minister of the International Business Development and Innovation Chief Trade Commissioner - BFM; the Director General of the Trade Sectors - BBD; the Director General of Investment and Innovation - BID; the Divisions responsible for administering and monitoring the performance of the program Aerospace, Automotive, Defence and ICT Practices (BBR), Trade Policy, Geographic Bureaus; Regional Officers; Trade Commissioners (TC); partners such as Business Development Canada Capital (BDC), The Industrial Research Assistance Program (IRAP), the National Research Council of Canada (NRC), and the Canadian public.

1.1 Background and Context

Established in 2009 as a pilot program of the Consulate General in San Francisco and Palo Alto—the Canadian Technology Accelerator Initiative (CTA) was designed to help high-growth, export ready Canadian Small Medium Enterprises (SMEs) at critical growth stages effectively expand their presence in international markets. For a period of three to six months, qualified Canadian SMEs received virtual and/or in-market entrepreneurial support in one of the 12 locations where the CTA operates. Managed by the Canadian Trade Commissioners Service (TCS), the CTA assists Canadian companies in the Information and Communications Technology (ICT), Life Sciences, Clean Technology, and Digital and Social Media sectors.  The CTA Initiative complements traditional TCS services which are offered at 150 sites around the world. Through the tools, partners and programmes, the CTA Initiative provides the following:

• Free office space in market within an existing international business accelerator;
• Mentorship from a variety of stakeholders such as industry leaders, partners, and Trade Commissioners in market;
• Support in accessing financial resources, engaging strategic partners, and refining business models for global market opportunities;
• Greater exposure to global partners and opportunities to pursue clients^{Footnote 1}

1.2 Rationale for the Initiative

SMEs represent the backbone of the Canadian economy, accounting for approximately 89.9% of the Canadian labour force in 2012.^{Footnote 2} Yet, research has shown that Canadian firms, and SMEs in particular, face an uphill struggle expanding into international markets, where navigating the business culture and regulatory environment can be a challenge. Of the more than 1.09 million SMEs operating in Canada, just 41,000 are currently exporting, well below the OECD average. With such a small number of companies taking advantage of the global marketplace, the potential for growth is vast for both SMEs themselves and the Canadian economy at large. Facilitating increased export activity of Canadian companies and growing the overall number of Canadian exporters requires a significant investment.

The complex challenges faced by companies looking to expand their businesses abroad are mitigated by the valuable insights of the TCS and other programmes like Export Development Canada (EDC). However, Canadian SMEs require more support to expand and be successful in the burgeoning technology sectors in the U.S., the U.K. Europe and India. Recognizing that SMEs need time in market to build trust and develop a network of local customers, investors and partners, the CTA was designed to address this need.

1.3 Program Objectives and Resources

The objective of the CTA Initiative is to assist with the development and expansion of Canadian companies in international markets. By supporting Canadian SMEs in their efforts to access global market opportunities through unique business acceleration programs that are tailored to opportunities in their local market, the CTA aims to increase the export opportunities and the competitive advantage of Canadian firms abroad, while creating new jobs and revenue in Canada. In foundational documents, GoC support through the CTA Initiative was expected to result in 500 high-growth SMEs benefiting over three years with increased successes. These successes were to be benchmarked against the 210 participants that benefited from the CTA from the first pilot in 2009. Outlined by the Treasury Board, the impact and results the CTA has on participating companies are measured through the following:

• 1. The increase in the number of employees,
• 2. The rise in annual revenues,
• 3. The amount of capital raised,
• 4. And the number of new strategic partnerships and global customers

The CTA achieves these outcomes by providing an intensive immersion experience of entrepreneurship; customer acquisition, supply and strategic partnerships; and, venture capital formation of a calibre not available in Canada. Through an in-market address, mentorship, and accesses to business development consultants and the TCS service, the CTA provides participating firms with a platform for immersion in global markets during their critical growth periods which enables them to succeed by building the international channels required for expansion while adjusting their business strategies and products to remain competitive.

The CTA Initiative is distinct in its structure and program offerings. In particular, while more traditional accelerators focus on the development of market-ready products and leadership teams, the CTA Initiative has, since its inception, focused on cultivating business opportunities and facilitating introductions to potential investors, buyers and/or partners for companies that already have a mature product and management team.

From the pilot stage in 2009, the program was funded from a variety of sources, including the Government of British Columbia and BDC. In 2012, with funding from BDC, North American Platform Program (NAPP), Client Service Fund (CSF), and the Integrative Trade Strategy Fund (ITSF) and the Government of British Columbia, the CTA pilot expanded to several additional US cities, including a Life Sciences CTA in San Francisco, a Clean Tech CTA in Denver and an ICT and Clean Tech CTA in New York. In an effort to further expand the Initiative and support Canadian SMEs seeking to access to the competitive and challenging global venture capital markets around the world, in 2013 the CTA received central agency funding from the GoC.

Following central agency approval in 2013, the CTA received $5 million over three years, until March 2016. BDC also allocates additional funding to TCS in Boston, Silicon Valley and New York. As a result, the CTA expanded to 12 CTAs operating in eight cities (San Francisco and Palo Alto, New York and Philadelphia, Boston, Denver, London, Paris, Delhi) and 4 countries (the United States, England, France and India). The central agency funding also resulted in the Initiative receiving two co-ordinators at headquarters and funding to hire four locally engaged staff. These positions were officially filled in the winter and fall of 2014 respectively. In FY 2015/2016 there were 11 CTA programs in nine cities.

1.4 Governance

The overall responsibility and accountability for the CTA resides with the Minister for International Trade. The TCS manages the program with support from the investment branch of BDC Capital. The TCS is accountable to the Deputy Minister of International Trade (DMT) for the overall operations and management of the CTA as it aligns in accordance with GoC policies and priorities.

2.0 Evaluation Scope & Objectives

This evaluation assesses the reference period from the pilot in 2009 until FY 2014/2015 with a focus on the years 2013 to 2015. In 2016 the budget allocation is due to sunset. Prior to 2013, the Initiative did not have a centralized governance or administrative structure, nor did it have dedicated funding. As a consequence, the systematic tracking of results did not occur until after November 2013. Thus, while the evaluation is based on approximately two years of performance data, this information has been augmented by data obtained from informant interviews and evidence from site visits which, collectively, provides sufficient data to report on the Initiative’s relevance and performance.

The specific objectives of the evaluation were to:

1. Evaluate the extent to which the CTA Initiative:

• Continues to respond to the needs of Canadian companies and other stakeholders; and,
• continues to be aligned with GoC foreign policy priorities and the strategic outcomes of DFATD and, in particular, the TCS;

2. Evaluate the performance of the CTA initiative as a whole, and the results in the four main areas of performance (described below) achieved from 2013-2015;

3. Determine the extent to which the whole-of-government implementation of the CTA has been efficient, effective and adequately resourced; and

4. Derive lessons learned and best practices that could inform future initiatives or contribute to the review and update of the CTA.

3.0 Evaluation Approach & Methodology

The evaluation was based on the objectives of the CTA, namely: to grow the business of a large number of Canadian SMEs by providing access to unique resources found in key U.S. and international tech clusters. The design of the evaluation was also guided by the expected results and measures for the CTA outlined in the Initiative’s 2013 foundational documents. These measures for success included: whether a company had increased the number of employees; increased its annual revenues; increased its capital; and finally increased the number of strategic partners and global customers. Notably, the measures outlined in the 2013 foundational documents differ from the 2009 Logic Model of the CTA which also includes increasing the commercial operations at Post while increasing quality service to customers and stipulates an incremental number of jobs created in Canada as well as Leads, Prospects and investments made into Canada as key outcomes. Finally, the evaluation was guided by the need to assess the continued relevance, performance and cost-effective delivery of the Initiative as stipulated by the Treasury Board Policy on Evaluation.

The evaluation employed a mixed-method approach using multiple data sources, including data from key informant interviews with CTA leaders, stakeholders and participants, site visit observations, and document reviews. These sources were triangulated to formulate findings, conclusions and recommendations regarding the CTA Initiative. Additional information such as activity proposals and reports, and data on jobs created, revenue generated, capital invested and networks added were also used as was data on the number of applicants, participants and drop outs to improve the accuracy and validity of the conclusions. Preliminary findings were then validated and confirmed with stakeholders and program operators of the CTA.

Details on data collection methods used in this evaluation follow:

Document and File Review

Literature related to the Canadian SMEs sector, venture capital, accelerator space and investment in Canada and around the world were reviewed to examine the relevance and continued need for the CTA and its locations. These documents included civil society and business reports on emerging trends in the technology startup industry and similarly minded programs created by governments around the world. In addition, the evaluation team reviewed corporate documents, including those related to the development of the CTA and the feedback of participating firms, press releases to assess the extent to which the CTA continues to be aligned with the GoC's foreign policy priorities, and departmental mandates and objectives. Internal documents, such as budget allocation with spending, Strategia requests, and TRIO inputs on participating firms, were also reviewed. Finally, performance data was accessed to determine the results of CTA clients at various CTA locations.

In addition to the above, the evaluators examined documents including, but not limited to:

• Foundational documents
• Correspondence
• Financial documentation detailing the budgetary allocations for each year, as well as the actual expenditures at each CTA location
• CTA web documents
• Participant ‘Exit Forms’
• Participant application files

Key Informant Interviews

In-person and telephone interviews were conducted with approximately 60 key informants who were either knowledgeable about the CTA or were directly involved in its operations at HQ or at missions. Informants included internal and external stakeholders, such as BBR personnel, TCS and ROs, CTA participants, BDC Capital, NRC-IRAP, mentors, SME consultants and other informed individuals at similarly interested organizations, such as NGOs responsible for assisting Canadian companies in foreign markets.

Field/ Site Visits Observations

Field visits were conducted at four CTA locations. They included the CTAs at Silicon Valley (Plug and Play), San Francisco (Rocket Space), Boston (Cambridge Innovation Centre) and New York (The Grind). At each location, the evaluation team visited both the mission as well as the space where CTA participants are hosted. Useful information was collected in terms of design, location, and functionality of the different accelerators. Site visit locations were selected to create a representative sample of the larger CTA program. The visit focused on CTA locations in the United States because they are comparable in size, activity, structure, age.

Survey Analysis

Surveys were distributed and conducted by BBR in coordination with the CTA locations. These surveys were not conducted on behalf of the evaluation team but rather were independently conducted by the BBR for their own reporting needs. Despite some of the limitations of the data, the results provided key quantitative information on the success of the Initiative with regard to job creation, revenue generation and capital raised by company and CTA location. They also served to provide significant qualitative data on company and CTA performance and how company expectations and CTA offerings align.

Regarding limitations, since 2013, these surveys were administered at the end of each CTA cohort for a total of 41 group cohorts. They were again administered 6 months following a company’s participation in the Initiative. Participation in the surveys is voluntary. When all reported data is considered from the exit forms and the 6 month intervals following the end of a company’s participation in the Initiative, there is data for 59% response rate or 161/274 participants. When only survey forms are measured, the number of respondents drops to 44% or 121/274 participants.

Significantly, in FY 2013/2014, some of the performance data was collected via surveys while other data was collected through self-reporting by posts. Through this self-reporting, which occurred before the creation of a standardized form, data was collected by the Initiative in spreadsheets and often had inconsistent measurements with not all metrics collected. Mid-way through FY 2013/2014 a standardised survey was implemented. The survey proceeded to evolve over the course of the subsequent fiscal years, but much of the data remains comparable.

Data Analysis

The evaluation team conducted a review of the raw data provided by the BBR team in spreadsheets, exit forms, Strategia budgetary request and TRIO to assess the objectives of the CTA: job creation, the generation of capital and revenue, and the development of networks over time. The data was triangulated with data from interviews and with mission staff and CTA participating companies to assess the results achieved and the ongoing implementation challenges.

4.0 Limitations to Methodology

The evaluation methodology utilized, while comprehensive, did encounter some limitations. These limitations, along with the actions taken to mitigate them, are briefly described below.

The availability of performance data related to the continued need for the Initiative was limited for the CTA due to its newness and the short time period under study. Notably, there was no centralized administrative apparatus prior to 2013 responsible for collecting data in any systematic way. Thus, at the time of the evaluation, and despite efforts from administrators in Ottawa, the Initiative had not yet adopted a clear and consistent performance measurement framework. As a result, the CTA lacked a systematic method to track the activities and outputs of the CTA and its economic impact on participating companies on a year over year basis. Despite the creation of some mechanisms such as exit surveys for participants and a uniform application template for companies, the lack of consistent data reporting from Posts on the CTA activities, outputs and outcomes also presented a challenge to evaluators when assessing the impact of the Initiative.

Often the data was simply not reported. Notably, data collection and the adoption of performance measurement frameworks is not specific to the CTA, but is rather endemic to the entire business accelerator and incubator industry which is reported to have had little success assessing the economic impact that support services have on early-stage companies. While the evaluation team undertook significant steps to overcome these limitations through coordination and consultation with the CTA administrators responsible for collecting the data, the evaluation team was limited in the extent to which it could incorporate quantitative analyses. Due to numerous reporting irregularities this was particularly true of analyses pertaining to the number of applicants and participants at individual CTAs, completion rates, activities implemented at Post, and their resultant outputs and outcomes from each CTA location.

In addition, the availability of performance data was limited particularly with regard to assessing the consequences to Canada’s commercial interests if the program is cancelled in whole or in part. While data was collected on key outcomes, such as the number of jobs created, the amount of revenue generated and the capital invested, the CTA is too young to determine the significance of these numbers and no comparable data source exists either within the government or across Canada with similarly focused programs. Finally, it is impossible to accurately determine what would have happened to participants had they not participated in the CTA, as there is no comparable performance data generated on impact metrics by domestic accelerators across Canada with GoC investments or even with privately run accelerators. Over time, as more companies graduate from the Initiative and if data continues to be collected and tracked, it may prove possible to identify whether outcomes listed under GMAP, such as an increase in the amount of Canadian exports from SMEs, or larger CTA outcomes, such as meaningful Canadian job growth increases, can be attributed to the CTA.

Program Administrative Systems

At the time of this evaluation, consistent with department operations, financial and program data on the CTA was not consolidated and analysed in any single data repository. Furthermore, direct allocations from partners to posts are not tracked by HQ thereby limiting the ability of the evaluators to access key program and performance data. In response to this limitation and risk, the evaluators worked in cooperation with BBR personnel to assemble key data (financial, program, performance) that is managed at HQ, which was cross-checked against other sources. The evaluators also took steps to discuss any financial and program reporting irregularities with program managers. The explanations provided were to the satisfaction of evaluators.

Recipient Informant Selection

Program recipients contributed to this evaluation in two ways. First, their exit interviews conducted by the CTA Initiative administrators were used and aggregated to draw findings, conclusions and recommendations for the Initiative. With a response rate of approximately 44% for exit surveys alone the evaluators are confident that information provided by the exit interview forms is representative. However, since the voluntary nature of the survey could result in bias, additional companies that did not complete the exit surveys were targeted for interviews. In depth interviews were also conducted with a sample of companies that did complete the survey. The opinions of the additional companies selected were largely reflective of the larger sample.

Site Visits

Site visits were used to gain a better understanding of the CTA context. However, the evaluation team was limited in the number of site visits that could be undertaken. This meant that not all CTA locations were examined in person and in depth. The sites that were selected were deemed to represent the core of the CTA Initiative and to most accurately reflect the original objectives of the CTA Initiative as outlined in the logic model and the subsequent 2013 foundational documents. These CTA locations also had the added advantage of being operational for more than two years, thus allowing for the evaluators to track the activities, outputs and outcomes over a longer period of time.

To compensate for the limitations, telephone interviews with representatives from additional CTAs were conducted. Doing so expanded the sample size and ensured that the findings found from the initial sample were representative of the larger population.  

5.0 Evaluation Findings

5.1 Relevance Issue 1:  Continued Need for the Program

Finding #1:  The assumptions that led to CTA’s creation remain valid, with the Initiative addressing a distinctive need in the SME sector in Canada.

SMEs represent 99% of all businesses in Canada and account for nearly 90% of jobs in the private sector. Research has shown that companies actively engaged in exporting their products and services are more dynamic competitors and more likely to experience above-average growth and productivity. This is especially true in comparatively smaller markets such as Canada where companies must extend themselves internationally to access vital opportunities for growth.^{Footnote 3} Yet, Canada’s export profile remains dominated by large companies. Approximately 8% of Canadian SMEs are exporting their products and services internationally.^{Footnote 4} In this context, programs intended to boost the international exposure of SMEs are particularly relevant to Canada’s economy. Reports have found that a strong public role, preferably harmonized across levels of government, is necessary for the growth of SME’s.

Although success and failure rates for Canadian SMEs are difficult to ascertain, approximately 75% of venture-backed firms and 90% of technology start-ups in the U.S. fail and do not return investors’ capital.^{Footnote 5} The CTA aims to mitigate potential failures by reducing the risks of internationalization for Canadian companies through mentorship and other entrepreneurial support services. The proactive efforts of the CTA and the market support services provided reduces the risks faced by SMEs when internationalizing while also helping them identify their market niche. The Initiative furthermore serves to reduce psychological or perceptual barriers to internationalization through its in market support. Thus, the assumptions that led to the creation of the CTA that Canadian SMEs need more time in market with targeted support to launch their businesses, remains valid.

Through its unique public offering, the CTA Initiative addresses a distinctive need in the Canadian economy and SME sector in particular by providing a large number of high-growth SME’s looking to internationalize with market immersion in a local ecosystem, mentorship, and programming. With CTA locations in key markets, including, the U.S., UK, Europe and India the Initiative has become actively engaged in establishing a competitive advantage and export opportunities for Canadian SMEs in a variety of sectors and markets where Canadian companies have proven their ability to succeed. Thus, the mandate of the CTA addresses a distinctive need in the Canadian economy.

Further evidence in support the CTAs validity include the fact that CTA companies have noted that the CTA accelerated their business years ahead of where they would have been if they had not participated. This is a key measure and outcome of the Initiative. Furthermore, of those companies reporting in FY 2014/2015, 93% believed the CTA improved their understanding of the local market. This was an increase from 89% of responding participants in FY 2013/2014. Over half of participants also found the support provided through office space, mentorship and tailored services to be positive.

Finding #2:  While the CTA’s activities are generally viewed to be a relevant government instrument for engaging Canadian SMEs, challenges remain in aligning the services offered by the Initiative with the demands of companies.

The CTA Initiative offers a collection of services designed to facilitate better access for Canadian SMEs in their chosen international market. Of particular note are the mentorship offering, the TCS support and guidance as well as the office space and the advantages that come from having a location and address in-market. In the exit forms for FY 2014/2015, 90% of participants noted receiving value from mentors and 66% from the office space. Interviews with relevant stakeholders and CTA clients echoed this feedback, stating that the mentorship and guidance was the most significant value offering and most helpful in orienting companies to the market, preparing their pitches, and providing introductions. CTA participants further remarked on the value of having an address in-market to work from and schedule meetings and network. This was believed to have improved their access to prospective clients and enhanced market credibility. The positive responses of companies to these offerings highlight the relevance of the Initiative’s activities for engaging SMEs and addressing their needs.

Notwithstanding the foregoing there have been some challenges in aligning the services offered by the CTA with company expectations and demands. Since its creation in 2009, the CTA has continued to refine its offerings to better align the same with the demands and needs of Canadian SMEs. CTA stakeholders noted that in the early iterations of the Initiative there was not a lot of communication between the Posts and the companies coming to participate in the CTA, and as a result there was often confusion or mixed expectations about what the company would receive while in-market. Notably, companies which were too junior or were expecting a more traditional accelerator with a significant amount of structured programming and hands-on assistance were found to be less satisfied with their overall experience. In contrast, where company demands or expectations are aligned, the company has been documented to have had a more positive and successful experience in the Initiative. Based on interviews and exit forms, companies also appear to be more likely to complete the Initiative, and more likely to express satisfaction with the CTA overall. In response to these findings, in 2014 a virtual program was added in several CTAs to help align company expectations with what the CTA can provide.

The majority of the CTAs have now adopted the virtual program which has increased the awareness among TCs at Post and CTA participants on what both can expect during the cohort. Virtual programs vary by location but range from a week to a month in duration, and consist of discussions with the TCS and mentors and in some cases a series of modules, before the company arrives at the CTA. During this time the company also develops a strategy and business plan for their time in market. This has been noted to improve communications between the CTA and participants thus allowing the Initiative to tailor its offering more closely to SME needs prior to arrival.

Participating companies, however, continue to seek services that the Initiative simply cannot provide. Several administrators and participants interviewed noted the significant strain that in-market housing costs put on their finances. In many cases, these costs led to the companies commuting back and forth between their home and the host market, despite an understanding that doing so would lessen their ability to take full advantage of the Initiative’s offering. Suggestions for rectifying this problem have ranged from the CTA creating a “Canada House” at each location to establishing a list of affordable housing options for participants, to providing a living stipend or travel expenses. While opinions were mixed regarding the possibility of providing additional financial support, all CTA administrators agreed that supplying affordable housing is not the responsibility of the TCS and not an offering the CTA can provide. Furthermore, many participants believe participating companies should have a financial stake in their future and that no additional financing is necessary.

Finally, it was noted that the monetary value offering of the CTA through the provision and access to free accelerator space, mentorship, business development consultants, pitch coaches and Intellectual Property lawyers, is consistent with similar accelerator offerings and sufficient for meeting the needs of Canadian companies.

Finding #3: Continued demand for the CTA provides evidence that the Initiative is responsive to the needs of Canadian SMEs and CTA participants.  

From the initial pilot in 2009 until FY 2014/2015, the CTA has grown from 6 companies participating in its first cohort to a high of 140 participating in FY 2014/2015. Approximately 75-100 participants are projected annually in FY 2016/17. With almost 274 participants completing the Initiative since the central agency approval in 2013, by the end of FY 2014/2015, demand for the CTA across Canada remains strong as awareness of the Initiative grows. At the end of FY 2014/2015, the Initiative had 11 CTAs in 9 cities and the majority of applicants were targeting Silicon Valley and San Francisco CTAs (combined 38%), followed by Boston (20%) and then New York (12%)^{Footnote 6}.

Since the central agency funding allocation in 2013 and tracking applicants and participants began, the Initiative has seen a reduction in the number applicants from 438 in FY 2013/2014 to 298 FY 2014/2015. This has occurred despite the addition of 3 CTA locations between FY 2013/2014 and FY 2014/2015.

On its face this would suggest a decreased demand for the CTA; however, when the evolution of the Initiative is taken into account, a different picture emerges. First, refinements to the selection criteria towards more mature, high performing SMEs and away from younger start-ups has resulted in a smaller pool of potential clients. Second, a reduced number of cohorts now exist at CTA locations to reflect the various demands and capacities at Post. Finally, the adoption of a standardized application procedure that did not exist for the majority of FY 2013/2014 was used for the entirety of FY 2014/2015 and has resulted in more reliable reporting.

It is also important to keep in mind that 25% of participants in FY 2014/2015 have participated in more than one CTA. These companies are classified as “repeats”, meaning they either reapplied to stay within the same CTA or they were accepted to a new location. The high incidence of repeats reflects both the additional demand within Initiative as well as a potential challenge: many CTA companies are simply not ready to be completely on their own following their participation. Ideally, when a company graduates, if they are not already, they should be ready to become a TCS client but, this is not always the case. Significantly, some of the CTA’s most successful companies have in fact participated in more than one CTA cohort, taking advantage of the different offerings provided around the world.

5.2 Relevance Issue 2:  Alignment with Government Priorities

Finding #4:  The CTA is well aligned with the objectives of the GoC and in particular the TCS.

The CTA Initiative is well aligned with Government priorities to help the Canadian economy become more innovative, competitive, and successful. Additionally, the efforts of past and current governments to significantly expand support to incubators and accelerators and business innovation in Canada are strongly aligned with the ambitions of the CTA.

The CTA assists Canadian companies in their efforts to meet the demands of an increasingly complex and competitive global economy. Accelerating the business activities of a large number of SMEs in international market hubs aligned directly with the GoC’s aim to improve Canada’s competitiveness and to support Canadian firms as they pursue opportunities in the international marketplace. More specifically, the CTA initiative aligned with GoC’s ambitions to drive further improvements to the trade promotion and support service efforts of the TCS. It has done so by helping SMEs access global value chains and promote inward investment and innovation partnerships in markets where the TCS could have the greatest impact, thereby helping to grow Canada’s SME footprint as well as create new jobs for Canadians. Notably, the CTA was cited as being central to expanding the capacity of Canada’s business incubators and opening access to international business accelerators for Canadian companies. Highlighting the synergies between the CTA and other government programs, since 2013, 66% of CTA companies were past recipients of NRC-IRAP advisory services while 16% companies had BDC-Related Affiliations. Many more were experienced alumni of domestic accelerator programs supported by Canadian Accelerator and Incubator Program.

5.3 Relevance Issue 3:  Consistency with Federal Roles & Responsibilities

Finding #5:  The CTA is consistent with the mandate of the GAC and in particular the TCS program.

Historically, the GoC has been focused on strengthening Canada’s competitive position in traditional markets, extending Canada’s reach to new emerging markets and pursuing deeper trade and investment ties with the same via preferential trade agreements. However, Canada’s export profile indicates that, Canadian SMEs continue to face barriers to exporting their products and services abroad.  While the GoC maintains a roster of effective trade promotion tools, resources and services, such as those provided by the TCS, SMEs and start-ups require a more intense and focused form of facilitation and support in order to benefit from these international arrangements.^{Footnote 7}

With a mandate to provide information, advice and support to Canadian businesses as they seek to expand internationally, the TCS, located in over 161 sites around the world and with a regional network throughout Canada, has the resources, knowledge and networks necessary to support and operate the CTA Initiative. While EDC and the Canadian Commercial Corporation (CCC) have strong linkages to business networks, they are focused on significantly larger companies and contracts than the CTA. 

The TCS helps Canadian companies, through their traditional service and their more tailored CTA activities, gain market intelligence and insight, and uncover business opportunities in international markets. The privileged access of the TCS to foreign governments, key business leaders and decision-makers as part of Canada’s embassies and consulates was an often noted unique value proposition of the CTA as compared to other accelerator initiatives.

Furthermore, the depth of business contacts, including potential customers, distributors, sources of finance or investment, technology partners and intermediaries of the TCS make it a natural fit with the CTA. Reflecting the significance of TCS outreach for the successful operation of the CTA, 62% of all companies in the CTA in 2013 came from TCS referrals and 50% in 2014. There is no other government department with the access to the appropriate business networks, both in Canada and abroad, necessary to successfully run the CTA.

In response to today’s highly integrated global economy, the TCS is re-aligning its services to meet the broader range of needs of Canadian companies.^{Footnote 8} This includes the CTA Initiative. Described by the RO’s and TCS involved as the “Trade Commissioner Service for the 21^st Century,” the CTA enables the TCS and RO’s to reach, and recruit new and more varied clients. In the 2013 foundational documents it was cited that the CTA’s have served as a powerful client recruitment tool enabling the TCS to reach and service up to 10 times more clients in addition to those participating in the program. While such claims are difficult to verify, interviews with ROs confirmed that where they previously felt they had little to offer start-ups and SMEs in Canada, the CTA has given them a reason to reach out and engage with younger and smaller companies. RO’s cited that even if a company does not fit the typical TCS and CTA client profile today, they might in the future which is a benefit to the TCS and to the company.

5.4 Performance Issue 4:  Achievement of Expected Outcomes

Finding #6:  The CTA increases the capacity of SMEs to compete in the global innovation economy.

The expected results and measures of the CTA as outlined in the 2013 foundational documents stipulates that the impact/ results of the CTA on a company is measured by, in addition to the increase in the number of employees discussed in the previous finding, the increase in annual revenues, in capital raised and in the number of strategic partners and global customers created.

Since the CTA began systematically collecting performance data in 2013, approximately 348 new jobs (190 in Canada) have reportedly been created, $62.43 million dollars in total capital generated, $27.08 million dollars in total new revenue acquired, and 341 new strategic partnerships identified. In relative terms, this means that from 2013-2015, on average, 62% of CTA clients reported that participation in the Initiative resulted in new job creation. Furthermore, relative to the size of the Initiative, these numbers represent the important direct and indirect contribution of the CTA to the Canadian economy. The initial outcomes of the CTA echo what international research has found: that internationally engaged companies make a significant contribution to employment, both internationally and domestically.^{Footnote 9} As the CTA Initiative continues to grow, and more companies participate, this will only continue to contribute to the growth of Canada’s innovation economy at home and abroad. Ultimately, evidence has shown that the CTA, in line with the GoC priorities, is helping to grow Canada’s SME footprint and to create new jobs for Canadians.

While much of the CTA’s positive contribution to the global innovation economy can be measured there are many positive externalities that are not visible through quantitative data. For example, output and outcome measurements do not capture the international training, exposure and skills developed by companies and their personnel as a result of their CTA participation. Yet, this is important knowledge which can then be transferred to future enterprises as well as colleagues in the Canadian technology sector. All CTA administrators and participants cited that these are notable outcomes that contribute in a more indirect way to Canada’s ability to compete in the global innovation economy.

Finding #7: The provision of in-market services, and guided mentorship, through the CTA and TCS network has been highly valued by Canadian SMEs in targeted technology markets, contributing to their in-market success.

As previously noted, the CTA has been cited by a number of participants as helping to accelerate their business years ahead of where they would have been had they not participated in the Initiative. With regard to the services offered by the CTA and in particular the guided mentorship, the reviews have been similarly positive. Overall, exit forms from the CTA found have found that in FY 2014/2015, 90% of CTA participants believed the CTA had helped them meet global partners and customers relevant to their business and greater company success. Furthermore, since 2013, 91% percent felt the CTA had helped improve their understanding of the local market while 78% believed they had reached their objectives while participating in the CTA. As seen in the table below, participant satisfaction ratings for the CTA are high and have increased over the two years for which data has been collected.

The access the CTA provides to mentors and the TCS network, as well as the in-market address and office space, are key offerings of the Initiative and highly valued by CTA participants. With the exception of the virtual CTAs in New York and San Francisco, India and Philadelphia, the access the CTA provides to free office space within existing international business accelerators is a key offering of the Initiative. This is the case as acquiring and paying for competitive office space in these markets is an otherwise noted challenge for small and young companies in foreign markets.^{Footnote 10} Some CTA administrators felt that in some locations office space could be rented relatively easily by the companies themselves on an as-needed basis instead of as part of the CTA offering. However, participants across all locations viewed the office spaces as important to creating direct and indirect business linkages and the networks needed to advance their business interests. In some cases this the in-market location has even resulted in joint business ventures by CTA participants. Overall, for year 2013/14 and 2014/15, 66% of participants felt the office space was useful for their time in-market. While many interviewees reported that office spaces were commonly unoccupied or used only intermittently, this should not be viewed negatively as successful CTA clients are expected to be networking and making connections in market.

In addition to the in-market office space, mentorship and engagement with industry experts were cited by CTA participants as particularly beneficial to their CTA experience. From 2013-2015, approximately 90% of completed feedback forms rated this experience positively. Significantly, the variety and depth of the mentors linked with the CTA Initiative were cited as integral to the experience, providing a number of services including, local market knowledge, brokering key relationships and business opportunities, and orienting companies in relation to the competition, and finally advising companies on their compliance with domestic regulations. The DEEP Centre found and this evaluation confirmed that the role of mentors and business development consultants was especially important where the TCS lacked the industry networks entrepreneurs wanted, and/or where companies perceived the Trade Commissioners as unable to provide them with the personalized service necessary for them take full advantage of their time in-market.^{Footnote 11} In addition to the mentorship, the hired consultants such as pitch coaches and Intellectual Property lawyers were reviewed highly by participants and also cited for their role in helping CTA companies gain new customers or venture capital and/ or Angel investors.

While mentorship has been highly valued by the surveyed participants, there exists, however, significant variation in the depth and availability of the mentorship offerings across CTA locations. Among companies that have participated in multiple CTAs it was highlighted that the level of sector knowledge and assistance provided varied dramatically. In some CTA locations there is a roster of volunteer mentors while in others paid business development consultants are hired to mentor companies. This produces a notable variation in mentorship opportunities and experiences for CTA participants. These variations can be simplified into two mentorship structures and ideologies: paid and non-paid mentors. Paid mentors are described as being engaged as “business development consultants” where as in the un-paid model the mentors are all volunteers, sometimes Canadian expats and other times Americans with strong ties to the TCS or an interest in helping Canadian SMEs grow. Interviews with participants and stakeholders revealed mixed opinions regarding the payment of mentors. While some believed it is ethically wrong and signifies a transition from “mentorship” to “consultancy”, others maintained it made no difference or that it was impossible to engage sustained and dedicated support in certain markets without financial compensation. Participants were, however, consistent in believing that regardless of whether a mentor was paid or a volunteer the details of the arrangement should be disclosed to participants.

Ultimately, where mentorship and company interests did not align or the availability of mentors or consultants was unreliable this negatively impacted participant experiences. A best practice identified in Boston to overcome such challenges has been to have mentors sign a clear contract stipulating the terms of their involvement in the CTA. This mentorship contract in addition to investment guidelines is based upon best practices established by Massachusetts Institute of Technology (MIT) in conjunction with the Canadian Mentoring Service. This contract clearly outlines the roles and responsibilities of the mentors and CTA participants and adds structure to the CTAs overall operations. Currently, such guidelines do not exist at the majority of CTA locations.

Finally, in the delivery of CTA activities, the TCS performs an essential role in ensuring the success of the Initiative. The service aims to provide warm introductions to local investors, potential clients and partners and reduces the time companies would otherwise need to spend in market searching for such contacts. Many exit forms as well as internal and external stakeholders and mentors cited the key role of TC officers in making crucial introductions in all locations. The TCS was repeatedly cited by stakeholders as unique to Canada’s CTA model, distinguishing it from other international accelerator programs which lack such an extensive and dedicated network of officers. Despite the large reach of the TCS networks, the CTA remains reliant upon one or two key individuals in each location. There appears to be few mechanisms in place to ensure the consistent continuation of services offered upon the departure of a key individual involved in the CTA’s administration. There is, therefore, a question of the sustainability of the Initiative under such a model.

Finding #8: A key contributor to the achievement of the CTA’s successful outcomes is its flexibility and the ability of missions to tailor CTA programming to company needs. However, where this is not balanced with standardized and structured programming, significant obstacles to the CTA’s success emerge.

One of the most significant strengths of this initiative remains its adaptability, agility, and the willingness of staff to draw upon their networks in-market to address the needs of CTA participants. Such customization helps the CTA achieve its intermediate and longer term outcomes by matching participants with mentors and business leads that could generate new revenue, jobs and growth.

While the ability of the CTA to tailor their service offerings to clients based on the local market demands and culture is highly valued, the absence of minimum standards adversely impacts the CTA’s performance. In exit forms, one of the most significant critiques of the Initiative from participants was the lack of consistent service offering across CTAs. For example, it was reported in some locations there was no orientation for the participants upon arrival, few progress check-ins, few opportunities to showcase their business through the CTA, or CTA-organized networking events. In a few extreme cases, there were no mentors made available to participating companies while in other instances many participants found the consultants unapproachable or unresponsive to their needs. Such inconsistencies in service delivery across and within CTA locations were frequently reported to have negatively impacted company experiences.

Among program administrators at Post there are significantly mixed views regarding the need for such structured programming. While some have argued that they are not in the business of providing “mentorship 101” and that companies should be ready when they arrive at in market, other Posts have seized the opportunity to provide support to CTA clients in market through structured programming. Given the large variety of companies participating in the Initiative, all with different needs, some maintained that the development of any standardized service offering across all locations would be problematic. This said, participants, internal and external stakeholders and partners agreed that some consistency must be established, especially among the U.S. CTAs. Such a move is viewed as helping to ensure a basic level of service for participating companies while also providing market specialization.

Finding #9: Virtual pre-programming or complementary CTA programs ensure the market readiness of companies and the alignment of company expectations with deliverables.

As the CTA Initiative has evolved over the years, several Posts have now transitioned away from companies arriving shortly after admission to using an “on-boarding process” to ensure that companies are ready for their time in market. Such an approach helps to ensure that the expectations of the companies and the CTA offering are aligned. These range from the ‘48 Hour’ sessions and workshops run by partners in San Francisco, Palo Alto and Boston to virtual orientation programs administered by CTA personnel. These complementary projects connect CTA companies to the TCS and mentors prior to their arrival in market.

Interviews and the review of exit forms revealed that virtual programming or complementary pre-CTA programs such as “48 Hours in the Hub” in Boston and “48 Hours in the Valley” in Palo Alto have helped companies and Initiative administrators align their expectations with the CTA’s offering. They have also helped companies better prepare themselves for the local market. These programs furthermore appear to reduce the challenges younger companies face when confronted with the task of orienting themselves to a new city and environment while also making a strong start in the CTA that will produce a return on their investment.^{Footnote 12} Notably, while these services are often viewed as necessary precursors to the CTA by both administrators and participants, they have not been considered officially part of the CTA since the 2013 central agency approval.

Similarly, addressing participant concerns that they felt they were unprepared for the rapid start of the Initiative; the TCS in New York has develop a virtual on-boarding process. During the virtual process, which was developed by and has been administered by CTA personnel, the TCS and mentors, ensure a company is prepared and ready to come to market by reviewing their business plan and setting expectations on the kinds of support the company will receive while in market. In doing so, administrators confirm a company has identified key contacts for their time in market, and that the company’s needs and the mentorship offering are aligned with the CTA, prior to their arrival. Given the short duration of the CTA this has improved the ability of CTA participants to achieve successful outcomes and obtain a return on their investment.

However, participants through the exit forms and interviews suggested there was also a need for a pre-arrival information package with logistical details. More specifically, requests were made for information on housing, local business culture, background information on mentors, and guidelines on what materials participants should prepare in advance. Overall, information on the structure of the Initiative and CTA programming were suggested as areas where the CTA could improve communication with participants to ensure readiness.

Finding #10: Where initially the program accepted a large number of companies, it is now focused on attracting the most qualified firms and generating smaller and longer cohorts to ensure that companies that complete the program are ready for the market.

The financial support provided to the CTA from GoC was expected to result in 500 high-growth SMEs benefiting over three years with increased success in the CTA over time. This number was benchmarked against the 210 companies that had benefited from the CTA since the first pilot in 2009. Since the Initiative’s receipt of central agency funding 2013, 274 companies have participated in the Initiative. Thus, the initiative is unlikely to meet this benchmark. Instead it is estimated that up to 350 SMEs will have benefited from the Initiative when the funding sunsets in 2016.

A lower number of expected companies can be explained by a number of reasons. Over the course of the first two fiscal years each CTA location has accepted fewer applicants. Since the number of CTAs also increased over this period, the decline in companies per CTA was offset and thus the CTA Initiative as a whole consistently accepted the same number of companies from 2013-2015.

• FY 13-14: 20 group intakes in 8 CTAs
• FY 14-15: 21 group intakes in 12 CTAs
• FY 15-16: 18 group intakes projected in 11 CTAs

Notwithstanding the foregoing, the number or participants is expected to decrease in FY 2015/2016 as the majority of CTAs have planned to have longer cohorts with fewer intakes. In most of the locations the CTAs expect to take 2 cohorts per year for 5 to 6 months rather than 3-4 cohorts a year for 3-4 months. The notable exceptions are Boston and London, which have moved away from the standard cohort model to rolling admissions that will accept companies throughout the year.

As remarked earlier, the lower number of participants does not suggest a decline in demand for the CTA, but rather evidence of a shift in focus towards attracting more mature companies better able to benefit from the Initiative. In the early days of the Initiative, very early stage companies were recruited and participated. In hindsight CTA administrators realise these companies were not ready and were not very strong candidates for the Initiative. Today, documents note that the “ideal” CTA participant is a small to mid-sized Canadian company from ICT, Life Sciences or Sustainable Technology industries with initial traction in the market, differentiated technology, and a potential to scale their business. Reflecting the shift towards more mature CTA companies, in November 2014 the average client prior to joining the CTA was 5-6 years old, had $780,000 in revenue and $2.75 million in Capital. By the end of FY2014/2015, the average size of a CTA company had grown. The average company was 6-7 years old with 19 employees, 1.5 offices, $1.6 million in revenue and $3.5 million in capital. Notably, in FY 2014/2015 the addition of the CTA’s in London, Paris and India may have also increased the recorded average size of CTA companies as participants and applicants at these locations had on average, between 14 -36 employees, between $853,000- $2.78million in revenue and between $2 and $4.4 million in capital.

The recruitment of such high-potential companies is now the biggest challenge many CTA administrators at Post face, with many expressing concern that they will not be able to find enough companies to deliver on their key performance metrics and fill desk space.^{Footnote 13} CTA administrators also found that with four intakes per year and the need to recruit more highly qualified companies, they were constantly recruiting. The amount of time spent recruiting was cited as unsustainable and conflicting with their work as TCS officers. Thus, the longer group cycles and fewer intakes were established in an effort to alleviate the time and resource constraints administrators previously faced when attempting to complete CTA recruitment, selection and programming.

While the 4-month cohort model initially adopted by the CTA Initiative is aligned with private and international comparators, many CTA companies reported in their exit forms that their time in the CTA was too short. In year 2, for example, in response to client demands, CTAs increasingly extended their participants into subsequent group intakes, or accepted previous CTA alumni from different CTAs. The percentage of companies participating in multiple CTAs grew from 7% in FY 2013/2014 to 25% of total participants in FY 2014/2015. Overall, since 2013, 15% of CTA companies have participated in more than one CTA. Thus, the shift to longer cohort model is also a reflection of company needs and demands.

Finding #11: As the CTA matures in the US and expands to the UK, Europe and India it continues to offer tailored services to participants based on market opportunities, however challenges have emerged regarding consistency in CTA offerings.

The Initiative’s expansion to UK, Europe and India are viewed by some CTA administrators and stakeholders as sequential steps for Canadian companies at different stages of maturity, and with different market goals in mind. In this sense, the U.S. is seen as a first step for the potential growth of most companies. In the U.S. the focus is on the access to international technology scouts, venture capital and the ability to validate market appeal. The overseas CTAs, on the other hand, because of the increased financial barriers, are expected to host even more mature companies, perhaps with double the revenue and capital than those in the U.S. These companies are also expected to likely already have initial traction and success in the U.S. market. The CTAs outside of the U.S. are therefore often, but not always, seen as the next step in internationalization for companies seeking to enter other international markets and participate in European and Indian supply chains. Similar to the U.S. virtual program, the initiative in CTA in India, which is the first emerging market CTA, serves as a largely virtual CTA and soft landing spot for more mature companies seeking assistance to operate in a more complicated regulatory, legal and commercial environment.^{Footnote 14}

As the CTA expands, there are mixed opinions relating to the ability of the Initiative to tailor uniquely to specific companies. An ongoing challenge is how to provide consistent but flexible and customizable CTA offerings across locations, without deviating significantly from the CTA brand or placing unreasonable burdens on the TCS at Post. With such a variation in the offerings across CTAs in the U.S. and abroad, challenges have emerged with aligning company expectations and needs. Improved information dissemination prior to a company arriving at mission aims to close this gap. However, company exit forms highlighted mixed opinions regarding the ability of the Initiative to respond to their needs.

Finding #12: The CTA brand has strong value among participants and potential investors. However, weak communications efforts with internal and external stakeholders and administrators have resulted in confusion regarding the CTA’s offering. It has also hampered the Initiative’s ability to establish a clear and consistent CTA brand that can help the Initiative and its participants achieve successful outcomes.

Generally, participants value the CTA brand since participation in the Initiative is associated with success and an endorsement by the GoC. In many cases, companies felt that it was an added bonus to be able to inform potential investors that that they were selected by the GoC to participate in the CTA. In this sense, the CTA acted as a stamp of approval for many companies.

However, as the Initiative has grown, the lack of a standardized service offering has made projecting a consistent CTA brand a challenge. While each CTA provides important and tangible benefits to participating companies, the value propositions of each CTA differs, leaving companies and stakeholders unsure of the differences. This makes it a challenge to communicate the differing value propositions of the CTA sites to possible participants and interested parties. Many internal and external partners remarked that they were rarely clear on the differences between the CTAs and how to refer clients. For example, many of the CTAs focus on ICT start-ups. But some markets focus on Ad-tech or Fintech while others on Cleantech or digital health. In addition to these differences, there are also varied levels of service offerings and expectations from the administrators and companies.

While significant efforts have been made to clarify what the CTA offering is in each location, further specification of what the term “Accelerator” means to the CTA is not only important for aligning company expectations with CTA offerings, but also for ensuring that more mature companies do not overlook the initiative because of the term “accelerator”, which is more often than not associated with younger companies. Some participants for example came to the CTA expecting more intense, hands-on level of assistance based in large part on the “accelerator” brand. While in some cases, and at some sites, such assistance was available, the lack of a standardized model of delivery has meant that some sites fail to meet the service delivery expectations of some incoming companies. Additional concerns emerged over the term “Technology” as it was suggested in interviews with several TCS that the Initiative is accelerating businesses as opposed to technology.

Thus, while it is important to balance flexibility and market customization, interviews highlighted a strong importance should also be placed on communicating the basic offering of the brand. Once established, consistency across all locations regardless of the market specialization should occur. Thus, as the Initiative continues to refine its selection criteria the CTA should not only establish consistent requirements and expectations for use of the CTA name but also a clear, concise branding proposition. Improved communications with ROs, partners and potential investors and participants regarding the program offerings, deadlines and benefits of the CTA are needed to ensure this occurs.

5.5 Performance Issue 5:  Demonstration of Efficiency & Economy

Finding #13: The absence of a clear delineation of the different roles and responsibilities of all personnel involved in the Initiative has constrained its ability to administer the CTA.

The CTA’s foundational documents of 2013 stated that two term positions would be created at HQ to take on the tasks, associated with the transition of the Initiative from a pilot to a formal Initiative and to provide the required coordination. These positions were also charged with being responsible for ensuring consistency and communications across CTA Missions, Business Sector divisions and Geographic divisions and for establishing and maintaining an internal governance structure, reporting framework and performance measurement framework. Finally, they were tasked with managing cross-sector, multi-mission and stakeholder alignment and disseminating strategic intelligence on promising SMEs within the TCS network including making links to emerging markets.

Furthermore the foundational documents allocated funds for the hiring of four LES positions in Palo Alto, New York, Boston, the US and UK to support the transition from the pilot, and enable the TCS in global hubs to more effectively leverage the CTA Initiative in an effort to increase benefits to all TCS clients, including alumni. In practice, the discretion was left at Post to determine the job descriptions pertaining to these positions. In New York the LES position emphasized communications, while in Boston they were expected to contribute to the delivery of the International Business Development program at the mission and in San Francisco to contribute to the commercial program.

Despite the foundational documents of 2013 outlining the responsibilities of the two term positions at HQ and those of the four LES’ at Post, interviews with staff at Posts, ROs and at HQ frequently cited that identifying roles and responsibilities of those involved in the administration of the CTA was a challenge. This occurred for several reasons. First, with the CTA having been operational in many locations prior to the 2013 submission, many locations had already established modes of operating that were resistant to outside influence. Second, as a result of a delay in funding many of the positions established by the 2013 foundational documents were only filled in late 2013 or early 2014, leaving little time to establish a significant and clear governance structure with accompanying clarity regarding their roles and responsibilities. Third, there has been little clear guidance from HQ to TCS officers on how to balance their regular roles and responsibilities with those of the CTA.

A general inconsistent understanding of the roles and responsibilities of personnel appears to have produced inefficiencies in delivery of activities and at times may have even resulted in the loss of potential participants. More specifically, the lack of clear governance structure has produced: a fragmented delivery of services and an inconsistent reporting of the data necessary to assess the Initiative’s performance. It has also resulted in miscommunication with partners and stakeholders, as well as potential, current, and past clients.

Finding #14: Balancing traditional TCS responsibilities and methods of reporting with the operations of the CTA has created a tension for CTA administrators who expressed concern over their ability to meet traditional TCS demands and performance metrics.

Balancing traditional TCS roles and responsibilities with the operations of the CTA was a concern expressed almost unanimously among CTA administrators. Specifically, it was cited by many CTA administrators that the activities in the CTA and the clients served are not traditional for the TCS. Thus, much of the work done for the CTA Initiative such as recruiting and maintaining a roster of mentors or business development consultants, hosting orientation sessions and other Initiative activities, identifying high-quality accelerator spaces and finally assessing the admissibility of potential CTA clients are not accounted for in Key Performance Indicators (KPIs) or metrics for performance measurement found in TRIO.

Administrators expressed that their CTA roles are in addition to the traditional roles and responsibilities they have as TCs yet most successes are not attributed to them under the current model. Thus, while all CTA administrators saw significant value and commitment to the CTA, many administrators expressed that it was a significant drain on their time that does not yield the outcomes that are important for assessing their job performance.

Recognizing that TRIO is not designed to capture all the types of services that the CTA provides, there are nonetheless opportunities for staff at Post to record more of their activities in TRIO. This evaluation found that there was a significant gap between what could be reported by TCs and what was actually being reported. For example, while many at Post argued there is not a clear alignment with respect to the definition of “services delivered” or “leads” to a success under TRIO, interviews found that TCS officers involved in the CTA would record their “services delivered” in general terms as opposed to specifying what exactly the team was doing to tailor the services of a CTA to a specific client. As a result, the service would be downgraded at HQ to an “interaction” which is not used to assess the performance of the TCS outside of the mission.

Finally while the majority of CTAs have expressed challenges in integrating CTA activities with TCS KPIs, some CTAs have worked to ensure that where possible their CTA KPIs are aligned with those of the traditional TCS. While elements such as knowledge learning, company pivots and improved pitches, cannot be measured by TRIO, administrators felt this effort has resulted in a more harmonized system and less tension between their roles. In contrast, other Posts were found to be using TRIO incorrectly by inputting identical notations for different companies rather than specifying what was unique for this company in “services.” By doing so and not tailoring the action to what a TCS officer or team may have done for an individual client, the service becomes downgraded to an “interaction” which is not reported on to HQ. However, while some CTAs are using TRIO incorrectly, in other cases CTAs were not using TRIO at all, citing that it was not relevant for the CTA and that there was little time to focus on such inputting.

Finding #15: The decentralized nature of the CTA, which allows for important in-market specialization, is crucial for achieving the Initiative's successful outcomes. Yet, in many cases it has also resulted in a duplication of efforts by CTA personnel across locations.

The CTA administrators at Posts and in Ottawa were found to be highly reflective of and responsive to the needs of companies and their particular market challenges. Interviews highlighted the strong willingness of CTA administrators to learn and build upon previous cohorts or from other CTAs to ensure the development of responsive and successful programming. This held particularly true in company recruitment, the development of mentorship programs, and the development of networks with accelerators in Canada. In most cases, this resulted in the CTAs further refining their offering to match both company demands and the capacity of the CTA to deliver responsive services. In the case of newer CTA’s this learning was cited to have helped the programs avoid many of the challenges experienced in the early pilot years of the project in San Francisco and Palo Alto.

Newer CTAs in London and India for example were able to take advantage of several years of CTA experience and best practices and build and tailor their programming accordingly. Following the first cohort, in these locations like their counterparts in the U.S. continued to adapt as they learned more about their market demands and potential CTA clients. As a result, London, for example, has found that shifting to a rolling admissions rather than a structured admission process allows them to capture more CTA companies that are ready and coming to market than if they make these companies wait until the next cohort. Similarly, in San Francisco and New York these CTAs have found that extending the cohort length and allowing overlap between CTA companies at the accelerator breeds connections and even informal mentoring between CTA companies.

While each individual mission has demonstrated a clear understanding of their own respective market challenges, it was found that they often fail to view how their particular CTA is situated within the larger Initiative. It was found for example that the different CTAs as well as their partners were at times unable to refer interested CTA participants to the CTA location best suited to their needs. When this has occurred, evaluators observed that it could result in the same company being referred multiple times and to multiple locations before they were matched with the most appropriate CTA. This results in not only in a duplication of efforts by CTA personnel but also reflects poorly on the Initiative to the prospective client.

While some sites are cautious of standardization within the Initiative, templates and general orientation packages with information specific to the U.S. market were noted by the evaluators as well as external consultants to be something that would be of use to the Initiative. Furthermore, at Posts, the TCs have, in several cases, crafted or begun to craft a refined and structured offering that combines sector-specific insights with an introduction to the intricacies of the host market. In doing so, the CTAs have responded to the demands of companies, for more structure in the Initiative while also striking a balance between providing a consistent set of high-quality program deliverables and a more flexible array of local and sector-specific services.

Such efforts also have the added benefit of building in a small degree of positive redundancy into the system which ensures that even when key CTA officers rotate to new posts or leave the TCS, there are still some structures and mechanisms in place to ensure fewer challenges in transition. This has become increasingly relevant as the initiative has grown and some coordination and standardization has become necessary to ensure the continued achievement of the CTAs successful outcomes across locations through an efficient use of resources.

Importantly, in recognition of the need to avoid duplication, CTAs have made significant efforts to enhance coordination and prevent duplication. A notable example of this came early on in the pilot with the CTAs in New York City and Silicon Valley building a joint life sciences virtual program that allows companies to pick the services and connections that best suit their needs. The impetus for this collaboration came after both CTAs separately piloted independent programs in 2011 and quickly realized that it did not make sense to duplicate efforts given the overlapping networks of investors and end clients across the U.S. and the fact that there were not enough Canadian companies to warrant two separate CTAs.

Finding #16: While efforts to refine CTA participant requirements have been made, the Initiative has yet to cultivate a consistent and strategic recruitment and application process.

Since the initial pilot project in 2009 the methods for recruiting CTA companies have evolved significantly. Where the initial pilot project involved the recruitment of companies known to the mission and ‘trade show’ style recruitment by TCS officers across Canada, later iterations have involved referrals from regional officers and partners, and recruitment from domestic accelerators. Currently, there is no strategic outreach plan for how to recruit CTA companies, a problem which was highlighted by multiple CTA stakeholders in interviews. Presently, potential CTA companies are engaged via the TCS networks, including ROs, other CTAs and Canadian missions. Additional recruitment is done through partners such as BDC Capital and NRC-IRAP as well as non-profits such as the Canadian Digital Media Network.

With regard to current recruitment efforts, a sample analysis of the applicants for the winter 2014/2015 cohort found in the table below demonstrates that approximately 39% of applicants heard about the CTA through the TCS service either domestically or abroad. Other government contacts accounted for approximately 20% of applicants while social media accounted for approximately 10% and CTA alumni and domestic accelerators and incubators accounted for a further approximately 6% respectively.

Furthermore, while ‘Social Media’ was less than 10% and “Alumni” less than 7% in this sample, some of the CTA’s most successful companies as measured by jobs created, revenue earned and capital generated, heard about the Initiative via social media such as Twitter and through word of mouth from CTA alumni. It was further noted by partners and former participants that entrepreneurs in the tech world mostly listen to other entrepreneurs and find these to be the most credible references in applying to programs to accelerate their business. With less than 20% of companies coming to the CTA via social media and alumni networks combined these findings therefore suggests that the CTA is potentially missing a large and viable pool of potential participants.

Furthermore, much of the current outreach has historically focused on domestic accelerators and while this outreach is important, graduates of Canadian accelerators are typically not ready for the CTA Initiative. Most interviewees agreed that there is usually an 18-30 month gap between when a company comes out of a domestic accelerator and when it is truly ready to go to an international market to participate in a CTA Initiative. As such, there is a need to ensure that recruitment efforts extend beyond existing domestic accelerators.

With regard to the recruitment of companies, overall, it was determined that establishing a common outreach strategy remains a problem and external partners felt there was room for improvement. It was frequently referenced by CTA administrators that they had already accepted the companies known to them and that they would now have to spend more time on recruitment. With regard to how to recruit companies, some CTAs and personnel advocated for casting a wider net through broad advertising and marketing, while others were concerned about their capacity to process a larger amount of applicants and whether the Posts had the staffing availability to do so. It was also suggested by CTAs and mentors that the “CTA tours” be revived, allowing TCS officers from Post to go to Canadian accelerators and recruit viable companies. Other partners suggested pitch competitions to recruit, attract and vet Canadian companies in major cities in Canada could also be of U.S.

With regard to the current application process, despite the use of a uniform template, each CTA is allowed to establish its own selection committee and criteria as well as their own application deadlines. In some cases there were large selection committees made up of volunteer mentors, the TCS and even a BDC representative where as other CTAs had smaller selection committees or referred companies on an ad-hoc basis to relevant partners for input on the viability of a company. While it is important that the final selection of companies be left to the discretion of individual CTAs, the selection process was frequently criticised by many partners as not being transparent or responsive to the applicants who may have been denied entry into the program.

While individual missions have developed matrices to track and evaluate applicants, there are no standardized criteria for evaluating CTA candidates and as a result assessment criteria vary between missions. Furthermore, ROs frequently reported that they were not advised of the outcome of a company’s application, nor were they given feedback on how unsuccessful companies could improve their prospects of participating in the CTA.  This makes it a challenge for ROS and partners to provide counsel to prospective applicants.

Another concern with the application process is the fact that each CTA has a different submission deadline, which is often extended.  While such extensions may occur in response to a lack of applications, in-market or mission demands, all partners and external stakeholders interviewed uniformly found this ad-hoc application process to be a significant weakness of the Initiative. The lack of communication regarding application deadlines combined with a lack of knowledge of the various CTA offerings makes recommending or referring CTA locations to clients a challenge. Frequently, internal and external CTA partners, including ROs cited that they did not possess enough information regarding the different CTA locations to appropriately refer an interested company. Often, RO’s would simply refer on the basis of the strength of their relationship with a particular TCS as opposed to company needs.

The lack of information available to applicants on each CTA was cited as a roadblock to determining which location to apply.  The weak understanding of the different locations among partners and ROs has reportedly only increased with the expansion of the Initiative to Europe and India.

An additional concern raised with respect to the recruitment and application processes relates to the integrity of these processes and the need to ensure that funding partners cannot influence or ‘buy’ seats in the Initiative for their respective companies. While there is no evidence of funding partners such as the provinces or federal entities systematically securing seats in the program in exchange for funding, concerns were raised over this possibility. It was believed that any move away from a completely meritorious recruitment and selection process would be exceptionally harmful to the program. The data supports this belief. In a review of all recorded CTA dropout companies, companies that the CTA had initially rejected based on lack of merit, but were then admitted later because of encouragement or pressure by a funding partner, failed to complete the program, or even sometimes failed to show up in market at all. A competitive process therefore remains central to ensuring that the most deserving companies are accepted and that the CTA Initiative can deliver the greatest economic benefits to Canada.

Finding #17: The expansion of the CTA to new locations has resulted in some confusion over the selection criteria and program offering required of each CTA and the amount of funds available to each.

The foundational documents for the CTA of 2013 state that the selection criteria for a CTA would include, as a minimum, the in-market accelerator’s capacity to: provide dedicated office space; the success of past participants (in growth of revenue, investments, employees, and customers); active and influential network/mentors; and, programming that increases access to business development opportunities. The submission furthermore stated that the success of the CTA initiative is dependent on establishing a presence in key global markets to leverage the location and influential communities to increase opportunities for innovative and high-growth-firms. By situating the Initiative in some of the world’s largest tech markets or specialized markets, the CTA has largely remained true to the criteria outlined in the Initiative’s 2013 foundational documents. Furthermore, by ensuring that proposals for CTAs are aligned with CTA objectives and that funding is allocated to achieve results as outlined in GoC and CSF/ITSF guidelines, the CTA Initiative has had notable success thus far. This said, many CTA administrators at Post remarked that they were unaware of any specific eligibility criteria for establishing a CTA and for determining the quantum of funds required to operate one effectively.

For example,  a review of the Strategia Commercial Action Plans for the FY 2014/2015 (see table below)  showed there to be little alignment between TCS KPIs found in TRIO and the allocation of funds by HQ.  Administrators noted that they do not currently use TCS KPIs for (identifying locations) and allocating CTA resources. Rather, when determining the allocation of funds, CTA administrators have relied on more qualitative data for the establishment of CTAs such as market location, the number of projected participants, the mission’s capacity to deliver, and the proposal’s alignment with other ITSF initiatives such as sector practices.

With the Initiative’s success, Posts are increasingly requesting CTA locations. In this context communicating with posts a clear formula for selection will be of increasing importance and will ensure that CTAs continue to achieve the Initiative’s expected outcomes. This holds particularly true as over the course of the evaluation questions were raised regarding why some locations over others would warrant a CTA. Notably, there are other potential locations which consistently rank highly on several metrics for potentially hosting a CTA. In this regard, interviews with relevant stakeholders emphasized that expansion does not always mean better performance and that clear criteria should be established and a business case made for being permitted to host a CTA. It was highlighted that efforts should also be made to ensure that the demands of companies drive the creation of CTAs as opposed creating artificial demand by establishing CTA locations into which companies then try to fit.

The 2013 foundational documents and central agency funding guarantee that the initial pilots in San Francisco and Palo Alto, New York and Boston as well as the new location in London receive core, guaranteed funding. The remainder of funding is allocated via competition to CTA applicants. However, because of the limited budget and the effort of the CTA to collect year over year data to assess long term results, the same CTAs tend to receive funding year to year with no new locations added in FY 2014/2015 or FY 2015/2016. Thus, while officially the allocation of CTAs is a competitive process, in practice there are very strict limitations on how truly competitive the Initiative can be in its allocation of funding to new Initiatives.

Finally, through site visits and interviews, the evaluators found there is no “one-size-fits-all” model to ensuring the successful development of a CTA. A notable strength of the program is its flexibility and ability to tailor towards the local market. This however must be balanced with meeting the key CTA objectives and the desired outputs and outcomes of the program. Thus, the current annual review and competitive analysis of each CTA location is of significant importance.

A tension often emerged between the “original” CTA locations and the newer locations established overseas which target more mature companies. CTA administrators often questioned what the offerings of the new CTAs were and how they fit into the broader objectives of the CTA Initiative. Many expressed concern about ensuring a consistent brand offering and maintaining a clear value proposition for the CTA as the Initiative expands. While the evaluation found that in many cases, though not exclusively the CTA’s located overseas do appear to often target more mature companies with regard to the average number of offices, employees, revenue and funding raised they largely continue to fulfill the requirements of the CTA and in one location have even made dedicated efforts to learn from the best practices of the American locations.

Finding #18: The overall CTA budget remains fixed but the growth of the Initiative and fluctuations in the value of the Canadian dollar have resulted in decreased financial resources.

In the 2013, the CTA was allocated $5 million dollars with central agency approval from 2013-2016. The budget is largely fixed with approximately $2 million in funds made available to the Initiative each year. Additional funds have been made available to individual CTAs through provincial partners, BDC and NRC-IRAP. At the time of the evaluation, in the only full year of funding, FY 2014/2015, the Initiative has had a very small portion of lapsed funding. In FY 2013/2014 lapsed funding accounted for approximate 1.9% of the total budget.

With regard to allocation, BBR determines the Post totals based on requests. In the case of Posts with multiple CTAs, a block amount of funding is allocated with each post then allowed to determine how to allocate this money internally for each CTA. As indicated in the table below, the overall budget for the CTA has remained constant with minor fluctuations in the distribution of funding to each CTA Post. The notable exception is San Diego, where the CTA was closed following the 2014/2015 FY due to administrators at HQ determining that the cost of the Initiative relative to the demand by Canadian SMEs did not warrant a CTA. Including the funding from BDC, small decreases in funding occurred in Boston and London with larger decreases for San Francisco and Palo Alto from 2014/15 to 2015/16. While a significant decrease was seen in the funding allocation for San Francisco and Palo Alto in FY 2015/2016 this is largely attributed to the funding for the virtual CTA, shared with New York, currently being counted in New York’s funding allocation.

The majority of CTA locations expressed the desire for additional financial resources to support their particular CTA. This was particularly true in the case of CTAs located in the U.S. where the Posts felt they were being asked to do more activities with less funding due to the declining value of the Canadian dollar. CTA administrators noted that with this new challenge in FY 2014/2015, it has required all Posts to stretch their funds much further to provide and maintain activities. It was also noted by administrators that with the falling value of the Canadian dollar it also increases the costs to Canadian companies moving to the US market for the duration of the CTA.

Finding #19: The allocation of CTA resources are not presently aligned with the demands placed on the various CTA locations. As result this is limiting their capacity to deliver the Initiative.

A majority of CTAs expressed the concern that they did not possess the resources necessary to adequately support the CTA Initiative while also fulfilling their responsibilities as TCS officers. Many TCS attributed their resource constraints to their inability at times to provide responsive programming to CTA participants at their respective locations.

When the FTE to applicant and participant ratio is examined a significant variation appears. Specifically, as demonstrated in the table below, in New York, San Francisco and Silicon Valley and Boston it was found that funding and the number of FTEs hours allocated relative to the number of CTA applications and participants varied dramatically. Specifically, those Posts with the most applicants and participants tended have the least FTEs dedicated to the Initiative.  In one instance it is noted that almost double the number of FTE hours were allocated to a CTA that had approximately a quarter of the applicants and participants of another.

In response to concerns over resource constraints by those CTAs which reported being overburdened, it was recommended that the number of cohorts and seats be reduced. However, interviews at Post indicated that requests to reduce the foregoing prior to mid-2014/2015 were not accepted by HQ. This said, several CTAs viewed the 2015/2016 model of longer and smaller cohorts as a welcome change to balance such responsibilities.  This highlights the need to assess the capacity of each CTA individually to order to ensure an ideal allocation of time and resources based on the demands of the Post.

Finding #20: Reporting on CTA performance and outcomes remains ad-hoc, inconsistent and lacks standardized procedures.

There are several reasons for the lack of clear reporting and performance measurement data. First is that until FY 2014/2015 there was no consistent reporting mechanism to track applicants and participants. For example, the lack of standardized templates prior to late 2013 resulted in inconsistent and inflated numbers of applicants being reported by missions in FY 2013/2014. The application template introduced for FY 2014/2015 created a standardized form for all applicants. Notably, however, there continues to be a wide variation in reporting by Posts on “participants” and “drop outs”. As a result of a lack of clear direction on performance metrics, each CTAs use different performance metrics. For example, each Post uses a different definition for what it means for a company to “complete” or conversely “leave” or “drop out” of the CTA. Furthermore, there is currently no guidance on establishing standard criteria for what is necessary for a company to fall into any of these categories or be considered a “graduate”. Thus, the reliability of the reporting from Post on how to classify a company’s experience is also in question. Minimum standards and guidelines would result in better quality and improved accuracy.

Additionally, participant information was often found to be lacking. This included information on who the representative of a company at the CTA was as well as information such as their gender and age and if this was their first time participating in the CTA. This type of information would enable the CTA to develop a better understanding of their participating company profiles and possible positive externalities of the Initiative. It would also provide more detailed information on what company characteristics and policy interventions lead to success.

One of the most significant challenges for the CTA is measuring its outcomes. Since its inception as a pilot in 2009 the CTA has not focused heavily on quantitative and qualitative performance measurement. Further, a delay in the allocation of central agency funds until late in 2013 resulted in anecdotal data gathering from individual missions without independent corroboration for the majority of FY 2013/2014. While data on key metrics is listed in the CTA Logic Model and 2013 foundational documents have been collected through the CTA exit forms, this reporting only began with the central agency financing in 2013. Prior to this, data on key outcomes such as revenue generation, capital investment, new networks, leads, prospects, and jobs created, when collected, was done at Post with no rigorous or standardized form of tracking. Thus, there was very little reliable year of year data available for the evaluators to examine when looking at the quantitative outcomes of the Initiative.

Further, across all stakeholders and clients interviewed, it was agreed that the current method for measuring the outcomes of the Initiative is insufficient and does not reflect the reality of the SME business cycle in any of the locations or industries in which the CTA operates. Many deals in ICT for example are not finalized for a minimum of 16 to18 months following the program. In the health IT sector it sometimes takes 2-3 years for deals to close. Yet, there is no additional and continued formal follow-up procedure by the CTA to capture these outcomes. In this case again, the full financial impact of CTA will not be realized within just the 3-4 month period of the cohort or the 6 month exit form.

Currently, the CTA only captures results data through individual follow-up or if the company participates in another CTA cohort and the exit forms are tracked. Through these exit forms, data is collected on a variety of indicators such as whether a company was satisfied with the services offered, whether they established more strategic partnerships, whether they raised new funding or venture capital, whether they adjusted their business model, and whether the CTA helped them understand their new market and adjust their pitch accordingly. As demonstrated in the table below, these exit forms collect valuable results data.

However, the results data generated through the exit forms should be read with caution. CTA participants are under no obligation to fill-in these forms, and though the response levels from companies have been an acceptable 50%, the voluntary sampling runs the risk of, over time, skewing the metrics by only capturing outcomes in the reports from the companies that succeed. This is the case as companies that either fail or exit the CTA early are not currently considered in the metrics and reporting. Unlike Similar programs, the CTA does not currently have established a system where participating companies are not allowed to participate a second time if they have outstanding documents from their first participation. Furthermore, while there is a 6-month follow up form which captures similar metrics, there is no systematic follow-up assessment with companies after this point.

With regard to the administration of these forms, the initial concept was for Posts to administer them directly to companies and leverage their relationship with participants ensure a strong response rate. Yet, the support and reporting from Posts was inconsistent and a result HQ began to also administer the forms where the Posts needed support. The response rate has been mixed in both cases. As a result of this ad-hoc system of data collection the evaluators identified several challenges. Notably, it was found that additional time and follow up was required at HQ to ensure that the Posts reported information when they received it and it also required additional efforts by those at HQ to ensure that the information was stored in a reliable and easily accessible manner. Ultimately, this ad-hoc arrangement often resulted in a duplication of efforts at Post and HQ.

In addition to the challenges faced collecting data on important CTA metrics there are also concerns regarding the current method of aggregating and storing the data in easily accessible and usable formats. While the Initiative currently collects data for deck reports and other public reporting, the evaluators were unable to locate an easily accessible and up to date repository of the information collected both for and from these reports. Information was often located in multiple spreadsheets, decks, or only remained as raw data in the exit forms presenting a challenge for evaluators to verify the data and track progress over time.  Additionally, although the data is collected from individual Posts or CTAs, in many cases it has not been aggregated by location. As a result, over the course of the evaluation it was a challenge to measure the performance of each CTA on a variety of metrics and identify the most and least effective locations.

Ultimately, the CTA Initiative highlights a challenge that has been documented to exist across many accelerators. With little standardization across CTAs as to what metrics to collect and how, self-reporting itself is a challenge and inefficient form of data collection. With more granular data on the inputs of each CTA and their activities not collected or tracked in any standardized fashion in relation to the outputs and outcomes, the ability to build a meaningful analysis is made more difficult. While significant efforts have been made to strengthen performance management and data collection, the CTA is currently underperforming in this regard making it a challenge to draw conclusions based on the data collected.

As the Initiative matures and CTAs multiply and more cohorts graduate, the result will be more firms to track. Given the current method of measuring outcomes in the program, many outcomes will not be recorded unless systematic follow-up mechanisms are established to supplement the short-term evaluations.  The current ad-hoc system in place is both unsustainable and ineffective.

6.0 Conclusions of the Evaluation

Conclusion #1: In alignment with GoC, the CTA is a relevant initiative that fills an important and unique void in the SME sector for Canadian companies looking to internationalize.

Canadian SMEs, domestic and international stakeholders, and partners immensely value the services offered by CTA Initiative. The Initiative has been noted to have tremendous value in assisting SMEs achieve their international growth ambitions. In alignment with the GoC's objective to promote entrepreneurial services and help high-potential Canadian businesses maximize access to international business opportunities, the CTA has successfully helped Canadian businesses: (1) hire more Canadian and international employees, (2) increase their revenue, and (3) obtain more capital investment. As the Initiative enters its third year of central agency funding, many early participants continue to grow in all key performance metrics as business deals come to fruition and are finalized.

The relevance of the CTA extends beyond its alignment with government priorities. Through much-needed support to Canadian SMEs, the Initiative also fills a clearly identified void in the Canadian SME sector. By assisting Canadian SMEs’ navigate through what has been called the “Valley of Death”, after the completion of a domestic incubator program, the CTA provides support that is distinct in its design and offerings. The public offering of the CTA and the expansive networks of the TCS services are a notable offering available to Canadian SMEs exclusively through the CTA. Through access to entrepreneurial resources, partners, and tools in global market technology hubs, the CTA is an exceptional and necessary support system for Canadian SMEs looking to reach international markets.

Conclusion #2: The objectives of the CTA are adequately aligned with the TCS, making it the appropriate body to administer the Initiative. However, there is a need to better integrate the CTA with the operations of the TCS.

The CTA is a logical fit in GAC and in particular the TCS. Under the TCS, the CTA has the potential to utilize the RO networks in Canada and TC networks in international markets including global technology hubs. Through the TCS, CTA clients have the opportunity to reach relevant business partners and make important but otherwise challenging connections. The favourable reviews of CTA by clients highlight the value of the engaging the TCS network.

The ability of CTA personnel to leverage their own personal networks and those of business development consultants or mentors has also proven to be a highly successful strategy for ensuring the success of Canadian SMEs. In doing so, the TCS through the CTA is providing a crucial service and expands its reach by advancing Canada’s economic interests. Across many of the CTA’s however, there appeared to be few institutional mechanisms to ensure that the departure of CTA personnel would not result in the loss of important institutional knowledge and networks upon which the Initiative relies.

Furthermore, tensions over the amount of time officers spend on CTA activities relative to traditional TCS responsibilities, and the alignment of CTA and TCS KPIs, have emerged as a source of concern. The alignment of CTA and traditional TCS KPIs appears to be more easily achieved in locations which have endeavored to make the CTA an expansion of the TC Service as opposed to a unique program offering. Further consideration therefore should be given to how to align the CTA outcomes of with those of the TCS.

Conclusion #3: The CTA has yet to establish a clear delineation of the roles, responsibilities and requirements for all personnel involved with the Initiative. This creates a tension between Ottawa and the Posts and ultimately constrains the ability of the same to administer the Initiative and report on performance.

A tension exists in the governance structure of the CTA, comprised of an ad hoc arrangement with CTA locations, two personnel at HQ, BDC, and the Director General (DG) and four dedicated LES at San Francisco/ Palo Alto, Boston, New York and London. This is due to the lack of clear roles, responsibilities for personnel that are often not understood or accepted by stakeholders. A clear delineation of roles and responsibilities would improve the governance of the Initiative and its delivery. While both Posts and HQ make significant contributions to the Initiative, the lack of clear governance structure has produced: a fragmented delivery of services and an inconsistent reporting of the data necessary to assess the Initiative’s relevance, performance and effectiveness as measured against its objectives and final outcomes. It has also resulted in miscommunication with partners and stakeholders, as well as potential, current, and past clients.

With few clear channels for communication or responsibility, a patchwork of ad hoc and unsustainable arrangements have been made to implement activities, respond to communications queries, and collect the necessary performance measurement information. While notable efforts have been made to improve the governance through the use of new tools and templates, the overall management of the Initiative remains a challenge.  

Conclusion #4: The CTA’s decentralized structure creates a tension between the dual imperatives of maintaining flexible service offerings with the need for minimum standards in service delivery. While evidence of significant learning from best practices across CTAs has been observed, the challenge remains unresolved.

Establishing equilibrium between a consistent set of high quality program deliverables and a flexible offering of local and sector specific services, has been, and continues to remain one of the most difficult challenges for the CTA. Participants and those involved with managing the CTA value the Initiative’s flexibility to adapt to unique in-market demands and to tailor programming to company participants. However, the inconsistent offerings across CTAs have presented a challenge for promoting the CTA brand, recruiting companies and ensuring a consistent experience for companies. Examples such as the varied mentorship arrangements, the time and expectations of companies while in market and the availability of orientation programming all contributed to mixed experiences and levels of satisfaction among participants.

While some standardized program forms and components have been developed, resistance to centralizing too many components of the Initiative has been a concern of program administrators at Post. Yet, the development of minimum standards around mentorship for example and customizable templates can be used to build sustainability and consistency in the program, thereby freeing up resources to develop customizable programming.

Finally, what constitutes a CTA company’s “participation”, “completion”, and “dropout” varies across posts. For example some administrators and participants have maintained that requiring the company to stay in market and/or be active in the Initiative for its entire duration of the cohort does not allow flexibility for companies that might complete their objectives in two weeks or two months. Here, there is a lack of a consistent definition on what constitutes a “completion” under these circumstances. Examples like this have resulted in serious repercussions for tracking the performance of the Initiative over time and within individual CTAs. As the Initiative expands, this hinders the ability of the TCS to provide consistent and quality programming, track company activity, and report on the initiative’s achievement of objectives.

Conclusion #5: The CTA’s use of communication tools has not been adequate to support the activities of the Initiative. The lack of consistent and substantive information has hampered further synergies and created inefficiencies, in particular in the area of CTA promotion and recruitment.

The ability of the CTA to leverage its networks and partners has been hindered by weak communication among CTAs (and partners) and through a lack of coordinated tools such as an updated, easily accessible, and informative website. The challenges the Initiative has in communicating its offering has hindered its ability to recruit qualified clients, connect with partners, and promote the CTA brand. Currently, the Initiative has a central but very limited web presence hosted on the GAC website. One Post has constructed an independent web presence for their CTAs. Additional outreach to partners is done either by Posts or HQ, however there is little uniform messaging. Ultimately, this ad-hoc arrangement is not sustainable, and has already bred confusion and competition among CTA locations. With the growth in the number of CTAs, a more streamlined and coherent communications strategy would be to the benefit of internal and external stakeholders as well as interested CTA participants.

7.0 Recommendations

The following recommendations are derived from the findings and conclusions. They take into account the importance of the continued growth of Canada’s SME sector in the context of our current trade policy and that there is a need to assist Canadian SMEs with the internationalization process to solidify Canada’s stake in the innovation economy. Should the Initiative be renewed these recommendations aim to improve the Initiative’s current operations.

Recommendation #1: To meet programmatic objectives, the CTA should continue to refine and clearly define the target client and the selection and recruitment process. To ensure the recruitment of qualified companies, the process should be managed through a clear, coordinated and centralized two-pronged approach that:

• Includes a targeted and qualified recommendation process through ROs, partners (BDC, NRC-IRAP), TCS and previously successful CTA companies;
• Includes an open, informed and transparent application process via social media and through an easily accessible CTA website that specifies company requirements and expectations.

Building on best practices identified by the CTA, the Initiative should continue to refine its target client. Such a refinement should consider whether a company has: (1) demonstrated traction in their local markets; (2) have clear client leads; and (3) have international market potential. Doing so ensures not only that market-ready companies are recruited but also that the demands of the companies and the programming of the CTA are also closely aligned.

Ensuring that market-ready companies are recruited requires improved techniques via networks that are currently underutilized and through greater marketing of the Initiative. Coordinated and targeted outreach to domestic accelerators and incubators and through proven networks such as the ROs, NRC-IRAP and BDC and previously successful CTA participants ensures qualified companies are recommended and vetted for the Initiative. Such a strategy reduces the burden on TCS officers at CTA locations to constantly recruit while also generating awareness among partners about the CTA. However, this type of approach is only effective through the successful and clear communication of CTA requirements. Basic criteria should be established and made available on all public CTA documents and clearly highlighted in the application process to ensure efficiency and the attraction of the appropriate client base.

In addition to the targeted recruitment through proven networks, efforts should be increased to attract highly qualified companies that fall outside of these traditional networks. Evidence suggests that some of the CTA’s highest performing companies were not actively engaged through these traditional networks such as BDC or domestic accelerators but were actually made aware of the Initiative via social media. Thus, efforts should be made to expand the web presence of the CTA through a fully operational and up to date website as well as links to relevant social media platforms to attract new companies. While this type of an outreach strategy would likely result in an increase in applications, the adoption of a rigorous eligibility criteria should mitigate this risk.

Finally, the current confusion due to the lack of consistent deadlines for applications, poor information regarding recruitment, and questions about which CTA a company should apply to, could be addressed through a more centralized application process and harmonized deadlines. Such coordination efforts would also serve to reduce competition between CTAs over qualified clients and ensure that the client is matched with the most appropriate CTA. While some centralized administration of recruitment is necessary for the Initiative, discretion should ultimately be left with each individual CTA to determine whether a company is the right fit for their CTA.

Recommendation #2:  The CTA should clearly define minimum requirements and standards across all current and future locations/ missions to ensure the Initiative fulfills its programmatic objectives.

As indicated in the foundational documents of 2013 and affirmed in this evaluation, mechanisms must be put in place to maintain the CTA brand and ensure consistency in core service delivery as the initiative grows in popularity and size. More specifically, CTA locations should commit to the following best practices:

1. Defining the relationship between mentors or business development consultants through a clear contract which is based upon the best practices established by MIT and the Canadian Mentoring Service. This contract stipulates the expectations and boundaries between mentors/business development consultants and participants.

2. Commitment to providing minimum offerings, such as orientation days, dedicated mentor time, TCS “check-ins”, exit interview, etc. Such standardized aspects should be balanced with the customized elements that are unique to each location and allow each CTA to maximise its responsiveness to client and market needs;

3. Provide space in a high-performing local accelerator with strong in-market connections and opportunities for networking.   

Additionally, minimum standards should be put in place to identify where a CTA is viable. In order to establish a CTA Initiative, a business case should be made which highlights the alignment of the location with GoC priorities and SME demands. More specifically, the location should possess the following attributes: 

1. Clear global leader / centre for a respective markets sector
2. Strong mentor networks with an active interest in the development of a CTA locally (ideally Canadian)
3. Strong demand expressed by Canadian clients to go to that market
4. Ensure that BBR/BBD approves all CTA locations and use of the CTA name with a clear business case for a CTA (including justification / resources / structure etc, can also harmonize or guide the ‘reverse CTA’ and ‘virtual CTA’ and ‘fast runner’ to be separate from a classic CTA)
5. Ensure that the CTA services are branded as part of the TCS (need to align job titles and functions accordingly)

Establishing such minimum requirements for hosting a CTA will ensure that all CTAs are better positioned to offer a minimum level of service delivery in a receptive host environment. For the Initiative to deliver value, all current and future CTAs must be built where there is significant financial and personnel support and market demand. These requirements also ensure a consistent offering in global tech hubs that positions the CTA brand and therefore CTA companies under the TCS in a meaningful way.  

Recommendation #3: To ensure a consistent value offering the resources of the CTA should be better aligned in order to establish an optimal Trade Commissioner to company ratio.

The FTE hours in particular were observed to be directly tied to the amount and the quality of services the CTAs were able to deliver in each location. To ensure consistent offering, resourcing levels must remain tied to in-market needs and demands of participants. Doing so ensures similar performance and consistent ‘service’ delivery for CTA clients. In establishing and operating CTAs, the resourcing and the availability of staff should therefore be considered so as not to dilute the brand or offering. The creation of an alignment of optimal TC to company ratio would help ensure that the CTA does not overburden officers and encroach on normal TCS activities.

Recommendation #4: The CTA should regularly and actively leverage the extended networks of the entire TCS and improve coordination and communication/ information sharing with the regional offices and non-CTA missions during the recruitment, program and post-program phases.

There is a significant untapped opportunity for the CTA to leverage existing resources across the TC Service. Most notably, ROs are currently engaged in the CTA in an ad hoc fashion.  ROS have recommended companies to the CTA, have provided feedback on the application process, and at times provided additional services to companies once they have completed a CTA. Uniformly, however, poor information sharing with ROs on the various offerings of the CTAs and the Initiative as a whole limits their ability to engage with CTA and its selected companies, which limits their ability to contribute to the Initiative.

Engaging with the ROs and formalizing their relationship with the CTA would also address the demand that many CTAs had for creating a “CTA alumni”. Leveraging RO networks and keeping updated information tracking on the CTA can balance the increased assistance that some CTA clients may require following their CTA departure while still not creating a formalized alumni structure and maintaining the integrity of the TCS system.

Recommendation #5: Communications efforts between CTAs, partners and the public should be improved via the development of a single web presence which outlines the offering at each CTA location and the objective of the CTA as a whole. The web presence and all communications tools should be streamlined and updated regularly.

There is significant room for improved communications and information sharing. It is anticipated that greater information sharing will decrease indirect or irrelevant requests, poor referrals, duplication of tasks and reporting, and the inconsistent knowledge of the CTA among networks and partners. The key to such communication is the development of an informative and up to date web presence. The current web practices which outline the offering at each CTA location should therefore be streamlined and expanded into a single web presence that represents the entire CTA Initiative. A unified web presence that lists the following is therefore crucial:

1. Various CTA city sector focuses
2. Various CTA mentor skills and expertise and number of mentors
3. Various CTA consultant offerings (pitch coaches, IP lawyers etc.)
4. Each CTA’s processes and timelines / structure
5. Each CTA’s space / office offerings
6. Successful CTA ‘graduates’ list and contact info
7. Allowing all stakeholders (TC’s, BDC’s, RO’s etc.) to promote and pass consistent messaging
8. Clarify the ‘acceleration’ of businesses and not technologies
9. Clarify the program and services offered by the CTA (i.e.: mentor + office + TCS)

By establishing a single web presence that is easily accessible, the Initiative can not only promote the CTA to its target demographic more effectively and communicate clearly with partners and graduates, but also demystify and resolve misconceptions about the Initiative. This should then serve to support other recommendations in this evaluation which aim to improve the recruitment of qualified companies to the program without increasing the burden on TCS officers at posts. It will however, require consistent and clear updates from the posts on the offering at each location to ensure the web presence is accurate and up to date.

Recommendation #6: Improved reporting and recording of CTA outputs and outcomes is necessary from Posts and HQ. Company performance and outcomes should be tracked for an additional follow up with companies 12 to 18 to 24 months after graduation. Harmonized standards of reporting across CTAs are also required.

One of the biggest challenges facing the CTA is its ability to measure performance and the outcomes associated with the Initiative. As more companies graduate, an increase in tracking is required. Many of the outcomes from participation in the CTA are not apparent until 12 months or later following completion.  Thus, a systematic performance measurement strategy and evaluation must be put in place.

The current ad-hoc internal system for evaluating performance is not sufficient for capturing the output or outcomes. Without better and more consistent reporting, it will be a challenge for management to demonstrate its value. Many of the successes of the CTA are not captured because there is no consistent tracking in place.

Finally, the harmonization of standards, such as defining what constitutes company participation, completion and dropouts, ensures consistent reporting and creates a viable data repository for performance tracking and analysis. Presently, the various metrics used at the different CTA locations and that have evolved over the course of the Pilot Initiative are unreliable and give an incomplete and distorted picture of performance.

8.0 Management Response and Action Plan

The management of the Canadian Technology Accelerator Initiative (CTA) values the insights shared in this evaluation, including the best practices and the challenges that the evaluation team identified. Management strongly supports the continuation of the CTA Initiative as part of the services offered by the Trade Commissioner Service (TCS) to innovative Canadian small and medium size enterprises (SMEs). As the program matures CTA staff at headquarters and at posts will continue to explore novel approaches to delivering client services and will continue to share best practices across CTA sites.

In late 2013, the Department received funding to align existing CTA pilots, develop additional pilot programming in new markets, improve consistency and coordination, manage stakeholder relations, and to create a reporting framework. The findings of the evaluation are consistent with the observations of the management team for this early stage of the CTA Initiative. Management notes progress in several areas identified in the evaluation including data collection, communication and coordination, and also agrees that these are important areas for ongoing improvement.

Recently, the TCS has focussed efforts on establishing the foundation to coordinate and expand this multi-Mission initiative. The Department has: started to implement data collection processes to support impact analysis; put in place regular coordination and management meetings; formalized relationships with key partners (such as BDC Capital and NRC-IRAP); articulated common client and program definition;  identified and shared of best practices; implemented a streamlined web-based CTA application process; produced alumni profiles and testimonials that enhance awareness of Canadian capabilities; and increased coordination among Canadian stakeholders.

As noted in the evaluation, the CTA initiative is highly responsive and adaptable to the needs of its SME clients. Since late 2013, the Department has demonstrated flexibility, adjusting programming to support the growth of participants by analyzing and responding to results, market trends and company feedback. Management will continue to adopt a flexible approach that sets clear and common objectives while leaving space for flexibility and sensitivity to the differences between CTA sites and their target markets. Management will likewise continue to track and respond to client needs, with the goal of building strong client relationships and delivering exceptional client service.

Recommendation 1

Recommendation #1: To meet programmatic objectives, the CTA should continue to refine and clearly define the target client and the selection and recruitment process.

To ensure the recruitment of qualified companies, the process should be managed through a clear, coordinated and centralized two-pronged approach that:

• Includes a targeted and qualified recommendation process through ROs, partners (BDC, NRC-IRAP), TCS and previously successful CTA companies;
• Includes an open, informed and transparent application process via social media and through an easily accessible CTA website that specifies company requirements and expectations.

Associated Findings: 2, 3, 10, 13, 16

Management Response & Action Plan: The management of the CTA strongly supports the goal of recruiting high-potential companies and acknowledges the need for continued efforts to streamline the recruitment process.

While acknowledging the temporal scope of the   evaluation, Management notes that efforts have been made since Spring 2014 to increase engagement with Regional Offices and in-Canada partners. For example, in Winter 2015, Global Affairs Canada signed a Memorandum of Understanding with BDC Capital to define the target CTA client and selection process, and to leverage BDC Capital’s domestic network to recruit companies. In Autumn 2014, the Department began providing coordinated CTA recruitment material to NRC-IRAP’s Industrial Technology Advisors (ITAs) and in Autumn 2015, webinars were held for staff from Global Affairs Canada’s Regional Offices and for ITAs to enhance their understanding of the CTA Initiative and the target CTA client.

Further, in 2015 the TCS Marketing division, CTA officers and Missions began using social media to proactively recruit cohorts – a practice that is now encouraged and which will become common as the CTA moves forward.

As of Spring 2014, the standardised application form process has been centralised through the TCS website where there is a dedicated CTA web presence compliant with the Government of Canada’s online communications policy. The use of this publicly available common application has made the process accessible to a broader audience.

Since Fall 2013, efforts have been made to increase the rigor of the client selection process at Mission through the introduction of panels comprised of private sector experts, Trade Commissioners and stakeholders like BDC Capital. Many CTAs have adopted this panel approach, which provides a review mechanism for application forms to ensure applicants are ready for the program, have a high-potential product, and are a good match for CTA services.  Should the Initiative be renewed, HQ will work with all Missions in FY 2016-17 to ensure that selection committees are established at all CTAs and that relevant stakeholders, including partners and Regional Offices, are represented on these committees.

Responsibility Centre: BBR

Time Frame: April  2016 – December 2016

Management Response & Action Plan: In addition HQ, with support from the Missions and Regional Offices, will refine application assessment and selection procedures across CTAs.  Management will further take steps to ensure that applicants receive feedback on their application.

Responsibility Centre: BBR

Time Frame: April  2016 – March 2017

Recommendation 2

Recommendation #2: The CTA should clearly define minimum requirements and standards across all current and future locations/ missions to ensure the Initiative fulfills its programmatic objectives.

Associated Findings: 2, 7, 8, 9, 11, 13

Management Response & Action Plan: Management acknowledges the need to ensure that client expectations are met and that the purpose of the CTA program is clearly defined.

Responsibility Centre: Mentoring agreements: individual Missions

Time Frame: April  2016 – September 2016

Management Response & Action Plan: The evaluation recommends that agreements be put in place with mentors, to set expectations and boundaries. Management agrees with this approach. Should the Initiative be approved, Missions will be required to standardize and adopt best practices for MOUs with mentors.

Responsibility Centre: Mentoring agreements: individual Missions

Time Frame: Ongoing – March 2017

Management Response & Action Plan: Management acknowledges that all CTAs must adopt a consistent set of basic service standards. Since Spring 2014, HQ has facilitated and encouraged the sharing of best practices among CTA Missions through conference calls and exchange of documents such as sample event programs and templates on the Department’s wiki which can assist in the development of common service standards. Management will continue to refine mandatory standards on service offerings across all CTAs.

Responsibility Centre: Minimum program standards: BBR and individual Missions

Time Frame: April  2016 – March 2017

Management Response & Action Plan: In Spring 2014, Management initiated communication between CTAs, through regularly scheduled conference calls at the operational level, as well as quarterly calls to discuss governance strategy at the management level.

Management acknowledges the need for improved follow-up by TCS officers with CTA clients. With the addition of dedicated FTEs at four CTA sites (staffed by Fall 2014), and following adjustments to cohort size that aligned the program with capacity at the Missions, TCS officers are better positioned to engage in depth with each client. In addition, should the Initiative be renewed, HQ will seek guidance from the Regional Office network to identify best practices in follow-up with potential CTA clients, as well as alumni.

Finally, the evaluation recommends that the criteria for establishing a CTA be clarified and that a distinction be made between CTA programs that are structured around an in-market business accelerator and CTA programs that offer a short term “soft landing” at an in-market business accelerator as part of their CTA program offering. Management will continue to examine the benefits of and demand for U.S. and non-U.S. locations in order to have a good sense of the differences between structured CTAs and soft landing sites. Management feels that additional data is needed to more clearly define what these two models could look like in order to support sound decision-making.

Responsibility Centre: Refining the CTA model: BBR

Time Frame: April  2016 – March 2018

Recommendation 3

Recommendation #3: To ensure a consistent value offering the resources of the CTA should be better aligned in order to establish an optimal Trade Commissioner to company ratio.

Associated Findings:  13, 14, 15, 17, 18, 19

Management Response & Action Plan: Management acknowledges the challenge of aligning human and financial resources to CTA client needs. In order to address the additional requirements that CTAs impose on host Missions, dedicated FTEs have been hired in Boston, London, New York and Palo Alto. Since all positions were staffed (by Fall 2014), their presence has steadily improved the quality of service to CTA clients and consistency between CTA programs, while managing the demands placed on existing TCS staff.

Management notes that some adjustments have already been made to the staff-to-client ratio. This was initiated at Missions in Spring 2015 by changing the length and number of cohorts going through each CTA annually. These adjustments had the additional advantage of responding to client demands for a longer period of support as they established traction in market.

Management acknowledges the need to continue monitoring and adjusting the staff-to-client ratio at CTAs. Management believes that more data is needed to understand the optimal level of support per CTA participant. Going forward, management will assess TRIO data, CTA client feedback, and engage with Missions and the responsible Geographic Divisions to discuss methods to improve staff-to-client ratio, should the Initiative be renewed.

Responsibility Centre: BBR

Time Frame: April  2016 – March 2018

Recommendation 4

Recommendation #4: The CTA should regularly and actively leverage the extended networks of the entire TCS and improve coordination and communication/ information sharing with the regional offices and non-CTA missions during the recruitment, program and post-program phases.

Associated Findings: 4, 5, 12, 13, 14, 16

Management Response & Action Plan: Management acknowledges the value of leveraging the entire TCS network, including the Regional Offices, in support of the CTA recruitment, program and post-program phases.

Since April 2014, HQ has supported CTA Missions with coordinated engagement of Regional Officers in the recruitment of CTA candidates and will continue to give Regional Offices the tools they need to coordinate efforts with the CTA initiative such as maintaining an updated web presence on the TCS website, marketing materials, and calendars of upcoming events.

Since 2014, 50% of applicants reported that they were referred to CTA programs from the TCS. This has helped Missions and Regional Offices better assess the quality of applicants. However, management acknowledges that there is more work to be done to strengthen the engagement between ROs and the CTA.

Responsibility Centre: BBR

Time Frame: Ongoing

Management Response & Action Plan: Should the CTA Initiative be renewed, starting Spring 2016, management will develop a clear engagement strategy for CTAs to leverage the extended TCS network.

Responsibility Centre: BBR

Time Frame: April 2016 – July 2016

Recommendation 5

Recommendation #5: Communications efforts between CTAs, partners and the public should be improved via the development of a single web presence which outlines the offering at each CTA location and the objective of the CTA as a whole. The web presence and all communications tools should be streamlined and updated regularly.

Associated Findings: 2, 12, 13, 15, 16

Management Response & Action Plan: Management acknowledges the importance of presenting a coherent CTA brand, and a single coherent and consistent CTA web presence hosted on the TCS website. Management notes improvements made to the content of the CTA web presence on the TCS website, the coordination of CTA Mission recruitment outreach with stakeholders (ROs, BDC Capital, NRC-IRAP), the implementation of regular calls between CTA Missions, the development of promotional videos, as well as flyers and calendars to facilitate more coordinated recruitment efforts since Spring 2014. Headquarters will strengthen coordination with Regional Offices to improve the dissemination of these CTA communication tools to domestic partners across Canada. This will include consultation with the Department’s Public Affairs Bureau on the use of different tools available, such as social media, direct e-mail campaigns, and the continued publication of sample CTA success stories.

Management will continue to align all of our communications efforts to reach the domestic audience (such as; decks, one-pagers, banners, video testimonials, etc.) with TCS marketing guidelines, and follow the Department’s policy on a common look and feel.

Should the CTA Initiative be renewed, management will make further improvements on the TCS website in alignment with scheduled migration to the Canada.ca platform. Management will also seek guidance within the Department’s Public Affairs Branch on web presences that target foreign audiences and a co-ordinated proactive communications approach.

Responsibility Centre: BBR

Time Frame: Ongoing

Recommendation 6

Recommendation #6: Improved reporting and recording of CTA outputs and outcomes is necessary from Posts and HQ. Company performance and outcomes should be tracked for an additional follow up with companies 12 to 18 to 24 months after graduation. Harmonized standards of reporting across CTAs are also required.

Associated Findings: 3, 13, 14, 15, 20

Management Response & Action Plan: Tracking and reporting on results of the CTA Initiative has improved significantly over the life of the program, including through the introduction of a standardised survey for CTA graduates. Since Spring 2014, an improved result tracking system has been in place, though officers are still working to follow-up with all clients and track results due to resource constraints. Management acknowledges the need to work with Missions to improve their reporting and inputting of data to support HQ efforts to demonstrate the impact of the CTA Initiative.

The CTA is a program of the TCS. While some unique reporting elements like the client surveys will continue, management will explore means to capture the incremental reporting of the Initiative’s impact on participants will also aligning CTA reporting with existing TCS tools which measure impact on clients. This could include the use of TRIO to track deals signed by clients 12-24 months post CTA.

Responsibility Centre:

• TRIO tracking: Individual Missions
• CTA definitions: BBR and Missions

Time Frame: Ongoing – March 2017

Management Response & Action Plan: Management acknowledges the need to establish common definitions of what constitutes participation, drop-out and completion in the CTA program. Management will work with CTA Missions to refine these definitions.

Responsibility Centre:

• TRIO tracking: Individual Missions
• CTA definitions: BBR and Missions

Time Frame: April 2016 –  July 2016

Management Response & Action Plan: Management will ensure that the data collected is consolidated and stored in a consistent and up to date format that allows program management to sustain long term analysis and reporting.

Responsibility Centre:

• TRIO tracking: Individual Missions
• CTA definitions: BBR and Missions

Time Frame: Ongoing – March 2017

Management Response & Action Plan: Should the initiative be renewed, Management will develop and implement a revised performance metric strategy that addresses the aforementioned issues.

Responsibility Centre: 

• TRIO tracking: Individual Missions
• CTA definitions: BBR and Missions

Time Frame: April 2016 – March 2017

Appendix 1: List of Findings

Finding 1: Finding #1:  The assumptions that led to CTA’s creation remain valid, with the Initiative addressing a distinctive need in the SME sector in Canada.

Finding 2: While the CTA’s activities are generally viewed to be a relevant government instrument for engaging Canadian SMEs, challenges remain in aligning the services offered by the Initiative with the demands of companies.

Finding 3: Continued demand for the CTA provides evidence that the Initiative is responsive to the needs of Canadian SMEs and CTA participants.

Finding 4: The CTA is well aligned with the objectives of the GoC and in particular the TCS.

Finding 5: The CTA is consistent with the mandate of the GAC and in particular the TCS program.

Finding 6: The CTA increases the capacity of SMEs to compete in the global innovation economy.

Finding 7: The provision of in-market services, and guided mentorship, through the CTA and TCS network has been highly valued by Canadian SMEs in targeted technology markets, contributing to their in-market success.

Finding 8: A key contributor to the achievement of the CTA’s successful outcomes is its flexibility and the ability of missions to tailor CTA programming to company needs. However, where this is not balanced with standardized and structured programming, significant obstacles to the CTA’s success emerge.

Finding 9: Virtual pre-programming or complementary CTA programs ensure the market readiness of companies and the alignment of company expectations with deliverables.

Finding 10: Where initially the program accepted a large number of companies, it is now focused on attracting the most qualified firms and generating smaller and longer cohorts to ensure that companies that complete the program are ready for the market.

Finding 11: As the CTA matures in the US and expands to the UK, Europe and India it continues to offer tailored services to participants based on market opportunities, however challenges have emerged regarding consistency in CTA offerings.

Finding 12: The CTA brand has strong value among participants and potential investors. However, weak communications efforts with internal and external stakeholders and administrators have resulted in confusion regarding the CTA’s offering. It has also hampered the Initiative’s ability to establish a clear and consistent CTA brand that can help the Initiative and its participants achieve successful outcomes.

Finding 13: The absence of a clear delineation of the different roles and responsibilities of all personnel involved in the Initiative has constrained its ability to administer the CTA.

Finding 14: Balancing traditional TCS responsibilities and methods of reporting with the operations of the CTA has created a tension for CTA administrators who expressed concern over their ability to meet traditional TCS demands and performance metrics.

Finding 15: The decentralized nature of the CTA, which allows for important in-market specialization, is crucial for achieving the Initiative's successful outcomes. Yet, in many cases it has also resulted in a duplication of efforts by CTA personnel across locations.

Finding 16: While efforts to refine CTA participant requirements have been made, the Initiative has yet to cultivate a consistent and strategic recruitment and application process.

Finding 17: The expansion of the CTA to new locations has resulted in some confusion over the selection criteria and program offering required of each CTA and the amount of funds available to each.

Finding 18: The overall CTA budget remains fixed but the growth of the Initiative and fluctuations in the value of the Canadian dollar have resulted in decreased financial resources.

Finding 19: The allocation of CTA resources are not presently aligned with the demands placed on the various CTA locations. As result this is limiting their capacity to deliver the Initiative.

Finding 20: Reporting on CTA performance and outcomes remains ad-hoc, inconsistent and lacks standardized procedures.