Formative Evaluation of the Advancing Specialized Skills for Economic Transformation Project - Executive summary

Rationale - As the Advancing Specialized Skills for Economic Transformation (ASSET) project enters its final year of implementation, this evaluation focuses on accountability: to identify and assess the extent and quality of results it is achieving and to recommend any needed adjustments to the project. It will be used by Global Affairs Canada; World University Service of Canada (WUSC), as Canadian implementing organization; and the Sri Lankan Ministry of Skills Development and Vocational Training (MSDVT) to assess overall project performance. The purposes of the evaluation were threefold: to determine if ASSET is on track to meet expected results; to make recommendations for improved implementation; and, to a lesser extent, to inform the design of possible future Global Affairs Canada interventions in the technical vocational education and training (TVET) sector. Specific objectives were to assess effectiveness, relevance, efficiency and sustainability of the intervention; to assess the relevance of the approach to technical-vocational training; and to provide findings, conclusions, recommendations and lessons to inform implementation.

ASSET is a multi-year development intervention seeking to support vocational training reform in Sri Lanka. With a total budget of $12,279,108 (of which $819, 840 comes from WUSC), ASSET is aligned with Government of Sri Lanka (GoSL) economic and labour policy through its partnership with MSDVT and its operational agencies as focal points for TVET policy and programs. ASSET operates nationally in four regions (east, northwest, north and south) and in four high-demand trade and technology sub-sectors: construction, information and communications technology (ICT), hospitality and tourism, and automotive/light engineering. It collaborates with and promotes links between public and private agencies and between employers and training providers, to identify work opportunities for unemployed and underemployed marginalized youth, with specific emphasis on the inclusion of young women, persons with disabilities and ex-combatants. Its direct beneficiaries are the 3,600 young men and women who are expected to acquire vocational skills and be placed in private sector jobs (40% women) and 900 current private sector employees needing skills upgrading. The ultimate outcome of ASSET is increased income potential for key economic drivers in Sri Lanka through enhanced skills and national recognition of the value of the trades and technology sector and respect for the technical skills of tradespeople (male and female). It is realized through three intermediate and seven immediate outcomes:

1100 - Improved linkages between government, training institutions, private sector and civil society organizations to collaborate on skills for employment and workplace learning for Sri Lankan men and women:

• increased understanding of the needs and priorities of key stakeholders;
• enhanced collaboration on development and management of workplace-oriented training; and
• improved human-resource practices, promoting labour-force integration of women, persons with disabilities and ex-combatants.

1200 - Enhanced planning and management by training institutions for employment training for Sri Lankan men and women in high-demand sectors to be placed in job opportunities identified by the private sector:

• improved quality of employment-oriented training to meet identified private sector labour demands; and
• improved access for marginalized Sri Lankans to workplace-related training through scholarships/loans.

1300 - Increased acceptance of careers in trades and technologies as worthwhile occupations for skilled men and women:

• increased engagement from stakeholders in developing/managing a strategic communication plan; and
• increased awareness of the dignity of labour in the trades and technology sector.

WUSC (Colombo and Ottawa offices) is responsible for overall ASSET management and project delivery through three regional offices covering four regional zones. A Global Affairs Canada-WUSC project steering committee (PSC) meets annually. It does not include MSDVT, but these officers participate in annual project advisory committee (PAC) meetings. ASSET seeks to add value through the use of volunteers.

Scope and methods of the evaluation - Covering the entirety of the ASSET project as described above, the evaluation was set within a results-based framework aligned with standards set out by the Organisation for Economic Co-operation and Development’s Development Assistance Committee (OECD-DAC). Gender equality, as both a crosscutting theme and a core component of ASSET, was considered in depth; issues of environmental sustainability and governance played a more minor part in ASSET, and data collection and analysis reflect this. Ethical standards of confidentiality and the right of non-participation were applied.

The evaluation was conducted over 50 days within a six-month period. In-country data collection took place over 16 days and reached all regions covered by ASSET, including eight of its 11 districts and all major ethnic, linguistic, religious, socio-economic and war-experience groups. An ethnographic design and mixed methods were used. Qualitative and quantitative data were integrated to answer statement of work questions as reflected in the evaluation matrix, explore indicators and gauge both the extent and intensity of results. Data collection tools included document review, structured focus group discussions (FGD) and semi-structured and open-ended key-informant interviews (KII). Findings were analyzed using constant comparative triangulation of explanatory causes and their implications for overall progress on and toward outcomes. All data were disaggregated by sex as fully as possible where appropriate. Equitable gender representation in KII and FGD was emphasized, with females, in fact, over-represented in the sample compared to their actual participation in ASSET.

Limitations of the evaluation included a relatively brief fieldwork period leading to a partially purposive sample, favouring data breadth over depth; an inability to confirm the present employment status of trainees sampled in the 2015 tracer study that assessed the impact of the vocational training at the individual level; and a weak results-based management (RBM) structure, reducing confidence in results assessed.

Main findings

Effectiveness - Based on cumulative reporting of the performance management framework (PMF), ASSET appears to be reaching or exceeding most of its immediate and intermediate results. However, the project has not been well served by weaknesses in its logic model and PMF: the first, insufficiently differentiating intermediate and immediate outcomes to allow for clear results chains, and the second, allowing results to be reported as completed activities rather than change. These problems were not alleviated by the field data: reports and stakeholder interviews were clear that, at the micro level, a great deal is happening, but left uncertain whether, overall, ASSET is making a significant difference in the rigour and quality with which employment of vulnerable youth is being planned, prepared or assured.

Results chain 1100:

• Quantitative targets to create/modify business stakeholder forums (BSFs), a core expected result, are exceeded. However, data as to the quality and depth, and thus sustainability of linkages between employers and trainers, are limited.
• Signed memorandums of understanding (MOU) for service agreements between private sector employers and training institutions have met targets, and 48 vocational training institutions (VTI) have training plans reflecting private sector priorities. Less consistent were data on how employment-ready trainees are, especially in ICT.

Results chain 1200:

• ASSET is meeting its criteria of engaging economically vulnerable youth, an output since these are youth ASSET selects; graduates are being placed in relevant work, but with uncertain longevity; the target of marginalized PWD and ex-combatants reached is not met, the former unwilling to self-identify and the latter reluctant to come forward to often unwelcoming work settings.
• Effectiveness is difficult to assess insofar as the indicators of the intermediate outcome are not relevant to its immediate outcome precursor and its indicators.
• Outcomes on marginalized youth and student financing have not been met due to as yet undetermined GoSL policy and lack of interest from banks in this type of activity.

Results chain 1300:

• Indicators in this chain highlight gender targets, and progress here has been limited. While this is the case in general, gender equity targets are difficult to track and synthesize because they are dispersed across results streams without a reporting “space” for creating a holistic picture of advances—perhaps leading to an underestimation of impact.
• The number of expected women (and men) applying for training has been exceeded, but not targets for enrolment.
• There was no evidence of a major shift in public opinion about women’s role in trades, although there are promising signs.

Satisfaction with ASSET is generally positive across all stakeholder groups, albeit in all cases nuanced—most notably for GoSL partners. However, greater communication with GoSL partners would have increased satisfaction. The cost-efficiency of the separate activities based on outcomes realized could not be assessed: ASSET reports on activities, not on outcomes, and field time was insufficient for a comprehensive audit of budget allocations. Outputs are being achieved broadly as planned, meeting efficiency criteria in one sense, but ultimately, efficiency is tied to results, and confirming these is problematic. ASSET is proving efficient in its collaboration especially at district level through “tripartite” arrangements of ASSET, VTI and employers that allow both for matching project activities with the “realities on the ground,” and for partners to play a significant role in planning, executing and monitoring activities. ASSET management and administration have functioned in a reasonably positive and timely way; budgets have stayed within the planned allocations. There have also been challenges. Appropriate volunteers for planned ASSET tasks have not always been found; their short contract duration has not always allowed effective closure. To the extent ASSET seems to be implemented less as a funded project with well-defined scheduling parameters than as a more open WUSC program with results evolving as challenges and opportunities arise, it may be hindered from moving efficiently through to outcomes.

ASSET is clearly relevant to the priorities of the GoSL as expressed in the Skills Sector Development Program (SSDP) in seeking to enable young and socio-economically vulnerable women and men to gain access to entry-level employment in sectors deemed critical to moving the economy toward mid-level status. It also coincides well with the recent “Vision 2025” commitment to creating a knowledge-based economy. In supporting cross-institutional linkages with multiple government, donor and non-governmental organization agencies, it addresses well-recognized coordination constraints that are hobbling the TVET sector. ASSET conforms to GAC development assistance objectives in Sri Lanka in promoting equitable and sustainable economic growth and good governance within a human rights and poverty reduction framework. From a different angle, ASSET is assessed positively by stakeholders in its ability to maintain relevance by being flexible and working from both policy and training perspectives. Assessment of relevance of the training is also generally positive in terms of what is done. However, relevance is less certain with respect to the depth of learning being realized and the implications both of what is being left out (e.g. as yet limited application by VTI partners of soft skills curriculums) and a focus on entry-level standards that may limit trainee opportunities for a career path.

ASSET sustainability is less a matter of maintaining current training courses and work placement arrangements than of generating a continuing commitment across the sector and within the wider community to the importance of linking young adults professionally into the trades and technology sectors; evolving capacities for creating the training programs and employer networks for doing so; and through both of these, realizing incremental improvements in the value accorded TVET as a viable way into good employment. Given the limited evidence of project outcomes overall, assessment of what will be sustained and how well can only be indicative rather than definitive, and at the present point in time, seems likely limited. Over the next 12 months, actions to move toward sustained outcomes could be taken, e.g. a stronger emphasis on stock-taking of lessons learned and consolidating models of effective practice. It is unclear, however, that ASSET is moving in this way; rather, activities are continuing and new ones are being initiated.

Crosscutting themes - ASSET is making uncertain cumulative progress on gender equality in part due to the poor quality of result statements and indicators. The development of the Gender Equality and Social Inclusion (GESI) program has been positive in allowing a defined staff and budget, but perhaps also risks masking the specific exclusions of women as women (beyond being poor, a PWD or an ex-combatant). Overall, gender-linked results are weak:

• 29.5% of the planned 40% trainees are female, although this is in line with figures generally.
• 50% of expected private sector partners have set gender equality targets;
• No private sector partners have developed operational plans to implement GESI-linked HR policies; 19% sent senior managers to GESI training on the integration of GESI (target 50%).

There has been modest action on environmental sustainability: production of an environmental strategy, not yet implemented; training/exposure visits in the sectors; an MOU with the national apex body for construction for training on compliance; and identifying existing laws and regulations. ASSET’s principal governance activities concern the development of partnerships with government in support of employment-based training in the four core sectors; of private sector leadership in implementing such training; and of collaborations among government, the private sector, civil society organizations and VTI for sustained action on a TVET system. Promoting these objectives has been appropriate, and most actions continue with considerable competence to be supported by ASSET officers through technical assistance. However, there was limited evidence of this happening in a sufficiently in-depth and consistent way to enable consolidation, and thus, sustainable change. Not including it in the PSC is seen by MSDVT as problematic, limiting scheduled access to ASSET planning decisions and follow-up on actions. The PAC provides a platform for MSDVT input to the work of ASSET in the sector, but seems not to have fully compensated for the gap in joint decision-making.

ASSET has been designed and managed both as an operationally complex intervention concerned with catalyzing change more so than directly producing it. Together, these characteristics have produced breadth of reach, but more questionable depth, and a tendency toward the production of outputs rather than pushing toward outcome-level change. ASSET has been well served by a regional office design and management structure that has allowed regional teams of technically competent, committed officers an appropriate degree of discretionary authority to be responsive to context. Based on annual reporting, budgets have been adequate and on track, and managed with due diligence without undue delays.

Conclusions

ASSET operates within two quite distinct sets of activities across its three results streams: national policy engagement and outreach; and human resource development through training programs and institutional linkages. Operationally, the first concerns efforts to catalyze interest, commitment and policy action on TVET among government and donor agencies, largely in Colombo; the second concerns nationally dispersed mobilization and training activity, well-grounded in local partnerships. Both are clearly relevant to the needs of the sector; both appear to be of good technical quality.

However, while outputs are evident and well-reported in both, outcomes are less well reported, especially in the first, and it is not clear how outputs are moving to outcomes. Framing the project within a stronger RBM architecture (i.e. a clearer logic in its results chains and more appropriate outcome indicators) might have improved the potential of its showing progress on outcomes by forcing a regular stock-taking of the extent to which activities were moving toward them and taking appropriate action to ensure them. However, the emphasis of the PMF on activities (courses, media events, meetings and workshops) has tended ASSET decision-making toward generating evidence on the number and quality of those activities, less so on the difference they are making.

ASSET is making clearest progress in realizing results at the local, micro level: vulnerable youth are being brought into employment-focused training that is of reasonable quality in each of the four priority trades and technology sectors, largely at the entry level of work readiness, and most of these trainees are being placed into private sector jobs in which they receive a degree of on-the-job supervision and reasonable salaries. Tracer study data indicate most employed trainees remain in work, either in the placement or somewhere else in the sector. Much of this progress, then, concerns individuals. In terms of institutional linkages and changing perceptions of TVET, evidence of substantial and sustainable change at the systems level to a degree that might be reasonably and confidently considered contributions to intermediate outcomes was not confirmed for either policy or human resource development activities.

Based on Global Affairs Canada criteria, ASSET has not done as well as might have been expected in advancing gender equality. Outcome statements tend toward “adding ‘women and men’” rather than “explicitly demonstrating changes in gender inequalities;” indicators are quantitative, concerned with gender parity (e.g. numbers participating) less than qualitative, concerned with effects of policies and actions on/for women and the characteristics of their participation.

Based on the long-term, labour-intensive nature of the institutional and socio-cultural changes implied by immediate and intermediate outcomes, coupled with a relatively small human and financial resource base, there is little to suggest ASSET will have made substantial progress on realizing its results by termination. It could, however, make progress toward them through, in the final year, prioritizing action within the two activity areas: national policy and local training/placement and consolidating change. There were no apparent decisions to move in this direction.

Given its high profile for WUSC/ASSET, its large budget and the miscommunication it seems to have produced with at least two stakeholders in MSDVT, particular note is made of the mass media/video campaign. The value of the modality, especially as a platform for advancing the position and prospects for women in the trade and technology sectors, will only be known in the longer run, and its cost-effectiveness compared with the other more-localized, hands-on mobilizing approaches to reaching the most vulnerable. Importantly, a comprehensive evaluation of the campaign is expected to be in place in the first part of 2018, but its findings and their implications for ASSET are likely to be felt only very near or after termination.

Finally, it is important in drawing conclusions as to ASSET’s effectiveness and sustainability to differentiate it as a time-bound project with defined results from WUSC as an organization with a long-standing program of TVET activity in Sri Lanka. It is clear that the commitment and experience of WUSC is adding value to ASSET implementation. However, there is also potential risk, exacerbated by weaknesses in the RBM framework, that results chains will remain ill-defined and outcomes less well realized if/as agenda and actions shift between the two perspectives.

Recommendations

What Global Affairs Canada and WUSC (Colombo and Ottawa offices) could do in the near term:

1. Systematically and rigorously review the three results streams to confirm the theory of change (TOC) and the logic of the results chains, including a revision of the indicators, as appropriate for the outcome level.

2. Revise the membership of the PSC to include the MSCVT or one of its affiliates and plan a subsequent agenda specifically designed as opportunities for stock-taking and decision-making, recognizing that while the PAC provides a platform for MSDVT input to the work of ASSET in the sector, it does not fully compensate for the gap in joint decision-making.

What Global Affairs Canada could do in the medium term:

3. Remain open to supporting the TVET sector in Sri Lanka specifically as it affects women, recognizing gender equality as a priority for the country’s sustained and inclusive development and as immediately relevant to Canada’s commitment to addressing women’s persistent exclusion from viable employment opportunities, greater self-reliance and a way out of poverty.

What WUSC (Colombo and Ottawa offices) could do in the near term:

4. Revisit the project work plan for fiscal year 2017-2018 and into 2018-2019 to focus specifically on consolidating activities for which there is evidence of progress toward outcomes and for which planning their further consolidation or institutionalization is realistic.

5. Revise monitoring specifically to focus on assessment of outcomes: the degree, importance and sustainability of changes being realized.

6. Reinforce actions initiated by GESI to bring together the various strands of gender action in ASSET by at least notionally regenerating the “gender equality” results stream, including appropriate indicators.

7. Move quickly on plans both to evaluate the mass media/video campaign, specifically to measure changes in national perceptions about TVET and gender equality and to conduct a second tracer study.

8. Continue to work with VTI to develop and reflect the soft-skills curriculum in all training courses, as its promotion by ASSET was strongly supported by employers and trainees as a particular value-added activity.

What WUSC (Colombo and Ottawa offices) could do in the medium term:

9. Align Uniterra and ASSET WUSC volunteer terms of reference more specifically with those aspects of the ASSET program to which the volunteer is expected to contribute over the contract period.

10. Work with MSDVT and the ICT Sector Skills Council to undertake a thorough review of the existing ICT curriculums, including its scope, currency and relevance to employers of the competencies being produced and options for widening and diversifying its coverage in closer association with the private sector.

11. Continue to identify and further operationalize results of ASSET-commissioned research and analyses (e.g. on student financing, retention and women’s participation in TVET) that are likely to contribute to realizing project outcomes.

Response to the recommendations

Global Affairs Canada and WUSC have taken note of the consultant’s findings, conclusions and recommendations and have shared them with relevant stakeholders for consideration. In the near term, Global Affairs Canada and WUSC have agreed to review the PMF and revise indicators as appropriate. In the near term, WUSC will also prioritize taking action to consolidate activities to better contribute to achieving and assessing outcomes, ensure that the gender target of 40% women is achieved, conduct a second-generation tracer study to assess the impact of the vocational training at the individual level and carry out an evaluation of the mass media campaign through a targeted perception study.

Lessons

Several generic lessons would seem to pertain to the ASSET experience, as follows. Projects will not necessarily perform up the requirements of a good RBM architecture, but may be confounded by the limited requirements of a poor one. External funding aimed at significant social and systems change will be effective to the extent that it is reasonably narrow in focus, sustained in duration and sufficiently comprehensive to become consolidated and institutionalized. To consolidate institutional and systems change it is important go beyond simply recurring exposures to an innovation or activity. Distinguishing between “need” to see and “like” to see is key, by ensuring good principles are reasonably accommodated within parameters of what is feasible for, and acceptable to, intended beneficiaries.