Formative Evaluation of the Women’s Economic Linkages and Employment Development (WE-LEAD) (P002120) and Launching Economic Achievement Program (LEAP) (P002121) Projects in Jordan – Executive summary

Evaluation Type: Formative
Commissioned by: Global Affairs Canada
Consultant: Integrated International
Date: December 2020

Rationale and purpose of the evaluation

Global Affairs Canada (GAC) commissioned INTEGRATED International to undertake a formative (mid-term) evaluation of the Canada-funded Women’s Economic Linkages and Employment Development (WE-LEAD) and Launching Economic Achievement (LEAP) projects.

The mid-term evaluation is intended to serve as a means of validating or filling the gaps in the data obtained from monitoring. The mid‐term evaluation is being commissioned at this point in time to assess project progress and achievement of results and prompt necessary adjustments.^{Footnote 1} It should also provide insights into challenges due to project design and assumptions.

The purpose of the mid-term evaluation is to:

• Inform adjustments to the WE-LEAD and LEAP projects to improve their effectiveness, efficiency, and gender responsiveness
• Inform the design of future projects in the women’s economic empowerment sector

This external, independent formative evaluation will benefit and inform the implementing partners: World University Service of Canada (WUSC), Canadian Leaders in International Consulting (CLIC) and the Canadian Bureau for International Education (CBIE), as well as Global Affairs Canada. The evaluation results will also be made available to stakeholders in Jordan.

Objectives of the evaluation

The objectives of the evaluation are as follows:

• Assess the achievement of observed results^{Footnote 2} and efficiency of the WE-LEAD and LEAP projects
• Assess the sustainability of observed results from WE-LEAD and LEAP
• Assess the gender responsiveness^{Footnote 3} of the projects’ observed results
• Provide findings, conclusions, lessons and recommendations to fulfill the evaluation purposes stated above.

Scope of the evaluation

The mid-term evaluation covers 2 projects funded by Global Affairs Canada in Jordan: WE-LEAD and LEAP.

Development context

Women in Jordan account for almost half of the total population, but less than 18% of the total labor force. The International Labour Organization (ILO) estimates that the gender gap in Jordan’s labour force participation resulted in a loss of US$8 billion in the value of Jordan’s GDP in 2018.^{Footnote 4} Women in Jordan are an underutilized economic resource with the potential to contribute critically to the country’s economic growth by enhancing business productivity and competitiveness.^{Footnote 5} Currently, only 14% of women in Jordan are actively engaged in the labour market, a level well below regional and world averages. Employment prospects for Jordanian youth and women are considerably worse than for the average population,^{Footnote 6} and Jordan ranks 139 out of 142 countries in women’s economic participation— one of the lowest rates in the world.^{Footnote 7}

The most recent global economic crisis has slowed Jordan’s GDP growth, and the conflict in neighboring Syria has had a considerable impact on the socio-economic front. The COVID-19 pandemic has also negatively affected Jordan’s already challenged economy. In April, the World Bank forecasted the economy to shrink to negative 3.5% of GDP in 2020 due to reduced global and local demand.^{Footnote 8} Labor force participation is also expected to continue to decrease coinciding with a sharp rise in unemployment that reached 19% before the pandemic.^{Footnote 9}

There is a lack of an enabling environment to support women’s participation in the labour market, such as availability of affordable childcare, safe transportation, and harassment-free workplace.^{Footnote 10Footnote 11} Unsafe, unreliable, unaffordable, inaccessible, or non-existent (especially in remote areas) transportation is a binding constraint to women’s labor force participation; safety is also a key concern. Lack of capacity-building opportunities, salaries below the minimum wage, and lack of control over income and autonomous decision making further discourages women from entering or remaining in the private sector.

Canada’s development investments in economic growth in Jordan focus on enhancing women’s economic empowerment through skills development and entrepreneurship, in order to support women to be competitive in the labour market and to establish and sustain businesses.^{Footnote 12}

Intervention

Below are summaries of both projects.

CBIE: Launching Economic Achievement Program (LEAP) for Women in Jordan

Date: February 2017 – January 2022

Budget: Can$4.91 million

Project description: This project aims to support women’s economic empowerment in Jordan through programming that seeks to address the high unemployment rates among Jordanian women and improvements to the enabling environment and culture regarding entrepreneurship and women in the workforce. The overall goal of the LEAP project is to increase women’s empowerment for sustainable economic growth. Project activities include: (1) establishment and operation of a business incubator focused on women-led teams that includes training, mentoring, advisory services and networking; (2) development, enrichment and delivery of gender-sensitive entrepreneurship curriculum and initiatives at Jordanian universities and schools; and (3) development and implementation of a stakeholder capacity-building campaign.

WUSC-CLIC: Women's Economic Linkages and Employment Development (WE-LEAD)

Date: March 2017 – March 2022

Budget: Can$6.6 million

Project description: This project aims to remove barriers to women’s access to the labour market in Salt, Irbid and East Amman. It seeks to provide skills-based development in the private health care sector, find solutions to child-care challenges and identify safe and affordable transportation to and from work. Project activities include: (1) developing and implementing new vocational training programs in the health care sector through the Vocational Training Corporation (VTC); (2) working to reduce key barriers to women’s employment related to childcare and transportation, and within the workplace; and (3) linking public and private sectors to promote on-the-job apprenticeships in health administration for female graduates of the vocational training programs.

Intervention logic

Table 1: LEAP logic model

Table 2: WE-LEAD logic model

Stakeholders

Table 3: Direct beneficiaries of the LEAP and WE-LEAD projects

LEAP

Canadian Bureau for International Education (CBIE)

Table 4 LEAP cooperation partners

WE-LEAD

World University Service of Canada (WUSC)

Canadian Leaders in International Consulting (CLIC)

Table 5 WE-LEAD cooperation partners

Evaluation approach and methodology

The methodology applied in this evaluation adheres to the principles of impartiality, independence, credibility and usefulness as well as the OECD-DAC criteria for evaluating development assistance.^{Footnote 13} The evaluation employed a participatory approach using mixed methods of qualitative and quantitative data collection tools to examine WE-LEAD and LEAP projects and appraise progress and achievement of results in the women’s economic empowerment (WEE) sector.

The evaluation was completed in 3 phases described below:

1. In the Work Plan Phase, the main output was a work plan report that included an evaluation evidence matrix and data collection plan. The work plan constituted the guiding framework for the data collection, analysis and reporting phases.
2. In the Data Collection Phase, the evaluation team implemented the data collection plan designed in the work plan. Data collection was closely coordinated with the Jordan-based primary implementers to introduce the evaluation team, organize meetings, arrange logistics and ensure timely resolution of any emergent data collection issues. The data collection phase was concluded with a debrief session to validate data.
3. In the Analysis and Reporting Phase a systematic process of organization, comparison and synthesis of the collected data was undertaken. Data analysis was guided by the evaluation evidence matrix (EEM) triangulating in the process information generated through qualitative and quantitative methods. Data analysis comprised a systematic assessment of information and data as relevant to each of the main evaluation questions. Data analysis findings were clustered under the main evaluation criteria of effectiveness, impact, efficiency, sustainability and gender equality, which enabled the generation of conclusions and recommendations.

Data collection methods included a desk review, group or individual key informant interviews, and online or phone surveys of project beneficiaries.

Sample: Purposive sampling was deemed the most appropriate sampling approach to select and target stakeholders for data collection, to ensure representation of the various sub-sets based on a set of selection criteria projected to be relevant for the later data analysis. Criteria proposed for selecting specific population sub-sets varied depending on the specific beneficiary group but generally involved gender, age, regions, and type or extent of project involvement. After selecting the main population sub-sets, the evaluation team applied a random sampling approach to select and target beneficiary stakeholders within each specific population sub-set. The sample size of each surveyed group took into consideration the relative significance of each beneficiary group to the assessment of the 2 main drivers of the projects i.e. increasing women’s economic empowerment and gender equality.

Limitations: Whether for qualitative data or quantitative survey results, the evaluation’s main source of information was based on self-reported feedback by project stakeholders and beneficiaries. It is important to note that the evaluation was not designed to validate reported figures from project reports, especially the indicator numbers reported under each project’s immediate and intermediate outcomes. Hence, the reported statistical information should be considered potentially indicative rather than absolutely indicative of achievement of project outcomes.    

The main limitation of purposive sampling in the context of the evaluation of the WE-LEAD and LEAP projects is the risk for researcher bias. Researcher bias, also called experimenter bias, is intrinsic to the selection of the specific population groups to participate in the collection of evaluation data since it is based on the evaluators’ judgement of the relative importance of these groups to the research findings. Researcher bias is mitigated through triangulation of multiple data sources with the survey findings.

COVID-19 and the need for remote data collection imposed some restrictions on the initially planned data collection methods. Focus group discussions were changed to group interviews and/or key informant interviews to limit the number of beneficiaries/stakeholders present in close proximity. Site visits and observations were cancelled due to the travel restrictions imposed by the Government of Jordan, and all interviews were conducted online via media platforms. Nevertheless, the collection of evaluation data was possible for the different evaluation target groups, reaching the great majority of surveyed beneficiaries to identify and validate findings.

Despite repeated requests and reminders, the LEAP team was unable to secure the required permits from the Jordanian Ministry of Education to survey school students.

Key findings

LEAP

• Finding #1: (Immediate outcome 1110) A business incubator targeting women-led teams (WLTs) with business support services and infrastructure —excluding finance or linkages to financial institutions— is established within INJAZ operations in Amman. 
• Finding #2: (Immediate outcome 1110) Overall satisfaction of WLTs with the My Start-up incubator business support services with outstanding needs focused mainly on financial support and networking with investors.
• Finding #3: (Immediate outcome 1210) Increased capacity of INJAZ to provide gender-sensitive entrepreneurship training programs that inspire female and male students to pursue a future career in entrepreneurship.
• Finding #4: (Immediate outcome 1210) The LEAP project did not yet implement linkages with “communities or supportive environments for female students in their pursuit of entrepreneurship as a career path.” 
• Finding #5: (Immediate outcome 1220) BPWA and members of the project’s advisory committee report increased understanding of barriers but no measurable changes in capacity.
• Finding #6: (Intermediate outcome 1100) Project-supported increase in productivity and job opportunities generated by women entrepreneurs is tracked only during the incubation period with no further monitoring after WLTs graduate the incubator.
• Finding #7: (Intermediate outcome 1200) Limited and mostly indirect results were observed for “reduced gender-specific barriers to women entrepreneurship and to women entering and remaining in the workforce.” 
• Finding # 8: LEAP indirectly built INJAZ’s internal capacities for gender integration. 
• Finding #9: Project implemented as separate pillars with no tangible integration, benefits or learning shared across pillars. 
• Finding #10: LEAP is overachieving with regard to the type of business support services provided by the incubator and by the end of project will most likely achieve its planned results for WLT satisfaction with the incubation experience.
• Finding#11: LEAP has overachieved on the target number of students receiving entrepreneurship programs, is on target to achieve plans for revised and new programs reflecting international standards and is on its way to attaining “stakeholders’ understanding of gender and entrepreneurship.” 
• Finding #12: Variance exists between results reported by the project and those observed in this evaluation. While the target number of stakeholders has been overachieved, the evaluation observed increased awareness rather than understanding or capacity of stakeholders. 
• Finding #13: (Intermediate outcome 1100) LEAP is well on its way to achieving the target for this outcome.
• Finding #14: Reported and observed results reveal delays and challenges with regard to achieving intermediate outcome 1200. However, LEAP is likely to achieve some progress by end of project.
• Finding #15: Several factors positively impacted the achievement of expected results including INJAZ’s positioning in the youth development space, the niche focus of the incubator, and the representation of the private sector on the advisory committee.
• Finding #16: Several factors also negatively impacted the achievement of expected results including weak metrics for measuring awareness and start-ups’ success, limited linkages with other players in the ecosystem and limited financial support for emerging start-ups.
• Finding #17: Several international and local actors are active in the women’s economic empowerment development space. There is a lack of an overall sector vision to inform direction, and coordination is weak. 
• Finding #18: The project highlights the added benefits of financial and life-coaching support to female entrepreneurs in Jordan to help maximize their performance and draw participation from outside the capital Amman.
• Finding #19: Entrepreneurs and start-ups need a habitable ecosystem including supportive laws and regulations, favourable investment levels and tax system. Otherwise, entrepreneurship is questionable as a job creator.
• Finding #20: To encourage private sector companies to provide safe and gender-sensitive working environments, more strategic initiatives are needed to incentivize companies to adopt agreed-upon criteria.
• Finding #21: The number of WLT graduates (incubated teams), even with a high survival rate, would not significantly affect change on the long-term project goal of “increased women’s economic empowerment” because the level of the indicator is set too high.
• Finding #22: Reduction of gender specific barriers will not be significantly impacted by project end.
• Finding #23: LEAP’s overall project management is relatively efficient, but better horizontal integration between the 3 pillars would have improved efficiency.
• Finding #24: Results efficiency related to partners’ contributions and position within the sector is relatively good, but some results are difficult to ascertain beyond reported numbers.
• Finding #25: The COVID-19 restrictions on movement and subsequent economic uncertainties were the main factors that negatively affected project efficiency in terms of women’s economic empowerment.
• Finding #26: The sustainability of the incubator for hosting and supporting future cohorts of women-led start-up teams will depend on INJAZ’s capacity to raise new donor funds to cover the incubator’s operations.  
• Finding #27: The sustainability of LEAP gender-integrated entrepreneurship programs is contingent upon INJAZ’s ability to secure future donor support to maintain delivery of these programs to school and university students.  
• Finding #28: The preliminary results regarding increased awareness and understanding of stakeholders are not sustainable without a clearer vision of how such awareness can translate into effective action that can addresses barriers.
• Finding #29: At the personal level, the project engendered significant gains in confidence, agency and capacity as perceived by the participants.
• Finding #30: At the relational level, the participants perceived that the project enhanced their agency and decision-making power in relation to others and their ability to exert influence in the family. 
• Finding #31: At the environmental level, participants reported increased support from family members and immediate social circles and positive changes in social norms.
• Finding #32: Respondents reported reduced barriers in access to training and/or employment opportunities and control of resources.

WE-LEAD

• Finding #1: (Immediate outcome 1110) WE-LEAD has built the capacity of the Vocational Training Corporation and three VTC centres to deliver 2 new market-driven and gender-sensitive skills training programs for women in the health sector.
• Finding #2: (Immediate outcome 1120) Collaborative practices/linkages between employers and training providers were initiated for curriculum development and training. Permanent mechanisms to further integrate women into employment are a work in progress. 
• Finding #3: (Immediate outcome 1130) WE-LEAD created access for unemployed women to providers of market-driven and gender-sensitive skills training programs.
• Finding #4: (Immediate outcome 1210) Observed results for “improved access to more equitable, inclusive and safe working and training conditions” vary. While pilot child care facilities and cooperation with Health Care Accreditation Council (HCAC) is laying the ground for improved results, transportation issues remain challenging.
• Finding #5: (Immediate outcome 1220) Increased awareness on women’s economic empowerment is challenging to observe, especially with regard to communities and families.
• Finding #6: (Intermediate outcome 1100) The compounded effect of the challenges experienced in delivering on the project’s immediate outcomes resulted in low rates of women entering into and remaining in the workforce.
• Finding #7: (Intermediate outcome 1200) The pilot child care facilities were successful in addressing a main barrier to women’s access to training and employment, but other barriers, namely transportation and social norms, are still outstanding impediments.
• Finding #8: Unintended outcomes have been significant in terms of enabling additional and more advanced investments for women’s economic empowerment in the health sector.
• Finding #9: Unmet expectations in women’s economic empowerment led to some level of disappointment with the program, which can potentially affect the project’s credibility. 
• Finding #10: (Immediate outcome 1110) WE-LEAD is well on its way to achieving all planned targets under this outcome.
• Finding #11: (Immediate outcome 1120) WE-LEAD is overachieving on planned targets under this outcome.
• Finding #12: (Immediate outcome 1130) WE-LEAD is overachieving on planned targets for this outcome.
• Finding #13: (Immediate outcome 1210) Achievement of target numbers on most indicators does not fully translate to “improved access to more equitable, inclusive and safe working conditions” for all women beneficiaries. 
• Finding #14: (Immediate outcome 1220) Assessing progress was challenging due to the lack of appropriate appraisal measures. Observed results indicate some level of indirect change in social norms in support of women’s economic empowerment. 
• Finding #15: (Intermediate outcome 1100) WE-LEAD is unlikely to achieve this outcome by end of project due to the current COVID-19 situation, as well as other economic and geographic factors.
• Finding #16: (Intermediate outcome 1200) WE-LEAD is expected to achieve some level of reduced gender-specific barriers in the availability of child care and minimal improvement to safe working environment and availability of transport solutions by end of project.
• Finding #17: Several factors positively impacted the achievement of expected results including the project’s investment in a niche market with high-growth potential; alignment with national priorities, strengths of local partnerships; management by local staff; ability to leverage emerging opportunities; and donor flexibility.
• Finding #18: Several factors negatively impacted the achievement of expected results including transportation; certification delays; social norms; weak linkages with the private sector; inadequate understanding of the market landscape; leadership changes in government institutions; lack of associated regulatory reforms; and COVID-19.
• Finding #19: It is important to conduct market research to understand market demand needs and nuances prior to the identification of target areas for training.
• Finding #20: It is important to work with local partners who can navigate the local context and provide technical support.
• Finding #21: Regulatory and policy reforms and enforcement are needed to ensure market demand, minimum wage and to raise standards within the sector.
• Finding #22: Awareness raising efforts around new occupations should be targeted to families and immediate communities, as well as trainees.
• Finding #23: Better marketing of the project and earlier linkages with the private sector would have improved outcomes.
• Finding #24: Government ownership is critical for sustainability.
• Finding #25: WE-LEAD has created a new market potential for supporting women’s engagement in the workforce and has operationalized a viable approach to women’s economic empowerment.
• Finding #26: The likelihood of attaining a significant impact on “increased women’s empowerment” by project end is limited.   
• Finding #27: WE-LEAD’s management efficiency was rated mostly effective by the majority of operation partners, private and government stakeholders with noticeable improvements throughout implementation.
• Finding #28: With regard to results efficiency as it relates to partners’ contributions, the project was able to successfully engage well-established local partners and to secure significant contributions from them.
• Finding #29: Multiple contextual factors negatively impacted the project’s efficiency in achieving women’s economic empowerment.
• Finding #30: The VTC’s capacity to deliver 2 or 3 market-driven skills training programs is well institutionalized and enjoys government buy-in.
• Finding #31: Mechanisms for sustained collaboration with the private sector for future internships and employment are being established through the VTC Employment Hub.
• Finding #32: Various plans carried out with employers to enhance safe and equitable internships and working conditions are in the process of consolidation.
• Finding #33: Addressing the child care barrier is potentially sustainable. Supporting awareness-raising for women’s economic empowerment could be sustained through other actors active in the field. 
• Finding #34: At the personal level, the participants perceived the project engendered significant gains in their confidence, agency and capacity.
• Finding #35: At the relational level, women beneficiaries reported enhanced participation, influence and decision-making ability within family.
• Finding #36: At the environmental level, respondents reported more support from family and immediate environment.
• Finding # 37: Respondents reported reduced barriers to access to training and/or employment opportunities and increased ability to access and control resources.

Key conclusions*

LEAP

• Conclusion #1: The project achieved significant progress on its immediate outcomes and outputs. Findings #1, 2, 3, 4 and 5
• Conclusion #2: Current indicators under intermediate outcomes are not adequate to provide information on performance toward outcomes. Findings #6 and 21
• Conclusion #3: The likelihood that the project will achieve impact as operationalized by the project is highly unlikely. Findings #21 and 22
• Conclusion #4: The project is efficient, but horizontal linkages between the pillars are very weak affecting results efficiency. Findings #23 and 24
• Conclusion #5: The sustainability of interventions under the project varies depending on the pillar. Findings #26, 27, and 28
• Conclusion #6 The project enhanced gender equality at the personal, relational and environmental levels and reduced barriers to access to training and employment. Findings #29, 30, 31, and 32

WE-LEAD

• Conclusion #1: The project achieved significant progress on its immediate outcomes and outputs, but was challenged meeting its intended intermediate outcomes. Findings #1, 2, 3, 4, 5, 6 and 7
• Conclusion #2: Challenges are mostly due to the project’s limited accommodation of market needs and contextual variables including culture. Findings #6 and 18
• Conclusion #3: What should have been a central role for the Jordanian government in ensuring the project’s success turned out to be sub-optimal. Findings #18 and 21
• Conclusion #4: The early childhood development component has the potential for success if the project learns from the other 2 training and employment programs. Finding #7
• Conclusion #5: The impact of the project resides in the new approach operationalized by WE-LEAD for women’s economic empowerment. Findings #25 and 26
• Conclusion #6: Project management and operations are relatively efficient with room for improvement in better targeting of awareness activities and improved engagement with the private sector. Findings #19, 22, 23, 27, 28 and 29
• Conclusion #7: The sustainability potential of observed results varies but can be significantly bolstered. Findings #30, 31, 32 and 33
• Conclusion #8: Overall, the project enhanced gender equality at the personal, relational and environmental levels and reduced barriers to access to training and employment. Findings #34, 35, 36 and 37

Key recommendations*

LEAP

Recommendation #1: Improve monitoring management and framework to include metrics that track the potential achievement of outcomes prior to project end line.  Targeted party: LEAP and GAC
Link to conclusions #2 and 3

Recommendation #2: To boost growth, sustainability and impact, build linkages with financial institutions and other entrepreneurship players in the sector that can extend further support to project graduates. Targeted party: LEAP
Link to conclusions #1 and 5

Recommendation #3: Focus stakeholder capacity building activities on tangible, well defined and measurable changes. Targeted party: LEAP and GAC
Link to conclusions #1 and 2

Recommendation #4: Strengthen horizontal linkages to increase efficiency and optimize impact. Targeted party: GAC
Link to conclusion #4

Recommendation #5: Introduce research and advocacy initiatives with government stakeholders to support changes in the legal, taxation and investment ecosystem to provide a supportive environment for entrepreneurs and start-ups. Targeted party: GAC
Link to conclusion: #5

WE-LEAD

Recommendation #1: The private sector is a key stakeholder for attaining and sustaining project outcomes. WE-LEAD (or future women’s economic empowerment programs) needs to introduce new strategies and project tactics to engage the private sector. Targeted party: WE-LEAD and GAC
Link to conclusions: #2, 5 and 6

Recommendation #2: Address identified weaknesses in awareness-raising activities through more targeted interventions to reach the specific interest groups with messaging focused on women’s economic empowerment. Targeted party: WE-LEAD and GAC
Link to conclusions: #2 and 5

Recommendation #3: Build the capacity of the VTC Employment Hub ensuring a new, committed advisory committee and strong VTC buy-in. Targeted party: WE-LEAD
Link to conclusions: #3 and 7

Recommendation #4: Apply the lessons learned from medical office assistant and health support worker skills training and private sector employment to build a stronger foundation for early child development training and employment. Targeted party: WE-LEAD and GAC
Link to conclusion: #4

Recommendation #5: Consider and, if feasible, pursue regulatory reforms to sustain demand for 2 vocations and raise standards in the health sector. Targeted party: WE-LEAD and GAC
Link to conclusion: #3

Management responses

Department’s response: The Department took note of the consultant’s findings, conclusions and recommendations and has shared them with relevant stakeholders for consideration.

Cooperation Partner(s) response(s):

CBIE: The partner took note of the consultant’s findings, conclusions and recommendations and has shared them within the organization for consideration. The partner has committed to integrate specific actions into the remaining program delivery time frame that will address the accepted recommendations.

Language: If you would like a copy of the report, please contact info@international.gc.ca