Formative Evaluation of the Innovation Platform for Maternal, Newborn and Child Health (IP4MNCH) - Executive summary

Evaluation type: Formative
Commissioned by: Global Affairs Canada
Consultant: Deloitte LLP
Date: May 2020

Rationale and purpose of the evaluation

Global Affairs Canada (GAC) commissioned a formative evaluation of the Innovation Platform for Maternal, Newborn and Child Health (IP4MNCH). The formative evaluation was undertaken to report on IP4MNCH results achieved to date, inform adjustments to future programming, and inform GAC decision-making on whether to continue a partnership with Grand Challenges Canada (GCC).

Specific objectives of the evaluation

As stated in the terms of reference, the specific objectives of the formative evaluation were to:

• assess the relevance of transition to scale (TTS) innovations with regard to the needs and priorities of end users, including innovators supported by the platform, intermediaries using innovations, and beneficiaries benefiting from innovative products and/or services
• assess the effectiveness and efficiency of IP4MNCH
• identify key factors that facilitate or hinder the graduation of innovations from the proof of concept (PoC) phase to the TTS phase
• provide findings, conclusions, recommendations and lessons to fulfill the evaluation purpose as stated above

Scope of the evaluation

The formative evaluation covered the Contribution Agreement (CA) period from March 2016 to February 2020. Innovations across all IP4MNCH program areas were covered, including closed (n=184) and active (n=232) projects (as of December 2019), and innovations funded at the PoC and TTS phases. The evaluation assessed the relevance, effectiveness, efficiency and sustainability of the platform and its portfolio of innovations.

Development context

Despite progress in many areas of health, approximately 303,000 women, almost all living in low- and middle-income countries (LMICs), died due to complications during pregnancy and childbirth in 2015.^{Footnote 1} In 2017, 5.4 million children under 5 died, the majority of whom were also in LMICs.^{Footnote 2} Almost all of these deaths could have been prevented with proper medical equipment and access to health care. The Government of Canada is committed to supporting development innovations related to reproductive, maternal, newborn and children health (RMNCH). The partnership between GAC and Grand Challenges Canada (GCC) provides a platform for funding innovators to develop and test novel solutions to RMNCH challenges.

Intervention

The IP4MNCH is designed to enable innovations to catalyze improved RMNCH outcomes. The platform includes 4 programs that fund programs from LMICs, Canada and other developed countries: Saving Brains, Saving Lives at Birth, Stars in Global Health and TTS. Innovations are supported at the seed phase to establish PoC or at the TTS phase, which includes developing viable plans for scale and sustainability, testing scaling plans and beginning to scale innovations.

Intervention logic

The logic model below lists the immediate, intermediate and ultimate outcomes of the intervention.

Stakeholders

Key stakeholders engaged included:

• primary stakeholders: GCC, its Investment Committee and Scientific Advisory Board, innovators funded by GCC and their partners (e.g., accelerators /incubators, local government), intermediaries that use innovations (e.g., health providers, educators), and beneficiaries of the innovations
• cooperation partners: Canadian Institutes of Health Research
• donor organizations and ecosystem partners: GAC and its external monitors, the Bill & Melinda Gates Foundation, Grand Challenges Africa, the United States Agency for International Development (USAID), and the Norwegian Agency for Development Cooperation (Norad)
• interested parties: The Canadian Forum for Impact Investment and Development, and the Canadian Partnership for Women and Children’s Health

Evaluation approach and methodology

The evaluation was conducted from November 2019 to April 2020. The evaluation approach was aligned with and guided by the Organisation for Economic Co-operation and Development’s Development Assistance Committee (OECD/DAC) Criteria for Evaluating Development Assistance and United Nations Evaluation Group (UNEG) best practices. A gender-sensitive and mixed methods approach was used to obtain complementary insights from multiple data sources. This included a review of secondary data through comprehensive document reviews and collection of primary data through an online survey, case studies and semi-structured stakeholder interviews. Following the data collection phase, the Evaluation Team compiled, analyzed and triangulated data and developed a comprehensive evaluation report.

Comprehensive document reviews included IP4MNCH Performance Management Framework (PMF) data; programmatic and policy documents; relevant documentation for selected case-study innovations; literature on global challenge funds; national government plans; and other relevant GCC data including Results-based Management Accountability Framework (RMAF) reports from Fluxx, GCC’s grants management system.

An online survey was deployed to all funded and closed innovations and to 60 Smart Partners (i.e., strategic partners that provide innovators with matched funding or other value-add support) to assess innovator and partner perspectives on the relevance of innovations to user needs, the level to which gender equality (GE) is addressed, and results achieved. The survey also captured feedback on the value and complementarity of funding provided by GCC and other partners and perspectives on the sustainability of innovations. The response rate was 148 funded innovators (66%), 108 closed innovators (65%), and 22 Smart Partners (37%).

Case studies were conducted for six innovations across Brazil and Uganda. Case studies encompassed in-depth stakeholder interviews, focus groups and observations to explore the ways through which innovations are made available to, accessed or used by end-users and beneficiaries, and the benefits they provide to those groups. 151 stakeholders were engaged through interviews and focus group discussions using a snowball and convenience sampling approach. Case study innovations were selected using predetermined criteria and input from GAC and GCC, including type of funding, program area and scaling pathway.

Semi-structured interviews were conducted to gather insights from a wide range of stakeholders to provide diverse perspectives on the role of GCC within the innovations landscape. Global interviews with 29 stakeholders were held in two rounds, before and after case-study country visits.

The Evaluation Team identified several limitations, among them the fact that the majority of innovators are not yet reporting on outcomes—which impedes data analysis. There were limitations with regard to assessing the extent to which innovations have reduced gender-based inequalities in access to health products and services at the portfolio level and with regard to making direct comparisons between GCC’s efficiency and other challenge funds’ efficiency given GCC’s unique approach. The Evaluation Team sought to address these limitations and triangulate findings through other sources of data, including the online survey, interviews and country field visits.

Key findings

Relevance. The Evaluation Team found that the objectives of IP4MNCH are closely aligned with several Sustainable Development Goals, and innovations are closely linked with the national priorities of the countries in which they are operating. This was particularly true of innovations related to maternal health and early childhood development (ECD). Those related to emerging global health priorities (e.g., mental health) faced challenges as these areas are not always prioritized by national governments in LMICs; however, there are several cases where innovators have had success upon the creation of new policies, or by engaging and building momentum with municipal leaders. The evaluation found that alignment with national priorities does not necessarily ensure government support or facilitate public scaling pathways. Stakeholders noted that the platform could benefit from a formal process to engage government stakeholders at PoC and the early phases of TTS.

Funding provided to develop, test and implement new ideas proved to be important, especially for the 81% of funded innovators based in LMICs where access to finance is a major barrier. Despite significant relevance to innovators, several limitations stem from the funding agreements and structure under the CA with GAC, including that GCC cannot invest in equity and that $1 million cap is not enough for some innovators to advance through the stages of TTS. Innovators suggested the need for more technical support at PoC, greater fundraising support, and a reduction in the burden of reporting. There is also a desire for greater interaction with GCC post-deal and facilitation of information sharing and networking across innovations within a region and/or focus area.

Case study innovations were found to be relevant to targeted beneficiaries’ needs, and in some cases they contributed to the well-being of intermediaries such as medical doctors and nurses. According to innovators and their partners, a majority of innovations funded are designed to be accessible to marginalized groups.

Effectiveness. At the immediate outcome level, PMF data and stakeholder perspectives indicate that there is increased awareness of GCC’s innovation platform globally and of GCC’s contribution to knowledge of how to enable successful PoC and scalable health innovations in the development innovation community. GCC support was found to have increased innovators’ capacity to demonstrate health outcomes, attract investment and integrate GE considerations into their innovations. A small number of innovators indicated increased capacity to engage government stakeholders. While there is limited evidence of increased capacity to integrate human rights and environmental sustainability into innovations, this is not surprising given GCC’s focus on GE and the limited potential for environmental impact of most innovations in the portfolio. Analysis of PMF data shows that GCC has surpassed targets related to increased awareness and intermediary capacity to use innovations in the areas of essential health care and communicable diseases, but progress toward these targets in sexual and reproductive health and rights and Saving Brains has been slower.

At the intermediate outcome level, the platform is on track to meet or exceed targets to enhance its efficiency and effectiveness in supporting health innovations to achieve PoC and improve the pipeline of innovations. The same is true for targets to support innovations to achieve scale and sustainability at TTS. GCC support has improved innovators’ capacity to complete TTS and move toward sustainable scaling, with room to improve government engagement for greater public sector buy-in and commitment. Across programs, GCC is on track to meet or exceed targets for the number of intermediaries using innovative products or services. GCC is approximately halfway to reaching the targeted number of beneficiaries using innovations across focus areas, with the exception of innovations to improve ECD, which have reached one quarter of targeted beneficiaries. There has also been limited progress in improving innovators’ GE scores, despite a large number of innovators reporting integration of GE considerations into their innovations.

Several factors were found to support the achievement of results, including alignment with national and/or local priorities, innovators’ ability to leverage GCC funding to attract other partners, and GCC’s role in providing tailored support and connecting innovators globally. External enabling factors (those outside the control of innovators or GCC) included government buy-in/ownership and market demand. On the other hand, lack of technical expertise (e.g., financial management, fundraising) among innovators, limitations in the CA (e.g., $1 million cap to fund TTS innovations, funding vehicles), and limited GCC field presence and government engagement were identified as internal factors limiting the achievement of results. External factors such as changing and complex political environments and other national level contextual factors can also play a limiting role.

Several unintended results were identified at the innovation level. For example, investment in innovations has, in some cases, led to the development of broader innovation ecosystems and communities (e.g., success of the AIR device has been leveraged to build an innovation ecosystem at Mbarara University of Science and Technology, in Mbarara, Uganda, including at CAMTech Uganda, which has incubated 20 innovations). Similarly, some innovations have seen greater-than-expected market demand for innovative products and services, increased engagement of men and boys through innovations not targeted to that gender, and outcomes extending beyond innovations’ intended beneficiaries (e.g., to intermediaries).

Gender equality. Survey responses uncovered the 3 most common approaches for the promotion of GE: (1) training women and girls on health behaviour change; (2) analysis and reporting of results by gender and age; and (3) engaging women and girls to understand their needs. Other common approaches included developing and implementing a GE strategy, and working with men and boys to improve outcomes for women and girls.

Across programs, gender-disaggregated data reported in RMAFs provided supporting evidence of increasing numbers of intermediaries and beneficiaries using innovations, and increased access by women and girls. There is limited definitive evidence of increased control over decisions and resources to access innovative products and services, but case studies provided supporting evidence of improvements resulting from access to innovations that have enabled women and girls to seek care and secure the resources needed to support their families. Survey responses also indicated that a greater proportion of innovations are empowering women’s decision-making over time.

Innovators reported satisfaction with the availability of technical training and resources to integrate GE—including tool kits, sex-disaggregated reporting, and one-on-one support—but more evidence is needed to demonstrate the effectiveness of GCC’s support to do so. Very few closed innovations have demonstrated improved GE scores over the funding period. While GCC data provides limited evidence of improved GE scores, the Evaluation Team noted that the proportion of survey respondents reporting embedding GE “to a great extent” in their innovation was higher than those GCC classified as GE-2 or GE-3 projects at the time of signing the funding agreement. This could indicate a mismatch between innovator and GCC understanding of and expectations for GE, or that innovators have improved the integration of GE since project start. If the latter is true, a greater number of innovations would be expected to demonstrate improved GE scores upon project close.

Stakeholders consulted suggested a number of opportunities to review GCC’s GE strategy. Several are already being pursued by GCC, including a revised equality scoring rubric and revised gender analysis tool. Others emerged over the course of this evaluation, including the need for an improved understanding of gender outcomes achieved by innovators and a desire for greater support for innovators funded at PoC. There may also be an opportunity to review and increase the proportion of women-led innovations graduating from PoC to TTS, and innovations led by low- and middle-income countries and low-income countries innovators funded at TTS, which could inform future investment decisions.

Efficiency. The evaluation assessed the platform’s allocative efficiency (i.e., outcomes achieved compared to funds disbursed) and operational efficiency (i.e., outcomes achieved compared to operational costs). At the portfolio level, GCC operates the innovation platform with low overhead costs, with 84% of the GAC contribution amount allocated and invested directly in innovations.

At the program level, operational efficiency is highest for STARS due to GCC’s role primarily as a funder, without post-deal support to innovators. Allocative efficiency is greater for TTS as GCC offers targeted technical support (e.g., one-on-one support from venture advisors) for innovators to enhance their sustainability, scale and/or impact. While inputs are greater, TTS innovations are expected to drive results, particularly lives saved and lives improved.

Factors found to contribute positively to efficiency include integrating learning communities within targeted challenges, basing investment decisions at TTS 3 and TTS 4 on impact modelling, implementing a phased approach to TTS, and drawing on the expertise of external parties and partnerships. GCC’s position outside of government was also found to contribute to its efficiency, enabling the platform to be dynamic and to respond more quickly to shifting global priorities. Stakeholders suggested several opportunities to improve efficiency, including removing the $1M cap on TTS investments, improving the continuity of funding cycles (e.g., by bridging 5-year funding cycles or extending the timeframe for innovators to spend funding), providing greater flexibility to adapt and respond to learning over time, and allowing funding of catalytic grants (i.e., for an entire portfolio of a geographic cluster of innovations). Streamlining reporting requirements to reduce the burden on innovators, particularly those at PoC, and playing a bigger role in identifying ecosystem players to take TTS projects to scale and facilitate “soft handoffs” when GCC funding ends were also suggested.

It is difficult to compare cost efficiency per outcome with other challenge funds given differences in approaches to impact modeling between funds. In general, stakeholders viewed GCC as an efficient organization, albeit one that had room for improvement in certain areas, including the more timely disbursement of funds, a reduction in the amount of time required for administrative tasks, and—for innovators—streamlined reporting requirements.

Sustainability. GCC aims to have 7-10% of its PoC projects transition to scale with GCC funding, a target it achieved as of PMF fiscal year 2019/20. A further 10% of innovations are transitioning to scale through other partners. Several factors have affected likelihood of sustainability of project outcomes and graduation of innovations from PoC to TTS. Common internal factors enabling graduation included evidence of impact and demand, user acceptability (beneficiaries and/or intermediaries), innovator capacity and peer-to-peer knowledge transfer. On the other hand, short funding timelines, matched funding requirements and limited technical support at PoC were identified as common challenges hindering graduation of innovations from PoC to TTS.

External factors supporting graduation of innovations from PoC to TTS included public sector engagement, alignment with government priorities, and engagement of influential partners or champions. In contrast, common external factors hindering graduation of innovations included typical field research challenges (e.g., delayed ethics approvals), delays associated with public sector scaling pathways, complex or changing political environments, and availability of funding to begin the TTS process. Case study innovators have demonstrated integration of sustainability considerations into innovation design and implementation at both PoC and TTS. For example, innovator approaches have included early government engagement, identifying appropriate scaling partners, and dual market approaches (i.e., targeting two or more unique markets) for scale-up. These approaches improve the chances of sustained outcomes, but there is room to improve engagement of the local private sector and of municipal, regional or national governments for long-term scaling and sustainability.

Key conclusions

Relevance. GCC takes an innovator-centred approach, tailoring the support provided to innovators based on their stage of development and capacity, improving relevance to their needs and priorities. The investment approach contributes to bridging the “pioneer gap” (i.e., the burden shouldered by organizations pioneering new models in the early stages of growth not considered investable by many impact investors) by deploying capital that is targeted, flexible and patient. Funding and technical support provided by GCC is highly relevant to innovators’ needs and priorities, particularly due to focused investment in LMICs, with room to improve funding vehicles and structure, and greater technical support at PoC.

Effectiveness. IP4MNCH has demonstrated its effectiveness in achieving most of the desired immediate and intermediate outcomes. The platform is on track to meet, or has the potential to meet, the majority of its immediate and intermediate targets in fiscal year 2020/21 (Table 1).

Gender equality. GE remains a focus for GCC and its innovators, who use a variety of context-specific approaches to integrate GE considerations into their innovations. Innovators are satisfied with the support and resources available to integrate GE, but more evidence is needed to understand gender outcomes and GCC’s ability to improve GE integration.

Efficiency. GCC is an efficient organization and operates IP4MNCH with low overhead costs. Contributing to the platform’s efficiency, GCC takes a venture capital approach to investing, recognizing that not all projects in a portfolio will succeed and few will produce substantial returns. This informs program level approaches, as GCC invests broadly at PoC to build the innovation pipeline and then spends more time and resources providing tailored support to high-potential innovations at TTS, which are expected to drive results. At the program level, allocative and operational efficiency therefore vary due to differences in delivery approach.

Sustainability. GCC has been instrumental in enabling several factors for sustainability such as building innovator networks within targeted challenges and providing innovators opportunities and guidance to demonstrate PoC. Case-study innovators demonstrated integration of sustainability considerations into innovations’ design and implementation, supporting evidence that outcomes will be sustained, particularly for TTS innovations. There are opportunities to improve the engagement of local private and public sector officials.

Key recommendations

Based on the findings of this formative evaluation, the Evaluation Team proposed the following considerations for GAC and GCC:

Short-term (could be implemented within the current CA):

• improve the catalogue of support materials for innovators, including for GE and fundraising (Owner: GCC)
• develop virtual platforms to connect innovators by focus area and/or geography (Owner: GCC)
• take on a bigger role in identifying ecosystem players than can take TTS projects to scale once GCC funding has ended (Owner: GCC)

Medium-term (could inform design of the next CA):

• provide greater flexibility in the CA for GCC to invest in high-potential innovations over a longer timeframe, removing the $1M cap on TTS investments and the number of times GCC can reinvest in a single innovation (Owner: GAC)
• allocate greater resources for in-depth learning on factors affecting effectiveness and efficiency of the platform (Owners: GAC and GCC)
• conduct a cross-portfolio gender review to improve understanding of GE outcomes (Owners: GAC and GCC)
• review the funding life cycle with GCC to improve platform effectiveness and efficiency by avoiding starts and stops caused by 5-year cycles (Owner: GAC)
• leverage convening power with local government to improve chances for sustainable scale and efficiency of results (Owners: GAC and GCC)
• reduce the reporting burden on PoC innovators by aligning indicators with what is needed to demonstrate PoC (Owners: GAC and GCC)

Long-term (could be implemented during the next funding period):

• continue to invest broadly at PoC (high volume of smaller investments), but focus the funding approach for TTS to invest a higher dollar amount in a smaller number of high potential innovations (Owner: GCC)
• provide a greater level of post-deal support to innovators at PoC to improve the chances of graduating to TTS, while balancing cost efficiency through development of standardized toolkits and/or virtual support (Owner: GCC)

Management response summary

Global Affairs Canada and Grand Challenges Canada have taken note of the consultants’ findings, conclusions and recommendations and have agreed to implement the suggested recommendations outlined in this report.

Language: The evaluation report was originally written in English, and a French version is also available. If you would like a copy in either language, please contact info@international.gc.ca.