Summary report: Evaluation of the Maternal, Newborn and Child Health Initiative, 2010-11 to 2017-18

Why it is important?

Since 2010, Canada has played a leadership role in global action to end the preventable deaths of mothers, newborns and children. Canada’s Maternal, Newborn and Child Health (MNCH) Initiative is a ten-year (2010–2020) $6.5 billion commitment aimed at improving the health of women and children in the world’s most vulnerable regions. This represents a significant amount of international assistance resources spanning numerous geographic regions and functional areas.

What the evaluation assessed

Global Affairs Canada’s international assistance programming associated with the MNCH Initiative between 2010–11 and 2017–18. The evaluation combined a summative evaluation of MNCH 1.0 programming from 2010–11 to 2015–16 with a formative evaluation of MNCH 2.0 programming from 2015–16 to 2017–18.

Evaluation questions

Relevance

• To what extent were the activities funded by MNCH initiatives (including sexual and reproductive health) aligned with recipient country needs in low and middle-income countries?

Effectiveness

• To what extent did the activities funded under Global Affairs Canada’s MNCH initiatives achieve their immediate outcomes?
• To what extent did the activities funded under Global Affairs Canada’s MNCH initiative contribute to intended results at the intermediate outcome level?
• To what extent has programming in strengthening health systems helped to achieve greater results in other programming areas (disease, nutrition, and sexual reproductive health?)
• In which contexts were different types of partners able to achieve maximum results (e.g. Canadian partners, multilateral partners, bilateral partners, etc.)?

Gender Equality

• To what extent has programming contributed to gender equality results and the empowerment of women and girls?

Innovation

• Have any innovative practices, approaches, and new ways of working been integrated into Canada’s MNCH programming in recipient countries?

Sustainability

• What is the likelihood that the results will be sustained?

Efficiency

• Are there opportunities to improve the efficiency of programming?

What the evaluation found

Relevance

• Global Affairs’ MNCH programming was aligned with partner government priorities and local needs in all countries sampled, and it evolved to better target key gaps identified in the 2015 formative evaluation.
• Programming was relevant to international MNCH goals and there was a continued need for support of the broader determinants of health and adolescents.
• Health systems strengthening was an essential approach for achieving sustainable MNCH outcomes, and would be a valuable overarching theme for all MNCH projects.

Effectiveness

• MNCH projects have been achieving results and a high proportion of the projects examined made progress at the immediate and intermediate levels of corporate MNCH outcomes.
• While it is difficult to attribute success solely to Canadian international assistance, MNCH projects contributed to decreases in child mortality in Bangladesh, Mozambique, Tanzania and South Sudan.
• The Department could not fully report on the overall results of the MNCH initiative as a key thematic priority, primarily because of a misalignment of performance indicators between the project and the departmental level.
• Reporting challenges for effectiveness were compounded by reporting software that was not fit for purpose, a lack of adequate technical support for staff, weak country-level data systems, and a lack of attention paid to data projects designed to help Global Affairs report at the impact level.
• Project timeframes were frequently insufficient for implementation and achieving long-term results.
• Canada was seen as a respected donor with a positive track record in contributing to national-level policy dialogue and donor coordination in the health sector in all countries sampled.

Gender Equality

• Canada was recognized as a leader in advancing gender equality.
• Overall, there was an increased effort to ensure project activities and delivery approaches were gender sensitive, and observed and reported progress suggested positive changes towards increasing gender equality.

Innovation

• Global Affairs successfully supported some partners to engage in innovative work, but the Department did not yet have a culture that fully fostered innovation in employees’ daily work.

Sustainability

• Although no mechanism existed to assess the sustainability of MNCH projects after funding ended, evidence suggested that the effects of some MNCH interventions would likely endure beyond the duration of the project.
• Some projects incorporated key strategies to support sustainable results, such as promoting ownership and stakeholder participation at all levels, and integrating cross-sectoral considerations. However, many MNCH results could be at risk without ongoing donor support due to internal and external factors.

Efficiency

• Slow project and planning approval processes caused significant delays in project implementation, though project partners appreciated efforts to be flexible with extensions.
• Financial disbursements to MNCH projects were generally timely, though late disbursements for some health pooled funds disrupted the delivery of some planned health services.
• Insufficient internal specialist resources negatively impacted timeliness of program delivery, as well as Global Affairs’ ability to engage in technical and strategic discussions.
• Frequent turnover of Global Affairs staff made the management of MNCH projects less efficient and presented a potential reputational risk.
• Over the course of the evaluation period, there were several instances of a need for better communication and guidance on new policies and priorities including Sexual and Reproductive Health and Rights and the Feminist International Assistance Policy.
• Retroactively applying changes in reporting requirements put an unnecessary burden on implementing partners.

Summary of recommendations and management responses

Recommendation 1

For any future health and nutrition programming, consider continuing to integrate the needs of adolescents.

Response: Global Affairs Canada agrees with this recommendation. Adolescents face numerous health challenges and are often marginalized by health services and policy, while also facing legal and social barriers to accessing care. In line with the Feminist International Assistance Policy, Global Affairs Canada will ensure that greater focus is placed on the needs of adolescents in new initiatives, including those that will be part of Canada’s 10-year commitment to health and rights, as announced by Prime Minister Trudeau on June 4, 2019.

Recommendation 2

To facilitate the achievement and sustainability of results, ensure that health programming promotes an integrated, multi-sectoral approach which addresses the broader determinants of health.

Response: Global Affairs Canada agrees with this recommendation. The SDGs and Canada’s Feminist International Assistance Policy both require an integrated, multi-sectoral approach that puts the broad needs of women and girls at the center. In line with the Feminist International Assistance Policy, Global Affairs Canada will ensure multi-sectoral approaches in new initiatives, including those that will be part of Canada’s 10-year commitment to health and rights, as announced by Prime Minister Trudeau on June 4, 2019.

Recommendation 3

Strengthen and modernize corporate reporting systems, including both software and IT systems, which support all international assistance programming, especially thematic programming like MNCH. This includes:

• Ensure that staff receive direction from senior management along with technical support on the systematic use of indicators, guidance tools, and compliance with reporting requirements.
• Ensure that the sequencing of corporate reporting systems aligned to avoid retrofitting.

Response: Global Affairs Canada agrees with this recommendation. Global Affairs Canada continues to improve reporting guidance and quality of reporting throughout its corporate reporting exercises on Canada’s Feminist International Assistance Policy.  The performance and reporting functions in the department are a shared responsibility with the Corporate Planning and Reporting Division (SRD), the Evaluation and Results Bureau (PRD), and the International Assistance Operations Bureau (DPD), while the International Assistance Policy Bureau (PVD) leads on the ODA Accountability Act report.

Recommendation 4

For future health programming, ensure that projects have sufficient time for implementation and the greatest chance to achieve longer-term outcomes by considering project durations of up to ten years, where appropriate.

Response: Global Affairs Canada agrees with this recommendation. When assessing projects, Global Affairs Canada always considers their potential to achieve sustainable, long-term outcomes, as evidenced in due-diligence and project approval documents. In light of Canada’s 10-year commitment to health and rights, as announced by Prime Minister Trudeau on June 4, 2019, Global Affairs Canada will be well positioned to consider longer project timeframes where appropriate. In all projects, Global Affairs Canada will continue to work with implementing partners to develop projects that have the greatest chance to achieve longer-term outcomes.

Recommendation 5

Provide staff with sufficient and timely access to internal technical expertise, particularly health and gender specialists. Access to gender specialists would ensure that project staff have the technical support they need to best respond to policy guidance and emerging requirements, including more recent expectations stemming from the Feminist International Assistance Policy.

Response: Global Affairs Canada agrees with this recommendation. The Global Issues and Development Branch (MFM) has been leading a review of the specialist function based on the findings of the 2016 MNCH audit. In December 2018, the Deputy Minister of International Development (DME) approved 11 recommendations on specialist service delivery. Chief among them was to establish a specialist coordination function within MFM. This role has been assumed by MFMZ, the Branch’s recently established Business Intelligence Unit, which will lead the implementation of the remaining recommendations.

Considerations for future programming

Policy considerations

Health systems strengthening overarching theme: Given the widespread acceptance of health systems strengthening as a foundation for sustainable results, programs could consider adopting health systems strengthening as an overarching theme when proposing health investments. This would include the training of frontline health personnel, disease, nutrition, data, and other aspects of health programming. This would also help to ensure that health systems strengthening is considered in the design of all projects seeking health outcomes.

Communicate guidance on shifts in Departmental priorities and policies in a timely manner: Knowing that there will always be changes in priority commitments as contexts change, Global Affairs could embrace a more fulsome approach to change management. Specifically, the Department could review and develop streamlined processes to ensure that when commitments shift, the necessary guidance and support will be available to staff as soon as possible. These processes would increase the timeliness of implementing new departmental priorities and policies, such as Sexual Reproductive Health and Rights programming and the Feminist International Assistance Policy.

Programming considerations

Timeliness of project and planning approval: In line with efficiency recommendations from recent evaluations such as South Sudan (2017) and Honduras (2017), as well as the Audit of the Harmonization of the Administration of Grant and Contribution Programs (2017), MNCH programming branches could work towards streamlining internal processes to reduce the time required to complete project agreements, as well as further standardize requirements.

Building learning for sustainability into projects: Sustainability is concerned with measuring whether the benefits of an intervention are likely to continue after donor funding has ended. To support the sustainment of results, programs can incorporate sustainability considerations into the planning and design phase of projects to ensure the appropriate transition of resources, skills and knowledge at the end of the intervention. For high-risk or innovative projects, programs could also experiment with new approaches to monitor whether results are sustained after funding has ended.

Reporting requirements: To minimize disruption and burden on partners, Global Affairs could consider the value of applying any newly developed reporting tools or processes only to future projects than to existing projects.

Initiative mapping: To inform program design, facilitate coordination and avoid duplication, global initiatives, like MNCH, could map current and planned initiatives, along with funding from all donors, in recipient countries. This could be done at the country level. For example, KFM has been working with CanWach on Project Explorer, which includes mapping of 134 MNCH projects funded by Global Affairs.

Management considerations

Managing staff mobility and rotation: International assistance branches could identify best practices and develop procedures to ensure robust knowledge transfer to incoming staff.

Comprehensive Results Framework: Given that priority commitments change, a streamlined process could be established to develop and roll-out coherent Department-wide results framework (including program logic model, performance measurement framework with international best-practice aligned indicators, and methodology notes for partners). This should be done in a timely manner when new commitments are announced. The results framework should be in place as early as possible to inform the program plans that respond to priority commitments.