Evaluation of the Vietnam Country Program, 2009-10 to 2016-17 – Summary Report

Evaluation of the Vietnam Country Program Full report

The evaluation questions that were asked

Relevance:

• To what extent has the Global Affairs Canada Program responded to the needs and priorities of the target populations?
• To what extent is the Global Affairs Canada Program aligned with the country’s needs and national priorities?

Effectiveness (Results Achieved)

• To what extent did Global Affairs Canada’s Program achieve immediate and intermediate outcomes and contribute to achieving the ultimate outcome?
• To what extent did the Global Affairs Canada’s Program achieve results related to gender equality, environmental sustainability and governance?
• To what extent did policy dialogue contribute to the achievement of outcomes?

Sustainability

• To what extent have, or will, the results and benefits of Global Affairs Canada’s Program continue beyond GAC’s development assistance?

Efficiency

• Are there any opportunities to improve the operational efficiency of Global Affairs Canada’s programming?

Evaluation Scope

• Evaluation Period: 2009-10 to 2016-17
• Total aid disbursements 2009-10 to 2016-17: $208 million, with average disbursements of $30 million per year
• Data Methods:
  • key stakeholder interviews (n=134)
  • focus groups with project beneficiaries (n=15), with 6F/9M participants
  • site visits (n=11 projects)
  • project review (n=13 projects, assessment of implementation and results)
  • project and program financial analysis
  • document review
• Evaluation Completed: January 2018

What the evaluation found

Relevance

The Program was aligned with the official development needs and priorities of Vietnam. Key areas of focus for Vietnam were improved market efficiency and competitiveness, human resource development, and infrastructure, as well as gender equality and the environment. Capacity building was an integral part of programming and helped shape Vietnam’s plans, priorities, and strategies across levels of government and thematic sectors. The Program conducted sector-specific analyses and understood that there was a need to pay attention to certain higher-risk populations. In practice, the needs of target groups, such as women and ethnic minorities, were not well reflected in projects.

Effectiveness

• Engagement at multiple levels of government, long-term presence in the sector, and positive policy dialogue with key interlocutors allowed the Program to achieve results in the agricultural, environmental, and legislative reform sectors. In sectors where these success factors were not present, projects faced long delays and had poorer results.
• In the environmental sector, Canada’s reputation and expertise granted the Program a seat at the negotiations table and an ability to influence decision-making beyond the size of its financial contributions. Canada’s contributions helped shape and advance the national environmental plan and use of collaborative approaches in policy design and implementation that became known as the "Canadian Way".
• Identifying and communicating gender equality as an issue in Vietnam was a challenge, as stakeholders across all sectors did not perceive it as an issue. There were no gender-specific projects, however most projects attempted to integrate gender at some level.
• The Program integrated and implemented environmental sustainability into sectors where projects impacted or were impacted by environmental degradation and climate change. Results helped move progress on climate change and environmental degradation.
• As a result of the needs of the country, the Program focused on high-level governance work within institutional capacity-building and accountability, and did not focus on participation and inclusion.
• The Program demonstrated an in-depth understanding of how best to operate within the complex Vietnamese context. Long-term work and presence within the country has provided platforms to open further discussion on other priority areas of importance to Canada.

Sustainability

Projects helped the Government of Vietnam implement their plans in the sectors of environmental policy and legislative reform, agriculture and rural development, and SME development. Projects were successfully facilitated by relationships. The legacy of capacity building activities and training is evident in multiple sectors, particularly in the environmental policy and legislative reform sectors. In some cases, project theories of change were ambitious and required contextual changes well beyond project scope to achieve continued results.

Efficiency

• Working directly through and within Government of Vietnam (GoV) structures was appropriate, but complex administrative processes resulted in operational delays and required no-cost extensions in four out of six sectors.  The impact of Program decentralization was mixed.
• Key concepts that are central to Global Affairs Canada, such as Results Based Management and Governance, were difficult to translate across cultures and negatively impacted the timeliness of project implementation.
• Investment plans were not aligned with evolving program management tools, resulting in sub-optimal Program planning and reporting.
• Challenges with information and knowledge management in Canada and Vietnam contributed to operational inefficiencies.

Summary of Recommendations and Management Responses

Recommendation 1: Consider sun-setting projects in sectors where the achievement of results has been limited and find different ways of engaging in those sectors where there is mutual interest.

• Agreed. The Program will have regular policy and program dialogue with the GoV to deliver more development results in sectors of mutual interest including rural development and climate change. The Program will develop a programming paper to identify strategic orientations and action areas, consistent with the Feminist International Assistance Policy and the SEDP 2016-2020 and based on lessons learned. The paper will identify ways of engaging for optimal impact, including private sector involvement climate action.

Recommendation 2: Use analyses of the challenges that have made it difficult to advance gender issues to identify new programming opportunities. Where this is not possible, use policy dialogue to lay the groundwork for future programming.

• Agreed. The Program will leverage the resources of the FSS and Development Partners to refine analyses and to inform policy dialogue with the GoV on gender equality and women and girls. The Program will launch a review through the FSS of current Global Affairs Canada programming in Vietnam and identify opportunities to optimize the achievement of gender equality results in operational projects. The Program will discuss, in Hanoi, with the senior leaders of the GoV on opportunities to advance gender equality and results for women and girls in development cooperation.

Recommendation 3: Consider the length of time needed for societal change and lengthy approval processes when designing project timelines.

• Agreed. The Program will engage with senior leaders of the GoV to explore effective ways to address operational challenges. Besides building in appropriate time in project timelines to allow for the approval processes of both governments, the Program will seek to program in new ways that do not require lengthy approval and management processes -- to maximize development effectiveness. The Program will discuss with senior leaders of the GoV, operational issues, approval processes for projects and Memoranda of Understanding for each project, and mechanisms to deliver Official Development Assistance in the most effective way.

Recommendation 4: Articulate a coherent strategy in core documents, which are approved by senior management to serve planning and results management.

• Agreed. The Program will realign its strategy with the Feminist International Assistance Policy and take into account the feminist approach, the Action Area Policies that are under preparation, and additional policy guidance on innovation. The Feminist International Assistance Policy Implementation Plan includes initiatives to renew the strategic planning process to improve integration and coherence. This renewal process involves staged piloting of new strategic planning tools. The Program will apply these renewed strategic planning processes when called upon as part of this staged process.

Recommendation 5: Global Affairs Canada should pursue a solution for the management and sharing documents between Missions and headquarters.

• Agreed. Information Management (IM) Services will continue the development of an international, departmental IM system. A GCdocs technical pilot will run from 2018-04-01 to 2018-09-01.Information Services IM Support will use the IM Committees and the Information Management Improvement Plan to survey employees at missions and Canada to identify specific challenges, and will create ‘workarounds’ in unique situations to reduce workload and meet requirements for good IM. IM teams will develop a communication plan to explain the effective use of collaboration tools.

Considerations for Future International Assistance Programming

Considerations for Geographic Branch Programming

1. In-country experts, project stakeholders, and graduates from past projects can play a valuable role in building capacity, raising awareness, and improving the monitoring and achievement of results. There is an opportunity for Global Affairs Canada to play a role in continuing to connect these interlocutors beyond their immediate involvement in projects through the establishment of networking events or platforms. The benefit of continued involvement with these interlocutors can extend project results, bolster relationships, and improve Canada’s understanding of the longer-term impact of its interventions.
2. In countries that are rapidly changing, Global Affairs Canada should consider going beyond the traditional scope of development programming. This can include working in sectors that support government partners’ abilities to adapt to rapid expansion, particularly in the areas of industrial pollution control, public sector management related to industrial regulation, and labour rights. It can also include stronger links and cohesion between trade and diplomacy work within the country.
3. Achieving gender equality results, as the Feminist International Assistance Policy demands, will require shifts in how programming is designed, implemented, and monitored, as well as the nature of international assistance investments. This should involve changing the nature of some projects or project activities, and leveraging complementary, non-project mechanisms such as policy dialogue. Close relationships and ongoing dialogue with government partners, as well as access to gender expertise and its integration in all aspects of programming, will be required to support these shifts. Global Affairs Canada may also need to reconsider engaging in sectors where the unique needs of women and girls are not acknowledged by partners, or where there appears to be less need for intervention.
4. With regards to climate change, in countries experiencing rapid industrial expansion, Canada’s recognized achievements in the environmental sector can position it to be an advocate for the inclusion of mitigation programming alongside current adaptation work.

Considerations for International Assistance Programming across Branches

5. Global Affairs Canada should play a role in fostering information exchange and greater coherence through more explicit mechanisms and engagement between project staff and beneficiaries in various sectors. Project objectives and results often reinforce one another.