Formative Evaluation of Strengthening The People's Credit Fund Network Project (STEP) - executive summary

Table of contents

• Rationale and purpose of the evaluation
• Specific objectives of the evaluation and evaluation questions:
• Scope of the evaluation
• Development context
• Intervention
• Logic
• Stakeholders
• Evaluation approach and methodology
• Key findings
• Key conclusions
• Key lessons
• Key recommendations

Rationale and purpose of the evaluation

Strengthening the People’s Credit Funds Network (STEP) project is into year 7 of the 8 year-long project. The rationale of the evaluation was to report on the results obtained to date and to provide recommendations to stakeholders for the final stages of the project’s implementation.

The purposes of the evaluation were to inform key stakeholders of the progress towards results and whether the project is on track to meet expected results; inform the improvement of project implementation; and inform decision-making regarding the future programming in this sector in Vietnam. The evaluation was also to consider past initiatives supported by Canada to strengthen the network of the People’s Credit Fund (PCF) in Vietnam and take into consideration the impact of the COVID-19 pandemic on project activities and results.

Specific objectives of the evaluation and evaluation questions:

The specific objectives of the evaluation were the following:

1. Assess the effectiveness and efficiency of the development intervention.
2. Assess the effectiveness, relevance, and sustainability of project results.
3. Assess the relevance of the approach to support financial inclusion and economic empowerment, especially for women from ethnic minority groups and from rural areas, in Vietnam; and
4. Provide findings, conclusions, recommendations, and lessons to fulfil the above purpose.

The evaluation questions were:

• To what extent are results obtained relevant to the needs of primary stakeholders and other local partners, as well as the Government of Vietnam’s needs and priorities?
• To what extent has the development intervention achieved the expected immediate and intermediate outcomes and made progress towards the ultimate outcome as per the Logic Model? Are there unintended results, either positive or negative (e.g., impact of COVID-19)?
• Has the intervention contributed to the reduction of gender inequalities in access to financial products and services and to the economic empowerment of women, including women from ethnic minority groups and from rural areas? To what extent is the project’s gender equality strategy adapted to the needs of women, including women from ethnic minority groups and from rural areas?
• To what extent were the environmental risk mitigation and enhancement measures implemented? Were they effective in preventing negative environmental impacts and/or improving environmental management?
• How economically are resources/inputs (funds, expertise, time, etc.) converted to outputs?
• What is the likelihood that results/benefits will continue after DFATD involvement ends and are the appropriate financial and human resources committed to ensure sustainability?

Scope of the evaluation

The scope of the evaluation was to cover the entire development intervention and included evaluating the progress made by the project in achieving outcome targets in the Logic model. To this end, the scope included engaging with all the stakeholders to see the extent to which the project has addressed various dimensions of financial inclusion.

Development context

The PCF network includes 1,100+ PCFs and the Coop Bank of Vietnam (CBV). The network constitutes a financial system that serves over 2 million members, mostly in rural areas and agriculture and allied sectors in Vietnam. The PCF network is significant in the context of lower financial inclusion rates for women and particularly in rural areas. From this perspective, the Government of Vietnam (GoV)and the State Bank of Vietnam (SBV) would like to ensure the development, safety and soundness of the PCF network to contribute to their strategy to reduce poverty and promote financial inclusion.

The Canadian government, through previous initiatives, has supported the cooperative banking system since 1994. Central to Canada’s development focus is gender equality through its Feminist International Assistance Policy (FIAP) which promotes gender-responsive and environmentally sensitive initiatives. STEP was to contribute to this focus by improving the PCF network.

Intervention

STEP commenced in 2016 as a 60-month project. GAC approved two no cost extensions for the project to finish the field activities by November 2024 and close by April 2025. The CAD$18.1 million project aims to provide technical assistance to PCF institutions in 14 provinces to supply a range of financial products and services that meet the needs of the rural population. The project is being implemented by Développement International Desjardins (DID), in collaboration with its local partner, the CBV and aims to target 75 PCFs and to build the capacity of the CBV to reach the entire network of 1,182 PCFs.  

The project has two intermediate outcomes: i) Equipping the CBV to ensure its PCF network’s capacity building and improved performance of operations; and ii) Enabling the PCFs to inspire confidence, supply a range of secure financial products and services to meet the needs of rural clientele, in particular women, while ensuring its financial profitability.

Logic

The ultimate and intermediate outcomes of the project are:

Ultimate outcome

Increased financial security for Vietnamese men and women in targeted rural and agricultural areas by a more integrated and better performing Co-op Bank and PCF network.

Intermediate outcome

Improved delivery of support and oversight services by Co-op Bank to the PCF network

Improved delivery of sustainable financial products and services by PCF adapted to the needs of rural clienteles, especially women, provided in a profitable and a secure way.

Stakeholders

STEP’s stakeholders include the DID, which is the implementing organisation. The CBV, 75 PCFs and SBV are the local partners.

Primary stakeholders (direct beneficiaries) include the CBV headquarter and its branches, employees of 75 PCFs (located in 14 provinces) and their clients. GAC is the major contributor to the Project with contributions from DID and in-kind contributions from the CBV.

Evaluation approach and methodology

The evaluation took a non-experimental approach that included building arguments through plausible association between the interventions and observed changes/results achieved. This approach relied on project documentation that included the STEP Logic model, project management framework, various annual reports and other reports and documentations and collaterals produced in the project. Through a participatory approach, the evaluation team engaged the main stakeholders on the evaluation questions to bring out the achievements, challenges, and recommendations to improve the project implementation in the future.

The evaluation used a mixed methods approach to collect data and information from a range of sources. The evaluation evidence matrix notes the different sources of information, such as project data/reports, external documents, Key Informant Interviews (KII), Focus Group Discussions (FGDs), etc. The evaluation triangulated the findings using a mix of quantitative and qualitative data from the evidence matrix. The team used the internal discussions between the members of the consulting team as a source of triangulation. The evaluation included an iterative approach to consultations with most informative stakeholders covered twice or more through interviews.

Key findings^{Footnote 1}

Finding #1: STEP is relevant to the development context in Vietnam and adds to the GoV’s ongoing efforts towards financial inclusion. The project is relevant to the needs of the primary stakeholders and partners, though it lacks advocacy interventions to work on some sector level issues.

Finding #2: STEP is designed to contribute to gender equality and environmental concerns by applying a gender responsive and environmentally sensitive lens to finance interventions. The project complements the efforts of other multilateral and donors in this space.

Finding #3: STEP’s contribution is visible in the improvement of the CBV’s capacity in providing advisory services and improving oversight of PCFs. However, increased capacity has not yet been used to provide advisory support for the PCFs.

Finding #4: Project has improved and continues to strengthen the monitoring, surveillance, and risk management capability of the CBV by using technological solutions to access SBV gateway and automating the flow of information, and reporting. This would further strengthen the risk management of the PCFs and contribute to the broader goal of the project for “better performing PCFs and the CBV that would help to improve the financial security of Vietnamese men and women".

Finding #5: By working on developing an IT strategy, building a risk management framework and gender dashboard, the Project has laid groundwork for helping with strategic management of PCFs by the CBV. Once implemented, this will strengthen the PCFs’ ability to manage their risks better and take data driven business decisions.

Finding #6: STEP developed good products and services, but the uptake by the PCFs is low. Lack of an appropriate technical assistance delivery structure, inadequate product customisation and lack of a structured hand holding approach were a few reasons for low uptake.

Finding #7: Facilitated through coaching, there are nascent signs of uptake of some products like the revised credit methodology and internal audit, as evidenced by some progressive PCFs that have started to explore and formally use them. These PCFs indicated increasing the use of use of these tools in the future.

Finding #8: The project has enabled wider financial access through technology related interventions such as e-banking, which has created significant leverage to impact clients, given the CBV competitive advantage, and benefitted PCFs.

Finding #9: COVID-19 had a severe impact on the project. It also led to some innovations, as the project significantly amplified its online engagement with the CBV and PCFs.

Finding #10: Some indicators in the project’s Performance Measurement Framework (PMF) look only remotely connected to the project outcomes and/or are overly ambitious.

Finding #11: STEP has integrated gender-responsive measures in the development of the project’s products and services, which can contribute to the reduction of gender inequalities. The project has designed these interventions to address the pillar on access to resources. The transformative impact of these interventions is possible with greater adoption and rollout of products and services developed by the PCFs. STEP’s gender equality strategy is in sync with the needs of rural women. The project activities’ implementation plan on gender is on track.

Finding #12: At the strategic level, it appears that the CBV has not fully understood the importance of gender issues and has been hesitant to incorporate gender considerations into its strategic objectives. Due to a lack of focus on gender by the CBV STEP had to abandon efforts to integrate gender within the CBV. This also raises concerns about the extent to which the CBV can effectively support the PCFs in addressing gender issues.

Finding #13: STEP has integrated environmental risk assessment and management aspects in the development of the project’s products, such as revised credit methodology. The project plans to develop non-financial green products. Since the environmental risk mitigation and enhancement measures are part of these products, it will be possible to evaluate them once the PCFs widely adopt the revised credit methodology and non-financial green products.

Finding #14: The project was efficient at using the available resources to deliver quality outputs. Through market studies, strategic assessments and extensive consultations, STEP efficiently identified and developed appropriate products and services. Of these, tech enabled products provide high leverage in terms of impacting the CBV and PCF network business and risk management. The project could use a more efficient PCF engagement by initially focusing on a few but progressive PCFs

Finding #15: Overall, the CBV has made progress on systemic risk management and e-banking rollout, which are in alignment with its supervisory role and business objectives. These are most likely to be sustainable, as the CBV uses PRMS (PCF Report Management System) to perform its role in PCF risk management and expand e-banking to add more growth to its business. However, because of the lack of an appropriate institutional delivery structure within the CBV, there has been less emphasis on providing advisory services to PCFs, which may lead to sustainability concerns for the future development of PCFs.

Finding #16: SBV appreciates products like PRMS and internal audit tools for the PCFs and would like to adapt these for its own use.

Finding #17: The CBV-PCF relationship is not clear, and both the institutions differ in terms of their business interests. This could have a strategic impact on the project in terms of sustainability of project results.

Key conclusions

Conclusion #1: STEP is relevant to the needs of primary stakeholders, partners, and the Government of Vietnam, as it contributes to financial inclusion, gender equality, and brings awareness to environmental concerns, while complementing the efforts of other multilateral and donors. The project lacks advocacy interventions to work on sector level challenges.

Conclusion #2: STEP has effectively enhanced the capacities of the CBV in providing advisory services to the PCFs and managing risks related to the PCF network, in line with the project’s desired outcome (outcome 1100).

Conclusion #3: Despite setbacks due to COVID, STEP has been effective in developing a suite of transformative products and services for the CBV and PCF network, which if adapted and used can increase their competitiveness and financial access for women in contribution to outcome 1200.

Conclusion #4: The project has not been very effective at using the capacities built to deploy broadly some products, services, and solutions to the PCFs.

Conclusion #5: Although STEP’s blended approach, which includes coaching and training both offline and online, has been effective in disseminating knowledge and skills and in implementing the products at some progressive PCFs, it has not fully addressed the need to change attitudes, behaviour, and practices in some interventions.

Conclusion #6: For some outcomes, the STEP’s Performance Measurement Framework may be tracking indicators in isolation or not closely related to the project interventions.

Conclusion #7: While the project has gender equality concerns integrated into its products and services; transformative changes are not yet visible due to low uptake of the interventions by the PCFs. The CBV’s reluctance to prioritize gender issues could hinder their support for PCFs on this front.

Conclusion #8: STEP has made significant efforts to integrate environmental risk assessment and management into its products and interventions. The environmental interventions will be visible upon the uptake of credit products. 

Conclusion #9: STEP has been efficient in utilizing available resources to deliver quality outputs but could be more efficient by focusing on providing rigorous coaching to a select progressive PCFs and leveraging VAPCF platform to reach out to greater number of PCFs.

Conclusion #10: Sustainability of PCF specific interventions like cash flow based credit methodology, internal audit tool and PRMS usage at the PCFs is in question without the CBV’s commitment to engage more through a dedicated institutional support structure for the PCFs.

Conclusion #11: Sector level structural issues will continue to undermine the project investments in the CBV and PCF network without a resolution of the larger CBV, PCF network issues.

Key lessons

Lesson #1: For long-term projects with regulatory and structural constraints, it is important to identify macro structural factors that could influence the project’s sustainability. Conclusion #11 supports the lesson.

Lesson #2: To ensure fulfilling the social and environmental objectives, a thorough stakeholder analysis is important to understand the motivation of each partner and what each actor could bring to the table. It may be necessary to negotiate and secure commitments from the partners and adjust the project design and scope, if required, at the inception stage of the project. This will ensure greater commitment to social (such as gender) and environmental outcomes. Conclusion #7 supports the lesson.

Lesson #3: The choice of piloting with a smaller number of strong and willing partners in the development stage can conserve resources and bring more success. It is easier to scale the success achieved with such partners. Conclusion #9 supports the lesson.

Lesson #4: The projects like STEP should include policy and advocacy interventions in the project design to make stakeholders aware and enable them to take necessary policy and regulatory decisions to create an enabling environment. Conclusion #1 and #11 supports the lesson.

Lesson #5: Project outcomes and indicators should be logical and realistic, and if possible, changed based on the experience in early years for better monitoring and evaluation of project results. Conclusion #6 supports the lesson.

Lesson #6: Projects such as STEP should include stakeholder engagement and consultation strategy in the project design, include joint working groups for product design and piloting, and handholding. It should include support to build capacity, ownership, accountability, and success of products and services. Conclusion #5 and #10 supports the lesson.

Key recommendations

Recommendation #1: The project should focus on quick-win solutions, such as e-banking, add more services on the platform, and leverage the VAPCF platform to expand these solutions to more PCFs. STEP should identify a few “early adopter” PCFs for revised credit methodology and internal audit toolkit, tailor these products to their context, and showcase their successes to expand the reach of the project. (Targeted party: DID/STEP)

Recommendation #2: To provide ongoing support for the adoption/adaption of products and services developed under STEP, and for post-project support, CBV should focus on creating a training/advisory delivery structure to support the PCFs. (Targeted party: GAC and DID/STEP)

Recommendation #3: To support PCFs in using cash flow credit methodology and internal audit tools, STEP should improve the existing coaching to a rigorous handholding approach. (Targeted party: DID/STEP)

Recommendation #4: STEP should involve Vietnam Association of PCFs (VAPCF) in the process of tailoring the cash flow-based credit and internal audit tools to fit the operational reality of the PCFs. It should also engage with SBV to ensure easier adoption of the tailored products. (Targeted party: DID/STEP)

Recommendation #5: GAC and STEP should work to make the project outcome indicators more realistic, measurable within the project context, and attributable to the project interventions. (Targeted party: GAC and DID/STEP)

Recommendation #6: GAC and STEP should undertake advocacy efforts with the SBV, the CBV, and the VAPCF towards larger sectoral goals of PCF consolidation and network integration. (Targeted party: GAC and DID/STEP)

Management Response

The Department took note of the consultant’s findings, conclusions and recommendations and has shared them with relevant stakeholders for consideration.