Bangladesh Country Program Evaluation

2008-09 to 2012-13

Final Report

Acknowledgments

The Development Evaluation Division would like to thank all those who contributed to this evaluation. Staff from Global Affairs Canada’s Global Issues and Development Branch, Geographic Program Branches, Partners for Development Innovation Branch and Corporate Planning, Finance and IT Branch provided invaluable support throughout the process. We especially thank those who hosted the field missions and facilitated data gathering.

We would like to acknowledge the work of the team of consultants from Geospatial/SALASAN Consulting Inc.: Keith Ogilvie, Peter Hoffman, Rafiq Jaffer and Michelle Munro as well as Julia Ahmed who worked as a local consultant in Bangladesh.

From the Development Evaluation Division, the evaluation was initially managed by Muhammad Hussain, succeeded by Deborah McWhinney. Denis Marcheterre and then Pierre J. Tremblay, supervised the evaluation.

The Development Evaluation Division retains the authorship of the final report.

David Heath
Head of Development Evaluation

List of Abbreviations

Executive Summary

This report presents the findings, conclusions, lessons and recommendations stemming from the evaluation of Canadian development programming^{Footnote 1} in Bangladesh for the 2008-2013 period.

Rationale, Purpose and Objectives

In accordance with Canada’s Financial Administration Act and the Treasury Board of Canada’s Policy on Evaluation requirements, this evaluation has been undertaken for accountability and learning purposes and to support:

• evidence-based decision making by the ministers, deputy heads and development partners on policy, expenditure management and program improvements; and
• accountability of the government to Parliament, Canadians and development partners for results achieved with resources invested in this Program.^{Footnote 2}

The objectives of the evaluation are to assess the Bangladesh Country Program’s overall performance in achieving its expected results, including lessons from implementing the Program, and provide Global Affairs Canada with relevant information for future development programming. The primary audience for this evaluation is Global Affairs Canada management and programming staff. Program partners, including the Government of Bangladesh (GoB), non-governmental organizations engaged in project delivery, and other development agencies working with Canada in Bangladesh are also considered key users of this evaluation. (See Annex A for the summarized Terms of Reference.)

Evaluation Object and Scope

Bangladesh, a country with 158.5 million people (2014), has maintained a uniform annual decline in poverty rates of 1.74 percent between 2000 and 2010. This is likely due, in part, to a sustained GDP growth rate of more than 6 percent. However, according to the most recent data (2010), Bangladesh continues to suffer from high poverty: 43.3 percent of the population was still living with less than US$1.25 per day. Governance issues remain challenging in Bangladesh. The World Bank Governance Indicators show performance against five of the six indicators in the bottom quartile, with little change in ranking since 2003.^{Footnote 3}

Bangladesh has been one of Canada’s largest bilateral aid recipients since 1971, the year Bangladesh gained independence from Pakistan. The period under review was guided by the 2003-2008 and the 2009-2013 Country Development Program Framework (CDPF) for Bangladesh. Programming areas of focus shifted between CDPFs from education, health, democratic governance and private sector development to children and youth (including activities in education and health) and sustainable economic growth (SEG), with governance recast as a cross-cutting theme. During this period, xCIDA rolled out a “bigger and fewer” project delivery strategy, a global maternal, newborn and child health initiative as well as children and youth and sustainable economic growth strategies, all with implications for the Bangladesh Program. xCIDA also shifted management of its Bangladesh Program from headquarters to the field.

The evaluation covered fiscal years 2008/09 to 2012/13 and considered the different channels of cooperation and program delivery, particularly bilateral programming, which constituted 65.5 percent of xCIDA’s overall investments and partnerships. Although the review did not evaluate xCIDA’s long-term institutional support for multilateral agencies, it did evaluate bilateral funding of the Bangladesh Program to these agencies and the demands that xCIDA’s multilateral programming placed on staff in the field.

Methodology

The evaluation used a theory of change approach, which meant that the design of the evaluation methodology was based on an analysis of the Program logic and the contextual factors that may have had an effect on program implementation. The evaluation applied the OECD/DAC criteria for evaluating development assistance: relevance, effectiveness, efficiency and sustainability.

The data used to generate the findings and conclusions were gathered from secondary sources and supplemented by interviews with more than 200 respondents in Canada and Bangladesh and reports based on observations made by evaluators during field visits to health sector projects. From the overall portfolio of 61 projects with disbursements exceeding $250,000 within the evaluation period, a representative sample comprising 24 individual projects that covered all sectors, delivery channels, and investment types within the country program, was examined. Annex D describes the methodology, including the data sampling methods and limitations, and presents the evaluation matrix and evaluation questions.

Key Findings

Relevance

Both at the sector and project levels, the Program’s results aligned well with the needs of Bangladesh as set out in a) the Government of Bangladesh’s declared priorities (Five Year Plans and Poverty Reduction Strategy) and b) international development metrics, including the Millennium Development Goals. Similarly, the Program was highly aligned with the Government of Canada’s development priorities, xCIDA planning documents and corporate policy documents. Given the existing focus on maternal, newborn and child health (MNCH) in Bangladesh, the Program was able to quickly respond to the elements set out in the 2010 Maternal, Newborn and Child Health Initiative (the Muskoka Initiative). The bilateral project portfolio fit well with the activities of other donors and enjoyed a high degree of internal coherence within and between priority sectors.

Effectiveness

Health—Health sector programming contributed to measurable improvements at the sector and project levels. Parallel health projects have complemented core sector investments and had a good record of achieving their intended results.

Education—Education sector programming contributed to improvements in governance and management of the formal education system as well as improved learning outcomes. The two major consortia to which xCIDA contributed achieved positive results. The Program successfully supported complementary programming to develop life skills of at-risk adolescents and improve access for children with disabilities, reinforcing its image as an active player in promoting continuous and second chance education. Thus, Program support to both formal and non-formal sectors was effective.

Sustainable Economic Growth—This sector, introduced with the 2009 CDPF, included income enhancement projects for the ultra-poor and poor that were successful in meeting their outcomes, suggesting a good record in achieving results through activities formerly categorized as private sector development. Public Financial Management (PFM) projects—moved from the governance to the SEG portfolio as part of the “enabling environment”—had a mixed record of results achievement. PFM is an area of critical importance, being fully governance oriented. The results that were achieved in spite of the challenges resulting from the overall state of the governance environment in Bangladesh were important. They contributed to the functioning of other donor arrangements including the SWAps in education and health sectors.

Governance—Governance was first a sector with a portfolio of projects (2003-2008 CDPF) and then repositioned as a cross-cutting theme that became part of almost all projects, either as the main theme or as a key component of project activities. Governance played a central role across the Program. Investments addressed overarching issues of governance rather than a particular service delivery sector. As a result, they faced more challenges due to the unpredictable class="table table-condensed table-bordered" Bangladesh political context, which included extended periods of political instability. The projects were therefore varied in their effect.

Cross-Cutting Themes—The Bangladesh Program consists of a large number of multi-donor initiatives. Given that these initiatives are designed through negotiation with several other actors, xCIDA was limited in the extent to which it was able to fully incorporate xCIDA’s cross-cutting themes across such investments.

Efficiency

The use of resources, as measured by Program disbursements against number of staff, revealed a very small increase during the first four years of the evaluation period followed by a drastic increase (35 percent) in the last year. One cannot spread the last year’s gain over the period covered to support the premise that a program consisting of “fewer, larger” projects was automatically more efficient and easier to manage. Indeed, while large multi-donor initiatives that employed sector wide approaches (SWAps) offered efficiencies for the Government of Bangladesh, these initiatives consumed a significant amount of the Program staff’s time. Field staff were also at times required to respond to unanticipated demands to support partnership and multilateral programming.

Results-based management (RBM) tools were extensively used for reporting at project and program levels, in conformance with xCIDA requirements. However, their value in the planning and monitoring of programs, especially those within multi-donor mechanisms like SWAps, was uncertain, as they are in many cases a duplication of systems already agreed to by multi-donor consortia. Evidence demonstrated that the xCIDA’s tools were not used for program or project management, but for internal reporting. At the program level, the evaluation identified a number of weaknesses in the relevance and suitability of indicators.

Sustainability

Sustainability was a significant concern for the Bangladesh Program, with under half the projects in the sample falling short of the desired level when assessed by the evaluation team. Many of the structural elements, regulations and laws, processes and allocation of responsibilities which would have been required for projects to be sustainable were often within the purview of the central agencies of the GoB and even Parliament itself, and outside the purview of the project.

Prospects for sustainability in PFM and public accountability have been compromised by uncertainty over future donor funding to support related programs and projects. Sustainability received limited attention even when raised in joint Government of Bangladesh-donor forums. Ongoing policy dialogue efforts by donors have been more effective in improving implementation than in changing policy.

Conclusions

The evidence reviewed in this evaluation depicts a program that successfully maintained programming responsive to the evolving needs of Bangladesh. Canadian development programming in Bangladesh has been consistent over the long run in terms of both priority sectors and the approach it has taken. The Program has done so using delivery mechanisms (SWAps in particular) that have given Canada a prominent place in discussions concerning its social development priorities that have integrated well with the initiatives of other donors. As a member of and contributor to several major donor consortia, Canada can take credit for having contributed to many of the changes that have occurred over the evaluation period.

Based on the Program’s experience in its priority sectors, including health and education, the evaluation team concludes that governance is a challenge and is key in achieving and preserving results in all priority sectors. In the health sector, efforts to strengthen health systems faced these challenges. While the investments in MNCH in Bangladesh led to steady reductions in maternal and child mortality, there is a need to ensure that this progress does not overshadow work to address the strengthening of health systems. The focus in health sector support should be on marginalized populations that are less likely to realize the benefits of broader development.

In the education sector, Canada has been successful in contributing to the formal education system, while contributing to efficient and effective initiatives that parallel the formal system with support to hard-to-reach children and in efforts that mainstream these children into the formal system. Given that it is ultimately the responsibility of government to provide quality primary education for all, through two parallel systems, there are opportunities to foster closer linkages between these systems.

In the SEG sector, the portfolio of projects, and in particular PFM projects, had positive pervasive effects through policy dialogue, which influenced governance at many levels in various government bodies. These efforts enable systemic and most likely sustainable changes.

Finally, the evidence suggested that SWAps and similar large, multi-donor projects were not necessarily more efficient in terms of the demand they place on Global Affairs Canada Program staff. To be effective in sector and sub-sector working groups and at donor table class="table table-condensed table-bordered"s requires an optimum number of qualified resources otherwise these groups must be populated with staff chosen strategically from the current limited roster of resources using the complementarity of skills and sector technical knowledge, in particular with respect to PFM, governance, and gender equality.

Lessons

In delivering its programming, the Bangladesh Program experienced challenges in ensuring the sustainability of results as a result of systemic governance weaknesses. Even in cases where development interventions had political or high level policy support from the Government of Bangladesh, the influence of vested interests and the scope/span of a development intervention can have a significant impact on its success. Greater success (as indicated by effectiveness, efficiency and sustainability) is achieved when: a) vested interests are minimal, which enables a constructive policy dialogue and a collective sense of local ownership; and b) the logic of the development intervention is focused. Less success is achieved in situations where: a) vested interests are making policy dialogue difficult and local ownership is not equally shared among stakeholders and; b) the logic of the development intervention is multi-faceted.

In delivering its programming, the Bangladesh Program’s coherent portfolio demonstrated strong reinforcing linkages among projects. In particular, the projects concerned with governance or projects incorporating governance as a cross-cutting theme. For example, the Project Financial Management project contributed to the management of education and health SWAps, while the Parliamentary Strengthening and Media Support Project strengthened the role of the Bangladeshi parliament in overseeing the PFM project. Evidence suggests that country programs adopting a mix of concurrent and reinforcing governance projects enable catalytic and positively pervasive changes and contribute to greater effectiveness.

Recommendations

1. Maintain a strong presence in the sectorsin which Global Affairs Canada is presently involved but adjust some elements of the Program for greater effectiveness.
   1. In health, emphasize improving health systems governance and provision of health services to remote and disadvantaged communities.
   2. In education, emphasize governance of the education system and closer linkage between formal and non-formal education systems.
   3. In SEG, work on ensuring continuity of engagement in both program funding and introduction of systemic change through policy dialogue.
2. In the Bangladesh context, where weak governance has been a central determinant of the success of programming in all sectors, strengthen the focus on governance elements of all Global Affairs Canada programming by putting into place appropriate agreements with relevant central ministries to ensure the necessary budgets and personnel allocation are or will be in place for sustainability.
3. Staff the Program to meet the Program’s needs. A priority should be placed on engaging field advisors for gender equality, environment and governance. Canada-based staff in the field should have an appropriate mix of technical and analytical skills in priority areas.
4. Improve the utility and use of the Program’s Results-based management (RBM) tools. Improve the quality of Program level indicators and targets, and, in multi-donor programs and projects, link Global Affairs Canada systems directly with the RBM tools collectively agreed on to avoid duplication or unnecessary data collection.
5. Improve coordination with partnership and multilateral programs with a view to better aligning not just programming but also demands on field resources. In their own planning efforts, managers of these programs should anticipate and communicate their expectations regarding field support.

Management Response

This management response reflects the departmental priorities as expressed at the end of the evaluation period. Going forward, management responses will reflect new Departmental priorities as they evolve.

Introduction

This report presents the findings, conclusions, lessons and recommendations of the evaluation of Canadian development programming in Bangladesh from 2008-09 to 2012-13.

In June 2013, Foreign Affairs and International Trade Canada amalgamated with the Canadian International Development Agency (xCIDA) to become the Department of Foreign Affairs, Trade and Development. The latter was renamed Global Affairs Canada in November 2015. Since the evaluation covers the period just prior to amalgamation, the evaluation report refers to the former xCIDA but refers to Global Affairs Canada to describe recent decisions or issues arising since the amalgamation.

1.1 Evaluation Purpose and Objectives

In accordance with Canada’s Financial Administration Act and the Treasury Board of Canada’s Policy on Evaluation requirements, this evaluation has been undertaken for accountability and learning purposes and to support

• evidence-based decision making by the ministers, deputy heads and development partners on policy, expenditure management and program improvements; and
• accountability of the government to Parliament, Canadians and development partners for results achieved with resources invested in this Program.^{Footnote 4}

The objectives^{Footnote 5} of the evaluation are to assess the Bangladesh Country Program’s overall performance in achieving its expected results including lessons from implementing the Program; and provide Global Affairs Canada with relevant information to improve future programming. The primary audience for this evaluation is Global Affairs Canada management and programming staff. Program partners, including the Government of Bangladesh (GoB), non-governmental organizations engaged in project delivery, and other development agencies working with Canada in Bangladesh are also considered key users of this evaluation.

1.2 Evaluation Object and Scope

1.2.1 Bangladesh Context

Bangladesh, a country with 158.5 million people,^{Footnote 6} maintained a 1.74 percent annual decline in poverty rates between 2000 and 2010.^{Footnote 7} The country made significant progress toward many of the Millennium Development Goals, including those related to gender equality, education and maternal, newborn and child health. The country’s ranking on the Human Development Index has undergone modest but steady improvement with a 2 percent increase between 2008 and 2013. According to the World Bank, it is reasonable to assume the decline in poverty rates have continued given a sustained GDP growth rate that exceeded 6 percent, 3.3 percent growth in the agriculture sector and an average 6.7 percent growth in remittances since 2010.^{Footnote 8},^{Footnote 9} However, Bangladesh continues to suffer from high poverty. As recently as 2010, 43.3 percent of the population still survived on less than US$1.25 per day.^{Footnote 10}

Bangladesh’s industrial sector, in particular a rapidly growing garment industry, and remittances, were the primary drivers of the positive economic growth. In this context, a gap emerged between people engaged in traditional rural livelihoods and those who moved into new, more formalized, employment opportunities. Workers’ rights and protection became a more visible element of current donor programming, as has establishing linkages between non-governmental organizations (NGOs) and the GoB to facilitate the provision of basic social services to remote communities and to vulnerable people.

But these economic gains were limited by severe weaknesses in the governance system and by regular political instability. A conflict over election modalities in 2005 and 2006 resulted in large scale protests and eventually a takeover of government by a military backed regime. Bangladesh returned to a democratically elected government in December 2008, following an election held by the military backed regime and that was widely seen as free and fair. The Parliament elected in 2008 never fully fulfilled its role because the main opposition boycotted it. Following constitutional changes in 2012 that removed the provision for elections to be held under a caretaker government, conflict resumed over election modalities, culminating in the 2014 elections which were boycotted by the main opposition and in which 153 out of 300 elected seats were uncontested. In this fluctuating political environment, the Bangladeshi public service continued to use ineffective systems and processes that were not conducive to fundamental reforms with reform of the overall governance system.

The Transparency International 2014 Corruption Perceptions Index ranked Bangladesh 145th out of the 174 countries assessed and as one of 36 countries whose score declined from 2013.^{Footnote 11} The World Bank Governance Indicators show Bangladesh’s performance to fall within the bottom quartile for five of the six indicators, with little change in ranking since 2003. According to David Lewis, there was little evidence that efforts made by the GoB to tackle corruption were effective in curbing corruption. In his analysis, these issues are no longer seen as simply regulatory issues, but as having “hardened into a severe structural problem embedded in the social and political forces which govern the distribution of power and influence”.^{Footnote 12}

1.2.2 xCIDA Programming in Bangladesh

Bangladesh has been one of Canada’s largest bilateral aid recipients since Canada recognized its independence from Pakistan in 1971. Annual aid disbursements averaged $89.1 million between 2008 and 2013.^{Footnote 13} In 2009, Bangladesh was identified as a country of focus for Canadian development assistance programming.^{Footnote 14}

Over the evaluation period, a number of xCIDA policy and programming changes affected the nature and direction of the Bangladesh Program.

The Program included initiatives started under both the 2003-2008 Country Development Program Framework (CDPF), and the 2009-2014 CDPF. The 2003-2008 CDPF focused on social development (education and health, governance and private sector development). The 2009-2014 CDPF had two priorities: children and youth, with a focus on education and health; and sustainable economic growth (SEG). During this time, governance became a cross-cutting theme. The Program’s two governance-specific projects aimed at strengthening public financial management were retained under the SEG theme.During this period, the xCIDA sought to deliver programming through fewer, larger projects and to decentralize the management of bilateral programs from headquarters to the field.

1.2.3 Evaluation Scope

The evaluation covered fiscal years 2008/09 to 2012/13. It considered bilateral programming, which accounted for 65.5 percent of the xCIDA’s disbursements in Bangladesh, and partnership programming. Long-term institutional support for multilateral agencies and emergency humanitarian assistance was excluded;^{Footnote 15} however, bilateral funding through multilateral organizations was evaluated as was the effective integration of field staff into multilateral programming.

1.3 Methodology

The evaluation used a theory of change approach^{Footnote 16} and the OECD DAC key evaluation criteria of: relevance, effectiveness, efficiency and sustainability. These criteria are consistent with the Treasury Board of Canada’s Policy on Evaluation. The data used to generate findings and conclusions were gathered from secondary sources and supplemented by interviews with more than 200 respondents in Canada and Bangladesh and reports based on observations made by evaluators during field visits to health sector projects. From the overall portfolio of 61 projects with disbursements exceeding $250,000 within the evaluation period, a representative sample comprising 24 individual projects that covered all sectors, delivery channels, and investment types within the country program, was examined. Annex D describes the methodology, including the data sampling methods and limitations, and presents the evaluation matrix and evaluation questions.

2. Main Findings

The following sections are structured according to the evaluation’s key criteria and questions. Each sub-section presents a brief summary of the analysis used to support the finding.

Sectoral analyses are presented in the following order: Health, education, sustainable economic growth (SEG) and governance. In order to minimize duplications on governance elements, the analysis of the public financial management (PFM) project is at the end of the SEG section immediately preceding the Governance section. In addition, cross-cutting governance is included within the analysis of governance rather than with other cross-cutting themes. Projects started under the 2003-2008 CDPF are herein called legacy projects.

2.1 Relevance

The evaluation considered relevance in terms of the extent to which:

• the objectives of the Program were aligned with the needs of the beneficiaries and the country and xCIDA’s development priorities; and
• the Program’s strategy and the projects in the portfolio were relevant to meeting the intended results.

2.1.1 Alignment with the Needs and Priorities of Bangladesh

Canada’s development priorities correlated strongly with those of the Government of Bangladesh. The Bangladesh Program responded to the development needs reflected in the priorities set out in the Government of Bangladesh’s (GoB’s) five-year plans and its Revised 2009-2011 National Strategy for Accelerated Poverty Reduction. For example, the GoB’s sixth five-year plan identified the following priorities:

• accelerating economic growth and employment;
• improving human resource development (education, health and population);
• ensuring gender equality and environmental stability; and
• improving governance, enhancing administrative capacity, strengthening the public service and improving the planning and budgetary processes.

The Program focused on the key sectors identified in the 2003-2008 and 2009-2013 Country Development Program Framework, which aligned well with the GoB’s development priorities, including in treatment of cross-cutting themes of gender equality, environment and governance.

Programming in the Health Sector

Health sector programming was almost entirely focused on maternal, newborn and child health (MNCH). The value of the bilateral project health portfolio totalled $251.6 million. As indicated in table 1, five of the six legacy projects (in italics) addressed MNCH and reproductive health. For example, the first two projects provided parallel support to the country’s second and third health SWAps (Health, Population and Nutrition Sector Development Programs). The multiple components of these projects sought to:

• identify better service delivery models;
• support the procurement of essential MNCH and family planning commodities; and
• add Canada’s voice to policy dialogue in Bangladesh.

Health projects approved after 2009 reflected Canada’s commitment to MNCH announced in the Muskoka Initiative in 2010 and were prioritized in the Global Affairs Canada’s 2009 Country Strategy and CDPF.

*SWAps are indicated in blue. Legacy projects are in Italic. Budget figures include only the contribution of Canada.

The evaluation sampled five of the twelve projects representing 47 percent of health disbursements^{Footnote 17} from 2008/09 to 2012/13. In addition, three partnership projects were sampled: Maternal Reproductive Health, Community Led Health and Access to Health and Education for All (shared with the education sector). They all sought to improve MNCH in Bangladesh by strengthening the capacity of local civil society to improve health care.

Health sector programming remained well aligned with GoB health sector policies and strategies. The 2009 CDPF for Bangladesh was formulated during the implementation of the second health sector SWAp when the GoB’s 2011 Health, Population and Nutrition Development Sector Development Plan was also being formulated.

GoB and donor partners agreed that health sector programming was a strategically appropriate mix of sector investment and projects that targeted gaps in the health system or in service delivery levels, with a balance between demand and supply approaches. Other donors acknowledged Canada for taking on initiatives aimed at strengthening health sector human resources and supporting immunization. A number of interview respondents noted that Canada’s support to the International Centre for Diarrhoeal Disease Research supported Bangladesh MNCH and wider health research capacity, which has benefited Bangladesh (where the Centre is located) and other countries. The Adolescent Reproductive Health project filled an important niche, a fact that was highlighted by donors as a gap in the 2014 Midterm Review of the health SWAp. Canadian support for the Joint GoB-UN Maternal, Newborn and Child Health Initiative influenced the decision to add earmarked funding for community skilled birth attendant training to the Human Resources for Health project.

While the health sector portfolio included projects aimed at strengthening the health system, evidence suggests that the donor focus on meeting the MDG health sector targets, while needed, resulted in less attention being paid to wider efforts to strengthen the health system.

Health projects financed through the Partnership channel were not all aligned with the health ministry’s policies/guidelines for the sector. The evaluation noted that the process of reviewing partnership projects (review of calls for proposals) against the reality in Bangladesh sometimes failed to identify or anticipate issues that should have been taken into account. For example, the Community Led Health Project and the Maternal and Reproductive Health Project did not adhere to the Ministry of Health and Family Welfare (MoHFW) guidelines on Traditional Birth Attendant training, which had been discouraged since 2005.

Programming in the Education Sector

Education sector programming was mainly focused on primary education, with additional support provided to programs for hard to reach children (i.e. those living in remote areas) and disadvantaged youth (i.e. disabled or living in poor urban areas). The project portfolio delivered through the bilateral Program had a total value of $152.4 million. All projects of the Primary Education Development Program (PEDP II)^{Footnote 18} started under the 2009 CDPF. It is a large SWAp supporting the formal education system. The first four projects in the table were large multi-donor initiatives that supported NGO-delivered primary education initiatives working in parallel and complementary to the formal system. The remaining two legacy projects were designed to meet the educational needs of working urban youth.

*SWAps are indicated in blue. Legacy projects are in Italic. Budget figures include only the contribution of Canada.

The evaluation sampled four of the seven projects, representing 67 percent of education disbursements from 2008/09 to 2012/13. In addition, it also included one partnership project shared by the education and health sectors: Access to Health and Education for All.

Education sector programming was directly aligned with the GoB’s Education for All National Plan of Action and its policy framework for the primary education sub-sector initiative. As well, education sector programming supported the national Non Formal Education Policy Framework, which stipulate that basic education programming should be a continuous and progressive process. All five of the education sector projects reviewed had a focus on poor and disadvantaged children, particularly girls, with poverty alleviation being the ultimate goal.

Programming in the SEG Sector

The value of bilateral SEG sector programming delivered totalled $62.8 million. As indicated in table 3, SEG programming included legacy projects initiated under the rubric of Private Sector Development, which aimed to provide self-employment, skills upgrading and access to finance and training opportunities for the rural and urban poor. These projects shared a number of similarities, particularly in targeting the ultra-poor and emphasizing rural economic development projects. The portfolio also included two public financial management (PFM) projects transferred in the 2009 CDPF from the former governance sector on the basis that sound PFM is essential to support sustainable economic growth.

The value of bilateral SEG sector programming delivered totalled $62.8 million. As indicated in table 3, SEG programming included legacy projects initiated under the rubric of Private Sector Development, which aimed to provide self-employment, skills upgrading and access to finance and training opportunities for the rural and urban poor. These projects shared a number of similarities, particularly in targeting the ultra-poor and emphasizing rural economic development projects. The portfolio also included two public financial management (PFM) projects transferred in the 2009 CDPF from the former governance sector on the basis that sound PFM is essential to support sustainable economic growth.

*SWAps are indicated in blue. Legacy projects are in Italic. Budget figures include only the contribution of Canada.

The evaluation sampled five of the ten projects, representing 80 percent of SEG disbursements during the evaluation period. In addition, two projects funded through partnership programming were sampled: Partnering for Microfinance Development and Impact and Local Enterprise Investment Centre.

The SEG sector projects were fully or mostly aligned (except for one project^{Footnote 19} ) with the needs of the beneficiaries and with GoB priorities, as identified in the GoB’s five-year plans and in its Poverty Reduction Strategy Papers. For example, the PFM project, SPEMP, was noted in the Bangladesh SPEMP Annual Independent Review (2013) as having received “strong support” from Bangladesh’s finance minister and the finance secretary.

Programming in the Governance Sector

The value of governance sector programming delivered through the bilateral program totalled $65.6 million. As indicated in table 4, all projects were developed under the 2003-2008 CDPF. Governance sector programming covered a range of activities and included two relatively large projects.

The first, Chittagong Hill Tracts (CHT), was a grant to a multi-donor UNDP program with the aim to ensure the rights of the CHT indigenous peoples by strengthening the system of indigenous government while also supporting the GoB’s administrative and local government system.

The second was a major, long term intervention on legal reform that focused on promoting access to justice, particularly for women, children and other vulnerable groups.

The portfolio also included three projects aimed at improving elections and engaging the country’s mass media institutions, civil society and grassroots organizations to use the country’s new Right to Information Act to hold local government bodies and the national parliament to account. Finally, there it included three other initiatives:

• a project aimed at capacity building in Bangladesh’s Department of Environment;
• a local fund geared toward small, gender-equality projects; and
• a gender- equality policy support project.

Overall, the sector portfolio was designed to address the importance of strengthening political as well as institutional processes, with the understanding that governance itself has a political component.

*SWAps are indicated in blue. Legacy projects are in Italic. Budget figures include only the contribution of Canada.

The evaluation sampled four of the eight projects, representing 60 percent of governance sector disbursements during the period being evaluated. In addition, one partnership project was sampled: Inter Pares - Enabling Leadership.

Governance sector projects were found to have explicitly addressed the government’s policy objective of improving governance, enhancing administrative capacity, strengthening the public service and improving planning and budgetary processes.

2.1.2 Relevance to xCIDA Intended Results

The evaluation found the Program was highly aligned with xCIDA’s policies, strategies and programming priorities at the corporate and program level across the sampled projects: 21 of the 24 sampled projects met all or most expectations against the relevance criteria. See Annex E: Project Rating Summary.

Health programming directly contributed to the core objectives of the Program’s Country Strategy to create opportunities for children and youth, and to the xCIDA’s Children and Youth Strategy. The Program’s existing health program was strongly aligned with Canada’s commitments made under the Muskoka Initiative. All Program respondents agreed that the Initiative complemented rather than changed priorities and in fact brought additional resources to an already well positioned sectoral program.

With respect to education programming, the evaluation found a high level of coherence and linkages among the sampled education projects and between these projects and other donor activities at the sector level. xCIDA’s involvement in the large overlapping consortia for the Primary Education Development Program II and the BRAC Education Programme (BEP) increased the opportunity to engage in education sector policy dialogue since other key donors supported both these programs as well.

Significant programming synergies were revealed among sectors as a result of the implementation of the Program during the period being evaluated. For example, the focus on public financial management (PFM) in the SEG sector played an important supportive role in ensuring the integrity of the education and health SWAps to which xCIDA contributed. Similarly, the governance sector projects generated opportunities to cooperate with the projects in other sectors. For instance, as the Parliamentary Strengthening and Media Support Project became more focused on strengthening the role of parliament in overseeing PFM, closer links were developed with the two PFM projects, SCOPEand SPEMP. Finally, the Bangladesh Environmental Lawyers Association, a local partner in an Inter Pares project, provided training to the Department of Environment staff in the area of law and environmental regulatory compliance, which complemented the BEISP.

The shift of governance programming from an area of focus to a cross-cutting theme served to increase, rather than diminish, its relevance. It highlighted its importance to the successful implementation of education, health and SEG programming, both in terms of state building and state-civil society relationships. In addition, there was considerable complementarity between xCIDA and other donor governance programming, as illustrated by the following examples:

• The European Union’s governance sector programming focused on human rights, particularly in the Chittagong Hill Tracts, election reform and standing parliamentary committees;
• USAID’s governance programming focused on PFM and local government;
• DFID focused on strengthening the GoB’s revenue collection capacity and community access to legal services; and
• UNDP focused on strengthening the judiciary system, election management, civil service change management, devolution to local government and improved access to information.

2.2 Effectiveness

In assessing effectiveness, the evaluation considered the extent to which the Program contributed to, or achieved, expected results as set out in the CDPFs and project logic models.

With respect to effectiveness, 21 of the 24 projects in the evaluation sample (88 percent), fully or mostly achieved their intended results. Three projects fell short, two for reasons outside the control of the respective projects and one (of small value, 0.6 percent of portfolio disbursements) due primarily to an inappropriate design. ^{Footnote 20}

However, the overall effectiveness of a program is more than the sum of each project’s effectiveness. Program outcomes are also achieved as a result of program’s “non-investment” efforts such as policy dialogue, donor coordination, harmonization, alignment and synergy between delivery channels. It is also influenced/challenged by the country context.

The following sections present an assessment of a) each of the four key sectors’ (health, education, SEG and governance) effectiveness preceded by a short summary of their specific context and challenges, b) cross-cutting themes’ effectiveness and c) Program’s “non-investment” efforts.

2.2.1 Health Sector

Health Sector Context and Challenges

The 2009 Bangladesh Country Strategy cited three key obstacles to health sector improvement:

• persistent marginalization of poor and rural populations;
• slow progress in reducing maternal mortality and high rates of neonatal death; and
• shortages of human resources in the health sector, in particular skilled birth attendants.

Health sector projects faced challenges in improving health sector governance due to:

• the centralized nature of the MoHFW, especially with respect to health financing decisions and public service appointments: planning, funding and service delivery modalities could not efficiently respond to local health needs;
• key barriers to institutional strengthening: solutions to systemic staffing issues lay with central agencies of the GoB and were outside the purview of the MoHFW;
• fragmentation of the MoHFW into directorates: these functioned as silos for health care and family planning alongside parallel GoB systems for jurisdictions characterized as urban, rural or with high populations of ethnic minorities;^{Footnote 21}
• inadequate private sector regulation;^{Footnote 22}
• a system with a ratio of two nurses for every five doctors; and
• complex donor funding arrangement with numerous operating plans, which made it difficult to hold the GoB accountable.

Health Sector Effectiveness

In the health care sector, effectiveness, as measured by performance measurement framework (PMF) indicators, showed good progress. For example, the trend toward fewer girls being immunized was reversed while the trends toward childhood immunization and the use of contraceptives became positive. At the immediate outcome level, the number of births being attended by SBAs increased from 18 percent to 42 percent against a target of 50 percent. Canada contributed about a tenth of pooled fund and parallel project investments in the health SWAp for 2011-2016.

(See Annex F PMF Outcome Indicators and Performance Data)

The evaluation found that the sample projects in the Health portfolio met all or most of the stated outcomes.^{Footnote 23}

2.2.2 Education Sector

Education Sector Context and Challenges

The following systemic education challenges were identified in 2003 when the Asian Development Bank designed PEDP II:

• incorporation of excluded groups, including urban slum children, children with disabilities, and children in remote areas, including tribal (ethnic minority) children in the formal education system;
• high rates of repetition and dropping out and low rates for completion of primary education;
• poor quality of education characterized by poorly trained, low paid and poorly motivated teachers, high student-teacher ratios, double shift schools, shortage of classrooms, and lack of a system to monitor learning ; and
• governance issues related to a highly centralized education system, limited monitoring and evaluation and analytical capacity, and high transaction costs on the government due to a large number of uncoordinated donor partners.

Education Sector Effectiveness

In the education sector, programming effectiveness, as measured by PMF indicators, and based on data reported in the recent Bangladesh Primary Education Annual Sector Performance Report (BPEASPR) 2014 and the Household Income and Expenditure Survey (BHIES) 2010, revealed excellent progress on primary school completion rates, net enrolment rate, girl-boy ratio, mean scores of Grade 5 students, and teacher qualifications. Some of the targets set out in the PMF (e.g. ultimate outcome indicator – literacy rate or immediate outcomes indicators – single shift schools and student-teacher ratio) were unrealistic and were therefore not achieved. However, other results showed overall excellent progress toward their achievement.

The data show that progress against most program-level education outputs was positive and linked to the portfolio of projects’ results and in particular to PEDP II and BEP II.^{Footnote 24} The evaluation found that the four projects in the sample education portfolio met all or most of their outcomes. In particular, the more narrowly focused non-formal (only indirectly covered by the data presented in the table above) LEAD and BEHTRUWC projects made significant contributions to life skills development among poor adolescents and working children who had missed opportunities to participate in the formal education system. For example, some 21 percent of LEAD graduates were mainstreamed into formal education, with a follow-up survey of first and second cycle graduates showing a retention rate of 63 percent and 82 percent, respectively. See Annex G for project-level effectiveness substantiation notes.

(See Annex F PMF Outcome Indicators and Performance Data)

2.2.3 SEG Sector

SEG Sector Context and Challenges

In spite of Bangladesh’s positive economic record in recent years, a survey of relevant literature suggests that the country continues to face significant challenges to economic growth. These include the need to:

• address increasing risks to macroeconomic stability;
• revitalize trade liberalization efforts;
• improve the efficiency of the financial sector;
• improve infrastructure;
• improve land use management; and
• stimulate labour productivity through skills training.

SEG Sector Effectiveness

The 2011 SEG update to the Program PMF changed some of the indicators initially included in the CDPF. Many of these new indicators are limited in terms of their utility for the purposes of sector and program management. Although some indicators showed improving trends (e.g. overall employment, efficiency of investments and contribution of manufacturing to employment), these trends were especially difficult to directly attribute to xCIDA activities in this sector. In addition, no specific outcome-level indicators for two of the three priority areas: public-private partnerships (PPP) and skills for development were included. Coverage of these areas was limited at the output-level with the underlying assumption that it would logically contribute to outcomes.

(See Annex F PMF Outcome Indicators and Performance Data)

Given that the Program PMF did not provide a clear link to Program results, project level information on effectiveness is provided below. The evaluation found that five of the seven projects in the SEG sample portfolio met all or most of their outcomes.

The income generation projects within the sampled projects contributed to the 2003 CDPF objective of improved employment, improved economic opportunities, sustainable livelihoods for women and men through the promotion of entrepreneurship, access to financing and business development services.

The sophisticated results measurement system^{Footnote 25} used in the Katalyst II project indicated that the project enabled 2.358 million farmers to increase their incomes.

The BRAC/CFPR II projects lifted the members of some 600,000 households in poor rural areas out of extreme poverty and introduced them to sustainable livelihoods.

The SLUP project came close to its target coverage of 33,000 households, with results indicating that household incomes had increased by 42 percent for female-headed households and 33 percent for male-headed households, against a target 30 percent increase.

WRC’s Partnering for Microfinance Development and Impact included a $1.15 million component in Bangladesh that significantly exceeded its original targets and resulted in the development of a sustainable micro-lending institution led by WRC’s local partner. See Annex G for a summary of key results achieved in sample projects.

While the LEIC project addressed an important need in the private sector, several design flaws, including a short timeframe, compromised the ability of the local executing agency to make more than very limited progress toward achieving project objectives.

The PFM project SCOPE improved the functionality of the Office of the Comptroller and Auditor General (OCAG) by introducing gender and environmental auditing, facilitating the office-wide adoption of computers as working tools (for computer assisted audit techniques (CAATs) and piloting the public announcement of audit results. For a variety of reasons including overly ambitious objectives and a lack of enabling policy changes, the flagship PFM project, SPEMP has achieved only some of the outcomes initially established for it.

2.2.4 Governance Sector

Governance Sector Context and Challenges

As articulated by Lewis, two combining factors have contributed to a “chronic and persistent problem of instability at the heart of Bangladesh’s political institutions - class interests and patron-client networks”.^{Footnote 26} Governance challenges are also reflected by the World Bank Governance Indicators.^{Footnote 27} For example, on average, for the period 2008-2013, the percentile rank among all countries (ranges from 0 (lowest) to 100 (highest) rank) for Bangladesh is: 36 for Voice and Accountability, 25 for Government Effectiveness, 21 for Regulatory Quality and 24 for Rule of Law.

Governance Sector Effectiveness

A review of progress achieved against indicators at the Program level was impossible because, unlike the other sectors, there were no PMF indicators given that governance was recast as a cross-cutting theme. Instead, the World Bank World Wide Governance indicators were used to assess the contribution of projects results in the governance sector, as shown in table 8.

Individually, all five legacy projects listed in table 8 achieved diverse results that contributed to the overall objective of supporting good governance. The Parliamentary Strengthening and Media Support project was only partially successful, failing in its key objective of building capacity amongst parliamentarians because of the hiatus in parliamentary activity over the project period while the military-controlled interim government was in place. Nonetheless, it was able to make considerable headway in strengthening the strategic planning and public financial management capacities of the parliamentary secretariat.

The two PFM projects in the SEG portfolio were fully governance oriented, aimed at institutional and individual capacity building within government structures to improve governance and increase accountability. These two projects were credited by donors as having contributed to the functioning of other donor arrangements, including the SWAps in education and health sectors. (See Annex G, for project-level effectiveness substantiation notes).

The transformation of governance from a priority sector to a cross-cutting theme for xCIDA took place in late 2009. Projects were assessed as performing well in this area: 22 of 24 projects in the evaluation sample were judged to meet nearly all or most governance indicators.

The evaluation sample projects can be characterized as follows:

• augmenting or supporting the delivery of a public service, for example in health and education sectors SWAps;
• fostering government regulation and oversight;
• improving government support mechanisms;
• cooperating closely with various government ministries and local government bodies;
• strengthening the capacity of local NGO and civil society organizations to relate to government;
• working with communities of interest to affect government policy; or
• using a sub-sector wide approach to facilitate a single policy and program framework.

Examples for each sector are presented below.

xCIDA’s increasing investment in the SWAp was perceived as increasing its governance influence. This was strengthened by parallel “on-SWAp” investments such as the HRH project as well as projects outside the SWAp. Many of the MNCH initiative projects did not use the SWap.

In terms of governance, the education SWAp helped finance a significant expansion in the country’s primary education system and created a single policy framework for that system that encompasses both state and non-state actors. Both the LEAD and BEHTRUWC non-formal education projects cooperated with the formal system.

Nearly all of the SEG sector projects in the sample (in addition to the PFM projects) were found to have been linked to and influencing governance issues as part of their implementation approach to various government bodies: agricultural extension services, local government bodies like union councils and District Commissioners, local parliamentary representatives, or line Ministries.

2.2.5 Effectiveness in Relation to Cross-Cutting Themes:

Gender Equality

At the Program level and in terms of related policy dialogue, xCIDA succeeded in maintaining a strong record on gender equality over the evaluation period. The Program had a well-conceived and articulated gender strategy included in the 2009-2014 CDPF and a foundation of gender-specific programming in the past on which to build. However, during the implementation of the Program, the difficulties in obtaining GE expertise from xCIDA headquarters may have limited the achievement of gender outcomes. This issue is further discussed under efficiency. The absence of a dedicated GE specialist notwithstanding, Program staff in the field have been strong advocates for GE at decision making tables, with significant results. For example, Canadian representatives played a prominent role in the design of the third education sector SWAp and were credited by other donors with having raised the visibility and resources devoted to gender issues in this next phase of that program.

Over 80 percent of the sample projects met most or all of the criteria for gender equality. In only four projects (three in the SEG sector) of the sample of 24 was performance on GE found to be less than adequate, i.e. significant improvements and/or changes are required.

Environmental Sustainability

Few headquarters or field-based staff with environmental technical expertise were available to the Program. This was offset to some extent by the high proportion of projects implemented through multilateral organizations (e.g. World Bank, United Nations bodies), which adhered to their own environmental policies, standards and regulations that were comparable to Canadian standards. Thus, 11 of the 24 sample projects were judged as having met all environmental sustainability related performance criteria, and seven were judged as having met most of them. xCIDA was consistent in meeting its obligations for assessing planned projects against the Canadian Environmental Assessment Act requirements and involving headquarters expertise at least at the approval stage. In those areas where infrastructure was involved, for example, in the education sector, project managers respected requirements. Disaster risk reduction (DRR) was identified in the CDPF as an element to be taken into consideration. However, attention to DRR was ad hoc.

2.2.6 Effectiveness of “non-investment” efforts

Bangladesh had well-established donor coordination structures created under the Joint Coordination Strategy that was signed in 2008 by 18 donors and the GoB. The choice of program-based approach and SWAp delivery mechanisms to deliver much of the Program, together with the strength of the associated donor coordination structures, conformed to the principles of the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action. New programming using these modalities was set out in the Bangladesh Country Strategy and 2009 CDPF and discussed with the GoB and other donors using a sector and sub-sector working groups. At the Program level, these working groups were the primary vehicle through which xCIDA staff in the field engaged in high-level policy dialogue in its priority areas of programming. By contrast, at the project level, joint donor bodies (especially in education and health) offered opportunities to engage in policy dialogue on specific issues.

xCIDA had a consistent presence at the donor table^{Footnote 28}, made its views known and influenced key areas of large initiatives. For example, in the health sector, xCIDA used policy dialogue to promote health systems development, gender equality, SWAp management and financing and health system human resourcing issues. As a result, xCIDA expertise was seen as having a positive impact on the strategic direction and transparent management of that health SWAp.

In the education sector, skills brought to the project managing bodies by xCIDA representatives were reported by other multi and bilateral donors to have had a positive impact on gender equality and PFM/governance. In particular, xCIDA was credited with getting the World Bank involved in the PEDP II, strengthening its GE and performance measurement framework elements and supporting the design of PEDP III. xCIDA also contributed to the dialogue processes around the high priority issue of PFM through its involvement in SPEMP. Finally, the UNDP project implementation team was highly appreciative of the role that xCIDA staff played in supporting project renewal and revision processes in the Chittagong Hill Tracts Project.

The engagement on policy dialogue has been more responsive to project circumstances than to program level priorities. At the program level, beyond the CDPF and project needs, there was no overall plan that identified key issues on which xCIDA wished to engage on policy dialogue and how these issues were to be managed with the GoB and with other donors. In general, participation in multi-donor, project-level structures and in key working groups enabled good administrative and management coordination with other donors on a project and sectoral level.

The evaluation identified two instances where decisions made at xCIDA headquarters changed what had been negotiated in multi-donor engagement. One was the cancellation of a planned investment to develop MoHFW capacity for procurement of health commodities. The other was an investment in the Joint GoB-UN Maternal, Newborn and Child Health Initiative approved at headquarters. Deadlines to identify investment opportunities to support Canada’s pledges under the Muskoka Initiative were short. While the xCIDA office in Bangladesh provided input on the potential project for consideration, a greater participation in the decision-making process did not take place. Both instances were mentioned by xCIDA’s office in Bangladesh and HPNSDP donor partners as awkward in the context of plans that had already been developed and agreed among donors. Nevertheless, there was no evidence of a negative impact on relationships.

2.3 Efficiency

The evaluation considered efficiency in terms of the:

• extent to which resources were appropriate to the results achieved, considering context;
• suitability of the management approach to achieve intended results; and
• extent to which management tools were appropriately applied and used for decision making.

Overall Program Efficiency

The 2009 Country Strategy for Bangladesh refers specifically to the decision to move toward “larger, fewer projects mainly developed in concert with other donors”. This came at the same time that the Bangladesh Program was beginning to be decentralized to the field. The following table illustrates how staffing level and program disbursements evolved over the 2008-2013 evaluation period. It is important to note that the staff number includes only staff dedicated to the bilateral Country Program and excludes individuals from Partnership and Multilateral Branches operating from headquarters (HQ). These data show that disbursements remained almost constant until an increase occurred in 2012-13. The number of professional staff was slightly reduced in 2009-10 and again in HQ during the last year. Comparing the first and last years of the period, this is a change of 42 percent in the ratio. However, it is not possible to comment on the effect of this change on efficiency or on its effect on the quality of outputs since the change was concentrated in the final year of the evaluation period.

Sources: Staff figures from Program records; historical disbursements from Chief Financial Officer Branch.

There was an assumption that a program consisting of “fewer, larger” projects is more efficient and easier to manage. However, the evaluation did not find evidence to support this assumption. Canada participated in multi-donor project forums in order to ensure that attention is paid to its priorities. While xCIDA staff found SWAps to be effective delivery models, they also found them to be highly time consuming. For example, the second education sector SWAp required the full-time effort of a dedicated field officer, particularly during the project’s first year when Canada was the Chair of the donor consortium.

Increasingly, new programming over the evaluation period used responsive mechanisms. That is, the Program responded to initiatives proposed and designed by development partners such as the UNDP, the Asian Development Bank or the World Bank, which had projects at the ready. While this offered some efficiency in terms of xCIDA’s approval processes and disbursements, this evaluation did not find evidence that it translated into better achievement of results. In addition, significant effort was often needed to ensure that xCIDA’s cross-cutting themes were adequately represented in working groups.

Additional work was also required to “translate” the products of collective program monitoring and evaluation systems into products that were compatible with xCIDA’s own systems. This was corroborated by the project completion review mission for PEDP II, which reported that “overall, PEPD II reduced transaction costs for the GoB, but transaction cost for [Development Partners] remained high, due to differences in procedures among agencies.” The evaluation noted that the participation in the governance of the SWAp projects stretched xCIDA’s limited field resources. There were numerous sector and sub-sector working groups and it was not possible to attend all of them.

There were also efficiency issues related to delays in approval and contracting processes. For example, Adolescent Reproductive Health took three years to start, and Human Resources for Health experienced delays in start-up and its midterm evaluation due to slow contracting processes.

The Program was able to make only limited use of HQ-based professional services support in gender equality, the environment and governance, except for the review and sign-off of project and program planning documents. The reorganization of these specialist functions at HQ during the evaluation period, the reduced number of professionals working in these areas and the heavy demands on their time meant that they were not in a position to consistently provide project level advice.

Access was also severely limited to cross-cutting themes specialists in the field. A gender equality specialist position was established in the Program Support Unit (PSU) but this position has experienced a high rate of turnover and problems competing with positions offering higher wages and better job security, usually with other donors. Consequently the position was vacant from 2011/12 to the end of the period covered by the evaluation. In the PSU, there was no position for an environmental sustainability specialist or a governance specialist despite the prominence of governance elements within the Program. However, some xCIDA professional staff in Dhaka had specialized backgrounds in PFM and governance, which they have been able to bring to the Program, but this has been the result of circumstances rather than systematic planning.

Staffing levels were planned based on expected bilateral programming activities. In practice, however, field-based staff spent a significant amount of time supporting partnership and multilateral programming in addition to bilateral programming. One example is the Micronutrient Initiative, which had an office at the Bangladesh PSU during the evaluation period and for which the xCIDA office in Bangladesh coordinated meetings and provided project support.

Canada’s partners in Bangladesh, such as the GoB or other donors, did not distinguish between the different channels of Canadian programming (i.e. bilateral versus partnership or multilateral) therefore the Bangladesh Program had little choice but to provide support for all Canadian funded activities, regardless of accountabilities at HQ.

2.3.2 Efficiency of Sample Projects

The evaluation found that the majority of projects were efficient, i.e. 20 of the 24 sample projects met all or most expectations against the indicators set for the efficiency criteria. For more details, see Annex E Project Rating Summary. Most of the Program’s SWAps and PBAs performed satisfactorily in their use of resources and in coordination with other donors. These mechanisms offer a high level of confidence in the efficient use of resources given the extent of collective oversight they usually require.

Performance Management

Generally, there were limitations in the use of the Program PMF. At the ultimate and intermediate outcome levels, determining attribution was a challenge. For example, ultimate outcome indicators reflected changes in the economy and society that were not specifically attributable to Canadian development funding. At that level, Canadian investments can only be considered to have contributed to the achievement of outcomes.

This evaluation used the Program PMF at the intermediate level to measure the effectiveness of the programs’ sectors with mixed results. The link between the measured outcomes at the Program level with the underlying contributing projects was clearer for the health and education sectors than it was for the SEG sector.

In the education and health sectors, Canada strategically contributed to change in the overall systems, rather than to a specific part of these systems, through its engagement in major SWAps and the high degree of complementarity of other programming to these main SWAps.

As previously noted, SEG sector PMF provided indicators for outcomes that related only to the Program results statements indirectly, for example, “contribution of manufacturing to employment”. Thus, the means to measure Canada’s contribution to results in that sector were poorly defined, necessitating the use of project level PMF information.

Results based management (RBM) tools were extensively used for reporting at the project level. Overall, projects in the education, governance and SEG sectors followed the practices set out in xCIDA’s policies on the use of RBM, with varying degrees of success, while projects in the health sector had more performance management issues.

In the health sector several projects employed results and indicators frameworks that were found to be unnecessarily complex for the purposes of management and reporting. In addition, in the SWAps and the projects delivered by UN agencies, the logic models and PMFs developed by xCIDA for approval purposes were not used by the implementing agencies. For example, the HPNSDPproject results framework was clear, its performance indicators were measurable and all donors had both collaborated on its development and agreed to use it to guide their contributions. However, xCIDA’s PMF for this project was far more complicated, including 12 immediate outcome and 38 output indicators. Few indicators were reported on in the Management Summary Reports. The value of collecting and tracking the data was unclear. The BIRCHProject Approval Document set out separate outcome and output results indicators, at the urging of headquarters-based RBM specialists, that were also were not reported on. IDRF’s Maternal and Reproductive Health was the sole project in the health sector that reported using the RBM approach to resolve a number of project challenges arising from design limitations.

In 2007-08, the GoB started to take ownership of the education SWAp, PEDP II, while the Ministry of Primary and Mass Education adopted the results framework, which it applied to the whole sector. BEP II had a strong monitoring and evaluation system, which has been consistently acknowledged in donor reviews. Since BEP II and PEDP II encompassed the major portion of the education sector, the sector as a whole could be considered to have sound systems in place to monitor both quantitative and qualitative performance indicators.

Most SEG projects performed well with respect to their project-level management strategies and use of management tools, with the exception of the SPEMP project, which had difficulties in its use of management tools as reported in the Bangladesh SPEMP Annual Independent Review (2013). Katalyst IIwas noteworthy for having employed an unusually detailed system of oversight based on the Donor Committee for Enterprise Development (DCED) Standard for Results Measurement, which was acknowledged to have been useful but costly in the 2013 report on Added Value of Using the DCED Standard in Bangladesh from a Donor’s Perspective.

Finally, in governance, the Bangladesh Environmental Institutional Strengthening Project’s (BEISP’s) technical committee used the project’s logic model and PMF effectively to track results. The Parliamentary Strengthening and Media Support Project, implemented by the World Bank Institute (WBI), shifted from an output to a result based project management system, a move which WBI acknowledged facilitated a more robust tracking of results.

2.4 Sustainability

The evaluation assessed the extent to which the Program increased the probability of continued benefits from the portfolio of xCIDA’s development investments.

Out of the 24 sample projects, ten fell short of achieving sustainable results. The evaluation identified a range of reasons underlying this situation. One key concern, evident across all sectors of the Program, is the lack of engagement on the part of the GoB to make the significant changes (reforms) in established systems, structures and processes that were required to consolidate progress achieved - notably with regard to staffing, organizational structure and legislation. This compounds issues of limited initial capacity and budgetary resources needed to sustain newly introduced systems and infrastructure. Individual enthusiasm and willingness on the part of specific partners to work under the conditions needed to make change permanent and successful has not been enough to guarantee sustainability in all cases nor has the continual representation by xCIDA and other donors in policy dialogue forums.

Where the reform process seems to work best was in situations where vested interests were minimal, policy dialogue was constructive, the sense of local ownership was high and the task was relatively straightforward, as is the case with PEDP II.

2.4.1 Sustainability of Health Sector Projects

In the health sector, well over half of bilateral disbursements were made either to SWAps or to parallel projects to the SWAps. By contributing broadly to the public health system, the Program expected that sector results would be sustainable, i.e. embedded in the system. However, many of these initiatives were pilots whose scaling-up toward sustainability was questionable: GoB policy and commitments had not keep up with recommendations from pilot initiatives. Community clinics and Community Skilled Birth Attendants (CSBAs) were important exceptions to this as they represented a financial commitment made to extend services. However, CSBAs only delivered 0.2 percent of reported births in 2013, causing many stakeholders to urge a rethinking of the strategy. The GoB invested heavily in community clinics, which is encouraging, but this investment was through a project within HPNSDP that was supported through the GoB’s development fund rather than the revenue budget. The latter funding route would be more reliable for long-term funding and less prone to be influenced by current (and changing) development priorities, and thus more sustainable.

2.4.2 Sustainability of Education Sector Projects

The smooth evolution from the pooled-funded PEDP IIto direct budgetary support in PEDP III was a clear indication of the GoB’s commitment to sustainable education financing. Other donors and GoB representatives acknowledged the prominent role xCIDA played in the design of this transition, based on the experience with PEDP II. PEDP II has successfully institutionalized a Primary Completion Examination system, implemented a new teacher recruitment model, reformed teacher training, and built a robust monitoring and evaluation system with key performance indicators, the only Ministry to do so.

BRAC education programs (BEP) were sustainable in that they are simple and inexpensive enough to be repeated in the same location, replicated elsewhere and adapted to different contexts. One example is the BRAC pre-primary classes, which operated in government schools. Their graduates performed significantly better than regular primary students. As a result, the GoB is starting 40,000 pre-primary centres across the country. BEP II has developed a large network of teachers, trainers, monitors and NGOs, while community capacity is built to support and monitor classes. BRAC pulls out of communities when their students are able to access government schools. However, pre-primary centres were not self-sustaining in terms of funding. It is ultimately the responsibility of government to provide quality primary education for all, and until this is the case it is reasonable for donors to subsidize BRAC programs as well as the formal sector.

What was needed was a discussion between the NGOs, the government and the development partners on strategic cooperation and developing a strategy for government to assume full responsibility. The BEP IImid-term review recommended the establishment of formal structures for dialogue between BRAC and its key government partners (including ministries of Primary and Mass Education, Women and Children, Youth and Sports, the National Academy of Primary Education, National Curriculum and Textbook Board and others) to promote information sharing as well as strategic cooperation on matters of sustainability, a possible model for other sector SWAps. Already PEDP III encompasses several areas of schooling initially started by BRAC. They include pre-primary education in government primary schools, second-chance education, outreach to children with special needs, and enhancing the role of school management committees.

2.4.3 Sustainability of SEG Sector Projects

SEG sector projects had a divided record with regard to sustainability. The two projects that present the greatest challenge to the sustainability of results were PMF-support projects undertaken directly with the GoB: SPEMP and SCOPE. Both were hindered in achieving a sustainability threshold because of the GoB’s reluctance or inability to enact the reforms required to allow the operational changes introduced by them to take full root. For example, in the case of SCOPE, the failure by the government to revise the Audit Act hampered the project’s ability to strengthen Parliament’s financial oversight committees. These were long-term institutional change projects that demanded a long-term commitment to have a reasonable chance of success.

The rural economic development projects had far better results on sustainability. All four, including the partnership project, WRC Partnering for Microfinance, were rated as having good prospects for continuation and long-term benefits. Katalyst II is continuing into a third phase (albeit without xCIDA funding) in which further investment in market development will build on existing behaviour change identified in the second phase completion report. The BRAC CFPR II graduation model has been picked up by other donors (DFID, Ford Foundation) and is being applied elsewhere by them as well as by BRAC itself. Although no further xCIDA funding was allocated to SLUPand WRC Partnering for Microfinance, the local partners in both cases continued to work with funding from other sources. The WRC project resulted in the registration of a local microfinance institution that will be fully self-sustaining.

2.4.4 Sustainability of Governance Sector Projects

Only two of the five sampled projects have fully met the sustainability criteria. While the Chittagong Hill Tracts (CHT) Foundation has been turning some of the education and health facilities it has built over to the government, this process was highly dependent on the government’s fiscal capacity to maintain them. More importantly, the sustainability of the whole project is highly dependent upon a positive conclusion to the Chittagong Hill Tracts Peace Accord amendment process. The Chittagong Hill Tracts project presented a particular dilemma for donors considering further funding. The CHT project objective was not intended to be sustainable in itself. Yet, it was unlikely that the GoB will assume responsibility for all the capacities that the project has built over the last ten years, given the political turmoil currently prevailing in Bangladesh.

Continuation of linkages between Inter Pares and its Bangladeshi NGO partners in advocating for the rights of the poor is being put at risk by recent changes in xCIDA’s NGO funding modality, which has been moving away from NGO core funding support (e.g. to Inter Pares) to defined project support. While the Fair Elections and Institutional Reformproject was able to strengthen the Bangladesh Electoral Commission and use a consortium of NGOs (the Election Working Group) to update voters’ lists and establish community-based committees to monitor the 2008 national elections, both the United States and the European Union (EU) refused to send observers to the 2014 elections, citing the non-inclusiveness of the polls. On the other hand, the evaluation found that regulatory changes and training materials developed for BEISP remained in use. The organizational structures, new positions and new offices created during the project also remained in place.

3. Conclusions, Lessons and Recommendations

3.1 Conclusions

The evidence reviewed in this evaluation depicts a program that successfully maintained, over the period covered, programming responsive to the evolving needs of Bangladesh during a period with a number of organizational and thematic priority changes within xCIDA that affected the nature and direction of its program. It has done so using delivery mechanisms (SWAps in particular) that have given Canada a prominent place in discussions concerning its social development priorities - health, education and gender equality - that have integrated well with the initiatives of other donors and responded directly to the Government of Bangladesh’s priorities. As a member of and contributor to several major donor consortia, Canada can take credit for having contributed to many of the changes that have occurred over the evaluation period. Canadian development programming in Bangladesh has been consistent over the long run in terms of both priority sectors and the approach it has taken. This has given added strength and depth to the results that are evident over the period of this evaluation.

3.1.1 Relevance

The analysis of the findings demonstrates that both at the sector and project levels, the Program responded very well to defined needs of Bangladesh. Similarly, the Program was highly aligned with the priorities set out in xCIDA strategies, planning documents and corporate policies. It was able to quickly respond to the Muskoka Initiative given the Program’s existing focus on MNCH. The Program also adapted well to changing needs, for example, by increasing its attention to systemic issues, such as health sector human resources and policy dialogue, to address the centralized and fragmented governance of the health system, as these underlied many of the challenges faced by the Program.

The bilateral project portfolio fit well with the activities of other donors and sustained a high degree of internal coherence within and between priority sectors. There were several examples of projects in one sector that linked to other sector initiatives, and projects within sectors were highly complementary.

Partnership projects were found to be somewhat less aligned to national policies/guidelines than bilateral ones. While it was understood that partnership projects have a different mandate than bilateral programming and separate project selection mechanisms (call for proposals), the proposal review process and/or project proponents sometimes failed to identify irrelevant components within proposals.
The Bangladesh Program portfolio had a pattern of useful internal coherence, with strong linkages between projects within sectors of concentration and the management of governance as a cross-cutting theme.

These findings lead the evaluation team to conclude that the Bangladesh Program was clearly relevant to needs and priorities of Bangladesh and that its evolving strategies and project portfolio were relevant to meeting the intended results.

3.1.2 Effectiveness

The evaluation assessed the extent to which the Bangladesh Program contributed to and/or achieved the expected results, including the integration of cross-cutting themes.

The evidence showed that Canadian health sector programming contributed to measurable improvements at the sector and project level such as immunization rates and the number of births being attended by skilled birth attendants. Parallel health projects have complemented core sector investments, reinforcing specific elements of particular importance. Health projects across the board had a good record of achieving their intended results.

Education sector programming focused on primary education. It contributed to improvements in governance and management of the formal education system as well as improved learning outcomes. The two major consortia to which xCIDA contributed have achieved positive results. The Program successfully supported complementary programming to develop life skills of at-risk adolescents and improve access for children with disabilities, reinforcing its image as an active player in promoting continuous and so-called second chance education. Thus, Program support to both formal and non-formal sectors was effective.

The SEG sector (introduced with the 2009 CDPF) included income enhancement projects for the ultra-poor and poor that were successful in meeting their outcomes, suggesting a good record in achieving results through activities formerly categorized as private sector development. PFM projects - moved from the governance to the SEG portfolio in the 2009 CDPF as part of the “enabling environment” - had a mixed record of results achievement. PFM was an area of critical importance, being fully governance oriented. The results that were achieved in spite of the challenges resulting from the overall state of the governance environment in Bangladesh were important. They contributed to the functioning of other donor arrangements, including the SWAps in the education and health sectors.

Governance was first a sector with a portfolio of projects (2003-2008 CDPF) and then repositioned as a cross-cutting theme that became part of almost all projects, either as the main theme or as a key component of project activities. Governance played a central role across the Program. Yet, because these investments addressed overarching issues of governance rather than a particular service delivery sector, they faced more of the challenges arising from the unpredictable Bangladesh political context and as a result varied in their effect. All the governance projects in the survey achieved specific results, some of which had identifiable effect, but for some, political instability and the challenges of an entrenched system of public administration have prevented the original outcomes from being fully realized.

In terms of gender equality, the nature of the Bangladesh Program, with a high proportion of multi-donor initiatives, has meant that while staff advocated for the incorporation of gender equality it was not always prominent in investments. This may also be linked to the very limited access to sector specialists from xCIDA headquarters.

In terms of environmental sustainability, many project plans committed to actively seeking opportunities to address environmental concerns but these were more commonly incidental to project activities than deliberately planned. xCIDA was consistent in meeting its obligations for assessing planned projects against CEAA requirements. In those areas where infrastructure was involved, projects fully respected screening requirements.

Program outcomes were also achieved through the Program’s “non-investment” efforts. The evidence showed that the involvement of xCIDA staff in the field in high-level policy dialogue in priority areas of programming on specific issues was seen by GoB and other donors as having a positive impact on project direction and implementation. The commitment to ongoing participation in large multi-donor initiatives and key donor working groups has also allowed the Bangladesh Program to have a consistent voice in policy dialogue with the GoB.

3.1.3 Efficiency

The evaluation examined the utilization of resources, the management approach and the decision making tools to achieve intended results. The analysis of the findings on efficiency are multi-faceted and do not lend themselves easily to provide an overall conclusion on this criteria.

During the period covered by the evaluation, programming was delivered through fewer and larger projects, and bilateral program management was decentralized from headquarters to the field. The use of resources by the Program, as measured by disbursements against staffing levels, revealed a very small increase during the first four years followed by a drastic increase (35 percent) in the last year. Thus, these data do not contribute to confirming the assumption that a program consisting of “fewer, larger” projects was automatically more efficient and easier to manage.

Large multi-donor initiatives like SWAps offered efficiencies for the GoB but consumed significant Program staff time. For example, Canada participated in multi-donor project forums in order to ensure that attention was paid to its priorities. There were numerous sector and sub-sector working groups, and it was not possible to send staff to attend all of them given xCIDA’s limited field resources. Although there were transactional efficiencies, the evaluation team could not find clear evidence to state that the move to fewer, larger projects significantly reduced the amount of time required to deliver the Program. Field staff were also at times required to respond to unanticipated demands to support partnership and multilateral programming.

In terms of specialized human resources, the evaluation findings indicate that the Program was able to make only limited use of headquarters and field based professional services support in gender equality, environment and governance.

RBM tools were extensively used at the project and program levels. However, their value to this Program was questionable as they duplicated the multi-donor systems that had been agreed to. At the program level, the evaluation identified a number of weaknesses in the relevance and suitability of indicators.

3.1.4 Sustainability

Sustainability was a significant concern for the Bangladesh Program, with under half the projects in the sample falling short of the desired level when assessed by the evaluation team. Many of the structural elements, regulations and laws, processes and allocation of responsibilities which would have been required for projects to be sustainable were often under the purview of the central agencies of the GoB and even Parliament itself.

Prospects for sustainability in PFM and public accountability have been compromised by uncertainty over future donor funding to support related programs and projects. Sustainability received limited attention even when raised in joint GoB-donor forums. Ongoing policy dialogue efforts by donors have been more effective in improving implementation than in changing policy.

3.2 Discussion

Two major elements emerge in this evaluation: Governance and SWAps.

Governance

The enduring challenges of governance in Bangladesh supports the evaluation’s conclusion that governance was key to achieving and preserving results in all priority sectors. In the health sector, efforts to strengthen health systems faced challenges as a result of the governance environment in Bangladesh and, in particular, decentralization, privatization and regulation. While the investments in MNCH in Bangladesh have led to steady reductions in maternal and child mortality, there is a need to ensure that this progress does not overshadow work to address health systems strengthening. The focus in health sector support needs to be on marginalized populations that are less likely to realize the benefits of broader development. Thus, significant opportunities for future sectoral investment lie with improving services to marginalized populations and those who are less likely to realize the benefits of broader development in Bangladesh, while continuing to support general improvement of the health system.

In the education sector, Canada has been successful in contributing to the governance and management of the formal education system, while contributing to NGOs implementing efficient, effective - yet, not self-sustaining in terms of funding - initiatives that parallel the formal system with support to hard-to-reach children, and in efforts that mainstream these children into the formal system. There are opportunities to foster closer linkages between these systems.

In the SEG sector, the portfolio of projects, and in particular the PFM projects, had positive pervasive effects through policy dialogue, influencing governance at many levels such as SWAps’ management, Parliament (financial oversight committee) and various government bodies, agricultural extension services, local government bodies and District Commissioners, local parliamentary representatives, or line Ministries. These efforts enabled systemic and most likely sustainable changes.

Building state institutions and supporting governance reform processes is complex, long-term and demands local ownership. The evaluation noted that it was difficult to obtain a full commitment by stakeholders that also had weak institutional capacity and uncertain resources (budget and personnel allocation). The evaluation is positive in terms of the Program’s efforts in implementing strategies, particularly for SWAps that addressed such questions as the balance between sector wide reform and institutional capacity building and the role of NGOs as partners in service delivery.

3.2.2 SWAps

In theory, SWAps involving multiple donors can be successful despite the challenges involved provided there is good donor harmonization, strong government ownership and a strong commitment from donors to make the SWAp work. However, the evidence suggests that these and similar large, multi-donor projects and programs are not necessarily more efficient in terms of the demand they place on xCIDA Program staff. To be effective in sector and sub-sector working groups and in donor tables requires an optimum number of qualified resources otherwise these groups must be populated with staff chosen strategically from the current limited roster of resources using the complementarity of skills and sector technical knowledge, in particular with respect to PFM, governance, and gender equality.

There were various arguments for a balance between larger, multi-donor mechanisms and more traditional projects. On the one hand, the utilization of large SWAps and/or responsive mechanisms through international agencies, though increasing transactional efficiency may also reduce the visibility of Canadian contributions and reduce Canada’s ability to include cross-cutting themes or Canadian priorities. Smaller bilateral projects, on the other hand, offer opportunities for Program staff to get into the field and learn firsthand about context, to bring Canadian expertise to bear on local problems, and to experiment with pilot projects or innovations that can be scaled up by the GoB.

3.3 Lessons

In delivering its programming, the Bangladesh Program experienced significant governance issues as a result of systemic governance challenges in the country. In some instances, government partners were unable to make the required significant reforms in established systems, structures and processes to consolidate progress achieved. The cumulated experience suggests that two important factors can significantly influence reform process: vested interests by the stakeholders for, or against reforms, and the scope/span of a development intervention.

• Greater success (in terms of effectiveness, efficiency and sustainability) is achieved in situations where a) vested interests in maintaining the status quo are minimal, enabling a constructive policy dialogue and a collective sense of local ownership and b) the logic of the development intervention is straightforward.
• Lesser success is achieved in situations where a) vested interests in maintaining the status quo are making policy dialogue difficult and local ownership is not equally shared among stakeholders; and b) the logic of the development intervention is multi-faceted.

In delivering its programming, the Bangladesh Program offered a coherent portfolio with strong reinforcing linkages among projects, in particular with governance projects or governance as a cross-cutting theme. For example, the PFM project contributed to education and health SWAps management while the Parliamentary Strengthening and Media Support Projectstrengthened the role of parliament in overseeing PFM. The cumulative experience suggests that country development programs adopting a mix of concurrent and reinforcing governance projects enable catalytic and positively pervasive changes that contribute to greater effectiveness.

3.4 Recommendations

1. Maintain a strong presence in the sectors in which Global Affairs Canada is presently involved but adjust some elements of the Program for greater effectiveness.
   • In Health, consider an emphasis on improving health systems governance and provision of health services to remote and disadvantaged communities.
   • In Education, consider an emphasis on governance of the education system and closer linkages between formal and non-formal education systems.
   • In SEG, consider working to ensure a continuity of engagement both in program funding and introduction of systemic change through policy dialogue.
2. In the Bangladesh context, where weak governance has been a central determinant of the success of programming in all sectors, strengthen the focus on governance elements of all Global Affairs Canada programming by putting into place appropriate agreements with relevant central ministries to ensure the necessary budgets and personnel allocation are or will be in place for sustainability.
3. Staff the Program to meet the Program’s needs. A priority should be placed on engaging field advisors for gender equality, environment and governance. Canada-based staff in the field should have an appropriate mix of technical and analytical skills in priority areas.
4. Improve the utility and use of the Program’s RBM tools. Improve the quality of program level indicators and targets, and, in multi-donor programs and projects, link Global Affairs Canada systems directly with the RBM tools collectively agreed on to avoid wasting time through duplication or unnecessary data collection.
5. Improve coordination with partnership and multilateral programs with a view to better aligning not just programming but also demands on field resources. In their own planning efforts, these programs should be anticipating and advising on their expectations regarding field support.