Executive Summary – Indonesia Country Program Evaluation

Acknowledgments

We would like to thank the independent team which undertook data gathering, analysis, and technical reporting for this evaluation exercise. It included Michael Miner, Jerry Rogers and Dr. Melinda MacDonald of the Results Based Management Group; Simon Latraverse of Econotec Incorporated; Marzuki Darusman, Lanna Lubis, Ario Putra and Tata Mustasya of Mazars Indonesia; and Gender Advisor Lily Purda.

We would also thank all who have contributed to this evaluation, including Indonesia Country Program staff at headquarters and in the field who provided the documentation and participated in interviews, and the Eastern Indonesia Knowledge Center (BAKTI) who provided coordination and logistical arrangements to optimize the evaluation team’s time in-country.

Many other stakeholders made themselves available for meetings, interviews and project site visits, including Government of Indonesia officials, donor representatives, Indonesian and Canadian implementing partners, numerous civil society organisations, and individual consultants who were generous with their time and provided the evaluation team with valuable insights into their programs and projects.

From DFATD’s Development Evaluation Division, Michèle Samné managed the process and made a major contribution to the preparation of this synthesis report. Denis Marcheterre had overall responsibility for the evaluation.

James Melanson
Head of Development Evaluation

Executive Summary

This evaluation assesses the results and management performance of the Indonesia country program over the eight-year period from 2005-2006 to 2012-2013. Data was gathered towards the end of 2013, from a representative sample of projects and related activities. Findings, conclusions, and recommendations are based on this evidence.

Since the 1970s, by invitation of the Government of Indonesia, Canada’s development assistance program to Indonesia has been focussed geographically on the island of Sulawesi in the eastern part of the archipelago. Sulawesi is Indonesia’s third most populous island, and one of its poorest. Since the country is so large and populous and the Canadian investment modest, it was agreed to focus on six of the poorer provinces on that island.

A fundamental aspect of Indonesia’s democracy, which came into being in 1999, is decentralization of government powers to the local level. Canada has supported the decentralization from the outset, and helped to build the capacity at the local level to design and deliver government services and to enable the private sector to grow.

CIDA implemented 80 projects in Indonesia and disbursed a total of $204.5 million during the 2005-2013 period. Most projects (67 out of 80) used the ‘responsive’ delivery model, accounting for 72% of total disbursements.^{Footnote 1} Under the Country Development Programming Framework (CDPF) for 2004-2009, the three areas of concentration were governance (48% or $78 million), the environment (21% or $34.6 million), and private sector development (12% or $18.8 million). Under the next CDPF for 2009-2014, the two areas of focus were economic development and natural resource management in targeted areas of Sulawesi. The CDPF of 2009-2014 was suspended in 2012, when a new Country Development Strategy was developed, de-emphasising private sector development and moving instead to addressing skills for employment. In addition, governance became crosscutting in 2009, rather than a program focus. The latest Country Development Strategy for 2013-2018 reintroduces the new corporate theme of “Advancing Democracy”, which includes governance.

Through this period, the Program was aligned with Canada’s aid priorities and Indonesia’s development needs. Synergies between environment programming with a focus on sustainable natural resource management and private sector development with strong environmental sustainability characteristics helped to manage changes in strategy and approach over the years. The Program’s management was generally efficient and supported aid effectiveness principles. Partners have underlined the relevance and usefulness of the Program’s geographic focus on Sulawesi for building strong relationships at the sub-national level and developing approaches that could be replicated in other parts of the country. The Program has been willing to take risks by supporting sensitive initiatives such as the Corruption Eradication Commission, and overcoming resistance to gender equality.

Substantial evidence points to the achievement of project-level outcomes, including successful examples of policy dialogue such as helping to develop the local government’s peatland strategy that influenced policy at national and even international levels, and introducing the concept of community-driven investments in “green” sub-projects. Through its portfolio of projects, the Program has helped to build the capacity of local governments, universities and community leaders, and improve the government’s transparency and accountability to prevent corruption at different government levels. Notable gender equality outcomes were achieved in policy dialogue, advocacy and influence. Some projects contributed to enhancing the capacity of a significant number of small and medium enterprises (SMEs) whereas others assisted in streamlining business regulations at the sub-national level to improve services to them.

Program-level reporting on the achievement of outcomes, however, could have been stronger. Shortcomings in CIDA’s Program-level reporting regime have limited the Program’s ability to report on its Program-level results, placing emphasis on project level reporting instead. Another corporate management issue concerned “surge capacity” - needed in cases of emergency assistance in disaster prone countries. The increased workload in the field at these times needs to be covered by more operations and maintenance funds so as to not slow down implementation of regular programming.

As a lower middle-income country, Indonesia’s needs and reality are evolving quickly. Decentralization on the scale and at the pace that took place in Indonesia does not automatically lead to sustainable and equitable growth. Building local government capacity where needs are greatest, combined with support for strengthening civil society organizations and for empowering communities to influence government, remains a relevant program strategy. Focus will have to be maintained on piloting reforms and replicating or scaling up successes. This requires a high level of policy dialogue and government buy-in.

With significant regional disparities in Indonesia, low capacity in the regions, unclear government regulations and lack of coordination among different levels of governments, it is challenging to ensure the sustainability of CIDA’s program. The Program worked best when it included buy-in from the highest level of authority in government (minister at national level or governor, mayor or regent at sub-national levels); focused on multiple levels of government (national, provincial and district); included national and local civil society partners in its initiatives; and when successful initiatives piloted in Sulawesi were replicated and scaled-up nationwide. Project level sustainability was enhanced by extended implementation timeframes, strong ownership and institutional capacity, and adequate funding resources of local implementing partners. There is additional scope for replication of initiatives nationally and in other provinces, and increased coordination and sharing of information among projects working to achieve similar results.

Anti-corruption, accountability and transparency are important elements of integrating governance in all projects given the context of Indonesia’s decentralization process. Indonesia’s national Corruption Eradication Commission is a success story worth communicating to other DFATD programs to encourage sharing of best practices in this field.

In private sector development, risk management appears to have been weaker than in other sectors in the face of natural disasters and market price variations which threatened results obtained by small and medium enterprises (SMEs) that depended on natural resources. Community-based approaches to small and medium enterprise (SME) development were more powerful in mobilizing communities through the empowerment of women and involvement of local governments than approaches that used foreign expertise to promote SME development.

Environmental sustainability was prominent as a cross-cutting theme in the two CDPFs covering the evaluation period. It was integrated into all programming sectors and was relevant to the sustainable management of natural resources. Environment projects were challenged by insufficient timeframes, the lack of reliable and sustainable market linkages for SMEs that are dependent on the natural resource base for their livelihoods, and recurring natural disasters.

Improved integration of the crosscutting themes of gender equality and environmental sustainability into the Program’s sectors of focus remains key. Enhanced tracking of the environment outcomes of private sector development investments would help enhance environmental sustainability as well.

The Program had gender equality (GE) strategies and some gender-specific projects. However, the Program’s gender mainstreaming commitment was less evident in the 2009-2014 programming period that emphasized economic empowerment, than in the earlier 2004-2009 period where the focus on governance and human rights helped to provide a stronger enabling environment for GE results. The governance projects showed a satisfactory understanding of GE in terms of increasing gender understanding within different government levels and civil society, but there were significant weaknesses in the GE performance of private sector development projects. The workload is large for the Program’s part-time Jakarta-based GE advisor especially in the context of Indonesia’s decentralization given that gender capacity is generally weaker at the sub-national levels

Evaluation Recommendations

1. Gender Equality

Canada has earned a justified reputation for constructive long term engagement on gender equality through its development assistance program in Indonesia. It should build on this engagement, and in particular ensure that Sustainable Economic Growth programming is undertaken with sufficiently detailed gender analysis, target setting, and performance measurement.

2. Environmental Sustainability

Given its geographic circumstances and demographic pressures, appropriate environmental stewardship will be particularly important for sustainable long term growth and poverty reduction in Indonesia. Canada and other donors continue to pay considerable attention to this theme. The Program could improve impact by linking their individual efforts, taking an integrated programmatic approach with appropriate policy dialogue objectives, and ensuring that sustainable economic growth initiatives have explicit environmental sustainability objectives and sufficient resources and timeframes to achieve them.

3. Transparency, Accountability, Enabling Environment for Business

Through its geographic focus in Sulawesi, and emphasis on building capacity at provincial and district levels, the Program has achieved improvements in public sector management and civil society engagement that have been recognised and replicated nationally. In its new emphasis on Sustainable Economic Growth (SEG), a key challenge will be to improve the enabling environment for business, including through transparent and efficient regulation. The Program should leverage its past success in governance, by incorporating specific transparency, accountability, and anti-corruption objectives in its SEG programming.

4. Performance Management

There are examples of very good performance management practices in some projects, and a Program level Performance Measurement Framework (PMF) was well articulated although not regularly tracked and reported on. There is scope for moving further in these areas. Specifically, the Program should:

(a) At the project level, make more systematic use of baselines, project monitoring and evaluation over the next CDPF period, setting targets for frequency, and allocating sufficient resources to achieve them;

(b) At the program level, calibrate a PMF to the next CDPF, with indicators that can be tracked and reported on to show sector and national level progress.

(c) Increase the use of already well-articulated risk management frameworks as management tools. Set specific targets and measurable indicators for policy dialogue activities, and report on them.

(d) As part of the longer term mutual accountability agenda, seek opportunities to develop partner capacity and involve local experts in monitoring and evaluation.

This is a Synthesis Report of the Indonesia Country Program Evaluation (2005-2013), which included a field data-gathering mission to Indonesia in November 2013. Indonesia is one of 25 countries of focus that receive Canadian development assistance from the Department of Foreign Affairs, Trade and Development^{Footnote 2}.

1. Evaluation Process

1.1 Evaluation Objectives and Methodology

The evaluation had three objectives: (1) take stock of the results achieved by CIDA’s Indonesia Country Program over the eight-year period 2005-2013; (2) assess the Program’s overall management performance in achieving these results; and (3) document and disseminate findings and lessons learned, and formulate recommendations to improve performance. The Summary Terms of Reference are presented in Appendix A.

The evaluation team was composed of Canadian and Indonesian specialists in governance, the environment, private sector development, and gender equality. The evaluation examined the Program’s relevance, effectiveness, efficiency, sustainability, coherence, aid effectiveness, performance management, and the crosscutting themes of gender equality and environmental sustainability. The sample projects were assessed using the evaluation criteria, and given nominal scores based on a five-point scale (highly satisfactory to highly unsatisfactory).

Given the high degree of variability among the projects, a randomly selected sample might have left out key investments and affected the evaluation’s usefulness, so a purposive sampling methodology was used. The sample of 32 projects (from an evaluation universe of 80 projects) disbursed $152,688,525 from 2005-2013, which covered 75% of CIDA’s overall disbursements to Indonesia ($204.5 million by all branches from 2005-2013) – see Table1.

The evaluation focused mainly on the bilateral channel: the sample included 25 bilateral projects (representing $134,601,456 million or 67% of the Program’s disbursements from 2005-2013). Five projects were from the partnership channel and two were multilateral. The sampled projects are listed in Appendix F. Interviews were conducted with 280 stakeholders in Canada and Indonesia, and 683 program and project related documents were consulted.

The evaluation did not cover tsunami-related investments, since the Tsunami Reconstruction and Rehabilitation Fund (that disbursed $147 million from 2004-2009) was evaluated in 2009, nor emergency assistance which was assessed as part of the 2011 Evaluation of CIDA’s International Humanitarian Assistance.

Data collection and analysis were conducted at the project and program levels. The evaluation used mostly secondary source data for closed projects and, where possible, primary data collected from stakeholders during the site visits was added for ongoing projects.

Four data gathering techniques were used to obtain multiple lines of evidence: (1) Document review of program and project level documents; (2) interviews and focus groups with CIDA staff and managers at headquarters and in the field, Indonesian Government officials, and key Canadian and Indonesian partner organizations; (3) selected project site visits in Sulawesi and Jakarta in November 2013; and, (4) focus groups with other donors.

Source: CIDA Chief Financial Officer’s Branch/Statistical Unit, Data as of 2013-04-19

1.2 Evaluation Challenges and Limitations

There was limited evaluative documentation. The Program as a whole had not been evaluated in the past ten years, and twelve of the 32 sampled projects had end-of-project evaluations. The number, availability and quality of project-level evaluations, mid-term reviews, and monitoring reports varied across the three sectors.  When the available data was insufficient, the evaluation relied heavily on interview data collected during the field visit.

The eight-year timeframe covered two Country Development Program Frameworks (CDPFs). Projects that began under one set of CDPF priorities were in many cases completed under other ones with concomitant changes in focus, terminology and reporting emphasis. Projects were therefore assessed against their internal purpose and objectives as well as the CDPF that was in effect at the time they were designed. 

Observation and ratings in this report are based on data available at the time of the evaluation. New data could change current ratings.

2. Country Context and CIDA’s Country Program

2.1 Country Context

Indonesia is a country of 17,000 islands, with 247 million people. It is the fourth most populated country in the world, the third largest democracy, and the world’s largest majority Muslim country. In 1945, after decades of Dutch colonial rule, Indonesia obtained independence under President Sukarno. After some years of authoritarian rule and eventual civil unrest, General Soeharto came to power in 1968 and ushered in a so-called “New Order” dedicated to neo-liberal economic growth. Rapid industrialization led to accelerated economic development and until 1982 Indonesia maintained an annual economic growth rate of at least 5%. By the early 1990s, Indonesia was considered an “East Asian Miracle”, and “Asian Tiger”. However, this liberal economic strategy also resulted in unsustainable growth characterized by over-exploitation of the country’s natural resources and an authoritarian governance style. In 1998, after a popular uprising, precipitated in part by the severe impact of the 1997 Asian Economic Crisis, the Soeharto regime was ousted from power after three decades.

2.2 A Transitioning Democracy

Indonesia initiated a range of political reforms aimed at democratization, starting with the general election of 1999. Decentralization was implemented to improve public governance and support the aspirations of the regions to have more authority. From this perspective Indonesia is considered a success story of democratic transition. However the legacy of authoritarianism is still felt. While the electoral process is functioning well, democratic values such as accountability, access to justice, respect for human rights and equality, are still evolving. Corruption remains a challenge – Indonesia was ranked 114^th out of 177 on Transparency International’s Corruption Perceptions Index in 2013.^{Footnote 3}

President Susilo Bambang Yudhoyono, who came to power in 2004 and was re-elected in 2009, has been praised for his support of the national Corruption Eradication Commission (KPK). Since it started in 2003, the KPK has investigated, prosecuted and achieved a 100% conviction rate in 86 cases of bribery and graft related to government procurements and budgets, including the sentencing of 29 mayors and district chiefs in the period of 2006-2011.^{Footnote 4} Given the extent of their work, the KPK has gained significant public support in Indonesia. The election of President Joko Widodo in July 2014 (a non-military leader who is not linked to the old order) is a positive indication that anti-corruption, transparency, and accountability will be pursued with renewed vigour.

There are concerns with threats to democracy and human rights from Islamic fundamentalism, an issue in Indonesia since early 2000. Nonetheless, Indonesia is considered to be one of the strongest democracies within the Association of Southeast Asian Nations and is often described as its democratic leader.

Important public services, such as education and health, are now provided by municipal governments. Mayors and regents (the difference lies in differing city demographics, size and economics) have greater autonomy and legitimacy to pursue different policies than those of the central government that now transfers over 60% of the national budget to sub-national levels. While decentralization has generally been positive for increased inclusiveness and decision-making, it has also led to challenges. Most sub-national administrations have little capacity for developing and implementing economic development strategies for their areas. They generally lack the knowledge and skills to work collaboratively with the private sector and civil society to create an enabling environment for pro-poor growth.

With the move toward democratization, a number of civil society organizations (CSOs) have played strong advocacy and protection roles in areas like human rights, anti-corruption, and natural resource management. Their capacity has increased in the area of analysis and advocacy for improved public policy and effectiveness in government service provision, whereas traditionally their strength had been in service provision. While there are a number of strong national-level (often urban-led) CSOs, many local rural CSOs and community groups lack the necessary capacity to lead advocacy and policy dialogue efforts to make sub-national governments more accountable to the rural poor on critical issues such as land use, economic growth, gender equality, and environmental sustainability.

2.3 Gender Equality Challenges

Indonesia’s transition to democracy had an overall positive effect on women’s rights and gender equality. The ratification of international agreements, the passage of laws, the formulation of policies that address women’s issues and the creation of the Ministry of Women’s Empowerment and Child Protection, have all been positive developments. Most Indonesian laws emphasize the importance of gender equality, but their implementation presents on-going challenges. Secular law co-exists with Islamic principles, which are sometimes interpreted in a way that discriminates against women.

Decentralization has had many benefits but it has also created many challenges for gender equality, especially at the local levels where capacity is lower. Lack of capacity has sometimes translated into less gender sensitive policies and practices at the local levels, and the central government has been largely powerless to deal with these. A 2013 study by the Commission on Violence Against Women reported that 342 discriminatory local laws and ordinances have been enacted against women and girls since 2009 in Indonesia. The United Nations’ 2013 Gender Inequality Index places Indonesia at 106 out of 148 countries.

2.4 Development and Economic Growth

Indonesia is a middle income country, member of the G-20 and the largest economy in Southeast Asia. Economic growth has lifted many people out of poverty, but there is still vulnerability to economic downturns with few safety nets. Public expenditures on health and education in 2012 remain about 1% of Gross Domestic Product, below the average for the East Asia-Pacific region which is about 2% for health and 3.5% for education, and below the average for other lower-middle income countries.^{Footnote 5} Over 50% of Indonesians have only primary school education, maternal mortality rates are among the highest in the East Asia-Pacific region, sewerage coverage is among the lowest in Asia, and inequality is on the rise. This is holding back Indonesia’s human development progress. Its rank of 108^th out of 182 in the United Nations’ Human Development Index (2014) is largely attributed to the fact that approximately 16% of the population – more than 36 million people – still live below the national poverty line (based on the consumption of 2100 calories/day).

Sulawesi is the world’s 11^th largest island, Indonesia’s third most populous, and one of its poorest.

2.5 Increasing Environmental Vulnerability

Although Canada and other donors have worked in environment and natural resource management for over 30 years in Indonesia, environmental sustainability continues to be under serious threat due to increasing pressure from Indonesia’s growing population and economy, and lack of proper management of environmental issues. Climate change is becoming one of Indonesia’s most significant developmental challenges in the 21^st century, with the possibility of increased frequency and intensity of extreme weather events, including droughts, floods and storms having adverse impacts on crop yields and water resources. Other environmental issues include loss of forest and terrestrial biodiversity; deterioration of coastal habitats (such as mangroves, coral reefs, sea-grass beds) and fisheries resources; degradation of water resources; untreated waste; and air pollution.

Approximately 50% of the labour force is dependent on natural resources and agricultural land use, while resource exploitation and processing account for 44% of Gross Domestic Product.

2.6 CIDA’s Country Program

CIDA’s total disbursements in Indonesia from 1954 to 2011 amounted to over $1.47 billion (excluding disbursements in support of humanitarian assistance and reconstruction efforts to Aceh and North Sumatra following the Indian Ocean tsunami in December 2004).

Indonesia became a CIDA core country in 1970, with assistance focused on large infrastructure projects in areas such as irrigation, transportation, electricity, and communications. The Program’s focus evolved over time. Discussions with Indonesia’s National Development Planning Agency led the Program to focus for the past 30 years on the island of Sulawesi in Eastern Indonesia, which is poorer than Western Indonesia and more in need of development support.

Under the Country Development Programming Framework (CDPF) 2004-2009, the three key areas of concentration were governance, the environment, and private sector development. The Program was structured “to support Indonesian efforts to reduce vulnerability to poverty” by focusing its programming on “improved governance, growth of the private sector, and sustainable use of natural resources”.

The CDPF 2009-2014 narrowed focus to economic development and natural resource management, aiming to contribute to “increased sustainable and equitable opportunities for poor men and women in targeted areas of Sulawesi”. Governance comprised the bulk of programming. Environment programming with a focus on sustainable natural resource management and private sector development with strong environmental sustainability characteristics also have been a predominant feature of the Indonesia Program.

CIDA implemented 80 projects in Indonesia and disbursed a total of $204.5 million during the 2005-2013 period. Geographic Programs Branch contributed about $162 million (79%), Partnerships with Canadians Branch almost $25 million (12%), and Multilateral and Global Programs Branch about $17.5 million (9%).

Source: CIDA Chief Financial Officer’s Branch – Statistical Unit, Data as of 2013-04-19 (projects greater than $250,000) “Other Sectors” include Strengthening Basic Education, Emergency Assistance, Improving Health, and sectors coded “Other.”

Most projects (67 out of 80) used the ‘responsive’ delivery model (which accounted for 72% of total disbursements).^{Footnote 6} The Program’s main disbursements were in democratic governance (48% or $78 million), environment (21% or $34.6 million), and private sector development (12% or $18.8 million).

2.7 Development Assistance by Other Donors

According to the Development Assistance Committee (DAC) of the Organization for Economic Co-operation and Development (OECD), multilateral and bilateral Official Development Assistance increased with the election of President Susilo Bambang Yudhoyono in 2004, peaked at US$ 3,679 million with his re-election in 2009, and then decreased in 2009-2012. In part, this can be explained by the fact Indonesia returned to its place as a middle-income country in 2007 after a long recovery from the 1997 economic crisis. The top five bilateral donors in 2012 were Japan, Australia, the United States, European Union institutions, and Germany (Figure 1). Canada was the 6th largest bilateral donor from 2006-2011. According to OECD statistics the largest multilateral donors in 2012 were the Asian Development Bank (US$ 886 million) and the World Bank (US$ 706 million).^{Footnote 7}

Source: DAC Information System (http://stats.oecd.org/Index.aspx?DataSetCode=REF_TOTAL_ODF#) extracted 2013-04-23. Aggregate geographical data; Aid (ODA) disbursements to countries and regions (DAC2a). Ranked by total ODA disbursement percentages of the period 2005-2011.

3. Main Findings – Development Results

3.1 Relevance

The evaluation examined the extent to which the Program and project objectives were consistent with beneficiary and country needs, and CIDA priorities.

3.1.1 The Country Program’s Focus on Sulawesi

Canada is a small donor in Indonesia, but a significant one on the island of Sulawesi where the CIDA Country Program has focused over the past 30 years.^{Footnote 8} Although other bilateral and multilateral donors operate in Sulawesi, CIDA’s sustained presence and development programming has helped to build strong relationships between Canadian partners and Indonesian government and civil society stakeholders. The strength of these relationships has strengthened the level of influence of a small donor like CIDA. Discussions with the National Development Planning Agency led to the Country Program’s long term focus on Sulawesi, which is less developed, has higher poverty and maternal mortality rates than in Western Indonesia, and is aligned with CIDA’s poverty reduction policy and Indonesia’s focus on least developed areas.^{Footnote 9}

The Program used its focus on Sulawesi to begin to test practices that could inform provincial and national government policy or be replicated in other locations. It intends to continue piloting reforms in Sulawesi for the next programming period 2013-2018. “The CIDA program and its Government of Indonesia counterparts have adopted Sulawesi as a laboratory to test and inform national government policy development and implementation” as a way to respond “to the Government of Indonesia’s own priorities (e.g. forestry, governance, education, private sector investment, attention to Sulawesi).”^{Footnote 10} Governance projects, such as BASICS (Better Approaches to Service Provision), PEACH (Public Expenditure), and SIPS (Islands of Integrity in Sulawesi) have been picked up by the Indonesian government and shown to be successful and sustainable. There was less evidence of achievements being replicated in other sectors.

In interviews, senior government officials in several national ministries in Jakarta expressed their strong support for CIDA to continue to lead initiatives in Sulawesi where the needs are greatest. They indicated that projects that work well in the provinces have been, and will continue to be replicated in other provinces. They emphasized that support is not needed at the national level as much as in the provinces where the institutional capacity is low, and that Canada can play a valuable role in strengthening governance, gender equality, civil society, human rights, and environment and natural resource management. These are areas where Canada’s experience and know-how are strong and recognized by Indonesian government officials, civil society organizations and other donors.

3.1.2 Other Relevance Considerations

The Program’s earlier projects were consistent with the Country Development Programming Framework (CDPF) of 2004-2009. The recent portfolio was also consistent with the 2009-2014 CDPF, but the Program had to make adjustments to operational projects to ensure consistency with CIDA’s new Sustainable Economic Growth priority theme (2009).

The CDPF 2004-2009 matched five of the ten guiding principles of Indonesia’s 2004-2009 Medium Term Development Plan, namely private sector-led economic growth, natural resources and environment, integrating the three tiers of government, partnership through strategic alliances, and intervention in least developed areas. The CDPF 2009-2014 included a focus on performance-based budgeting and improved fiscal management which was also consistent with objectives of Indonesia’s 2009-2014 Medium Term Development Plan.

The Program was in line with half of the Millennium Development Goals, especially those designed to empower women, ensure environmental sustainability, and develop a global partnership for development. The Program also addressed the three main pillars of the Jakarta Commitment’s aid effectiveness priorities: country ownership, more effective and inclusive partnership, and delivering and accounting for development results.^{Footnote 11}

The governance portfolio was linked to Indonesia’s emphasis on fighting corruption and improving the government’s fiscal management and accountability. Of particular relevance were the Program’s support of the national Corruption Eradication Commission’s efforts at several levels including district and provincial (Support to Islands of Integrity project) and work with Indonesia’s National Development Planning Agency to create the National Poverty Reduction Strategy (Governance Reform Support II project). The Public Expenditure (PEACH) project worked with the Ministry of Finance and the SILE/LLD project (Supporting Islamic Leadership in Indonesia/Local Leadership for Development) with the Ministry of Religious Affairs. Many other projects worked with the Provincial or District/City Development Planning Agencies.

The private sector development (PSD) portfolio addressed poverty reduction as the overarching theme and aimed to promote the economic well-being of the poor, directly and indirectly. The Program’s emphasis on the environment and sustainable use of natural resources that was prevalent throughout the PSD portfolio was directly in line with CIDA’s Sustainable Economic Growth priority.

The environment portfolio contributed directly to improved natural resource management and sustainable livelihoods of the rural poor at a community level, and indirectly by increasing public awareness and knowledge of natural resource management and environmental sustainability issues, as was the case for the World Wildlife Fund’s project. The Java Earthquake Multi Donor Trust Fund improved the livelihoods of people affected by the 2006 earthquake, and Indonesia’s environmental sustainability priorities were addressed by the Climate Change Peatlands project and the Forest Carbon Partnership Facility.

3.2 Effectiveness

The evaluation examined the extent to which the Program and the sampled projects had achieved their outcome-level results, or were expected to achieve them.

The degree to which the Program achieved the outcomes of its two Country Development Programming Frameworks could not be assessed due to gaps and challenges in Program-level monitoring and reporting (see Section 4.4 on Performance Management). The evaluation examined the effectiveness of the sector portfolios through the lens of the sampled projects.

3.2.1 Governance

The Program’s governance projects achieved a number of outcomes and have helped improve knowledge and build the capacity of local governments, universities and community leaders. The Program was particularly successful in advancing poverty reduction objectives through the Governance Reform Support II project that assisted the National Development Planning Agency (BAPPENAS) in developing a work plan for the national Poverty Reduction Strategy (PRS) that included drafting the PRS, connecting it to fiscal decentralization reform and coaching BAPPENAS to develop regional poverty reduction strategies with regional governments.

The Program is also helping to improve the government’s transparency and accountability, working with national authorities to prevent corruption at different levels but especially the sub-national level where the need is greatest. The SIPS (Islands of Integrity) project in Sulawesi has created diagnostic tools for the assessment of corruption and supporting the development of transparent client-oriented one-stop services to extend easy access to integrated government services. The Corruption Eradication Commission is working with SIPS to develop an electronic complaint mechanism to help combat corruption at different government levels.

Indonesia’s decentralization in 2001 created a new system of revenue sharing between levels of government and transferred responsibility to sub-national levels that lacked the capacity to manage significant additional resources. The Program has helped to address this through projects promoting policy dialogue with National Ministries in Jakarta that have provincial or sub-national partners in Sulawesi. Recent examples include: Corruption Eradication Commission in SIPS (Islands of Integrity); Ministries of Forestry and Fisheries in RCL (Restoring Coastal Livelihoods); Ministries of Education and Health in BASICS (Better Approaches to Service Provision); Ministry of Religious Affairs in SILE/LLD (Supporting Islamic Leadership/Local Leadership for Development); Ministry of Home Affairs in PEACH (Public Expenditure); Ministry of Finance, Ministry of Women’s Empowerment and National Development Planning Agency in B3WP (Building Better Budgets for Women and the Poor); Ministry of Finance and National Development Planning Agency with Multi Donor Trust Funds.

As an example of sub-national policy impact, the PEACH project’s Public Expenditure Analysis was used to inform evidence-based decisions in the province of North Sulawesi, which led to reductions of 40% in official travel expenditures by 2013 that amounted to approximately USD 9 million in savings that were reallocated to help women and children’s health among others.

CIDA helped to overcome the historical issue of little public input to government in Indonesia and fill the space created by decentralization for civil society and non-governmental organizations to play a larger role in democratic governance reform efforts. The Development & Peace Program 2006-2011 worked with six civil society partners in Indonesia, one of which, the Women’s Coalition, achieved notable gender equality outcomes in policy dialogue, advocacy and influence leading to changes in government policies and programs. The Women’s Coalition succeeded in passing the Political Party Law which guarantees increased women’s representation nationally and locally; influenced changes made in the Anti-Domestic Violence Law; succeeded in getting regulations adopted to enforce the law on the prevention of trafficking of women and children; and convinced the Government of Indonesia to establish the Integrated Service Centre for domestic violence victims at the district level.

Although the governance projects have engendered many advocates who have contributed to improved outcomes, more effort could have been made to link the projects into a fully integrated governance program, ensuring that successful governance initiatives were replicated nationally and in other provinces, and increasing coordination and sharing of information among governance and other projects working to achieve similar results. The opportunity for further integration was reduced by the change of program focus from governance to sustainable economic growth.

3.2.2 Private Sector Development

The private sector development (PSD) projects made contributions to the enhancement of the technical, marketing and managerial skills of a significant number of small and medium enterprises and family businesses that have created up to 15,000 direct and indirect jobs since 2005. The PSD projects selected industry clusters and areas that allowed targeting the poor and women’s groups as the major beneficiaries of CIDA’s support. The ongoing Restoring Coastal Livelihoods (RCL) project achieved impressive results in terms of capacity development and poverty reduction, since it made real progress in restoring mangroves and rejuvenating the economic base of poor communities by promoting seaweed farming, fishing and fish processing activities, through a community based approach characterized by the political and economic empowerment of women. Through a similar community-based local development approach, the Sustainable Livelihoods (SL) project also significantly improved the economic wellbeing of a poor community through the introduction of better tree-farming practices and passion fruit processing.

Conversely, efforts made since 2005 for the institutional strengthening of Business Development Service Providers (BDSPs) had limited success with several partner institutions that received technical assistance, since in the end they lacked interest in strengthening their capacity and effectively serving their clientele of small and medium enterprises (SMEs) in the way proposed by the projects. However, a few BDSPs showed significant results in supporting SMEs with the preparation of business plans and capacity building. For example, the PEP (Private Enterprise Participation) project worked until the end of 2007 on building the capacity of the Indonesia Women’s Business Association to promote women owned businesses and to lobby governments in favour of women’s economic rights and role, and was successful in creating over 12,500 net new jobs.  However this evaluation has not been able to assess the current capacities of IWAPI and interviews indicate that sustainability is problematic due to limited financial capacity of partner organizations.

Indonesia has one of the most cumbersome and corrupt business licensing systems in Asia. Decentralization has resulted in local governments taking a leading role in establishing the regulatory environment for the private sector, which has often led to increases in the number of regulations to raise revenues or impose controls on business. The Program has some documented examples of policy dialogue, advocacy, and influence that led to change in policies or regulations.

IBESS (Improving the Business Environment for SMEs) contributed to streamlining business regulations at the sub-national level in Sulawesi by supporting the one-stop shops that provided improved services to SMEs (simplified procedures, less requirements, speedier registration and permit processing) which led to a large increase in the number of business licenses (13,360) issued in 2008, 38% to women entrepreneurs. RCL (Restoring Coastal Livelihoods) engaged in policy dialogue at the community and local government level. Interviews confirmed that RCL strongly addressed the policy environment and has become adept at demonstrating the problematic nature of environmental degradation and of the need for mangrove management and restoration. Each project visit is made with the participation of government officials, and program review meetings are used to provide information to stakeholders. It is expected that, as the project advances and its results and lessons are systematized, national stakeholders will be informed and invited to discuss the issues of mangrove restoration and of overlapping jurisdictions.

In other projects, such as PENSA and PENSA-2 (Program for Eastern Indonesia Small and Medium Enterprise—SME—Assistance) there was partial success in achieving policy level objectives. The Indo M-Banking initiative (PENSA-2) engaged with the Bank of Indonesia to improve the regulatory framework related to mobile banking and contributed to changing the regulatory environment, however according to documents this was not sufficient for mobile banking to expand. In the case of the SME Banking initiative (PENSA-2), the banks involved preferred to improve their own model for providing services to SMEs rather than adopt the proposed SME banking approach. PENSA-2 aimed at increasing credit flows to small businesses through a series of initiatives, was mostly unsuccessful. There was limited policy dialogue associated with the design and implementation of tools and models intended to be replicated and achieve sector reform.

3.2.3 Environment

The environment portfolio has contributed to the Program’s objective of supporting local and regional economic development by promoting good management of renewable resources. The projects have improved livelihoods and contributed to better management of natural resources. They have enhanced the agroforestry, forestry and coastal livelihoods of coastal and rural poor populations; supported the protection of watershed systems; improved the awareness and knowledge of sustainable management practices of natural resources and the environment; and supported global efforts to mitigate the effects of environmental disasters, malpractices and climate change. They have also provided cost-effective, basic social infrastructure such as water and sanitation systems; and institutionalized participatory processes in local government including equal decision-making among women and men.

For example, the CCP project (Climate Change Peatlands) resulted in concrete livelihood gains for local communities, the restoration of degraded peatlands and improved peatland management. CCP was able to change community behaviour (for example by changing from illegal logging to farming) and to increase awareness of environmental conservation. The project resulted in measurable carbon benefits, with an estimated 2,5 million tons of carbon sequestered or gained in Central Kalimantan, and about 1,9 million tons being sequestered or gained in Jambi, a province of Sumatra.

The projects supported by the Java Reconstruction Fund led to the completion of over 15,300 seismic-resistant core house structures in earthquake affected regions, improved access to finance for more than 15,000 micro, small and medium enterprises, and increased incomes for at least 70% of beneficiaries in Central Java and Yogyakarta.

The villages and districts targeted by the SUSCLAM project (Tomini Bay Sustainable Coastal Livelihoods and Management) have adopted more sustainable mangrove resource use and management mechanisms and practices that improved the mangrove landscape and productivity. Before the project, few understood the critical role played by mangroves in supporting the integrity and sustainable development potential of Tomini Bay’s coastal ecosystem.

In terms of basic social infrastructure, the SWASH project (Sulawesi Water and Sanitation Hygiene) helped to construct over 9,000 household latrines and provide some 160,000 people with access to clean water.

There were successful examples of piloting reforms with sub-national partners in Sulawesi which informed the policy work of provincial or national level ministries. The Green PNPM was introduced with CIDA support as an environmental pilot-project under the Sulawesi component of Indonesia’s National Program for Community Empowerment (PNPM)^{Footnote 12}. Through it, the Government of Indonesia disbursed block grants to finance community-driven development investments in ‘green’ sub-projects, encouraging community buy-in for economic activity that was conservation friendly. Another successful example was the Climate Change Peatlands (CCP) project, which influenced the policy environment at local, national and even international levels. CCP helped to develop the local government’s peatland strategy, which supported the Government of Indonesia’s National Wetlands Strategy/Action Plan, the technical materials used by the Forestry Ministry, the United Nations’ Intergovernmental Panel on Climate Change, and the Association of Southeast Asian Nation’s Peatland Management Initiative.

As long as poverty remains, there will be growing pressure on the use of natural resources to meet basic human needs. This is contributing to environmental degradation, which is one of the most serious threats to health, food security, habitat, income and quality of life, since 70% of Indonesia’s poor live in rural areas and are dependent on the natural environment for their livelihoods and survival. The Program’s support of the World Wildlife Fund’s project (Strengthening the Outreach and Education Network for Natural Resources Governance in Sulawesi) helped to educate and promote natural resource management. Journalists were trained to improve factual reporting. Some 16,000 articles and 400 radio/TV programs were disseminated, and environmental curricula were included in schools. Over 6,600 participants (from government, media, academia, civil society) took part in roundtable discussions on key natural resource management issues, such as stopping cove reclamation, spatial planning and educating the public on becoming sustainable seafood consumers. Policy-making was positively influenced, for example when public intervention enabled the authorities to prevent a planned road from being constructed through a protected area in Southeast Sulawesi.

Notwithstanding the achievements of the environment projects, they have been challenged by insufficient timeframes, as natural resources management is a long-term endeavour that needs to be supported over longer periods than CIDA’s customary five-year implementation period, sometimes through multiple phases of CIDA-funded projects. Other challenges included limited number of bilateral directive programming initiatives that directly support CIDA’s strategic objectives in the country; lack of reliable and sustainable market linkages for individuals and micro/small and medium enterprises who are dependent on the natural resource base for their livelihoods; lack of synergies among CIDA projects supporting good governance, private sector development, and improved management of natural resources; and recurring natural disasters. The projects have also been affected by the limited human/financial resources and technical capacity of government institutions.

3.3 Sustainability

The evaluation examined whether the results achieved by the sampled projects were likely to be sustainable without CIDA’s ongoing financial support.

Indonesia’s financial capacity has improved in the past five years as it has emerged as a lower middle-income country. The Government is now able to support projects it views as valuable and has done so in several cases, such as PEACH (Public Expenditure) in which initiatives were well integrated into government programs. Donor representatives noted that increasingly “Indonesia doesn’t need our money but wants our technical knowledge and the opportunity to innovate and build spaces for experimentation with new ideas.”^{Footnote 13}

There is evidence that projects that supported change and strengthening of the governance system at multiple levels were sustainable, in particular those that were replicated in other locations and are likely to continue to have impact on multiple stakeholders on a wider scale, An example was BASICS (Better Approaches to Service Provision) that responded to requests from the Government of Indonesia to present their lessons learned in several other locations in the country.

Higher ratings for sustainability were associated with good targeting, strong ownership and beneficiary input, continuing support mechanisms, good monitoring and accountability, and in some cases joint funding, such as in the SIPS (Support to Islands of Integrity in Sulawesi) project.

Examples of completed governance projects showing enduring benefits included the Islamic Social Equity project that was integrated into the regular university budget and the Governance Reform Support II initiatives that were implemented at multiple levels of government. In other areas, challenges occurred either when ministries were unable to secure approval from local parliaments for ongoing funding, or when there was turnover of key personnel who had been champions for a particular initiative, such as in the gender focussed projects GBRI (Gender Responsive Budgeting initiative) and BASICS (Better Approaches to Service Provision).

Sustainability was more challenging for the private sector development portfolio. Projects aimed at improving the business environment produced sustainable results in terms of regulatory reform improvement, however the capacity of local governments to enact reforms and of business associations to advocate for them remains weak. There are indications that, aside from economic activities that were viable and well mastered by some small and medium enterprises (SMEs) and family businesses, the results of the PEP (Private Enterprise Participation) project and the CIPSED (Canada-Indonesia Private Sector Enterprise Development) project in the area of institutional strengthening and SME development are not sustainable. Prospects appeared stronger for projects that adopted community-based approaches, such as Restoring Coastal Livelihoods and Sustainable Livelihood. However, natural disasters seriously threatened the sustainability of results that depended on the exploitation of natural resources such as seaweed farms and tree crops, while wide price fluctuations jeopardized the continuous production of commodities such as coconut and cashews.^{Footnote 14}

The sustainability of environment sector investments was enhanced when projects were either extended beyond their initial five years, or designed from the outset with longer implementation periods of up to nine years. Strong local ownership and institutional capacity also contributed to sustainable outcomes. For example, the Java Earthquake Multi-Donor Trust Fund strengthened the institutional capacity of Indonesian government institutions to undertake rehabilitation efforts in response to natural disasters, and the SWASH (Sulawesi Water and Sanitation Hygiene) project demonstrated that, through capacity building initiatives, continuing support could eventually be provided by local communities and local government institutions to sustain project outcomes. In addition, the most recent wave of PNPM^{Footnote 15} investments integrates environment considerations throughout all projects based on the lessons and successes of the Green PNPM pilot-project supported by CIDA. Sustainability was challenged by weak institutional and financial capacity: local governments may not have dedicated funds for environmental issues, or the allocation of scarce funds within villages may not favour supporting environmental issues over competing demands like education and health. Other challenges included limited access to markets and investment capital.

3.4 Gender Equality (crosscutting theme)

The evaluation examined the extent to which the Program and the sampled projects had integrated the crosscutting theme of gender equality.

CIDA is a strong and well known proponent of gender equality in the donor community. The Program is aligned with CIDA and Indonesia’s gender equality policies. In 2011, CIDA partnered with Indonesia’s Ministry of Women’s Empowerment and Child Protection, and two other donors, to lead research and develop a series of policy documents on gender equality. More recently, Indonesia’s Ministry of Women’s Empowerment and Child Protection requested CIDA’s support for preparations for the 2013 Asia Pacific Economic Co-operation Forum (APEC) conference on women and the economy.

The Program included a strong commitment to gender mainstreaming especially in the Country Development Programming Framework (CDPF) 2004-2009. This commitment was less evident in the CDPF 2009-2014. The first CDPF period, with its stronger emphasis on governance, human rights and civil society advocacy, helped to provide a stronger enabling environment for gender equality results. The second CDPF’s emphasis on economic empowerment would have benefitted if such a gender-enabling approach had continued. Simply promoting economic empowerment, without sufficient attention to the complex set of conditions that enable gender integration, can lead to inappropriately gender-blind strategies.

The Program supported “gender-focused” projects such as B3WP (Building Better Budgets for Women and the Poor) and GRBI (Gender Responsive Budgeting Initiative) that effectively benefitted key actors by providing information to challenge discrimination, inefficiency and corruption that affect women and men differently. Although not a “gender focused” project, Health for the Poorest Quintile (supported by CIDA’s Multilateral and Global Programs Branch) helped to round out CIDA’s gender-focused programming in Indonesia and had an important primary emphasis on women’s reproductive health and on overcoming barriers for mothers and caretakers in receipt of maternal health services.

Projects that included clear priorities for gender produced more evidence of gender results. SILE/LLD (Supporting Islamic Leadership in Indonesia/Local Leadership for Development) is a model that demonstrates that culture and traditional values that discriminate against women and girls can be challenged effectively over time. In cases where it was necessary to challenge the root causes of gender stereotypes in cultural practices, norms and values, working with respected community leaders, programming incrementally over the long-term, acquiring in-depth knowledge of local context, and engaging civil society actors have been important success factors.

The governance projects showed a satisfactory understanding of gender with particular focus on increasing gender understanding within line ministries, local government sectors, parliament, and civil society; and increasing women’s access to economic benefits. Three projects stood out as having gender particularly well mainstreamed. The SILE/LLD project (Supporting Islamic Leadership/Local Leadership for Development) helped to increase women’s participation in university decision-making processes. Through the university, the project brought a new perspective to Islam, which is key because people—especially local leaders—are listening to what the university is teaching about Islam and gender. The PEACH (Public Expenditure) project integrated gender in its capacity building programs to enhance the sub-national government’s understanding of the need to incorporate gender-related issues into the public financial management process. PEACH also included gender in its outreach programs to ensure the broader community including media outlets and civil society organizations understand gender-responsive planning and budgeting and exert pressure on sub-national governments to address gender disparities. PPD-2 (Partnership Program II) implemented gender budgeting, created gender profiles, mapped the leadership potential of women and helped increase the participation and representation of women in village administration.

There were significant weaknesses in the gender equality performance of private sector development projects. The PEP project (Private Enterprise Partnership) tried to promote private sector development in a way that would reinforce the role of women-owned businesses in the economy and create more jobs for women. This was done, however, without sufficient analysis of the social and institutional constraints Indonesian women face in business and in taking on employment outside their own homes.

IBESS (Improving the Business Environment for Small and Medium Enterprises in Sulawesi) looked at these constraints from the angle of business registration and licensing but without getting to the root of the constraints to women’s empowerment. PENSA and PENSA-2 (Program for Eastern Indonesia Small and Medium Enterprises Assistance) neglected gender equality in the projects’ design and implementation. In contrast, RCL (Restoring Coastal Livelihoods) included a very well designed gender equality strategy that is exemplary and has already achieved some valuable results, such as improving women’s control of economic resources and rights of ownership. RCL’s rights-based approach saw gender equality as fundamental to achieving project objectives: it included a gender analysis and action plan as well as operational measures to support women to influence policies and practices that reflect their priorities and needs.

In the environment sector, the initiative implemented by Development & Peace supported the Women’s Coalition in protecting women’s rights and enhancing their decision-making roles in society. SWASH (Sulawesi Water and Sanitation Hygiene), SUSCLAM (Tomini Bay Sustainable Coastal Livelihoods and Management) and EGSLP (Environmental Governance and Sustainable Livelihoods) achieved notable outcomes in terms of women’s decision-making within their households and communities as well as increased access to, and control over, development resources and benefits. The World Wildlife Fund’s project was rated unsatisfactory as the absence of a gender equality strategy was reflected in the absence of any evidence of gender outcomes.

Although there was strong gender capacity at CIDA headquarters from gender advisors, the workload was large given the size of the Program. The investment of only one part-time Jakarta-based Indonesian advisor in gender support for all projects in the field has proven to be insufficient, especially since decentralization in Indonesia is a risk factor for gender equality, given that gender capacity is generally weaker at the sub-national levels than at the national level and there is evidence that decentralization is having a negative impact on women’s rights in several districts.

3.5 Environmental Sustainability (crosscutting theme)

The evaluation examined the extent to which the Program and the sampled projects had integrated the crosscutting theme of environmental sustainability.

Environmental sustainability as a crosscutting theme had a prominent place in the Country Development Programming Frameworks (CDPFs) over the review period. Both CDPFs had strategies and integrated environmental sustainability considerations into decision-making and activities that took place in all three of the Program’s sectors. The CDPF 2004-2009 focused on capacity building for local stakeholders to achieve the sustainable use of natural resources. The CDPF 2009-2014 focused on sustainable economic growth through strengthening local and regional planning and programming and promoting improved management of natural resources.

In the governance sector, the SIPS project (Islands of Integrity) made environmental impact assessment mandatory for securing business permits. The SILE/LLD project (Supporting Islamic Leadership/Local Leadership for Development) demonstrated significant environmental content by working with environmentally-oriented non-governmental organizations and by making the diversity of ecosystems a criterion for selection of project sub-activities. The three projects supported by CIDA’s Partnerships with Canadians Branch projects (CUSO, CUSO-VSO and IPAC/Institute of Public Administration of Canada) all had well developed environment strategies for their programming which were carried out effectively.

In the private sector development portfolio, environmental sustainability, although taken into account, was generally less important than gender equality. Little information is available about the environmental outcomes of the private sector projects. The RCL project (Restoring Coastal Livelihoods) achieved the most significant and well-documented results. At the policy level, there was substantial progress in establishing a common understanding with Indonesian authorities about the regulatory context and complexities of restoring and protecting mangroves while developing sustainable livelihoods for the inhabitants. In restoring 140 hectares, RCL tested a new approach for mangrove restoration that combined appropriate natural regeneration techniques with the promotion of economically viable activities (such as seaweed farming, salt-resistant rice growing, vegetable growing, duck raising) for villagers, especially women. This approach may now be ready for dissemination and replication to other areas. Peat marshes and mangroves are not only essential breeding grounds for fish and wildlife; they are buffers to flooding, storm surges and sea level rise. Indonesia’s Ministry of Forestry reported only about 1.2 million hectares of mangrove in 2007, compared to about 3.5 million hectares in 1988.

Canada-based environment and forestry specialists and two Indonesian environment and natural resource specialists have been advising the Program on environmental sustainability issues and helping to monitor natural resource management projects. Technical support to the Program, however, has been limited because technical specialist positions at headquarters and travel budgets have been reduced. Furthermore, the workloads of the few remaining specialists at headquarters and of the local experts in the field is considerable which limits their ability to travel to Sulawesi to provide technical support or monitor environment sector projects.

4. Main Findings – Management Factors

4.1 Coherence

The evaluation examined the Program’s internal coherence (between projects) and its external complementarity with multiple levels of government in Indonesia as well as in the context of international efforts and Canadian whole-of-government efforts.

In 2007, the field officer for CIDA’s “regional” Southeast Asia Program was relocated from Bangkok to Jakarta – where the Association of Southeast Asian Nations Secretariat (ASEAN) is headquartered – and now works alongside the Indonesia Program’s field staff. This improved coherence between CIDA’s country and regional programs, and between national and regional initiatives in anti-corruption, human rights, rule of law, and support for civil society input to governments. For example, CIDA’s regional “SEARCH” initiative (2004-2011) in human rights and the rule of law supported long time priorities common to Canada and Indonesia as well as to ASEAN. It also built on previous programs by CIDA’s Partnership with Canadians Branch that had sent many representatives from Indonesian civil society organizations to Canada for human rights training over the years and had supported capacity building on the ground in Indonesia.

Most of CIDA’s projects in Indonesia were multi-sectoral in nature, which helped to address and integrate the different aspects of a development issue. Examples included: CIPSED (private sector development and governance – to support small and medium enterprises); BASICS (governance and gender – to build capacity development of governments); Environmental Governance and Sustainable Livelihoods (environment and private sector development – to improve natural resource management and rural livelihoods); and Restoring Coastal Livelihoods (environment and private sector development – to improve natural habitat and livelihood).

The Program’s effort to build coherence in the governance sector included stationing a Canadian “Harmonization and Governance Advisor” in Sulawesi for three years (2009-2012) with a mandate to provide support to the CIDA Program and the World Bank’s Decentralized Support Facility (a multi-donor/government coordination effort in Eastern Indonesia). The documentation and interview data show that although this was a good idea and contributed to the governance projects, the facility needed to be better integrated into the overall program and supported over a longer timeframe for maximum effectiveness. The suggestion of multiple interviewees is that more could have been done to take advantage of this opportunity to enhance the overall program impact. For the three active bilateral governance projects – Islands of Integrity (SIPS), Public Expenditure (PEACH), and Supporting Islamic Leadership in Indonesia/Local Leadership for Development (SILE/LLD), the stakeholders reported that further efforts were needed to improve communication and coordination between local government and civil society stakeholders. This could have been facilitated by this harmonization role.

There is evidence of coherence in the Program’s use of past best practices to develop new projects. For example, in designing Governance Reform Support II, the Program relied on knowledge gained from prior CIDA governance projects in Indonesia. The key lesson was the importance of using “an iterative approach, i.e., to recognize the fluid institutional environment that exists, the need to have flexibility to ensure effective programming, and to undertake programming with a range of partners rather than dealing with one single partner”^{Footnote 16}. This flexibility and working with multiple partners became a hallmark of many of the Program’s present governance projects.

The Program’s private sector development (PSD) approach was coherent with CIDA’s Asia Branch strategy that advocated for capacity building at three levels: macro (national policies), meso (business associations), and micro (enterprises and cooperatives). In the context of Indonesia, it was highly appropriate for the Program to have these three levels of intervention. However, the evaluation found that achieving coherence was at times difficult because each Indonesian ministry has its own PSD unit that works independently or without much coordination with the others. The PENSA multi-donor trust funds (Program for Eastern Indonesia Small and Medium Enterprise Assistance) allowed Canada to participate in high-level PSD policy reforms and have access to a donor coordination forum that proved useful after the Indonesian Government put an end in 2007 to the Consultative Group^{Footnote 17}.

External coherence of the Program’s environment portfolio with other donors was noted. For instance, four of the sampled projects required a considerable amount of aid coordination effort as well as engagement with civil society organizations, namely the Java Earthquake Multi-Donor Trust Fund; the Climate Change Peatlands project (comprising a consortium of Canadian, Indonesian and international governmental, non-governmental, and research organizations); Green Community Development in Sulawesi - PNPM (managed by the World Bank and funded by several donor countries and the European Union); and the Forest Carbon Partnership Facility (a global partnership of governments, businesses, civil society, and Indigenous Peoples focused on Reducing Emissions from Deforestation and Forest Degradation – referred to as REDD+).

Two projects successfully demonstrated Canadian whole-of government efforts and the participation of other Canadian Government Departments. The Climate Change Peatlands (CCP) project, approved under the Government of Canada’s climate change strategy, was managed by CIDA and involved the Canadian Forest Service/Climate Change Network (forest management and forest fire control); Environment Canada/Canadian Wildlife Service (assessment and management of peatlands); and the Centre for Wetlands Research at the University of Waterloo (capacity building). The global Forest Carbon Partnership Facility (FCPF) involved coordination with Environment Canada (departmental lead on all environmental issues) and Agriculture and Agri-Food Canada (that shared its International Forest Model and forest management tools with donors and members at the FCPF meetings).

4.2 Efficiency

The evaluation examined the extent to which the Program and sampled projects’ human and financial resources were economically used in generating results.

The Program was particularly efficient in delivering the $222 million Tsunami reconstruction program that more than doubled annual disbursements from 2004-2009. Only minimal additional resources were provided (one locally-engaged staff to help with contracting, one extra headquarters officer and one coordinator in Jakarta) to put together 101 projects in five years. The 2009 Tsunami Fund Evaluation concluded that “Tremendous unexpected levels of effort were expended at Headquarters during the labour-intensive planning phases and even more so in the field as projects got underway […] The net effect was that while Tsunami programming was delivered, regular programming slowed down.”^{Footnote 18} The CIDA management response agreed that in future cases of “surge capacity”, the increased workload needed to be covered by more operations and maintenance (O&M) funds.

Shifting Canadian development priorities created challenges that held back communications and Program-level meetings with the Government of Indonesia for one and a half years until late 2013. The need to adjust operational projects, or those in the pipeline, to meet changing corporate requirements caused delays, for example with the EGSL project (Environmental Governance and Sustainable Livelihoods) which was approved after multiple adjustments. .

As for the appropriateness of CIDA’s delivery modalities relative to the Indonesian context, the “responsive” modality (that represented 72% of total disbursements from 2005-2013) received strong buy-in from a broad range of stakeholders. Through its responsive programming, in which the projects are identified and designed by the partners, CIDA worked with a range of implementers (private sector companies, non-governmental organizations, universities, multilateral organizations) that had established strong and trusting relationships with Indonesian government, civil society organizations as well as rural and urban communities; had in-depth knowledge of the country’s development policies; and had experience in delivering challenging development projects in the Indonesian context. The “directive” modality, in which CIDA is responsible for project identification and design, accounted for 28% of total disbursements from 2005-2013, and was useful for targeting strategic Program-level objectives, such as public expenditure, anti-corruption, and gender equality.

The Program also supported four Multi-Donor Trust Funds (MTDFs)^{Footnote 19}. The role of MDTFs has been declining in Indonesia either because of donors scaling down aid or offering new aid mechanisms, such as Norway that now ties its aid to the attainment of certain environment protection goals. As a result, there is less coordination of donor initiatives through MDTFs. Due to the proliferation of sub-projects within each of the two PENSA MTDFs (Program for Eastern Indonesia Small and Medium Enterprise Assistance) the economies of scale usually expected from MTDFs did not materialize. Nevertheless, value added was derived from the Java Earthquake MTDF and the Tsunami rehabilitation funds that provided a swift transition mechanism to help communities recover from natural disasters. Green PNPM and the Forest Carbon Partnership Facility helped to leverage CIDA funds and give Canada profile.

Factors contributing to project efficiency included strong management capacities; well managed financial systems; timely training of sub-partners to address weaknesses in standard operating procedures; strong teams of international and local partners; appropriate human resource utilization (match between project needs and knowledge, expertise and personal skills); minimal use of expatriate human resources; and the mobilization of local government resources.

Projects that addressed weak results-based management were able to improve the tracking and reporting of project results, from which best practices and lessons were derived and shared with stakeholders. Local monitoring and evaluation capacity was built when local government officials participated in monitoring CIDA projects. Teams of international and national monitors were often effective as they had both local understanding and international expertise. Joint analysis with local partners helped identify any structural, institutional or cultural problems so projects were appropriately designed, the right types of local partners were involved, risks were identified and managed, and results were replicated or scaled up.

Efficiency concerns included the high proportionate costs for management and international technical expertise. The SUSCLAM project (Tomini Bay Sustainable Coastal Livelihoods and Management) for example, notwithstanding its important achievements, had a rather unwieldy project management structure with responsibilities spread over six locations in three countries. The Program’s other efficiency issues are corporate and included the longer CIDA approval and contracting process for directive projects compared to responsive projects; program-level tracking hampered by shifting priorities, and the need for O&M resources to cover demand for emergency responses to natural disasters.

The Program was able to hire highly qualified and knowledgeable locally-engaged specialists, including two advisors in environment and natural resource management, one harmonization/governance advisor, and a part-time gender equality advisor. However, the Program could have made better use of local experts to monitor projects, encourage cross-fertilization and gather sector results to improve Program-level reporting.

It would have been helpful for the mandate of the Harmonization Advisor in the field to include integrating the projects into a programmatic approach and gathering evidence of sector and Program-level outcomes. The Program would have benefitted from more mid-term reviews and end-of-project evaluations to improve project oversight.

4.3 Aid Effectiveness

The evaluation assessed the Program and sampled projects’ performance in relation to the Paris Declaration’s principles of ownership (partner country sets the poverty reduction strategy, improves its institutions, tackles corruption), alignment (donors align behind local strategies, use local systems), and harmonization (donors coordinate, simplify procedures, reduce duplication).

4.3.1 Ownership

The Government of Indonesia (GOI) has taken on more responsibility for the development process in the country, particularly since 2008 when they first participated in the Third High Level Forum on Aid Effectiveness and took part in the Survey of Paris Declaration Indicators in 2008. CIDA supported the National Development Planning Agency (BAPPENAS) as Donor Focal Point for the conduct of the Survey.

In 2009, the Jakarta Commitment: Aid for Development Effectiveness 2009-2014 was signed by 26 development partners including Canada. This compact laid out a strategic vision or national roadmap for Indonesia and its development partners to implement the Paris Declaration and Accra Agenda for Action by 2014 and also led to the GOI establishing the Aid for Development Effectiveness Secretariat (A4DES). CIDA has been a key participant in supporting the implementation of A4DES and was lauded by both the GOI and other donors for its role in chairing the government-donor working group on aid effectiveness that resulted in the signing of the Jakarta Commitment.

CIDA is supporting the country’s efforts to tackle corruption at different levels, including the national Corruption Eradication Commission through the SIPS project (Islands of Integrity in Sulawesi) to help combat corruption at different levels of government.

4.3.2 Alignment

The Green PNPM project is a good current example of CIDA’s alignment with the Government of Indonesia’s poverty reduction strategy. Indonesia’s National Community Empowerment Program (PNPM) is the world’s largest community empowerment and poverty alleviation program: it operates in every sub-district throughout Indonesia and has mobilized resources from the international donor community. Canada became a donor in 2006 to support a “green” pilot-project focused on natural resource management, environmental conservation, and renewable energy in eight provinces on the Islands of Sulawesi and Sumatra, to demonstrate that there are potential investment opportunities for communities through which they can realize socio-economic benefits while working to conserve Indonesia’s natural resources.

As for alignment with Indonesian Government systems, it is improving but there are still fears of corruption which is being tackled through work with the Corruption Eradication Commission, for example through CIDA’s Islands of Integrity project. Some projects such as SILE/LLD (Supporting Islamic Leadership/Local Leadership for Development) used country systems in the Ministry of Religious Affairs to channel investments to ensure they would be implemented. Another illustration of alignment is provided by the PEACH (Public Expenditure) project that was recognized as a model for improving sub-national public financial management in decentralized Indonesia. As implementation of PEACH moved forward, experience demonstrated that the resulting relationships, data, and momentum formed the basis for other development programs. This information facilitated the alignment of development partners and both national and sub-national governments by providing a common ground to build on.

4.3.3 Harmonization

Donor harmonization in Indonesia was hampered by local political pressures leading to the Government of Indonesia dissolving the Donor Coordinating Group on Indonesia in 2007. In the absence of a formalized donor-government dialogue, Indonesia established the Aid Effectiveness Secretariat (A4DES) in 2009 that included six Working Groups on key issues intended to improve Indonesian government systems supported by international knowledge and best practices. The A4DES Secretariat is striving to improve coordination and avoid duplication. CIDA field staff continue to work with the A4DES and all six Working Groups.

The Indonesia Paris Declaration report stated that: “…harmonization is another hard principle to implement. Every development partner has its own mission to carry out, and it may significantly differ (sic) with others. Having such challenges, the current quality of harmonization could be considered as a (sic) significant achievement. The government’s leadership in coordination has improved according to some development partners. Still, it has not been proactive enough in leading strategic coordination and using development partners’ comparative advantages.”^{Footnote 20}

In spite of this, CIDA and other donors harmonized their efforts to support Indonesia’s decentralization efforts. Canada chaired the Decentralization Committee from 2004-2009, under which four donors (Canada, Australia, Germany and the Unites States) worked out a common Monitoring and Evaluation Framework with indicators for tracking progress of Indonesia’s decentralization process. Over the years, CIDA staff have been the chairs/co-chairs or members of 14 different donor-coordinating committees related to the National Development Planning Agency, the Ministry of Finance, Aid Effectiveness, South-South Cooperation, and others.

Donor harmonization was accomplished most effectively and efficiently through the Multi-Donor Trust Funds (MTDFs) to which Canada contributed. A notable example was the quick response the donor community made after the Java Earthquake in May 2006. The Java Earthquake MDTF provided a forum for policy dialogue between donors and the Government of Indonesia on the effectiveness and sustainability of rehabilitation and reconstruction efforts. The Government of Indonesia wanted to expand the MDTF to include housing and livelihood support to the survivors of the earthquake and tsunami in West Java that occurred two months later in July 2006. In the interests of harmonization, all donors to the Java Earthquake MDFT, including Canada, agreed to support this expansion to the tsunami-stricken West Java area. The other MTDF was the Forest Carbon Partnership Facility, a global partnership of 36 developing countries and 18 financial contributors (including developed countries and businesses) engaged towards a common goal to reduce climate change.

The Program’s Environment Working Group was an “internal” example of efforts to share information, avoid duplication of efforts and create synergies in the delivery of CIDA’s environment projects in Indonesia. The Working Group benefitted the Program by bringing together CIDA partners and executing agencies who may have otherwise implemented their projects in isolation from one another. The Environment Working Group has not been meeting in the past few years but interest was expressed in reviving it for the next programming period.

4.4 Performance Management

The evaluation examined whether the Program and the sampled projects employed appropriate results-based management, risk management, and monitoring and evaluation.

The Program prepared Performance Measurement Frameworks for both Country Development Programming Frameworks (CDPFs). The CDPF 2004-2009 included a Strategic Results Framework and a Results-based Management and Accountability Framework, which was a CIDA instrument of performance measurement at the time but was subsequently abandoned.

For the CDPF 2009-2014, a draft Performance Measurement Framework (PMF) was prepared in 2009 and revised in 2012 but never completed or reported on, hence its baseline data and targets are still listed as “to be determined”. There were shortcomings in CIDA’s corporate strategy for Program-level reporting, at the time, which required Programs to report on the notional results of their Country Strategies, rather than on expected outcomes from the logic model contained in the CDPF. This limited the Program’s ability to use the PMF as the tool for Program-level reporting, placing emphasis on project-level reporting instead.

The Country Development Programming Framework 2004-2009 had noted that an important issue to be addressed was how to develop a Monitoring and Evaluation (M&E) system that could support a regionally focused program, rather than building isolated project-specific M&E frameworks. At that time, the Program believed there was an opportunity to build on the Government of Indonesia’s systems rather than creating overlapping or competing machineries. This approach would have included CIDA’s efforts to strengthen the capacity of local governments to build and improve monitoring mechanisms that local administrations could use to plan and deliver services. There is no evidence that this was accomplished over the past two programming periods.

At the project level, capacity for local monitoring and evaluation (M&E) was reinforced when local government officials participated in monitoring missions with CIDA project team leaders. There was also evidence of joint performance management activities, notably when AGFOR (Agroforestry and Forestry in Sulawesi) and EGSL (Environmental Governance and Sustainable Livelihoods) that participated in a joint monitoring mission with the Program’s staff to enhance cooperation between the two projects. EGSL is also an example of how mutual accountability mechanisms were used to build local M&E capacity at the project level. EGSL supported systematic public audits and periodic community meetings, which provided for a continuing exercise in participatory monitoring of the project’s outcomes with and by stakeholders at the village-level and at the watershed-level.

Overall, the projects’ performance management was assessed as satisfactory, however monitoring and evaluation at project-level could have been stronger. There was a noticeable lack of dedicated project monitors and too few mid-term reviews and end-of-project evaluations. For the 32 sampled projects, there were six mid-term reviews and twelve end-of-project evaluations. Monitoring missions by CIDA headquarters/field staff and sector specialists have decreased during 2005-2013. Budget reductions have also limited staff travel, decreasing project monitoring.

Although the Program ensured all bilateral projects used the Results-Based Management (RBM) approach and tools as per CIDA guidelines, there were challenges. The RBM concepts, principles, and processes were not easily understood by some executing agencies and Indonesian stakeholders; articulating good results statements and performance indicators was a challenge for most stakeholders; and baseline data was often unavailable. Executing agencies/partners were challenged over the past eight years to understand and adjust to changes in CIDA’s RBM terminology, approaches and reporting requirements.

In terms of risk management there was a well-planned Program-level strategy but the evaluation team did not find evidence that it was updated or used. At the project level the risk strategies varied: some project risk strategies were well done but not updated regularly and several projects did not fully integrate Indonesian partners into developing the risk and mitigation plans or in monitoring the strategies.

A few projects received a highly satisfactory rating. SIPS (Islands of Integrity) fully involved partners in its RBM approach and in the design and implementation of an exceptional risk management system. RCL (Restoring Coastal Livelihoods) enriched the RBM process by applying the “Most Significant Change” methodology (a form of participatory monitoring and evaluation) and also devised excellent risk assessments and appropriate mitigating strategies. Some projects managed to improve initial weakness, such as SILE/LLD (Supporting Islamic Leadership in Indonesia/Local Leadership for Development) by hiring an RBM consultant to improve the RBM system and deliver training for partners, and PEACH (Public Expenditure) by setting-up a mid-term workshop to review results and plan improvements to the project.

The SUSCLAM (Tomini Bay Sustainable Coastal Livelihoods and Management) and World Wildlife Fund (Outreach, Education and Natural Resource Management) projects were weak in their use of baseline data and risk management assessments. Governance Reform Support II had a weak logical framework analysis and performance measurement framework that made it difficult to report on results. The risk assessment and mitigation efforts for PENSA-1 (Program for Eastern Indonesia Small and Medium Enterprise Assistance) were inadequate, but three independent reviews led to corrections that resulted in improvements to risk management in PENSA-2.

5. Conclusions and Lessons

The Program was aligned with Canada’s aid priorities and Indonesia’s development needs. Its management was generally efficient and supported aid effectiveness principles. Substantial evidence points to the achievement of project-level outcomes, including successful examples of policy dialogue. Partners have underlined the relevance and usefulness of the Program’s geographic focus on Sulawesi for building strong relationships at the sub-national level and developing approaches that could be replicated in other parts of the country. The Program has been willing to take risks by supporting sensitive initiatives such as the Corruption Eradication Commission, and overcoming resistance to gender equality.

Program-level reporting on the achievement of outcomes could have been stronger. Shortcomings in CIDA’s corporate approach to Program-level reporting over the review period have limited the Program’s ability to report on its Program-level results, placing emphasis on project level reporting instead. Further, in future cases of “surge capacity” (such as in cases of emergency assistance in disaster prone countries), the increased workload in the field needs to be covered by more operations and maintenance funds so as not to adversely impact the management of regular programming.

Building local government capacity where needs are greatest, combined with support for strengthening civil society organizations and for empowering communities to influence government, remains a relevant program strategy. As a lower middle-income country, Indonesia’s needs and reality are evolving quickly. Decentralization on the scale and at the pace that it took place in Indonesia does not automatically lead to sustainable and equitable growth. Focus will have to be maintained on piloting reforms and replicating or scaling up successes. This requires a high level of policy dialogue and government buy-in.

With significant regional disparities in Indonesia, low capacity in the regions, unclear government regulations and lack of coordination among different levels of government, it is challenging to ensure the sustainability of CIDA’s program. The Program worked best when it included local buy-in from the highest level of authority in government (minister at national level or governor, mayor or regent at sub-national levels); focused on multiple levels of government (national, provincial and district); included national and local civil society partners in its initiatives; and when successful initiatives piloted in Sulawesi were replicated and scaled-up nationwide. Project level sustainability was enhanced by extended implementation timeframes, strong ownership and institutional capacity, and adequate funding resources of local implementing partners. There is additional scope for replication of initiatives nationally and in other provinces, and increased coordination and sharing of information among projects working to achieve similar results

Anti-corruption, accountability and transparency are important elements of integrating governance in all projects given the context of Indonesia’s decentralization process. Indonesia’s national Corruption Eradication Commission is a success story worth communicating to other DFATD programs to encourage sharing of best practices in this field.

In private sector development, risk management appears to have been weaker than in other sectors in the face of natural disasters and market price variations which threatened results obtained by small and medium enterprises (SMEs) that depended on natural resources. Community-based approaches to small and medium enterprise (SME) development were more powerful in mobilizing communities through the empowerment of women and involvement of local governments than approaches that used foreign expertise to promote SME development.

Environmental sustainability was prominent as a cross-cutting theme in the two CDPFs covering the evaluation period. It was integrated into all programming sectors and was relevant to the sustainable management of natural resources. Environment projects were challenged by insufficient timeframes, the lack of reliable and sustainable market linkages for SMEs that are dependent on the natural resource base for their livelihoods, and recurring natural disasters.

Improved integration of the crosscutting themes of gender equality and environmental sustainability into the Program’s sectors of focus remains key. Enhanced tracking of the environment outcomes of private sector development investments would help enhance environmental sustainability as well.

The Program had gender equality (GE) strategies and some gender-specific projects. However, the Program’s gender mainstreaming commitment was less evident in the 2009-2014 programming period that emphasized economic empowerment, than in the earlier 2004-2009 period where the focus on governance and human rights helped to provide a stronger enabling environment for GE results. The governance projects showed a satisfactory understanding of GE in terms of increasing gender understanding within different government levels and civil society, but there were significant weaknesses in the GE performance of private sector development projects. The workload is large for the Program’s part-time Jakarta-based GE advisor especially in the context of Indonesia’s decentralization given that gender capacity is generally weaker at the sub-national levels.

6. Recommendations

1. Gender Equality

Canada has earned a justified reputation for constructive long term engagement on gender equality through its development assistance program in Indonesia. It should build on this engagement, and in particular ensure that Sustainable Economic Growth programming is undertaken with sufficiently detailed gender analysis, target setting, and performance measurement.

2. Environmental Sustainability

Given its geographic circumstances and demographic pressures, appropriate environmental stewardship will be particularly important for sustainable long term growth and poverty reduction in Indonesia. Canada and other donors continue to pay considerable attention to this theme. The Program could improve impact by linking their individual efforts, taking an integrated programmatic approach with appropriate policy dialogue objectives, and ensuring that sustainable economic growth initiatives have explicit environmental sustainability objectives and sufficient resources and timeframes to achieve them.

3. Transparency, Accountability, Enabling Environment for Business

Through its geographic focus in Sulawesi, and emphasis on building capacity at provincial and district levels, the Program has achieved improvements in public sector management and civil society engagement that have been recognised and replicated nationally. In its new emphasis on Sustainable Economic Growth (SEG), a key challenge will be to improve the enabling environment for business, including through transparent and efficient regulation. The Program should leverage its past success in governance, by incorporating specific transparency, accountability, and anti-corruption objectives in its SEG programming.

4. Performance Management

There are examples of very good performance management practices in some projects, and a Program level Performance Measurement Framework (PMF) was well articulated although not regularly tracked and reported on. There is scope for moving further in these areas. Specifically, the Program should:

(a) At the project level, make more systematic use of baselines, project monitoring and evaluation over the next CDPF period, setting targets for frequency, and allocating sufficient resources to achieve them.

(b) At the program level, calibrate a PMF to the next CDPF, with indicators that can be tracked and reported on to show sector and national level progress.

(c) Increase the use of already well-articulated risk management frameworks as management tools. Set specific targets and measurable indicators for policy dialogue activities, and report on them.

(d) As part of the longer term mutual accountability agenda, seek opportunities to develop partner capacity and involve local experts in monitoring and evaluation.

Appendix A – Summary Terms of Reference

The Country Program Evaluation will have the following objectives: (1) To take stock of the results achieved by the Indonesia Program of CIDA's Geographic Program Branch (GPB) over the seven-year period from FY 2005-2006 to FY 2012-2013, in relation to the objectives and expected results as approved in the Country Strategy of 2009 and the Country Development Programming Frameworks of 2004-2009 and 2009-2014^{Footnote 21}. (2) To assess the Indonesia Program's overall performance in achieving these results. (3) To document and disseminate findings and lessons learned, and formulate recommendations to improve the performance of the current or future Country Program Strategy and Country Development Programming Framework.

This evaluation will cover eight years, covering as much as possible the Country Program Strategy of 2009 and the Country Development Programming Framework (CDPF) of 2009-2014, and extending back to the previous CDPF of the 2004-2009 programming period that had not been evaluated in the past. It is understood that the priorities of the Indonesia Program have been evolving over the evaluation period. This will be taken into account during the evaluation process during which all efforts will be made to document the specific context for each period, and to do justice to the complexity and evolving nature of the CIDA programming cycles.

For the period covered by the CDPF 2004-2009, the Evaluation will take a summative evaluation approach, emphasizing results achieved and performance in achieving these results, for accountability and learning purposes. For the period covered by the CPS 2009 and the CDPF 2009-2014, a formative evaluation approach will be used to provide guidance to the Program for the remaining CDPF period. The evaluation will not review projects implemented in response to the 2004 tsunami, because the Tsunami Reconstruction and Rehabilitation Fund was evaluated in 2009. That said, the evaluation should examine the impact of this major undertaking (which more than doubled the Indonesia Program's annual disbursements) on the Program's staff and operations both at headquarters and in the field.

The three main sectors of disbursements will be assessed, namely: (1) democratic governance with a focus on decentralization; (2) private sector development with a focus on small and medium enterprises; and (3) environment with a focus on natural resource management for sustainable livelihoods. In addition, the evaluation will assess the two crosscutting themes^{Footnote 22} of gender equality and environmental sustainability. This evaluation will not focus on programming areas that represented less than 10% of the Program's disbursements to Indonesia, such as improving health, basic education, peace and security, and emergency assistance.

Appendix B – Evaluation Matrix

PROGRAM-Level Indicators

Relevance

Overall relevance of the Program including policy dialogue in relation to:

• The partner country's development context and needs
• CIDA's policies and the Government of Canada's interests and foreign policy objectives
• The evolving nature of Canadian and CIDA policy environment

Effectiveness

Overall extent to which the Program in achieving (or is expected to achieve) its outcome-level results in relation to:

• Poverty reduction
• Strategic objectives for the country (CDPF) or Canadian foreign policy
• Sector/thematic or cross-cutting objectives

Sustainability

Overall Program sustainability in relation to:

• Assessment of the partner country's financial capability (ODA/GNP, ODA/Budget)
• Partner country's policy environment and reform context (political, social, economic, administrative, environmental)
• Predictability of Program's ODA resources (3-5 year forward information on planned aid to partner countries) per Accra Agenda

Crosscutting Themes (Gender Equality and Environmental Sustainability)

Treatment of gender equality and environmental sustainability as crosscutting themes (CCI), at Program level in relation to:

• Place given to the CCI in the CDPF
• Existence of a strategy regarding the CCI
• Existence of monitor or dedicated specialist
• Existence of a Fund or dedicated resources
• Role of ClDA/Canada on this issue in the context of donor coordination

Coherence

Coherence at the Program level in relation to:

• Cooperation channels (bilateral, multilateral, partnership)
• Development actors (government departments, NGOs, international orgs)
• External coherence in terms of consistency with programs of other donors (donor coordination)
• Coherence with Canada's whole-of-government efforts, including Program’s policy dialogue, leverage, and leadership played by Canada

Efficiency

Cost-efficiency at Program level in relation to:

• The appropriateness of the Program's delivery modalities relative to the partner country's context
• Delivery models (directive, responsive, PBAs)
• Program's O&M/G&C compared to other comparable CIDA Programs
• Benchmarking with other donors (i.e. staff, O&M costs, decision making, knowledge related activities)
• Strengthened field presence

Aid Effectiveness

Overall performance at the Program level in relation to Paris Declaration Principles of:

• Ownership (developing countries set their own strategies for poverty reduction, improve their institutions, tackle corruption)
• Alignment (donor countries align behind these objectives and use local systems)
• Harmonization (donors coordinate, simplify procedures, share info to avoid duplication)

Performance Management

Performance management at Program level:

• Results Management per CIDA guidelines
• Roll up & measure development results
• Apply lessons learned & best practices
• Follow-up on recommendations of previous evaluations
• Risk management per CIDA guidelines
• CIDA monitoring & evaluation (M&E)
• Joint M&E activities with other donors (or division of labour between donors)
• Mutual accountability mechanisms or activities to build local M&E capacity

PROJECT-Level Indicators

Relevance

Relevance in relation to:

• Country needs: Country development plans (PRSP), Millennium Development Goals (MDGs)
• CIDA policy: Poverty Reduction, Sustainable Development
• Canada's priorities: CIDA Country Strategy (CDPF), foreign policy objectives (DFAIT)

Effectiveness

Effectiveness in achieving results at the outcome level (and impact if available based on secondary data) in relation to:

• Poverty reduction
• Project sector/institutional objectives
• Cost-effectiveness per result unit

Sustainability

Sustainability in relation to:

• Time needed to attain the results (life of the project: 3-5 years)
• Institutional capacity to maintain the results (after the project)
• Financial capacity to sustain the results (after the project)
• Policy environment
• Ownership/commitment

Crosscutting Themes (Gender Equality and Environmental Sustainability)

Treatment of gender equality and environmental sustainability as crosscutting themes (CCI) in relation to:

• Policy dialogue
• Preliminary analysis (disaggregated data)
• Implementation of activities
• Results achieved: (i) Institutional strengthening;(ii) Improving quality of life
• Resources invested (budget)

Coherence

Coherence in relation to:

• Internal coherence within CIDA (delivery models/mechanisms and channels)
• External coherence in the context of international efforts (donor coordination)
• Coherence in the context of Canadian whole-of- Government efforts

Efficiency

Cost-efficiency in relation to transaction costs:

• Efficiency in use of human resources
• Efficiency in use of financial resources
• Time needed to approve, manage, and monitor

Aid Effectiveness

Performance in relation to Paris Declaration principles of:

• Ownership (developing countries set their own strategies for poverty reduction, improve their institutions, tackle corruption)
• Alignment (donor countries align behind these objectives and use local systems)
• Harmonization (donor coordinate, simplify procedures, share info to avoid duplication)

Performance Management

Performance management:

• Results-Based Management per CIDA guidelines
• Risk management per CIDA guidelines
• CIDA monitoring & evaluation (M&E)
• Branch-led M&E activities at the project/sector/institutional level
• Joint M&E activities with other donors or division of labour between donors
• Mutual accountability mechanisms or activities to build local M&E capacity
• Timely corrective action

Appendix C – Indonesia Facts at a Glance

Sources: Statistics from World Bank, Asian Development Bank, OECD, UN Data, and Statistics Indonesia-Government of Indonesia.

Appendix D – Other Donor Programs in Indonesia

 Sources: OECD, www.oecd.org, and World Bank, www.worldbank.org.

Appendix E – Indonesia Country Program Logic Model

List of Operational Projects:

• BASICS – Better Approaches to Service Provision through Increased Capacities in Sulawesi (A032129)
• CIPSED – Canada-Indonesia Private Sector Enterprise Development (A032645)
• EGSLP– Environmental Governance and Sustainable Livelihoods (A031866)
• Green PNPM– Green National Community Empowerment Program (PNPM is the Indonesian acronym for Program Nasional Pemberdayaan Masyarakat (A033048)
• GRS II – Governance Reform Support II (A031155)
• LLD – Local Leadership for Development (A033116)
• PEACH – Improved Public Financial Management by Local Governments (A034002)
• RCL – Restoring Coastal Livelihoods (Mangroves project) (A034028)
• SIPS – Support to Indonesia's Islands of Integrity Program for Sulawesi (A033493)
• SUSCLAM – Tomini Bay Sustainable Coastal Livelihoods and Management (A032581)
• WWF/Lestari – Strengthening the Outreach and Education Networks for   Governance (implemented by World Wildlife Fund Indonesia and Lestari Indonesia)(A032458)

List of Planned Projects

• BBB – Building Better Budgets for Women and the Poor (A034433)
• ICRAF/CIFOR – Integrated Agroforestry and Forestry (A034792)
• LED-PP – Local Economic Development Planning and Programming (A034946)
• SINRG– Sulawesi Initiative for Natural Resource Governance (A034273)

Other Program Activities (e.g. A.E support activities by CIDA staff such as policy dialogue, donor coordination, analysis, crosscutting themes):

• Technical Assistance for Aid Effectiveness
• Policy dialogue through partners’ meetings and joint donor groups on both decentralization and environment

Appendix F – Project Sample

Source: CIDA Chief Financial Officer Branch (Statistical Unit) – Data as of 2013-04-19.

NOTE #1 – Adjustments made by the Evaluation Team to the coding of the sampled projects
to distribute the sector portfolios more evenly among the team members

Projects moved from PSD to Environment

• A031169 001 Climate Change Peatlands
• A034792 001 Agroforestry and Forestry in Sulawesi
• M01344 001 Forest Carbon Partnership Facility

Projects moved from governance to PSD

• A032448 001 Improving the Business Environment for SMEs in Sulawesi (IBESS). This project includes under the same contribution agreement the: Gender Responsive Budgeting Initiative (GRBI)
• A032645 001 Canada-Indonesia Private Sector Enterprise Development
• A035355 001 Indonesia SME Assistance (PENSA-2), (added to sample)

Project moved from Environment to Governance

• S064671 PRG CUSO-VSO 2009-2014 Program
• Project moved from Education to Governance
• A030584 001 Islamic Social Equity

Project moved from Health to Environment

• A031516 001 Water and Sanitation

Project moved from Emergency Assistance to Environment

• A033390 001 Java Earthquake Multi Donor Trust Fund

Projects moved from Governance to Environment

• S063452 PRG OCCDP - Dev. & Paix Program 2006-2011

Projects moved from Governance to Other

• A032129 001 Better Approaches to Service Provision (BASICS)
• A034433 001 Building Better Budgets for Women and the Poor.

Appendix G – Summary of Project Ratings by Sector and Evaluation Criteria

Appendix H – List of Organizations and Stakeholders interviewed during the Evaluation

** CIDA HQ – 24 and CIDA Post 26 = 50 (Actual 35 since Program level were the same people).

Other Donors: 7 donors attending donor roundtable included: World Bank, International Monetary Fund (IMF), Asian Development Bank (ADB), United Nations Development Program (UNDP), United Nations Children Fund (UNICEF), Australian Agency for International Development (AusAID), and New Zealand Aid Program (NZAID). In addition, separate meetings took place with some of these donors including with United States Agency for International Development (USAID) and the German Agency for Technical (International) Cooperation.

Implementing Organizations included senior representatives of 12 active projects in one to three locations in Sulawesi, in Jakarta or in Ottawa: Agroforestry and Forestry in Sulawesi Project (AGFOR), Building Better Budgets for Women and the Poor (B3WP), Better Approaches to Service Provision through Increased Capacities in Sulawesi (BASICS), Canada-Indonesia Private Sector Enterprise Development Project (CIPSED), Environmental Governance and Sustainable Livelihoods Project (EGSP), Green Community Development in Sulawesi (Green PNPM), Public Expenditure and Capacity Harmonization Program (PEACH), Program for Eastern Indonesia Small and Medium Enterprise Assistance (PENSA 2), Restoring Coastal Livelihoods (RCL), Supporting Islamic Leadership in Indonesia/Local Leadership for Development (SILE/LLD), Support to Islands of Integrity (SIPS), Cuso International (CUSO and VSO – Voluntary Service Organization), Canadian Cooperative Association (CCA) and Forest Carbon Partnership Facility (FCPF). In addition, interviews took place with representatives of closed projects including: Sulawesi Water and Sanitation Hygiene Project (SWASH), Eastern Indonesia Knowledge Center (BAKTI), Indonesian Civil Society Alliance for Democracy (YAPPIKA), Tomini Bay Sustainable Coastal Livelihoods and Management Project (SUSCLAM), Improving Business Environment for Small and Medium Enterprises in Sulawesi (IBESS), Gender-Responsive Planning and Budgeting (GRBI), and one start-up project Sulawesi Economic Development Strategy (SEDS). In summary, 28 of 55 implementing organizations were met.

Government of Indonesia and other Indonesian Partners included multiple national and local representatives of the National Development Planning Agency (BAPPENAS), Provincial or District/City Development Planning Agency (BAPPEDA), Regional Legislative Assembly, Ministry for Women’s Empowerment and Child Protection, State Ministry for Cooperatives and medium and Small Enterprises, Ministry of Finance, Rural Bank, Statistics Indonesia, Ministry of Home Affairs, University Alauddin, Haluleo University, Government Inspectorates, Government of Indonesia in Konawe, Unaaha Bantaeng and Tompobulu Districts. In addition, focus groups took place with beneficiary groups focused on: cashew nuts, solar power under Green PNPM project, fish farming around restored mangrove areas, and cash crops including mangos, charcoal, and cacao.

In addition, beneficiaries of the 12 active projects (noted above under implementing organizations) were met in multiple locations in South Sulawesi, Southeast Sulawesi and North Sulawesi. The map at the front of this evaluation notes which projects operated in each of these three provinces. Where there were multiple locations in which projects operated in Sulawesi, visits were made to all three locations, or at least two of the three locations.

Appendix I – Selected Bibliography

Key & example documents selected from 510 reviewed documents noted in Technical Report

ADB Institute Discussion Paper No. 46 Policies for Private Sector Development in Indonesia Thee Kian Wie, March 2006.

CIDA - Background Profile, Indonesia Program Evaluation, Annex 7 – Disbursements to Indonesia by Investment Type and Fiscal Year for 2005-2006–2012-2013, April 2013.

CIDA, Country Development Programming Framework 2009-2014, Dec. 2009

CIDA, Country Development Programming Framework 2004-2009, March, 2004.

CIDA Indonesia Program Annual Country Report 2011-2012, Geographic Programs Branch, Sept. 2012.

CIDA Indonesia Environment Program Review, Executive Report, 2002.

CIDA Policy on Gender Equality, 1999, updated in Gender Equality Policy and Tools, 2010, CIDA.

CIDA Policy for Human Rights, Democratization and Good Governance, December, 1996.

CIDA Policy on Private Sector Development, CIDA, July 2003.

CIPSED, End of Project Report for the Period June 2008 to April 2013, Whyte Reynolds International Inc, April 2013.

Country Diagnostics Studies, Indonesia: Critical Development Constraints, Asian Development Bank (ADB), International Labour Organization (ILO), and Islamic Development Bank (IDB), 2010.

Doing Business in Indonesia 2012, Comparing regulation for domestic firms in 20 cities and 183 economies The World Bank and the International Finance Corporation, 2012. www.doingbusiness.org/subnational.

Executive Summary Picking up the Pace: Reviving Growth in Indonesia’s Manufacturing Sector, The World Bank Office in Jakarta, September 2012.

End of Project Evaluations: Improving the Business Environment in Indonesia (IBESS) and Gender Responsive Budget Initiative in Sulawesi (GRBI), Michael Hatton, October, 2008.

Expanding Opportunities through Private Sector Development: Canada Making a Difference in the World. CIDA, July 2003.

Government of Indonesia and OECD, Joint Evaluation of the Paris Declaration, Phase 2, Final Report, August 2011.

Government of Indonesia National Medium Term Development Plan (RPJM 2010-2014). 

Government of Canada, Indonesia Way Forward T-24127, December 2012.

Indonesian Women’s Empowerment, Dr. Elly Sinaga, PPT Tokyo, 4th of September, 8th APEC Transportation Ministerial Meeting 2013.

Jakarta Commitment on Aid Effectiveness, October 21, 2009, Jakarta.

Ministry of Finance Regulation No.119/2009 on Guidelines for the Preparation and Review of Ministry/Agency Work Plans and Budgets  for Fiscal Year 2010.

PENSA Mid-Term Review Final Report, Nexus Associates, Nov. 2006, p. 12. Studies, Dec. 2008.

PENSA-2 End of Facility Review, Final Report, Oxford Policy Management, Vol. 1; Vol. 2 Case studies; June 2013.

Proceedings of a Program Coordination Committee Meeting: The 6th PCC Meeting “Stimulating Sustainable Economic Growth (SEG) in Partnership with National & Regional Governments in Sulawesi Challenges & Accomplishments, Hotel Imperial Aryaduta Makassar, 25 October 2011.

Review of Country Partnership Strategy Completion Report for FY 09 – FY 12 period, Independent Evaluation Group, World Bank.

The Gender Action Plan At-A-Glance - http://www.wunrn.com/news/2008/04_08/04_21_08/042108_world.htm

UNDP’s Gender-related Development Index (GDI),

Source: http://hdr.undp.org/en/statistics/indices/gdi_gem/ accessed 2012-05-16).

United Nations 2013 Fact Sheet, MDG Goal 7: Ensure Environmental Sustainability.

Vinay Kumar Bhargava, E. P. (2004). Challenging corruption in Asia: case studies and a framework for action. World Bank Publications.

Worldwide Governance Indicators, World Bank: http://info.worldbank.org/governance/wgi/pdf/c170.pdf

Appendix J – Management Response

This management response reflects the departmental priorities as expressed at the end of the evaluation period. Going forward, management responses will reflect new Departmental priorities as they evolve.