Executive Summary – ACORD Pan-Africa Agricultural Program

Summary

Overview, Rational and Purpose of the Evaluation

The summative evaluation of the Pan-African Agricultural Program (PAAP), implemented by the Agency for Cooperation and Research in Development (ACORD), was commissioned to strengthen local partner and Global Affairs Canada knowledge of the subject and enhance the mutual accountability of both partners.

Specific Objectives of the Evaluation

The specific objectives, of the evaluation, were to assess effectiveness, efficiency, relevance and sustainability of results achieved; assess results achieved for criteria of gender equality, environmental sustainability and governance; and provide findings; conclusions, recommendations and lessons learned that could inform the design and implementation of similar types of projects.

Context for Development Intervention

Global Affairs Canada’s objective for PAAP was to respond to the food crisis by supporting the African Union’s Comprehensive African Agricultural Development Program (CAADP) and Pastoralists Policy Framework (PPF), for Africa. Global Affairs Canada’s support through PAAP enabled ACORD to increase focus on strengthening community-based groups (CBO’s) and civil society organizations (CSO’s) capacity to better engage in CAADP/PPF processes with their own national governments and CAADP partner platforms.

Evaluation Object

PAAP was implemented through the ACORD Secretariat (in Nairobi) and in 8 country programs. The Contribution Agreement (CA) for PAAP (signed 23 March 2010) had a total budget of $3,250,000.00 (including $500,000 from ACORD) for the three years duration (March 2010 to April 2013). The project received two time-based no-cost extensions for a total of six months ending the project in September 2013. Intervention Logic: The Ultimate Outcome of PAAP was to contribute to increased food security and incomes among small-scale farmers and pastoralists, especially women, in Sub-Saharan Africa with regional focus on East, Horn and Sahel region. PAAP had three Intermediate Outcomes that addressed: 1) Increased utilization of improved agricultural practices and access to markets by small-scale farmers and pastoralist’s especially women; 2) Increased representation of small-scale farmers and pastoralists organizations (CSO’s) in the CAADP/PPF implementation process; and, 3) Improved functioning of ACORD’s Pan-African Agricultural Program. The Seven Immediate outcomes for the three respective intermediate outcomes were as follows:^{Footnote 1}.

• 1.1 Improved knowledge of livestock and crop production methods, sustainable land and water management practices, and existing agricultural policy frameworks by small-scale farmers and pastoralists.
• 1.2 Increased skills of small-scale farmers and pastoralists to access markets.
• 2.1 Increased capacity of small-scale farmer and pastoralist organizations to develop and disseminate agricultural information and practices to their members (women and men).
• 2.2 Strengthened capacity of small-scale farmer and pastoralist organizations to engage in CAADP and AU PPF processes.
• 3.1 Improved knowledge and capacity within ACORD in key management functions.
• 3.2 Improved knowledge management capacity and systems and processes of accountability.
• 3.3 Increased capacity for gender-sensitive monitoring and evaluation.

Methodology Used

The evaluation used a theory of change approach with emphasis on the intervention logic for PAAP. A mixed method design that used both qualitative and quantitative approaches to data collection and verification was used to meet the requirements and expectations set out in the terms of reference. An evaluation matrix was designed to systemize the methodology through key questions and sub-questions. The three main sources of data used for the evaluation were: project documents and files; beneficiaries in Canada, Kenya and the 17 country programs; and the field visits to 8 project sites in Kenya, Tanzania and Burundi. Reliability of data was good as data provided by ACORD and Global Affairs Canada was triangulated and cross-validated through on-site interviews and observations of actual outcomes. One limitation encountered during the field mission was that, two years after the completion of the main activities, there was some loss of corporate memory at ACORD and movement of some direct beneficiaries of PAAP to other positions. They were thus not able to contribute to the findings. Stakeholders: ACORD as the executing agency of the initiative and Global Affairs Canada as the key contributor as well as the AU on the role of Non-State Actors in the implementation of the CAADP and PPF framework.

Findings by Key Questions

Findings are presented for each of the 13 key questions (presented in the Evaluation Matrix and the Terms of Reference) for the 3 Intermediate Outcomes and their 7 respective immediate outcomes (presented above).

1) 1.0 Effectiveness

Question 1.1

Has development intervention achieved the expected immediate and intermediate outcomes and made progress towards the ultimate outcome? PAAP was able to reach the targeted number of beneficiaries in the 8 country programs, as ACORD already had a history of working with local farmer/pastoralist groups and CSO’s, in March 2010 when the project commenced. The project contributed to increase household food security in the targeted rural communities through the use of locally based CSO’s as intermediaries. Though the set numeric targets was met, the implementation of the planned interventions would have benefitted from a more systematic needs assessment of beneficiary capacity, skill gaps and attention to longer-term sustainability of the interventions to meet the duel PAAP objectives of increased food security and incomes. The planning of marketing strategies was weak as smallholder farmer groups identified access to markets and post-harvest losses as a major constraint. The application of value-chain analysis beyond production would have benefitted from more attention at the planning stage of the intervention. The linkages with non-farm level actors to support farmers were minimal, leading to reduction in financial benefits to farming communities as post harvest losses were estimated to be around 35 to 40% in some project supported areas.

PAAP was effective in supporting and facilitating the work of locally based CSO’s who lived and worked directly with the rural communities to improve farm productivity. ACORD staff accompanied selected CSO representatives from local, national and regional CSO’s to the various CAADP partner platforms to increase their knowledge and capacity to engage in CAADP and AU processes. ACORD’s key management functions were effectively targeted to improve its internal financial management and reporting systems, human resource policies and strategies for gender equality. The Monitoring and Evaluation (M&E) position was, however, not adequately resourced as planned and, this weakness was reflected in the quality of the M&E systems and progress reports, which tracked delivery of activities and not outcomes. Monitoring of gender inclusiveness was good and this remains a strong suite of ACORD as an organization.

Question 1.2

From the point of view of target communities and organizations, has the development intervention achieved the expected outcomes? The PAAP intervention was effective in working with local CSO’s and in bringing communities together and making them work collaboratively to improve their livelihoods through increased productivity at farm level. Communities did not get sufficient assistance in dealing with the cycle of seasonal surpluses, especially when all community members were growing and processing similar crops.

2) Efficiency

Question 2.1

How economically were resource/inputs (funds, expertise, time, etc.) converted to outputs? ACORD was somewhat efficient in using the grant to support all three intermediate outcomes in year 1 and 2. By December 2012, the project started to wrap up, resulting in the curtailment of field level activities for some farmer/ pastoralist groups and CSO’s. The final expense including the audit adjustments was $2,707,965, which was below the Global Affairs Canada allocation of $2,750,000 provided in the Contribution Agreement (CA). In line with the Contribution Agreement, ACORD contributed its share of $500,000 towards the balance of the remuneration costs of the staff positions at the Secretariat. The review of line item expenditures for Year 1 and Year 2 showed that the burn-rate of certain line items was very low (M&E position was grossly underspent against plan) and the budget for the Financial Officer, and Workshops/travel, was overdrawn (exceeding the total CA allocation for three years) by the end of March 2012 (Year 2). In late March 2013, ACORD revised the CA budget (approved by Global Affairs Canada on 3 June) to reallocate the underspent lines to cover the over budget lines to ensure that all line items were within the 10% variance allowed.

Question 2.2

Were outputs achieved on time and on budget? Looking at the implementation of PAAP by quarter, the actual costs against plans were lower by an average of 35% for the first 21 months. The project was completed within the overall CA budget. Reporting was for the most part on time. The quality of the financial reporting was weak as it tracked variance of expenditures in each quarter and not on an annual or cumulative basis. The Excel file contained the variance but the actual narrative reporting did not highlight or explain the rationale or reasons for being over/under budget.

3) Relevance

Question 3.1

To what extent did ACORD/PAAP results meet the needs of the target communities and national development and research programs? All three intermediate outcomes were highly relevant to the needs of the targeted community groups, CSO’s and to ACORD as a pan-African developmental organization.

4) Sustainability

Question 4.1

What is the likelihood that results/benefits will continue after the Global Affairs Canada involvement ends? Over two-thirds of PAAP activities are still operational for intermediate outcomes 1, 2 and 3. The likelihood of benefits continuing is high as ACORD and the country programs have institutionalized and built on work implemented under PAAP. Participation of national and regional CSO’s in CAADP forums has slowed considerably due to availability of committed funds but ACORD plans to continue the work through its Pan-African Program in future.

Question 4.2

Are there committed financial and human resources to maintain benefits and results? ACORD is a viable concern with a diversified portfolio of funding partners. It is applying the funds raised to continue many initiatives started under PAAP albeit not with the same momentum. The likelihood that these activities will continue beyond PAAP is high, as they have essentially become part of ACORD’s organizational functions both at the Secretariat and in the country programs. The Monitoring and Evaluation function is being upgraded to manager level and is expected to address some of the weaknesses that were noted under PAAP.

5) Gender Equality

Question 5.1

To what extent was the gender equality strategy developed and implemented by ACORD, particularly in ensuring that gender issues were addressed in national, regional or continental platforms, as part of the advocacy level elements of the PAAP initiative? The African Union (AU) has recognized ACORD, as a key expert on gender equality. Under PAAP, gender assessments were carried out and gender strategy and policies updated. These are being implemented along with new methodologies and tools (developed after PAAP) for mainstreaming gender in country programs and with partners.

Question 5.2

Has the intervention reduced gender-based inequalities in access to project resources and benefits? PAAP activities were highly sensitive in addressing constraints faced by rural women and ACORD staff was very proactive in targeting and ensuring women’s participation at all levels. Women groups indicated that there was a positive overall reduction in gender-based inequalities for access to resources and benefits in their respective communities. PAAP was not effective in ensuring equitable participation of women representatives from CSO’s for the CAADP forums but this could be a reflection of the low representation of women in CSO management.

6) Environmental Sustainability

Question 6.1

Has the investment achieved results in environmental sustainability? PAAP activities addressed environmental risks especially for agricultural and pastoralist communities in an adequate manner at the planning stage.

7) Governance

Question 7.1

Has the investment achieved results in governance? PAAP helped ACORD to improve its internal management functions to make it more accountable and transparent to its staff and stakeholders.  

Key Conclusions for Key Immediate Outcomes:

One of the key successes of PAAP was to work with local community based organizations to bring together farmer and pastoralist groups to work collaboratively towards a common purpose. The project encouraged individual farmers and pastoralists to form groups and associations to speak with one voice, have access to services and achieve economies of scale. PAAP was also effective in building capacity and knowledge, increasing incomes and, improving the livelihoods of farmer and pastoralist groups at the farm/household level. PAAP was somewhat effective in increasing knowledge of national and regional CSO’s on CAADP implementation but the knowledge gained did not often filter down, as the participating organizations did not have the required resources (funding and linkages) for outreach activities.

PAAP funding was used effective in increasing transparency and accountability within ACORD as financial systems were upgraded and all financial staff trained to use the new Sun system for uniform reporting between the Secretariat and the 17 country programs. Human resource policies were developed and are still being implemented. Annual retreats were used effectively to build capacity of the field staff from all the 17 country programs to exchange ideas and knowledge. ACORD participated in a number of CAADP forums with representatives from farmer and pastoralist groups, national and regional CSO’s. The CAADP partner platforms were used to co-host workshops on CAADP implementation processes for advocacy groups.

PAAP implementation was strong on inclusion of women in all key activities especially with farmer, pastoralist groups and staff development at ACORD. A participative gender assessment was carried out to develop gender mainstreaming tools and methodologies for internal use and for dissemination to partner organizations. Women were well targeted and had equal access to project resources at the farmer/pastoralist level. The CSO representatives selected and sponsored by ACORD for participation in the CAADP partner platforms were predominantly male. This could have been a reflection of the number of women in CSO management.

ACORD was effective in deploying PAAP funds as it already had the required organizational structure in place to implement the project. The annual planning process was deficient as actual annual spending for most line items was consistently 35% lower than expected for the first 21 months of implementation, while others were already showing a tendency of going over budget by March 2012 (Year 2). The annual planning process with financial forecasts, therefore, could have been better-planned and aligned with ACORD and country program’s absorptive capacity. The financial reporting tracked variance of expenditures by quarter and did not highlight differences (over/under budget) on an annual or cumulative year-to-year basis. The financial reporting, thus, could have been more robust especially in Year 2 as some line items had a substantially higher burn rate than planned and looked highly likely to use up the total 3 year contractual allocations before the end of Year 2. In this respect, ACORD did not comply with the signed contractual agreement to inform Global Affairs Canada and seek approval in writing for variances of over 10% in line item expenditures. ACORD revised the CA budget in March 2013 to enable it to complete and wrap up on-going commitments at the Secretariat before the end-of the project in June 2013, within the CA’s overall budget.

PAAP was highly relevant to the targeted communities, national development and research programs. ACORD is a well-structured organization that dovetailed PAAP funding into its on-going activities. Country programs have built on lessons learned from PAAP and up-scaled livelihoods and food security activities with funding from other donors. ACORD management has assumed responsibility for the financing and implementation of many key initiatives from outcome 1, 2 and 3 after the completion of PAAP. Environmental sustainability was given due attention in general agricultural conservation practices. Most outcomes for improved governance were achieved and ACORD as an organization has benefitted from the strengthening of the accountability mechanism and human resources policies. Country program staff appears dedicated and focused on both raising funding for future activities while implementing on-going projects.

Key Recommendations:

1. Improved livelihood strategies should focus on food security (increased productivity) as well as the potential for sustainable income generation opportunities that can lead to reduction in household poverty levels for selected communities. Develop and improve the element of strategic planning for work on livelihoods with famer and pastoralist groups prior to promoting specific perishable produce (dairy, fruits and vegetables). They should assess the availability of private sector resources and build the capacity of potential stakeholders, who could facilitate the movement of the produce to markets. The demand and marketing strategy should examine annual supply and demand for identified commodities especially at harvest time and address the issue of high post-harvest losses.
   

   Response: For ACORD’s internal organisational processing and implementation.

2. ACORD should strengthen leadership training for women in community groups especially those with mixed membership groups. The capacity of the members should be built to improve their knowledge of governance structures, leadership accountability, member rights and gender mainstreaming.
   

   Response: For ACORD’s internal organisational processing and implementation.

3. ACORD should develop a comprehensive database of African CSO’s and categorize them by size, functional niche, ability and their institutional capacity to link with farmer and pastoralist groups for transfer of knowledge. There should be clear written guidelines for Secretariat and country program staff on the basic requirements that must be met by selected participants who are sponsored by ACORD for participation in the national and regional workshops and CAADP forums. Key interventions in country programs must have a clear and sustainable exit strategy phase planned with the full understanding of the targeted community groups and CSO’s.
   

   Response: For ACORD’s internal organisation processing and implementation.

4. Have clear Financial Reporting System that tracks Actual Expenses against Planned Annually and against Planned Budget and not just Short-term (Quarterly). Train project and accounting staff on donor specific reporting systems and tracking variance against plan. This is important for good internal management practices and external reporting.
   

   Response: For ACORD’s internal organisation processing and implementation.

5. Continue to convene the annual retreat as a venue for team building and enhancing the capacity of its field staff. Country specific environmental policies and guidelines are needed to effectively address programming needs especially in the arid regions. ACORD should continue to engage at Pan African level and particularly in the CAADP Non State Actors coalition engagement with NEPAD, AUC, RECs to move forward the African Agriculture Agenda.
   

   Response: For ACORD’s internal organisation processing and implementation.

6. Global Affairs Canada should provide a guidance manual to first time recipients of grant funding on their contractual obligations and reporting requirements. For Results-Based-Management Global Affairs Canada should provide a refresher workshop around the end of year one of implementation to help new organizations better understand reporting on outcomes of projects. Global Affairs Canada should also build in mechanisms and processes for the provision of timely feedback and guidance to grant recipients on an annual basis. This is particularly important in the final year of implementation.
   

   Global Affairs Canada Response: Global Affairs Canada has rigorous financial management and work planning mechanisms for recipients of project and program funding, including risk mitigation measures.  Prior to project approval, ACORD underwent a financial risk assessment by Global Affairs Canada’s Fiduciary Risk Assessment Unit, as well as an institutional assessment.  After the signing of the Contribution Agreement with ACORD, Global Affairs Canada, in accordance with procedure, conducted an initial visit of ACORD to review the terms and conditions of the Agreement and ensure good understanding and capacity for compliance.  In addition to the risk assessment, institutional assessment and initial visit, Global Affairs Canada undertook two audits of the ACORD PAAP covering the full period of the Contribution Agreement.  Audit adjustments were needed and Global Affairs Canada is satisfied that ultimately contractual obligations were met.   

   Global Affairs Canada recognizes the importance of results-based management (RBM) and the importance of robust reporting from partners, especially for those who have entered into contribution agreements with Global Affairs Canada. The Department has developed guidance manuals for partners’ use: “Partner Guidelines for Narrative Reports under CIDA Contribution Agreements” (a previous version was rolled out in 2013; and an updated version was finalised in August 2015.) This document provides detailed guidelines and key elements for partners to address in their reports.

   As far as the broader issue of RBM is concerned, there are regular RBM training courses taking place each year for programming staff. Occasionally, and upon request, RBM specialists from the corporate RBM Centre of Excellence travel to deliver RBM training workshops to both mission staff and/or partners in the field. Global Affairs Canada programming staff is well versed in RBM and carry out similar tasks as well. Furthermore, in Sub-Saharan Africa Branch, as in other programming Branches, there is a dedicated Performance Management Analyst, responsible for providing support and advice to programs on RBM and other performance management related issues. As updates to the Department’s RBM Guide are finalised, Global Affairs Canada will continue to work to ensure that the latest developments on RBM are shared with partners and that they are familiar with the RBM requirements.

   Global Affairs Canada has mechanisms and processes in place for feedback and guidance to funding recipients.  The scope of these is based on the level of fiduciary risk assessed and the type of initiative/partner.  It can include ongoing exchange with the project officer, quarterly, semi-annual or annual financial and narrative report review and comments, project officer field monitoring visits, external consultant field monitoring visits, external evaluations and audits, annual workplan reviews and approval at project steering committees and other joint donor coordination platforms.

Key Lessons

1. Interventions in rural communities must be planned carefully and should address farm productivity with post-production support to ensure farmers have an outlet for their produce and not suffer losses after harvesting.
2. Given that farmers and pastoralist groups have limited opportunities, to receive funding and knowledge, they often become overly reliant/dependent on the support. It is important to manage the groups’ expectations early and in a realistic manner. Parallel efforts should be made to enable these groups to widen their connectivity with other value-chain actors and promote use of technologies like cell phones and radios.
3. The project conceptualization, design and pre-implementation planning should be realistically aligned with the developmental context of the intervention and availability of resources. Expected outcomes planned for PAAP were ambitious given the $2.65 million grant funding over a 3-year period and, the number of countries covered.

Global Affairs Canada’s Response

The ACORD PAAP project ended in September 2013.  Recommendations from the summative evaluation report were shared with the partner, ACORD, whose responsibility it is to analyze and integrate relevant lessons learned into its systems and practices.  Recommendations relevant to Global Affairs Canada will be shared internally with programming staff.