Meta-Evaluation of Global Affairs Canada’s Decentralized Evaluations

March 2017

Acknowledgments

The Global Affairs Canada Development Evaluations Division would like to thank all those that contributed to this evaluation, including Development Evaluation Working Group colleagues and others who provided support and advice throughout the evaluation process.

The evaluation was executed by an independent evaluation team from Itad Limited. The Development Evaluations Division provided oversight and management throughout the evaluation process.

David Heath
Head of Development Evaluation

List of Acronyms and Abbreviations

Executive Summary

Introduction

This report presents a review of decentralized evaluations over the period 2009–14. During this period, decentralized evaluations were managed by Global Affairs Canada’s Program Branches, and were intended to support accountability, learning and decision making, as well as lead to improvements in programming. Decentralized evaluations were commissioned and managed within Program Branches, alone or jointly with other donors and focus on Branch-level projects or investments.

The purpose of this study is to provide Global Affairs Canada with an assessment of the quality of decentralized evaluations in order to develop recommended actions for the Global Affairs Canada to improve the credibility, reliability, validity and use of evaluations; and to improve management information systems for storing and sharing evaluation knowledge.

Global Affairs Canada’s six Development Program Branches have full responsibility for commissioning and conducting decentralized evaluations. The role of PCE, and specifically its Evaluation Support Unit (ESU), has been to provide guidance, technical support and quality assurance of decentralized evaluations as an integral part of PCE’s oversight function. The Unit did not have any authority over the acceptance or approval of such deliverables as this is the role of the Program Branches.

The scope included examining the planning of decentralized evaluations, the quality of evaluation products: the terms of reference (ToRs), work plans, reports and management responses, the effectiveness of support services, and finally the utility of decentralized evaluations.

Quality was determined by how well Global Affairs Canada’s decentralized evaluations met the OECD DAC quality standards for evaluation.^{Footnote 1} These are important as they represent an agreed international framework and were adopted by 32 bilateral donors and multilateral development agencies in 2010. They focus on the management and processes for conducting evaluations and were intended to strengthen the contribution of evaluation work to development outcomes, as well as to form the benchmark against which OECD DAC members are assessed in DAC Peer Reviews.

The study was undertaken by a team from Itad Ltd, a UK-based consulting firm, on behalf of the PCE. The methodology involved six components: a quality review of 116 decentralized evaluations, an online survey of Global Affairs Canada Development Branch staff, a review of support services provided by PCE, an in-depth case study of eight decentralized evaluations, a review of Global Affairs Canada and four other peer agencies information management systems for evaluations, and a review of status of the recommendations from the previous 2008 meta-evaluation.

Findings

Quality of decentralized evaluations

Between 50–60 percent of the three major evaluation products (ToRs, work plans and evaluation reports) met OECD DAC quality standards, but few of them achieved a highly satisfactory standard.

There was some indication of improvement in quality over the review period, although this trend was not definitive. The type of evaluation (formative or summative) or the Branch conducting it did not influence quality substantially, but higher budget projects were better evaluated. Evaluation team composition or size of evaluation budget did not affect quality, but if the consultant team leader was also an evaluator this improved quality.

The evaluation products had a varied performance against different OECD DAC standards. They did well in most areas of evaluation planning and implementation, although fell short in documenting ethical standards, evaluability and methodology.

Although aspects of the management response reports prepared by the Branches were of reasonable quality, such as giving an immediate response to recommendations, the follow-up and dissemination aspects were of lower quality, as judged by the level of detail provided in these reports.

Good quality ToRs were associated with a good quality work plan and also with a good quality evaluation report. This suggested that investing in a carefully written and high standard ToR was worth doing. There was no link between the quality of these products and the quality of the management response as measured by the level of detail given in these responses.

The quality of evaluation reports was most closely related to four OECD DAC criteria: appropriate evaluation questions; robust methodology; logical and well-attributed findings; and clarity of analysis. Further analysis indicated that a sound methodology was the most important single factor that distinguished between good and poor quality ToRs, work plans and evaluations.

Quality of decentralized evaluation planning processes

Evaluations were generally planned well in advance, but broad evaluation purposes combined with too many evaluation questions made it difficult for consultants and their evaluation teams to produce high quality reports. Program managers in Branches came from a broad range of professional backgrounds that may not necessarily have included evaluation, and evaluation budgets and levels of effort tended to be insufficient to match scope and quality expectations. At the same time, the ESU within PCE faced difficulties in ensuring quality because its mandate was advisory and its involvement was insufficient or not tailored to improve the evaluation planning process and so contribute to better evaluation quality. At the same time, ESU did not always take a situational approach to quality, leading to an inappropriate application of quality standards. The supply of ESU staff was also not appropriately matched to meet the demand for its services. Providing advice in a timely manner was a key challenge.

Use of decentralized evaluations

The overriding use of decentralized evaluations was for operational purposes. Program Branch staff undertook such evaluations largely for accountability reasons to check on implementation progress and achievement of objectives. Mid-term evaluations were commissioned to improve project performance and final evaluations to assist with the design of a new phase of a project or to help decide whether to extend an existing operation.

Branch staff found decentralized evaluations more useful if they were timely; and use was also stronger if there was ownership, a clear purpose and expressed management interest. While poor evaluation quality could undermine evaluation use, good quality reports were not a guarantee of widespread dissemination and follow-through of management action plans.

Conclusions and recommendations were perceived to be the most useful parts of evaluations. Lessons were the least used, mainly because there was no comprehensive knowledge management system in Global Affairs Canada to facilitate broader learning beyond the individual intervention. Furthermore, very few decentralized evaluations were published on the Global Affairs Canada website, further limiting broader uptake and use and placing Global Affairs Canada behind its peer agencies in terms of transparency.

Review of Global Affairs Canada information systems

There is strong dissatisfaction among stakeholders with the current approach to storing and managing decentralized evaluations and its ad hoc nature was found not fit for purpose. The lack of a centralized system for storing decentralized evaluations is curtailing their value to the wider organization. The challenge that Global Affairs Canada faces is not unique. Other agencies are either tracking evaluations through placing them on an external website or setting up a dedicated evaluation database to provide a more comprehensive way to store, search and access evaluations. The latter requires investment in more than just system hardware, but also in incentives for the system to be used. By comparison, two UN agencies encourage uploading and use by applying a quality rating system, by tracking evaluations planned and uploaded, and by recognizing the best evaluations each year.

Implementation of 2008 meta-evaluation recommendations

The 2008 report recommended strengthening PCE and clarifying its function and responsibilities for decentralized evaluation work. Since then, with the merger of the Canadian International Development Agency (CIDA) into the Department for Foreign Affairs, Trade and Development (DFATD), no significant strengthening of staff in terms of filling the number of established positions occurred over the review period, and there was considerable staff turnover. The 2005 Departmental Evaluation Policy has not been updated, and while the Treasury Board Policy on Evaluation in 2009 set out the requirements for the evaluation function across government, its application to decentralized evaluations has not been clarified. Very limited progress has been shown around recommendations dealing with training, information management and communication.

Conclusions

The study concludes that the 2005 Departmental Evaluation Policy Framework, though a sound document at the time, did not reflect changes in international evaluation standards and practices such as the introduction of the 2009 OECD DAC standards, nor the significant changes in the Government of Canada’s institutional environment for development assistance such as the amalgamation of CIDA and DFAIT as well as new Treasury Board policies for evaluation. PCE’s role and status has not been strengthened to meet its role of helping to ensure that the effectiveness of development assistance is robustly measured.

Between 50–60 percent of decentralized evaluation products showed acceptable quality, although very few achieved a highly satisfactory standard. Over the period 2009–14, over 40 percent of projects examined by the review team (representing a program spend of $2.8 billion)^{Footnote 2} were not evaluated well, and hence their results and performance were not reliably assessed.

PCE’s role in supporting decentralized evaluation quality remained unclear. The ESU was not sufficiently resourced for the role it was expected to play, and while its support was viewed by users as valuable, ESU’s quality improvement work was not as effective as it could be. ESU’s quality assurance function appeared to be optional as Branch staff were not always requesting the Unit’s help, and bypassed ESU processes. In addition, who has the final say on quality and approval of such evaluations was not certain.

The quality tools and templates offered by ESU were perceived by users as broadly useful, but they did not solve the issue of generally low evaluation capacity among Branch staff. The support strategy adopted by ESU was not able to solve the conflict between, on the one hand, the need for better decentralized evaluation quality, and on the other, the need at branch-level evaluation for sharper timing and better use of evaluations.

The system currently still in use by ESU to track, store and learn from decentralized evaluations is assessed as not effective. Retrieving documents is difficult and aggregating findings and lessons is not happening.

During the period of the review, there was the perception that decentralized evaluations had less status and attention than corporate evaluations, even though they accounted for the bulk of evaluation spend. The body of knowledge being generated was substantial, and could have been more effectively used to inform Global Affairs Canada, the Government of Canada, the Canadian public and the international audience.

Recommendations

The decentralized evaluation system in Global Affairs Canada is intended to provide both operational guidance for individual projects and a source of wider learning for the organization. There is room for the quality of such evaluations to improve further, and for their planning, implementation, storage and use to be supported more strongly and effectively. Therefore the study recommends the following:

1. Strengthening support for decentralized evaluations.
   1. Strengthen the role of the evaluation focal point in Branches by giving them clear accountability and oversight for the use of evaluations and for promoting knowledge sharing in their Branches.
   2. Clarify roles and responsibilities of all stakeholders (Branch staff, ESU, senior managers, the Development Evaluation Committee, etc.), ensuring that there is adequate support for the improvement of evaluation quality (i.e. that is integrated and comprehensive), and that this is reflected (and communicated) in an updated evaluation policy.
   3. Increase the training on evaluation skills for Branch staff and consider rotating staff with evaluation skills to region/country offices, particularly to Global Affairs Canada’s countries of focus.
   4. Provide sufficient support to PCE/ESU in terms of appropriate evaluation staff and budgetary resources.
2. Strengthening the sharing and use of decentralized evaluations.
   1. Present a periodic statement as a standing item on the number, status, quality of decentralized evaluations planned, commissioned and completed by Branch, theme, Departmental priority and budget at the Development Evaluation Committee.
   2. Develop a comprehensive knowledge management strategy including fora for sharing learning from decentralized evaluations, and the publication of decentralized evaluation reports in line with international practice.
   3. Build a stronger culture for evaluation knowledge sharing and use through better communication and use of different media (web, social, networks), as well as rewarding and recognizing high quality / good practice. Work towards achieving publication of all completed evaluations in the medium term.
3. Strengthening planning and conduct of decentralized evaluations.
   1. Strengthen the role, capacities and resources for decentralized evaluation work at Branch level. Achieve this by developing and implementing a training strategy to provide basic evaluation skills and more practical guidance to Branches.
   2. Strengthen ESU’s role as a knowledge broker rather than focusing on QA.^{Footnote 3} To achieve this, consider the option of focusing the ESU on developing appropriate guidance and tools for decentralized evaluations, working more closely with Branches on selected evaluations, and developing better knowledge management systems. Build ESU capacity through staffing the unit with the right mix of evaluation skills and experience and organizational/contextual knowledge and expertise.
   3. Ensure that Branches plan decentralized evaluation on an annual basis with the guidance of ESU tools and staff.
4. Strengthening information management of decentralized evaluations.
   1. In the short to medium term, strengthen accessibility to decentralized evaluation. For example, PCE could maintain a consolidated list on the intranet and develop functionality that updates staff on completed evaluations, management responses and the key findings. Explore the use of existing databases (e.g. CRAFT) to this end.
   2. In the long term, create a platform for staff and external stakeholders to access all decentralized evaluations.

Management Response

Overview

The Meta-Evaluation of Global Affairs Canada’s Decentralized Evaluations for the period of 2009-2014 was undertaken by Itad Ltd, a UK-based consulting firm, on behalf of PCE (the Development Evaluation Division). The purpose of the evaluation was to provide Global Affairs Canada with an assessment of the quality of decentralized evaluations, in order to develop recommended actions aiming to improve the credibility, reliability, validity and use of evaluations; and to improve management information systems for storing and sharing evaluation knowledge.

Based on the findings and conclusions and in order to support improved levels of planning, quality and use of decentralized evaluations, the report recommended strengthening four areas: 1) support for decentralized evaluations; 2) the sharing and use of decentralized evaluations; 3) planning and conduct of decentralized evaluations; and, 4) information management of decentralized evaluations. Of the twelve recommendations formulated in the report, all are accepted.

The Development Evaluation Division of Global Affairs Canada takes note of the performance gaps identified by the evaluators regarding the Meta-Evaluation. Many of those gaps have underlined a need for a renewed process and updated tools to support improvement in the quality and use of decentralized evaluations conducted by Global Affairs Canada’s Development Branches. PCE management welcomes these recommendations as an opportunity to review and improve PCE’s Evaluation Services Unit’s resources and role in providing guidance to branch staff through the lifecycle of decentralized evaluation planning, to ensure that adequate support is provided to meet the demand for its services. PCE also recognizes the need to leverage Global Affairs Canada’s resources across multiple channels including: learning and training; communications; knowledge generation and transfer; results and delivery; data and information management; and, performance measurement.

The recommendations identified by this evaluation give branch management further impetus to update processes and leverage existing platforms in order to address persistent gaps currently impacting the planning, quality and use of a large proportion of decentralized evaluation products. As reflected in the commitments and action items proposed in the table below, PCE will lead on some aspects towards the strengthening of the processes related to decentralized evaluations and their dissemination; while the collaboration of the listed branches will also be essential to oversee and monitor the implementation of this management response.

1 Introduction

This report forms the final deliverable of a meta-evaluation^{Footnote 5} conducted for Global Affairs Canada.^{Footnote 6} The report focuses on the quality and use of decentralized evaluations^{Footnote 7}. Quality was determined by how well Global Affairs Canada’s decentralized evaluations met the OECD DAC quality standards for evaluation.^{Footnote 8} These standards are important as they represent an agreed international framework and were adopted by 32 bilateral donors and multilateral development agencies in 2010. They focus on the management and processes for conducting evaluations and were intended to strengthen the contribution of evaluation work to development outcomes, as well as to form the benchmark against which OECD DAC members are assessed in DAC Peer Reviews.

The study was carried out over the period October 2015 to September 2016, with an inception phase from October-December 2015, and data collection from January – June 2016 . It was undertaken by a five-person team from Itad Ltd, a UK-based consulting firm

This final report includes eight sections: introduction (section 1), objectives and scope (section 2), evaluability assessment (section 3), methodology (section 4), findings (section 5), conclusions, lessons and recommendations (sections 6, 7 and 8).

1.1 Decentralized Evaluations at Global Affairs Canada

Decentralized evaluations are evaluations that are commissioned and managed by the Program Branches,^{Footnote 9} and are intended to support accountability, learning and decision making, and to lead to improvements in programming. They are conducted by private sector evaluators or evaluation firms that are commissioned through the use of Government of Canada procurement vehicles. The decision to conduct a decentralized evaluation, its scope, design and the specific issues addressed are based on the needs and requirements of managers within Program Branches and those of project/program partners. The Program Branches have full responsibility for commissioning, conducting and ensuring the use of decentralized evaluations.

Decentralized evaluations are also the building blocks for corporate evaluations that are conducted within the Policy Branch as they are a key source of data, particularly in the assessment of program effectiveness. Decentralized evaluations are different to corporate evaluations, which address broader thematic issues and are commissioned and managed directly by the Development Evaluation Division (formerly known as DED but now PCE) of Global Affairs Canada.

PCE has a role to support the quality of decentralized evaluation. More specifically, the Evaluation Support Unit (ESU) within PCE, over the period of the review, provided guidance, technical support and quality assurance of decentralized evaluations as an integral part of PCE’s oversight function. The Unit comprises a small team who review evaluation deliverables, such as terms of reference (ToR), work plans and evaluation reports. It is a demand-led activity, depending on requests submitted by the Program Branches. The Unit has provided report templates and quality standards, based on OECD DAC principles, for use by those commissioning and authoring decentralized evaluations. With the exception of ToRs that require ESU approval before going to tender, the Unit does not have any authority over the acceptance or approval of such deliverables as this is the role of the Program Branches. The ESU is also not involved in assessing consultant bids on evaluations.

2 Objectives and Scope

2.1 Purpose

The purpose of this meta-evaluation of decentralized evaluations is to provide Global Affairs Canada with:

1. An assessment of the quality of decentralized evaluations;
2. A set of recommended actions for PCE to improve the quality of Global Affairs Canada tools, guidance and planning processes to increase the credibility, reliability, validity and use of evaluations; and
3. Recommendations to improve management information systems (MIS) supporting the storing and sharing of evaluation knowledge.

The users of this meta-evaluation are PCE and the Program Branches.

The underlying rationale for the study is based on one of PCE’s stated roles. According to the Development Evaluation Policy (CIDA, 2005), the Development Evaluation Division “is responsible to report to the Evaluation Committee on the quality of decentralized evaluations by performing periodic quality assessment and meta-evaluations.”^{Footnote 10}

A previous meta-evaluation exercise was conducted in 2008.^{Footnote 11} A review of its recommendations forms a further rationale for this current study.

Since then, new quality standards (based on the OECD DAC Quality Standards for Development Evaluation)^{Footnote 12} were formally introduced into the 2009 Corporate Standing Offers Arrangements for evaluation services, and these standards now provide a fresh framework for assessing evaluation quality.

This study provides a valuable opportunity to assess how far Global Affairs Canada’s decentralized evaluations have improved over the past seven years, how well they match the OECD DAC’s guidance, what barriers there are to quality, and how PCE can strengthen its support to Program Branches in this area.

Finally, with the formation of Global Affairs Canada (then DFATD) by amalgamating CIDA and DFAIT, the period under study has seen significant organizational and personnel changes compared to the pre-amalgamation era. Merging the functions of foreign affairs and trade with that of development assistance, and the moving of evaluation into a policy department, has created potential for synergies but has also necessitated the reframing of development work within this new structure. Incentives have changed, with a greater emphasis on the scrutiny and the value added of aid following the Federal Accountability Act. The recent change in political leadership in Canada in 2015 has also initiated a new policy direction for aid as well as for results measurement and accountability, and the implications for evaluation are yet to emerge fully.

2.2 Specific Objectives

The ToR require the meta-evaluation to address eight specific objectives:

1. Assess the quality of the decentralized evaluations in relation to OECD DAC Quality Standards for Development Evaluation;
2. Determine the quality aspects or factors in which Global Affairs Canada evaluation documents excel and where they fall short;
3. Examine strengths and weaknesses of decentralized evaluation planning processes;
4. Determine whether decentralized evaluations benefited from the support, expertise and tools developed by PCE;
5. Examine whether the findings, conclusions, recommendations and lessons of decentralized evaluations have been used, for what purpose and where they were not used, why not;
6. Determine the extent to which the then CIDA 2008 meta-evaluation recommendations were implemented;
7. Review the existing MIS for storing and sharing decentralized evaluations and explore opportunities to improve or upgrade it;
8. Recommend measures needed to enhance the quality of evaluations.

2.3 Evaluation Object and Scope

The ToRs cover a comprehensive set of four objects for this meta-evaluation. Together they provide a basis for assessing the entire lifecycle of decentralized evaluations from their planning to their eventual use. These are set out in Table 1.

Given the differing nature of these four objects, the study team designed separate methodologies and tools for each. These are described in Chapter 4. Evaluation utility is not only a result of the quality of the evaluation product, but depends on the institutional setting and incentives. The scope of this meta-evaluation therefore also includes an assessment of these factors.

Stakeholders: As part of the scope, the study recognizes the need to include various sets of stakeholders. These are well defined in the ToR, and include three main groups: the commissioners of decentralized evaluations; the executors of decentralized evaluations; and the users of this meta-evaluation. The work plan sets out how the study team would reach these different actors.^{Footnote 15}

3 Evaluability Assessment

The work plan recognized and sought to mitigate potential constraints to the evaluation process and design. The study approach, discussed in detail in the work plan, sought to address limitations arising from the methodology, the practical and logistical constraints the team might face, the availability of data and people, and any conflicts of interest arising from our team’s past experience. The experience of the previous meta-evaluation conducted in 2008 also informed the work plan. For example, the 2008 study faced serious challenges in accurately identifying and retrieving evaluation documents. To overcome this, PCE took over a year to prepare an inventory of evaluation deliverables for this current study. The study team then chose a purposive sampling method to obtain as representative a coverage of the inventory as possible, and to replace cases where there was incomplete documentation. The data gathering strategy was designed to allow the team to engage with key stakeholders in different ways. Thus, in order to overcome potential gaps in evidence, the methodology used multiple sources of evidence to mitigate these gaps and establish more robust findings.

To avoid compromising the independence of the evaluation, Itad also assessed the team members’ past relationship with CIDA/Global Affairs Canada, and removed any areas of conflict of interest, such as where team members had previous involvement in evaluations. Data from interviews and documents has been held in a confidential and secure manner.

4 Methodology

4.1 Evaluation Questions

The ToR set out six main questions to be answered. These address: (i) the quality of decentralized evaluation documents against OECD DAC and Global Affairs Canada standards; (ii) where these documents exceed or fall short on these standards; (iii) the strengths and weaknesses of decentralized evaluation planning processes; (iv) the utility of the decentralized evaluation reports; (v) whether the 2008 meta-evaluation recommendations were implemented; and (vi) what opportunities exist to improve existing Global Affairs Canada information management systems.

The six questions are broken into 20 sub-questions and these in turn form the structure of Chapter 5 on findings. A detailed evaluation design matrix was prepared to show how the study would answer each of these sub-questions.

4.2 Methods

The scope of this assignment allowed the Itad team to conduct a comprehensive study of evaluation quality for Global Affairs Canada. While past meta-evaluations have concentrated on only selected aspects of determinants of quality, in this case the study has examined the quality of evaluation products, as well as the processes, institutional context, capacities and supporting systems that all play a role in determining quality. To achieve this, the meta-evaluation involved six components: (i) a desk-based quality review; (ii) an online staff survey; (iii) a review of PCE/ESU processes; (iv) a set of in-depth case studies; (v) a review of Global Affairs Canada’s information systems for evaluations; and (vi) a review of the 2008 meta-evaluation. The relationship between these components and Global Affairs Canada’s decentralized evaluation processes is shown in Figure 1. The study work plan, approved by PCE, describes the methodology in full.^{Footnote 16}

Figure 1: Evaluation processes and study components

4.2.1 Desk-based quality review

The quality review covers ToRs, work plans, draft and final evaluation reports and management responses. The templates are based on PCE tools and OECD DAC standards. A four-point quality rating score was applied to each of 72 individual quality criteria and to 16 overall quality ratings. While a sample of 125 evaluations were initially selected,^{Footnote 17} the actual sample was 116.^{Footnote 18} Quantitative and qualitative analysis was performed on the review data with a view to exploring: how representative the sample was, the characteristics of the evaluation documents, patterns of variation in quality against a range of independent variables, and associations between quality across the four deliverables.

4.2.2 Online survey

A web-based survey of Global Affairs Canada Branch staff gathered perceptions on the planning and use of decentralized evaluation and the utility of ESU support. This helped to identify issues for follow-up during the in-depth case studies and in interviews with senior management. Some 114 staff responded and usable results were obtained from 90 staff.

4.2.3 In-depth case studies

The case studies explored the barriers and enablers of good evaluation planning and identified how and why decentralized evaluations are used or not used. A purposive sample of 10 evaluations was chosen to reflect different levels of quality and use across the six Branches. Only eight could be completed due to difficulties in contacting relevant personnel and interviews were held with 25 people.

4.2.4 Review of PCE processes for supporting decentralized evaluations

The focus was around the work of the ESU, which provides both quality assurance (QA) throughout the evaluation process and technical advice on evaluation concepts and methods to Global Affairs Canada Branch staff. The review examined ESU staff capacities and the tools and processes used in ESU’s work. Evidence was drawn from several sources: the quality review, interviews with 23 staff and consultants, the in-depth case studies, and additional interviews with senior management and experienced Global Affairs Canada consultants.

4.2.5 Review of the Global Affairs Canada’s information management for evaluations

This activity consisted of examining existing Global Affairs Canada systems for storing and sharing evaluation knowledge and a comparative review of four other agencies to identify good practices.^{Footnote 19}

4.2.6 Review of 2008 meta-evaluation

The recommendations from the previous meta-evaluation in 2008 were reviewed in the light of evidence emerging from the other study components. This allowed an assessment of how far things had changed over the review period, and whether recommendations from the 2008 study had been followed up.

4.3 Limitations

There were a number of limitations arising from the evaluability assessment and the meta-evaluation design of the study.

This study was required to define quality in terms of adherence to an international standard: i.e. those proposed by the OECD DAC. While these are comprehensive and internationally recognized, there may be other ways to assess quality that depend on the stakeholders’ perspective. Indeed the guidance document notes that the DAC standards should not exclude the use of other standards, and should not supplant specific guidance for particular types of evaluations.

In any assessment of factors affecting evaluation quality and use, some areas will remain hard to assess when conducting the work remotely or from secondary material. These include the sometimes unpredictable relationships between evaluation managers, the evaluators and those being evaluated, and situations where conflicts or biases occur between actors that may not be reflected in documents. While this was difficult to address in the quality review, the study team was able to uncover information around such sensitive issues in the case studies, and indeed these were very useful in building a richer appreciation of the challenges faced in the execution of decentralized evaluations.

The lack of a system in Global Affairs Canada to monitor the implementation of management responses, as well as the small number of such reports available and their format, resulted in a very weak proxy for use. Additionally, while assessing the management response tool was a key part of the study’s assessment of evaluation use, it did not capture all the formal and informal outcomes of evaluation and, on its own, was not a complete measure of utility (see further discussion in section 5.4). The study was able to mitigate for this to some extent through the key interviews.

The study was not able to explore the OECD DAC ‘overarching’ standards around transparency, partnership, coordination or capacity development in detail in the review, since these criteria were not captured in the quality review.

Practical issues arose concerning access to documents and informants for the quality review and the case studies. Locating all the relevant reports proved challenging, and where documents were not found, replacements were made with the help of PCE. The smaller sample obtained for ToRs, work plans and management responses restricted coverage. Global Affairs Canada’s staff had in some cases been re-posted or retired, and it was therefore not always possible to find the most relevant individuals for interview. Many evaluations happened five years ago and interviewees found them hard to remember. Nevertheless, for the majority of case studies and management interviews, the appropriate staff were contacted and were able to recall relevant details.

The sample of respondents who completed the online survey was acceptable at 114, representing around 20 percent of the potential universe of over 500 staff invited to take part.^{Footnote 20} The profile of the respondents reflected fairly well the branches and the staffing categories. The quality review sample of 116 was sufficient for basic comparative analysis, but the use of more advanced correlation tools was limited by the sample size and the study findings notes that the results from this work should be treated with caution.

5 Findings

This chapter presents the main findings from the meta-evaluation against each of the evaluation questions and sub-questions set out in section 4.1.

5.1 Quality of Decentralized Evaluations

5.1.1 Representativeness of the quality review sample

The sample of 116 evaluation reports reviewed for quality was evenly distributed across the Branches according to the proportion of evaluations in the inventory. The most represented Branches were Asia, Africa and Americas, with fewer examples from Global Issues and Development (MFM), Partnerships for Development Innovation (KFM) and Europe Middle East and Maghreb (EGM). The majority of evaluations were either formative (46 percent including mid-term evaluations) or summative (41 percent including final project evaluations) and this maps well against the proportion in the inventory. Average project budgets were slightly larger in the sample reviewed, with a mean of $20 million, compared to a mean of $15 million in the inventory.^{Footnote 21} All Global Affairs Canada priority themes and OECD DAC sectors were covered by the sample. The most represented areas were related to democratic governance and economic development. The average evaluation budget where the data were given is $103,000. The distributions for sector, theme and evaluation budget unfortunately cannot be compared to the overall inventory since these data were not captured there. However from discussions with Global Affairs Canada, the coverage by theme and sector, and level of evaluations cost would appear representative.^{Footnote 22}

5.1.2 What was the quality of decentralized evaluation documents when compared against OECD DAC standards?

To answer this question, the study examined the single overall quality rating given to each evaluation product (Figure 2). The review found that 59 percent of 116 evaluation reports met OECD DAC/PCE standards (based on an overall quality rating of 3 or 4), while 52 percent of 81 ToRs and 52 percent of 58 work plans met such standards. Of 28 management responses, 79 percent also had a satisfactory quality rating, but the quality criteria used were substantially less challenging (relating to such questions as the degree of commitment to follow up on recommendations), while the sample of such reports was much smaller and therefore possibly less representative.

Although the majority of products therefore were rated as satisfactory for quality, it should be noted that only a small percentage were rated as highly satisfactory, so that the majority in this group still have deficiencies in some areas that might limit their reliability or usefulness. The strengths and weaknesses of each product are explored in the next sections.

Figure 2: Distribution of quality ratings across evaluation products

Factors explaining overall evaluation quality

The proportion of evaluation reports achieving an overall satisfactory quality standard varied year-on-year, but appeared to reflect a peak in quality in 2012 (Figure 3), although the data did not allow the analysis to test this definitively. In 2013 and 2104, for example, 71 percent (of 21 reports) and 100 percent (of seven reports) achieved satisfactory ratings. Overall a simple trend line superimposed on the graph indicates a modest improvement in quality over the period.^{Footnote 23}

Figure 3: Mean rating of evaluation reports’ quality over the period 2006–14^{Footnote 24}

Ratings vary from 1 to 4 with a mean quality level of 2.5

There was no major difference between the quality of evaluation reports by type or by Branch (Figure 4 and Figure 5), although there were slightly higher quality ratings for those commissioned by the partnerships and the multilateral Branches than by the three regional Branches.

Figure 4: Distribution of evaluation report ratings by evaluation types^{Footnote 25}

Figure 5: Distribution of evaluation report quality by Branch

There was a strong positive relationship between the higher cost of a project and higher evaluation quality (Table 2).

There was also a moderate but still significant relationship between the budgeted cost of a project and the evaluation budget committed for that project.^{Footnote 26}

There was no clear pattern of association between report quality and the size of the evaluation team in the 86 cases where data were available. There was a link, however, between whether the team leader was also an evaluation specialist and the quality of the work plan and evaluation report. Three quarters of work plans and evaluation reports were satisfactory where the team leader was an evaluator, while where the team leader was not an evaluator just a quarter of work plans and a half of evaluations were rated as satisfactory.

For the 31 cases where information was available, the size of the evaluation budget did not show a clear link to either work plan or evaluation report quality.

5.2 OECD DAC Quality Aspects where Global Affairs Canada Evaluation Documents Excelled or Fell Short

First, the study has drawn together the evidence from the study’s different tools used in the meta-evaluation and made a judgment about whether the quality standard was met for each of the OECD DAC standards. The assessment was partial because of the lack of evidence in some areas.

For DAC standards related to overarching considerations around evaluation, there is not enough evidence to comment on four of the criteria, since the quality review and case studies were not able to explore the issues of transparency, partnership, coordination or capacity development in sufficient detail. The extent to which the evaluations describe if they followed relevant ethical principles in conducting their work (such as confidentiality, privacy, informed consent and respecting human rights), however, was met in only very few of the evaluation documents (e.g. in only 6 percent of evaluations).^{Footnote 27} Adherence to the quality control standard was mixed, being met in 61 percent of the ToRs but just 48 percent of the work plans.

For the DAC standards related to evaluation planning, the picture was reasonably good, with 7 of the 12 quality standards achieved in the majority of documents. The exceptions were for evaluability, and approach and methodology, where the quality the standard was met by only approximately a third of cases.

For DAC standards concerned with evaluation implementation and reporting, 9 of the 15 standards were met. Stronger areas of quality relate to standards on report clarity, intervention logic and stakeholder consultation. The documents performed less well around methodology and limitations.

Finally for quality standards related to evaluation use, the picture was weakest. While the management response tool generally met the quality standard that was applied, when the views expressed from interviews and the online survey were taken into account, the standard on usefulness was assessed as being partially met, and the standards on response and follow-up and dissemination as not being met.

The following section examines the quality of the four evaluation products against the different OECD DAC criteria.

5.2.1 Terms of reference

Quality was evenly split between been satisfactory and unsatisfactory (52:49 percent) across the 81 ToRs assessed; with 6 percent awarded a highly satisfactory rating. An analysis of performance over time suggested that the quality of ToRs was increasing since 2009, and a general decrease in the proportion of ToRs rated overall as not meeting OECD DAC standards.

The four main sections of the review template indicated that ToRs generally have higher quality in relation to section 3, covering management and processes, and section 2, covering purpose, objectives and scope. Over 80 percent of ToRs were rated as satisfactory for the quality of their rationale or purpose of the evaluation, for defining specific objectives, and for defining required deliverables clearly.

ToRs were weaker in the quality areas covered by section 1, structure (layout and annexes) and section 4, related to overarching factors (gender, ethics, limitations). The weakest areas were related to ethics (which it should be noted was not included in ESU guidance), defining the expected limitations to the evaluation, identification of findings and recommendations from previous evaluations, and describing the context.

5.2.2 Work plans

For the 58 work plans assessed, quality was again evenly split with 52 percent rated as satisfactory and 49 percent unsatisfactory. Work plans were most frequently satisfactory in regard to describing the purpose and objectives of the planned evaluation, along with describing adequate management arrangements for the evaluation process. The weakest area was with regards to specifying appropriate methods and tools, for which only 44 percent of work plans met OECD DAC standards.

Longitudinal data suggested that work plans improved significantly in 2013 and 2014 after a period of lower quality in 2012 and before. This may have represented the beginning of a positive trend, although the number of cases was relatively few to confirm this.

In terms of individual criteria, the assessment of work plans revealed a number of strengths. Over 70 percent were rated as satisfactory for the following criteria: good specification of objectives for the evaluation, defining an appropriate scope, ensuring an adequate level of stakeholder involvement, and describing an appropriate team and management process. Unsatisfactory quality criteria included an inadequate description of the context of the evaluation object, poorly specified data analysis methods, and not applying standards for ethics and gender. There also tended to be lower quality with regard to ensuring quality assurance processes and specifying appropriate data sampling.

5.2.3 Evaluation reports

The overall quality of the 116 evaluations reports assessed was slightly higher than ToRs and work plans, with 50 percent rated as satisfactory and 9 percent as highly satisfactory. This finding may relate to the greater scrutiny as well as level of effort and time applied to the final evaluation report compared to ToRs and work plans.

The four main sections of the review template indicate that the strongest quality areas were in terms of logical and relevant analysis leading to adequate findings, conclusions, recommendations, and lessons learned. The weaker quality areas related to lack of detailed and appropriate methods – for which only 39 percent of reports were found to be of the required standard. This has an implication in terms of the quality of evidence that was being used in the later stages of the evaluation process to develop robust findings.

Analysis of longitudinal data revealed a similar pattern to the observations about work plans, with an improvement in quality for 2013 and 2014 after a dip in 2012.

In terms of individual quality criteria, the assessment of evaluation reports found that over 70 percent of reports were rated as satisfactory with regard to identifying specific objectives, describing the evaluation object, assessing the quality of the monitoring and evaluation system, and developing detailed and robust findings. Quality was weaker with regard to integration of gender and ethics standards, incorporating evaluability assessments or presenting data from previous evaluations, and where there was a weak description of methods (or specified methods of limited appropriateness and/or significant limitations).

5.2.4 Management responses

Based on the availability of documents, a relatively small sample of 28 management responses was assessed for quality. Overall, where available, management responses were found to be strong with 79 percent meeting the desired quality standard. However, the format of the report was not a particularly effective tool for judging the application of OECD DAC criteria.^{Footnote 28}

Parts of the management response were of higher quality, particularly in terms of giving an immediate response to recommendations (and whether to accept, decline or adapt them). But the commitments to specific actions, identification of responsible persons, and specification of target dates were somewhat lower, though still of good quality.

5.2.5 Associations between evaluation products and quality characteristics

In addition to undertaking frequency analysis, the ratings data was used to undertake a correlation analysis across and within the four products.^{Footnote 29}

First, a comparison between the overall quality ratings for four products found some significant associations as shown in Table 3.

The results indicated that there was a high probability of a link between the quality of ToRs, work plans and evaluation reports. If a good quality ToR was produced it was very likely to lead to a good quality work plan and that a good quality work plan was very likely to lead to a good quality evaluation. There was a slightly less strong link but still a significant one between the quality of the ToR and the quality of the eventual evaluation report.

While the first three evaluation products showed an important degree of association, no significant correlation was found between the quality of the management responses and the quality of any of the other products. Hence, evaluation quality, in terms of adherence to an industry standard, did not influence the use of the evaluation findings or recommendations, as measured by the management response tool. This is worrying if it implies that evaluation use, in terms of, for example, adoption of recommendations, does not take into account the reliability of those recommendations.

Second, a correlation analysis of all the evaluation report criteria against the overall quality rating for the evaluation report was undertaken. This revealed that the overall quality of an evaluation report is most closely related to four criteria: high quality evaluation questions, robust and appropriate methodology, logical and well-attributed findings, and clarity of analysis.

5.2.6 Principal components analysis

In addition to correlation analysis, a principal components analysis of the ratings data was undertaken in an attempt to identify the most important combination of criteria (or ‘components’) that were associated with high quality evaluation products. Overall, this analysis was handicapped by the small number of cases and the narrow rating scale, and hence revealed a complex set of associations in which a large number of components were produced.^{Footnote 30}

Nevertheless, examining the relationships between the criteria contained in the ToR, work plans and evaluation reports suggested that the methodology criteria from across all three products (specifically methodological appropriateness and robustness from the evaluation report, data analysis methods in the ToR, and data collection and sampling in the work plan) had the greatest explanatory effect on the variance found in the ratings. This was quite a revealing finding and seemed to emphasize again the importance of having a robust methodology in all three products if the overall quality was to be high.

When the principal components analysis was re-run to explore the relationship between all the quality criteria found in just the evaluation report, the analysis found that just over half of the variance was explained by two components. The first component could be categorised as ‘methodology’ and accounted for 41 percent of variance. Six of the top 15 criteria were associated with this dimension (with strong loadings for methodological robustness, sources of evidence, limitations, and description of the design). The second component, accounting for 10 percent of the variance was an unclear mixture of quality criteria, from recommendations to report style and structure, and scope.

5.2.7 Qualitative analysis

The study also undertook a qualitative analysis of the five reviewers’ comments across the different evaluation products.^{Footnote 31} The analysis extracted characteristics of evaluation products that were most frequently associated with reports rated satisfactory or unsatisfactory. The results found that the descriptive terms associated with more highly rated evaluations included clarity, comprehensive coverage of appropriate details, sufficient team and budget, soundness of design and methods, and strong recommendations and conclusions. By comparison, features associated with less well-rated evaluations included weak context, insufficient management information, insufficient treatment of ethics and gender, inadequate data and analysis, and generic findings.

5.2.8 Summary of the quality review

1. Between 50 and 60 percent of the three major evaluation products – ToRs, work plans and evaluation reports – were satisfactory when judged against the OECD DAC quality standards; few achieved the OECD DAC highly satisfactory standard. The study team did, however, highlight three examples of good practice.^{Footnote 32}
2. There was some indication of a modest improvement in quality over the period of the review though this was not definitive. The type of evaluation or the Branch conducting it did not influence quality substantially but higher budget projects were better evaluated. Evaluation team composition or size of evaluation budget did not affect quality, but if the team leader was also an evaluator, this improved quality.
3. The evaluation products had a varied performance against different OECD DAC standards. They did well in most areas of evaluation planning and implementation, although they fell short in describing adherence to ethical standards, evaluability and methodology. Although management responses were of reasonable quality, the follow-up and dissemination aspects were poor.
4. Good quality ToRs were associated with a good quality work plan and also with a quality evaluation report. There was no link between the quality of these products and the quality of the management response.
5. The quality of an evaluation report was most closely related to four criteria: high quality evaluation questions, robust and appropriate methodology, logical and well-attributed findings, and clarity of analysis. Further analysis indicated that a sound methodology was the most important single factor associated with good overall quality of ToRs, work plans and evaluations.

5.3 Quality of Decentralized Evaluation Planning Processes

The conceptual framework in the work plan recognized that the evaluation planning process was embedded in the relationship between the evaluation commissioner, ESU and the evaluation team and their respective capacities, as well as the wider institutional environment in which the evaluation was being conducted. Evaluation quality and use depended to a large extent on this interplay.

Evidence for this section was drawn from the quality review, online survey, case studies, interviews with Global Affairs Canada staff in ESU and Branches (notes of these are kept on Itad’s confidential web file), and an analysis of the ESU tools and templates designed to support quality of decentralized evaluations.

5.3.1 Which quality aspects and factors related to the planning process were associated with overall evaluation report quality?

From the quality review, the analysis found a strong and significant correlation between ToR quality and work plan quality and also a strong and significant correlation between work plan quality and evaluation report quality (section 5.2.5). This was especially so where the work plan contained a good evaluability assessment, an appropriate team and clearly specified deliverables. There was also a moderate yet significant correlation between ToR quality and evaluation quality. This supports the logic that a sound ToR is highly likely to lead to a good work plan, and in turn to a good quality evaluation. It implies that for evaluation commissioners, investing in a carefully written and high standard ToR is worth doing.

A triangulation of different types of quantitative and qualitative analysis indicated that detailed and clear explanation of evaluation methods appropriate to the purpose, object and context of the evaluation was a key contributor to overall quality. Whether this was provided in the ToR or in the work plan, it was a critical dimension given that the specification of methodologies remained the weakest aspect.

The case studies indicated that better quality evaluations were generally planned well in advance, allowing for sufficient time to ensure a quality process and product (true in five out of eight case studies). Typically, evaluations were planned and budgeted for when a project was designed, ensuring resourcing and appropriate timing. In contrast, there were also cases where evaluations were delayed and could not serve their main purpose anymore, i.e. some of the decisions they were meant to inform had already been taken.

The composition and technical skills/experience of the evaluation team was also associated with the quality of the evaluation report, particularly when the teams mobilised had the rights skills and expertise, and especially when the team leader had evaluation skills. The standing offer^{Footnote 33} was often seen as a helpful way of sourcing quality consultants. There was a concern with the process of selecting consultants located in the country where the evaluation took place, which was often done separately. In some cases, this led to dysfunctional teams or where team member skills did not complement each other.

A weakness in the planning process of decentralized evaluations was the lack of a clear and specific purpose to the evaluation. In six out of eight case studies, the evaluation purpose was either too broad or unclear, or there was a second hidden purpose that undermined evaluation quality. It was found that broad evaluation purposes combined with too many evaluation questions made it difficult for evaluation teams to produce high quality reports.

Another weakness identified was the comparative lack of evaluation experience among some evaluation managers/program staff for branch-led evaluations. Such staff possessed a wide range of development experience, but many were new to the subject matter and had to manage the process with little assistance. The lack of evaluation skills was evident in the allocation of insufficient budgets for evaluations, in the planning of evaluations at inappropriate points in time, and in the development of ToRs with weak content, at times copying evaluation questions from templates without tailoring them to the specific evaluation. Some of this may also be due to time constraints, since evaluation management was just one of their many responsibilities.

Third, evaluation budgets and levels of effort tended to be insufficient to match scope and quality expectations (in five out of eight cases). With relatively broad purposes as outlined above, and limited budgets, some evaluations had no choice but to cut back on quality. Some interviews indicated that there was a lack of guidance in Global Affairs Canada on how to resource evaluations and that such guidance would be useful to ensure sufficient resourcing and better quality. When reviewing ToRs, ESU did not seem to pay much attention to the resourcing question either. Global Affairs Canada staff is also limited by the procurement vehicles and resulting processes that are made available through Public Service and Procurement Canada.

Finally, the study found that ESU involvement was often insufficient or not sufficiently tailored to improve the evaluation planning process and contribute to better evaluation quality (three out of five cases) given their limited mandate and role in decentralized evaluations. ESU capacity to support decentralized evaluation in a broader way was limited and very much focused on providing comments on evaluation products. Given the capacity challenges among program staff, a more hands-on involvement would have been welcomed by many evaluation managers. It was also mentioned that ESU did not take a situational approach to quality, leading to the inappropriate application of quality standards. From the perspective of evaluation commissioners, ESU involvement was sometimes perceived as bureaucratic rather than facilitating, and response times fell short of expectations, thereby limiting the utility of ESU support.

5.3.2 How did capacity considerations influence evaluation planning processes?

There were two main areas where capacity to conduct and support decentralized evaluations was relevant: within ESU and in the Program Branches.

While ESU played no management role in such evaluations, its function was to provide QA and technical assistance to ensure that evaluation processes and products were as credible and robust as possible. Annually, the ESU estimated that it engages with 50-60 decentralized evaluations, and over the 5-year period of the review (2009–14) 467 evaluations with associated deliverables have been documented.^{Footnote 34}

Throughout the period of this evaluation, there were not enough staff in the ESU to meet the demand for its services based on current practices. Formally, ESU should have 3.5 full time staff.^{Footnote 35} While this structure had been achieved at certain points in the Unit’s history, ESU’s staff capacity was more diluted recently.^{Footnote 36} This was far from ideal given the workload. Study interviews and the web survey consistently indicated that stakeholders in both the Program Branches and PCE characterised ESU as ‘short of capacity,’ ‘overworked’ and having ‘too few resources’ to effectively deliver on its mandate, especially considering the expectation from evaluation commissioners that ESU provide a more situational and tailored approach to their work.

ESU also requires greater experience in conducting evaluations to effectively support Branch staff. While the web survey indicated that 63 percent of Branch staff felt the ESU’s technical capacity was high, key informant interviews revealed a more nuanced picture. One of the issues raised was the perception that there is a lack of practical evaluation expertise in the ESU. This meant some ESU staff were less comfortable using their judgment and applying the QA templates flexibly, which fostered a tick box approach. This was corroborated by the views within the ESU.^{Footnote 37}

Providing advice in a timely manner has been a key challenge for the ESU. Despite staff being generally positive about the technical support provided by the ESU, the overall level of satisfaction with the Unit is low. Out of the 57 survey respondents who answered this question, 40 percent were satisfied with the Unit’s support services. Both the survey and key informant interviews revealed a fairly widespread frustration at the time it takes ESU to provide feedback. This may potentially affect the usefulness of evaluations, given the importance of timeliness to assuring use.

Branch staff capacity for evaluation work was of course mixed, but overall less than ideal. From the online survey, respondents were asked if they felt that they had the necessary knowledge and skills to manage a decentralized evaluation. For the 88 that answered, just over half (55 percent) felt their knowledge was poor or none, while 30 percent felt it was fairly good and just 14 percent felt fully competent.^{Footnote 38}

ESU itself has also not had the capacity to provide sufficient evaluation training, which had once been considered a potential avenue for ESU to improve the quality and profile of evaluations across the department. Some training has been provided in the past, with good results, but this has not been systematic. Lack of resources has been an impediment to designing and rolling out more comprehensive evaluation training. This is not to say that ESU has not been able to support skills development of Branch staff, they have sought to do this through the engagements around evaluations. However, this is a very resource intensive approach. There is a clear need within Global Affairs Canada for the ESU to play more of a role in building internal capacity. A number of informants spoke of how they would like the ESU to not be solely focused on QA and to take on the wider more strategic role of advocating and building capacity for decentralized evaluation in Global Affairs Canada.

5.3.3 How did incentives and systems influence evaluation planning processes?

The provision of QA and advice were not only affected by capacity but also by systems and incentives.

The quality of ESU’s templates were generally good. The study reviewed the five templates that have been developed by ESU for structuring its QA process. In general the templates and tools were considered to be well-structured and comprehensive. In the case of the ToR template there was also good evidence to suggest it was appreciated by Branch staff. While there were small areas for improvement for each of the templates, the major challenge was in how they are applied.

ESU did not take a situational approach to quality. While the ESU had a common set of standards that it applied across the evaluations it reviews, the scope and budget of these varied considerably. Some were project-level evaluations costing around $100,000, others more complex program evaluations with budgets of $400,000. While it is understandable that ESU should seek to apply a minimum set of evaluation quality standards, a common issue that emerged from both the interviews and web survey was the perception that ESU was not flexible enough in how it applied quality standards across evaluations of different sizes. For many, the issue was around ESU matching their expectations around quality to the resources available for the evaluation. Responses from the online survey illustrated this: “ESU are more familiar with large scale program evaluations. They need to be more aware of what’s possible in a smaller evaluation” and “[ESU’s] application of the templates is too rigid and didn’t take into account the resources and time that were often available for conducting decentralized evaluations.”

For many Program Branch staff, there was a perception that the multiple rounds of revision around an evaluation product was inefficient and lengthened the QA process.^{Footnote 39} Interviews revealed a number of examples of ToRs undergoing three to four rounds of iterations.^{Footnote 40} This iterative process was a source of frustration for many who, while wanting to assure quality, also needed to meet their operational needs.^{Footnote 41} One informant said: “The fact that the ToR must be perfect before accessing supply arrangements means they [ESU] are effectively gatekeepers for evaluations. This has caused programs to abandon the idea of doing evaluations for many projects.” For example, one survey respondent commented that the ESU’s process for engaging with a decentralized evaluation was “cumbersome, onerous, and detached from what I needed to get out of it for the project” and that eventually, because of the time that was being spent on the process they made “a program decision to abandon the evaluation.” Another remarked that “the process is too long and rigorous and is not serving the operational needs of the program.”

As well as causing frustration among the users of the ESU services, the multiple revisions around outputs also created a significant amount of work for the ESU. Given that ESU was de facto serving as a gatekeeper to evaluations, the process needed to be more efficient and bounded.

There seemed to be a mismatch of expectations between what the ESU thought the users of its services wanted and what they actually wanted. This was especially the case during the development of the ToRs. ESU adopted an approach that was very hands on because they wanted to not only provide QA but also help build the evaluation capacity of Branch staff through the process. While this was in line with the Unit’s objective, which is to provide guidance in order to improve quality of decentralized evaluations against OECD DAC standards, the approach was very time intensive, and often, when a project officer approached the ESU for support, they wanted to finalize the ToR as soon as possible. Project staff were often not entering into an engagement with the ESU for the explicit purpose of building their capacity, they had more pressing operational needs. The ESU has not had the resources to conduct evaluation training, so they should be commended for trying to integrate capacity building into their support services. However, trying to combine the functions of QA and capacity development was not practical and created a bottleneck in the system.

More widely, the study interviews and web survey revealed that some PCE and Branch staff perceived that there was a lack of commitment from senior management to support evaluation work and particularly decentralized evaluations. Some 80 percent of surveyed staff identified that they perceived a lack of such support or interest especially over branch-led evaluation work. Furthermore, 72 percent indicated that senior management did not stress enough the importance of following up and implementing decentralized evaluation recommendations. This finding should be tempered by the responses that some senior management gave in interviews that indicated that there was in their view a serious commitment to strengthening evaluation.

5.4 Use of Decentralized Evaluations

Evidence for this section is drawn from the quality review, the case studies, the online survey, and an analysis of the use of decentralized evaluations in corporate evaluations.

The survey found that uptake and use of decentralized evaluations was mixed (Figure 6), with 57 percent of the 74 staff who answered saying that they found them useful, and 21 percent saying not useful. Branch staff did say that they found decentralized evaluations more useful than other types of evaluation, such as program or centralized evaluations, largely because decentralized evaluations were more relevant to their immediate operational needs.

“Most of the time, we don’t learn much from program evaluations (that we did not already know). Project evaluations are far more useful (if they are done in a timely fashion)”

Figure 6: Staff survey responses to usefulness of decentralized evaluations

5.4.1 Which quality aspects or factors were associated with evaluation use?

From the quality review, it was difficult to associate quality criteria with use, because the management response was a weak proxy for use and the number of cases available for analysis was too small to allow a reliable assessment. All the same, the earlier correlation analysis has revealed that there was no obvious association between the quality of either ToRs, work plans or evaluation reports and management response quality^{Footnote 42} (section 5.2.5).

From the case study evidence, several factors could be associated with greater evaluation use:

• Ownership by all program staff and implementers of the evaluation^{Footnote 43}
• Good timing to inform decision making
• A clear purpose of the evaluation
• Senior management interest in evaluation

The main finding was that, given the lack of correlation between evaluation report quality and commitments in management responses, good quality reports were not a guarantee of high use.

5.4.2 Which elements of evaluations reports were used and which ones not?

The online survey asked staff which parts of the evaluation reports (findings, conclusions, recommendation or lessons) were most useful. For the 63 who answered, all of these elements were felt to be useful but conclusions and recommendations were the most useful (over 80 percent said useful or very useful for these).

Lessons were the least used, mainly because there was no comprehensive knowledge management system in Global Affairs Canada to facilitate broader learning beyond the individual intervention, and few evaluations were published on the Global Affairs Canada website. Overall, use was mixed for specific programs, but virtually non-existent for the wider organization and the international development community.

In terms of the use of the management responses, 62 staff answered this question in the survey with opinion evenly divided: 47 percent said not useful and 53 percent said useful. Some comments were critical of the lack an institutional culture to integrate evaluations into programs, and the absence of any systematic follow-up for the management responses.

5.4.3 How were evaluations used and for what purpose?

The overriding use of decentralized evaluations was for operational purposes. Program Branch staff undertook such evaluations largely to check on the level progress of implementation and achievement of objectives. Mid-term evaluations were commissioned to improve implementation and final evaluations to assist with the design of a new phase of a project or to make a decision about whether to extend an existing operation (Table 4). The case studies confirmed this, with two evaluations assisting with current implementation and two informing a new phase. Any learning therefore tended to be confined to the individual project and to those directly involved.

Beyond this, corporate evaluations used decentralized evaluations as a standard source of data. A specific analysis was undertaken of six recent studies, and conclude that although such decentralized evaluation is routinely used, there was little serious assessment of the quality of the evidence provided.

5.4.4 What was achieved when evaluations were used?

It was difficult to assess what was achieved when evaluations were used. This was very dependent on the nature of the evaluation recommendations and the level of uptake and use. In some cases, evaluations helped projects achieve more results, in other cases they helped achieve results more efficiently, and sometimes they helped make results more sustainable. Tracing the chain of causality between an evaluation and its use more thoroughly was beyond the time and resources available for this meta-evaluation.

A key hindering factor to evaluation use highlighted in the case studies was the lack of a comprehensive knowledge management system in Global Affairs Canada. Where evaluations were used, use remained limited to the specific interventions. Learning was rarely shared with other programs and no decentralized evaluations are published on Global Affairs Canada’s website to inform either internal or external audiences, so seriously limiting broader uptake and use. In this respect, the lack in investment in a comprehensive knowledge management system covering decentralized evaluations represents a significant divergence between Global Affairs Canada and the four peer agencies that the study compared Global Affairs Canada with (section 5.6).

5.5 Implementation of 2008 Meta-evaluation Recommendations

This section examines the extent to which the recommendations from the 2008 meta-evaluation were implemented and the underlying factors influencing Global Affairs Canada’s response.^{Footnote 44}

5.5.1 2008 recommendations

Evidence from a variety of sources has been used to explore progress towards or challenges faced by Global Affairs Canada or PCE in reference to each recommendation; including this study’s assessment of PCE/ESU services and tools, its review of Global Affairs Canada’s information systems, and the online survey.

Alongside asking stakeholders about the level of implementation of the 2008 recommendations, the study also explored the underlying factors that either have enabled or prevented recommendations being taken forward. For those recommendations that are corporate in nature, evidence from a variety of sources has been obtained to provide the corporate context within which PCE and ESU have been functioning.

Table 5 provides a review of the extent to which each of the five main recommendations from the 2008 meta-evaluation have been addressed. The 2008 report recommended strengthening PCE and clarifying its function and responsibilities for decentralized evaluation work. Over the period that this study covered, no significant strengthening of staff occurred, and there was staff turnover due to the rotation of ‘mobile’ staff out of the Unit.^{Footnote 45} The Departmental Evaluation Policy framework issued in 2005^{Footnote 46} has not been updated in the past 10 years, and while the Treasury Board Policy on Evaluation in 2009^{Footnote 47} clarified the evaluation function, its application to decentralized evaluations was not specifically addressed. Since then, the policy has been replaced by the Treasury Board Policy on Results in 2016, which will undoubtedly bring new requirements and changes. Very limited progress has been shown around the 2008 study’s recommendations dealing with training, information management and communication.

5.6 Review of Global Affairs Canada Information Systems

5.6.1 What are the strengths and weaknesses of Global Affairs Canada’s MIS?

The issue of how best to store and communicate decentralized evaluations has been a long-standing challenge for Global Affairs Canada.^{Footnote 48} Both the 2004 and 2008 meta-evaluations recommended that better systems needed to be put in place to track decentralized evaluations and improve learning from them.^{Footnote 49} The Corporate Memory System used to be the central repository for project and program reports and evaluations. Its functionality was reduced over the years as: (i) more information was stored at Branch level rather than centrally;^{Footnote 50} and (ii) the enterprise document and records management system (EDRMS) was introduced and assumed many of its roles. The corporate memory system no longer exists.^{Footnote 51}

While ESU are doing their best with the tools that they have available, Global Affairs Canada’s current approach to managing and storing evaluations is not fit for purpose. Global Affairs Canada does not have a coherent system for storing and making accessible decentralized evaluations. What exists is a set of ad hoc efforts to bring them together. These include:^{Footnote 52}

• ESU guidance requiring that evaluation commissioners ensure the publication of executive summaries of decentralized evaluations on the Global Affairs Canada website. In practice, this has been done in only two cases.
• ESU maintaining a repository of decentralized evaluations in a spreadsheet that links to EDRMS. The inventory exists but is not easily accessible or known to the rest of the organization.
• PCE’s 5-Year Rolling Development Evaluation Work Plan focuses on the corporate evaluations, but it contains a list of planned decentralized evaluations and is updated yearly drawing on Branch evaluation plans.^{Footnote 53} What is actually commissioned, however, does not always align with what is planned.

Unsurprisingly, there is strong dissatisfaction with Global Affairs Canada’s current approach to storing and managing decentralized evaluations. The survey indicated that 92 percent of respondents were dissatisfied with the way that decentralized evaluation products were stored and shared, and 86 percent found it difficult to access evaluation products. This is a message which has been repeated through two Global Affairs Canada studies since 2007.^{Footnote 54}

Despite the view among staff that decentralized evaluations should be made public there are major challenges to realizing this within Global Affairs Canada. One of the easiest is simply to make the high quality reports available on the Global Affairs Canada external website. Key informants raised a number of challenges of doing this. The first practical issue relates to translation.^{Footnote 55} Second, there seems to have been a greater degree of caution within the Government of Canada compared to other peer agencies towards making evaluations public.^{Footnote 56}

The lack of a centralized system for storing decentralized evaluations is curtailing their value to the wider organization. Currently, it is virtually impossible for someone to find a decentralized evaluation unless they know which project to go to, or they contact ESU directly. For an organization of Global Affairs Canada’s size and given the amount of resources that are invested in decentralized evaluations, this is a suboptimal approach. The learning from such evaluations is being limited to the project or program that commissioned it, and undermining evidence-based decision making across the wider organization.

5.6.2 How do other agencies address similar weaknesses?

To inform the study’s recommendations to Global Affairs Canada on how to improve its current approach to storing decentralized evaluations, a comparative assessment of how other agencies approach this issue was undertaken. Four agencies were reviewed: the United Kingdom’s Department for International Development (DFID), Australia’s Department of Foreign Affairs and Trade (DFAT), United Nations Women’s Agency (UN Women) and United Nations Children’s Emergency Fund (UNICEF).

Based on this comparative analysis, there are two broad approaches to how these agencies store and use decentralized evaluations:

1. Managing a separate database for storing decentralized evaluations. This is the approach taken by UNICEF and UN Women.
2. Integrating evaluation outputs into wider project management systems. This is the approach taken by DFID and DFAT.

A number of findings emerged from the comparative analysis:

First, the challenges that Global Affairs Canada are facing in creating a consolidated repository for decentralized evaluations are not unique. DFID and DFAT are also grappling with how best to make decentralized evaluations accessible to the wider organization. Both are hampered by the limitations of their existing project-level MIS and the organizational barriers that can often exist to modifying such core organizational data systems. As a way of side-stepping these challenges, both DFID and DFAT have opted to list all of their decentralized evaluations on their external website. While both recognize this is not an ideal solution and has limitations, it at least puts in one place all decentralized evaluations for staff and the public to access and use.

Second, setting up a dedicated evaluation database is the most comprehensive way to store evaluations, but it requires investment in more than system hardware. Both UNICEF and UN Women have opted to develop independent databases for their decentralized evaluations.^{Footnote 57} The databases are accessible for both internal and external stakeholders and include a range of evaluation outputs, including executive summaries, full reports and management responses.

Importantly, alongside investing in system hardware, both UNICEF and UN Women have instituted incentives for the systems to be used. One of the key challenges to creating a functioning evaluation database in a decentralized system is ensuring that evaluation products are uploaded. Both organizations have sought to address this through a range of strategies. Both have an externally contracted QA function that reviews all reports and assigns them a quality rating. This rating is then made public alongside the report. These quality scores are shared internally and used to raise standards. Alongside this, key performance indicators track practices such as the number of evaluations planned versus the number of evaluations uploaded. In UN Women, they also recognize the best evaluations of the year. In addition, both organizations have dedicated staff overseeing the database and the QA function. In these cases, there is clarity about who is responsible for the publication of evaluation, and organization structures and processes largely support this role.

5.6.3 What other opportunities exist to improve Global Affairs Canada’s Management Information Systems?

Based on an analysis of Global Affairs Canada’s current approach to storing and managing decentralized evaluations and the comparative review of other partners approaches, the study has identified a number of actions. Recognizing resource constraints and some of the practical challenges to reform the study has divided its proposal into three stages: immediate, medium term and, if resources are made available, the longer term. These include: (i) in the short term modifying search functions in the database platforms currently in use; (ii) in the medium term, placing evaluation reports in the intranet and regularly updating them; and (iii) in the longer term, create a dedicated searchable platform for staff and external stakeholders to access decentralized evaluations and make decentralized evaluation reports public.

6 Conclusions

The following paragraphs summarize the findings in relation to the main evaluation questions:

1. There is room to improve the quality of decentralized evaluations at Global Affairs Canada. Between 50–60 percent of decentralized evaluation products show acceptable quality, although very few achieve a highly satisfactory standard. There was some evidence that more recent evaluation products showed a slight improvement in quality, although this trend was not definitive. The type of evaluation (formative or summative) or the Branch conducting it did not influence quality substantially, but higher budget projects were better evaluated. Evaluation team composition or size of evaluation budget did not affect quality, but if the consultant team leader was also an evaluator this improved quality. Evaluation reports were assessed as fair by this review (with 59 percent of sampled evaluation reports meeting OECD DAC standards). If this assessment is accurate, then over the period 2009–14, 41 percent of projects – representing a program spend of $2.8 billion out of the $6.9 billion (Table 1) – may not have met satisfactory quality standards. The quality of ToRs and work plans were lower with 52 percent meeting a satisfactory standard.
2. In terms of which OECD DAC evaluation standards were being met and which were not, there was mixed performance. Standards that were met well included defining the objectives and rationale properly, and setting out sound management processes. Standards that were not met well included defining appropriate data collection and analysis methodologies, assessing evaluability and documenting if ethical standards were followed. Although management responses were of reasonable quality, as judged by the level of detail around response to recommendations, follow-up and dissemination aspects were poor beyond the immediate initiative.
3. In terms of the quality of decentralized evaluation planning processes, there was good evidence that:
   1. The lack of systematic planning of decentralized evaluations across the Department was a contributing factor to several barriers to achieving high-quality evaluations including: clarity about the purpose of evaluations, ensuring adequate time for evaluation planning and conduct, ensuring appropriate time for engaging with technical experts in ESU, and ensuring buy-in from senior management.
   2. The quality of ToRs, work plans and evaluation reports were related, and good quality evaluation planning was likely to produce good quality reports. The data also indicated that defining evaluation methods appropriate to the purpose, object and context of the evaluation was a key contributor to overall quality.
   3. PCE’s role and responsibilities for supporting decentralized evaluation quality was unclear. While it had a QA and support role, and its tools and templates were perceived by those who used them as broadly useful, following ESU advice appeared to be optional as Branch staff were not always requesting help. ESU’s largely advisory role meant that it had limited influence on signing off on ToRs before procurement, and was not able to influence the quality of the final report.
   4. ESU was not sufficiently resourced to meet its QA role and its work was not as effective as it could be. There was a disconnect between what ESU was expected to do (i.e. improve the quality of decentralized evaluations) and what ESU had control over. While ESU struggled to always provide timely advice or to apply standards that met the situational needs faced by the evaluation commissioners or the time frames and resources available, it was also the case that Program Branch staff needed greater technical support, and a better centralized information system would have improved lesson learning.
4. In terms of decentralized evaluation use:
   1. Decentralized evaluations were mainly undertaken for operational needs, and where they were completed on time, they were deemed useful by Branch staff. However, their level of quality was not closely associated with their level of use, for example the adoption of recommendations may not always have reflected the reliability of the evidence base for those recommendations. Furthermore, the management response tool was not a sound means to gauge utility, as the format takes a rather “tick box” approach, and also was not available for the majority of decentralized evaluations.^{Footnote 58}
   2. The competing demand, on the one hand, for better decentralized evaluation quality, and on the other, for Branch-level needs for prompter timing and better use of evaluations, were not being met by the Global Affairs Canada approach for QA. The requirement to assure quality of every ToR and to achieve this by a process of iteration and then final ESU approval before contracting was not feasible given ESU staff capacity. It was also unrealistic to expect such intervention to be able to have a significant influence on evaluation quality considering the much larger role evaluation commissioners and private sector teams they hire have in ensuring high-quality products.
   3. Decentralized evaluations were perceived by evaluation participants to have less status and attention within Global Affairs Canada than corporate evaluations, even though they accounted for the bulk of evaluation spend. The 50–60 decentralized evaluations conducted per year were not strategically chosen or prioritized, and this reduced their potential value to the organization and beyond. While routinely used in corporate evaluations, the quality of decentralized evaluation evidence will need to be sufficiently assessed in these studies.
5. In terms of the implementation of the 2008 meta-evaluation recommendations:
   1. The 2008 report recommended strengthening PCE and clarifying its function and responsibilities for decentralized evaluation work. Since then, no significant strengthening of staff has occurred. Very limited progress has been shown around recommendations dealing with training, information management and communication.
6. Review of Global Affairs Canada information systems:
   1. The current system used by ESU to track, store and learn from decentralized evaluations is not an effective proxy for sharing evaluation across the department. Retrieving documents is difficult and aggregating findings and lessons is not happening. As a result, Branch staff commissioning new evaluations cannot easily draw on past experience to design the evaluation or choose the best team. Furthermore, wider corporate knowledge learning is not being supported. Global Affairs Canada is significantly behind other peer agencies in its current approach to evaluation transparency, even though some other agencies also face difficulties in this regard.
   2. Finally, an overarching conclusion is that the Departmental Evaluation Policy needs updating (as noted in section 5.5). It has not been revised since 2005, and there have been significant institutional and wider policy changes since then. For example, the policy needs to incorporate Global Affairs Canada’s adherence to the OECD DAC evaluation standards as well as to other international standards of good practice. Internally, the shift over the period of this review towards an emphasis on accountability rather than learning has reduced opportunities for the evaluation function to build greater understanding of the impact of aid programs,^{Footnote 59} and this needs to be rebalanced. The 2013 merger of CIDA and the Department of Foreign Affairs and International Trade Canada (DFAIT) and the creation of Global Affairs Canada in 2015 have affected PCE’s role and status, while the restriction on staff recruitment has prevented PCE and ESU from building the necessary capacity to meet its QA and support role for decentralized evaluations.

7 Lessons

1. Sound evaluation methodology is a key determinant of the quality of decentralized evaluation reports.
2. Investing in a high quality ToR is likely to lead to a high quality evaluation report.
3. Having quality assurance done by a unit that limits its involvement to assessing evaluation products against a quality standard is not sufficient to improve quality of evaluations; a more integrated and comprehensive approach that includes contextual awareness, improved capacity within Branches and building on lessons learned is more likely to yield an improvement of quality.
4. While OECD DAC evaluation standards provide an internationally agreed and comprehensive basis for guiding decentralized evaluation work, their application needs to be sensitive to the needs at field level and the project’s life cycle.
5. In an environment where there is no explicit requirement or oversight on how senior management should respond to decentralized evaluation findings or recommendations, there is unlikely to be a link between a high quality evaluation report and its effective use.

8 Recommendations

The decentralized evaluation system in Global Affairs Canada is intended to provide both operational guidance for individual projects and a source of wider learning for the organization. There is room for the quality of such evaluations to improve further, and for their planning, implementation, storage and use to be supported more strongly and effectively. Therefore this study recommends the following.

1. Strengthening support for decentralized evaluations.
   1. Strengthen the role of the evaluation focal point within Branches by giving them clear responsibility and oversight for promoting knowledge sharing on evaluation in their Branches.
   2. Clarify roles and responsibilities of all stakeholders (Branch staff, ESU, senior managers, the Development Evaluation Committee, etc.), ensuring that there is adequate support for the improvement of evaluation quality (i.e. that is integrated and comprehensive), and that this is reflected (and communicated) in an updated evaluation policy.
   3. Increase the training on evaluation skills for Branch staff and consider rotating staff with evaluation skills to region/country offices, particularly to Global Affairs Canada’s countries of focus.
   4. Provide sufficient support to PCE/ESU in terms of appropriate evaluation staff and budgetary resources.
2. Strengthening the sharing and use of evaluations.
   1. Present a periodic statement as a standing item on the number and status of decentralized evaluations planned, commissioned and completed by Branch, theme and budget at the Development Evaluation Committee.
   2. Develop a comprehensive knowledge management strategy including fora for sharing learning from decentralized evaluations, and the publication of decentralized evaluation reports in line with international practice.
   3. Build a stronger culture for evaluation knowledge sharing and use through better communication and use of different media (web, social, networks), as well as rewarding and recognizing high quality / good practice. Work towards achieving publication of all completed evaluations in the medium term.
3. Strengthening planning and conduct of decentralized evaluations.
   1. Strengthen the role, capacities and resources for decentralized evaluation work at Branch level. Achieve this by developing and implementing a training strategy to provide basic evaluation skills and more practical guidance to Branches.
   2. Strengthen ESU’s role as a knowledge broker rather than focusing on QA.^{Footnote 60} To achieve this, consider the option focusing the ESU on developing appropriate guidance and tools for decentralized evaluations, working more closely with Branches on selected evaluations, and developing better knowledge management systems. Build ESU capacity through staffing the unit with the right mix of evaluation skills and experience and organizational/contextual knowledge and expertise.
   3. Ensure that Branches plan decentralized evaluation on an annual basis with the guidance of ESU tools and staff.
4. Strengthening information management of decentralized evaluations.
   1. In the short to medium term, strengthen accessibility to decentralized evaluations. For example, PCE could maintain a consolidated list on the intranet and develop functionality that updates staff on completed evaluations, management responses and the key findings. Explore the use of existing databases (e.g. CRAFT) to this end.
   2. In the long term, create a platform for staff and external stakeholders to access all decentralized evaluations.