Internal Audit of Results-Based Management

Final Report
December 2012

Acronyms and Initialisms

Executive Summary

In accordance with the 2011-2014 Risk-based Audit Plan, the Office of the Chief Audit Executive at the Canadian International Development Agency (CIDA) conducted an internal audit of Results-Based Management (RBM). The objective was to provide reasonable assurance that the processes and controls in place to support RBM at CIDA are adequate and effective in support of decision making, accountability, and oversight.

RBM is a life-cycle approach to management that integrates strategy, people, resources, processes, and measurements to improve decision making, transparency, and accountability. The approach focuses on implementing performance measurement, learning, and adapting, as well as reporting performance, in order to achieve outcomes.

Strategic Policy and Performance Branch (SPPB) is the functional lead for RBM at CIDA, while the three program branches—Geographic Programs Branch (GPB), Multilateral and Global Programs Branch (MGPB) and Partnerships with Canadians Branch (PWCB)—are the main users of RBM to enable the achievement of development results.

The Agency's 2008 RBM Policy, Performance Management Agreements, and certain oversight bodies provide clear accountability with respect to RBM. Along with these robust accountability mechanisms, RBM leadership and understanding are indicative of a well-engrained RBM culture at CIDA.

At the Agency level, alignment of the Performance Measurement Framework (PMF) with the Program Alignment Architecture (PAA) provides assurance that the Agency's reporting to Parliament is in line with the expected results committed to in the Report on Plans and Priorities. Performance Management Strategies for CIDA's three thematic priorities exist, and although they can generally be traced to the PAA, they are not explicitly aligned with the PAA.

RBM instruments at the investment and intermediate levels are in use in GPB, and are used to inform decision making and reporting. In MGPB and PWCB, RBM takes place to inform decision making and reporting principally at the investment level from which performance information is not systematically channelled upward to the Agency level, although compensating measures exist to support external reporting. In all three program branches, opportunity exists to enhance the efficiency and effectiveness of RBM instruments.

Although some instruments are designed to provide management with all facets of performance information, including risk and financial information, in practice, integrated reporting is not being done at the Agency, intermediate, or investment levels. Furthermore, the lack of a standardized classification of indicators and the inconsistent use of the Agency's repository of performance information, combined with the varied and frequent requests for information, lead to an inefficiency of data collection and reporting.

Performance-related information is used by all three program branches to support decision making and course corrections at the investment level. However, an opportunity exists to strengthen linkages between Agency-level performance reporting and planning.

Conclusion

CIDA has implemented a number of RBM processes that are reinforced by a well-ingrained RBM culture. Although the audit found there were areas for improving RBM controls, compensating measures are in place to support accountability requirements, such as external reporting.

There are a number of initiatives underway that may help address some of the weaknesses noted in this report. Included among them are the work on "headline results and indicators," the review of the PAA, and a streamlining of the Monitoring and Reporting Tool (MRT).

Statement of Conformance

In my professional judgment as the Chief Audit Executive, this audit was conducted in conformance with the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing and with the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program. Sufficient and appropriate audit procedures were conducted, and evidence gathered, to support the accuracy of the findings and conclusion in this report, and to provide an audit level of assurance. The findings and conclusion are based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed upon with management and are only applicable to the entity examined and for the scope and time period covered by the audit.

Chief Audit Executive

1.0 Background

CIDA's strategic outcome is "Reduction in poverty for those living in countries where CIDA engages in international development." The Agency achieves this outcome by pursuing three priority themes: stimulating economic growth, increasing food security, and securing the future of children and youth. The multidimensional nature of CIDA's results is complex, and is further complicated by the three crosscutting themes of environmental sustainability, equality between men and women, and strong governance practices.

The Agency has implemented RBM to help deliver on these expected results and priorities. CIDA's RBM policy, practices and tools are intended to support sound corporate, program, and project planning and to enable program implementation, monitoring, and evaluation, as well as the Agency's reporting to Parliament, Canadians and international stakeholders.^{Footnote 1} RBM has been implemented in keeping with the Treasury Board Policy on Management, Resources and Results Structures, and the principles of aid effectiveness as outlined in the Paris Declaration on Aid Effectiveness and the Official Development Assistance Accountability Act. More than just a compliance requirement, RBM supports sound management and resource-allocation decisions, thereby contributing to effective stewardship, accountability, and value for money.

The means by which CIDA's RBM objectives are operationalized varies among the program branches in accordance with their particular operating realities.

Geographic Programs Branch comprises five regional directorates, each providing development assistance in a series of specific countries or regions. These individual country or regional programs are typically aligned with a development plan created by the government of that country. Of GPB's funding, 80 percent goes to 20 countries of focus. GPB's partners may include recipient governments, multilateral institutions, non-governmental organizations, universities, associations, private corporations, and other donors.

Multilateral and Global Programs Branch focuses its efforts on building strategic relationships with multilateral and global partners to achieve development results and to address the immediate needs of crisis-affected populations. The Agency is one of many donors to a given multilateral institution, each of which maintains its own RBM regime. CIDA leverages these institutional frameworks and works through the governance bodies of these organizations to influence institutional results. CIDA contributes to the achievement of multiple thematic priorities through its investment in a given multilateral institution.

Partnerships with Canadians Branch supports development projects and programs that are designed and implemented by Canadian organizations in cooperation with their counterpart organizations in developing countries. PWCB programming is responsive in nature. That is, through a formal call-for-proposal process, Canadian civil society organizations submit proposals for funding that align with the broad objectives CIDA wishes to obtain. Projects can be multi-country in nature, and will often contribute to more than one thematic priority.

The adjacent graphic provides a breakdown of grant and contribution funding by branch. The materiality of funding heightens the importance of managing for results. The many requirements for results reporting, driven in large part by the high degree of public and political scrutiny faced by CIDA and its partners, also increases the importance of RBM.

Chart description: CIDA 2010-2011 Grants and Contributions
Approved Budget by Branch - in $ thousands

GPB-$1,214,045 (54 percent)
MGPB-$822,911 (36 percent)
PWCB-$234,749 (10 percent)
Source: 2011-2012 Branch Integrated Business Plans

The inherent complexities and materiality of CIDA programming, coupled with the significance of RBM, led the Office of the Chief Audit Executive to identify RBM as an area of high priority. Accordingly, an audit of RBM was approved in the 2011–2014 Risk-based Audit Plan.

2.0 Audit Objective, Scope, Approach, and Criteria

2.1 Objective

The objective of the audit was to provide reasonable assurance that the processes and controls in place to support RBM at CIDA are adequate and effective in support of decision making, accountability, and oversight.

2.2 Scope

The scope of the audit was limited to 2010–2011; however, to ensure the relevance of audit findings, recent RBM practices that continue to evolve were taken into consideration. The audit examined RBM practices of SPPB as the functional lead for RBM at CIDA. As well, program branches—GPB, MGPB, and PWCB—were also included in the scope in light of their roles for practising RBM.

2.3 Approach and Methodology

The audit was conducted in accordance with the Treasury Board Policy on Internal Audit and the International Professional Practices Framework of the Institute of Internal Auditors. The audit's scope and testing protocols were determined based on an analysis of risk. Interviews and documentation review were conducted during the planning and examination phases with all program branches as well as with Communications Branch and Chief Financial Officer Branch (CFOB). As well, 22 project files, selected from all three program branches using judgmental sampling techniques, were examined. Once findings were developed, they were validated with representatives of all branches. These then formed the basis of this report.

2.4 Audit Criteria

The following three lines of enquiry were used that collectively set the standard against which the audit was conducted:

1. There is appropriate and effective leadership, accountability, and oversight of the Agency's performance-management regime.
2. RBM tools and measures are effective, and where appropriate, are aligned with one another.
3. CIDA leverages information collected through the established RBM tools to effectively support reporting.

3.0 Main Audit Findings and Recommendations

3.1 Governance and Accountability

One of the foundations of any key management practice, including RBM, is a robust governance regime that includes clear accountability for the function. In recognition of the importance of this, the audit sought to determine whether roles, responsibilities, and accountabilities related to RBM were well articulated, communicated, and understood.

The audit found that mechanisms to ensure clear accountability were robust and well entrenched. Of note is CIDA's RBM Policy (2008) developed by the Strategic Planning, Performance and Evaluation Directorate of SPPB, and readily accessible to all staff on the intranet. Accountability for the practice of RBM is clearly articulated in this policy. The policy includes a section entitled Accountability, which states that "CIDA will ensure that RBM is featured in Performance Management Agreements of its executives."

The audit also confirmed Performance Management Agreements were used to embed expectations for the implementation of RBM. Through this mechanism, executives are not only held accountable for RBM, but also for enabling the achievement of development results. While not responsible for the direct achievement of development results, the Performance Management Agreements provide a reasonable linkage between the enabling results against which managers are held accountable and the achievement of development results by CIDA's partners.

The clear documentation and guidance that define expectations with respect to RBM are complemented by a well-ingrained RBM culture across the Agency. All staff interviewed understood RBM, their RBM-related roles, as well as its importance to CIDA's operations. RBM leadership, understanding, and accountability for managing for results are all indicative of a strong RBM culture at CIDA. This culture in turn reinforces the more formal controls in this area.

Beyond individual accountability mechanisms, the audit also examined corporate governance structures used to oversee, and thus hold the organization accountable for, RBM. The audit concluded these mechanisms are appropriate and robust. Specific strengths include a strong degree of corporate leadership, process, and structures for the review and redevelopment of CIDA's PAA. Similarly, appropriate processes were found to be in place for the review and challenge of expected and actual results in CIDA's key accountability documents (i.e. Report on Plans and Priorities and Departmental Performance Report).

As well, the audit noted that various oversight structures were in place at the Agency and branch levels that contribute to accountability for RBM. At the Agency level, the members of the Management Board play an active role by reviewing and challenging the results outlined in the Annual Country Reports for CIDA's 20 countries of focus. At the branch level, the Branch Investment Review Committee within GPB reviews and challenges expected results prior to including them in project proposals submitted for approval. As part of PWCB's call-for-proposal process, subject-matter experts review and challenge expected results outlined by partners as part of their proposal submission. MGPB's Branch Oversight Committee plays an active role in reviewing institutional strategies and other strategic/policy directions, including relevant expected results.

3.2 Results-Based Management Instruments

The practice of RBM at CIDA is enabled by a wide range of instruments that allow for the identification of expected results; the measurement, capture, and analysis of actual results achieved; and the reporting of outcomes. A full compendium of these instruments is provided in Appendix B. The audit examined key instruments to determine whether they were adequate and effective in enabling RBM to be deployed across the Agency.

Further, the audit sought to determine whether there was vertical alignment of these instruments and results at the investment, intermediate, and Agency levels.^{Footnote 2} Vertical alignment is needed so that Agency-level expected results inform the development of lower-level expected results such that the latter are in line with the higher-order outcomes. This is necessary to avoid divergence of planned outcomes, and thus, of activities and expenditures. This alignment is also critical to ensure that results, once achieved at the lower levels, can be effectively and efficiently analyzed and aggregated in support of meaningful decision making and reporting. Simply stated, management needs to be able to answer the following questions:

• Are the investments being made on the ground the right ones, and are they relevant in the context of the higher-level outcomes?
• Are the results being achieved on the ground meaningfully and positively contributing to these higher-order outcomes whether at the intermediate or the Agency level?
• If not, how should resources be reallocated to better achieve the outcomes?

Agency-Level Instruments

At the Agency level, the corporate PMF, the highest-order RBM instrument of the Agency and the basis for reporting to Parliament, is aligned with the PAA. This alignment provides assurance that at the highest level, the Agency's accountability reporting is in line with the expected results committed to in the Report on Plans and Priorities. However, the audit also noted that, in practice, other Agency-level instruments are in use and are more predominant in operations. Of note, Performance Management Strategies for CIDA's three thematic priorities exist and are highly referenced across the Agency's operations. They are also integrated into program-related decision-making and oversight mechanisms. While thematic priorities can be generally traced to the PAA, the thematic Performance Management Strategies and the corporate PMF are not explicitly aligned. This means that thematic results, once realized, must then be translated into a different frame—that of the corporate PMF—for accountability reporting purposes. Current efforts are underway to develop a set of Agency-level headline results and indicators, which will be anchored on the thematic priorities. These headline results are being developed to clearly articulate CIDA's performance through understandable, plain language statements and targets. A review of potential changes to the PAA is also underway and includes consideration of the headline results. It was noted that over the last three years, the Agency has implemented or modified a number of its Agency-level instruments (e.g. corporate PMF, thematic priority Performance Management Strategies). While some level of evolution is appropriate, the use of a stable, common PAA for the depiction of expected results is necessary to regularize RBM.

Results-Based Management Instruments at the Intermediate and Investment Levels

Working from the Agency-level frames and expected results, the audit identified and assessed the adequacy and effectiveness of RBM instruments at various levels. Further, the audit sought to determine whether there was alignment from the highest level to the lowest level of operations: in other words, whether Agency, intermediate, and investment-level frames and results were mutually aligned.

The audit found that, within branches, a range of instruments exists to identify expected results and measures. In GPB, RBM instruments at the investment and intermediate (i.e. program) levels exist. They are aligned with one another, and with the PAA and corporate PMF. In PWCB and MGPB, the principal level at which RBM takes place is the investment level. Although expected results are articulated in the PAA at the subprogram level (i.e. the intermediate level, as outlined in Appendix C), they were not used by PWCB and MGPB as the basis of management, monitoring, or decision making as both branches have stated that they did not add value.

MGPB's investments are fundamentally different from the projects funded by GPB or PWCB in that CIDA is one of many donors providing long-term institutional support to these institutions, along with earmarked funding for global initiatives. To this end, the branch considers the alignment between an institution's expected results and CIDA's goals. By approving CIDA funding for an institution or global initiative, CIDA shares ownership of the expected results. MGPB plays an active role in the governance and RBM regimes of the multilateral institutions, and works with them to oversee and influence them in the achievement of expected results. In this way, programming supported by MGPB aims to contribute to the achievement of CIDA's expected results. The audit noted that the process for reporting how an institution's results have contributed to achieving CIDA's development goals could be strengthened. As a result, MGPB should look for opportunities to enhance the identification, measurement, and reporting of results.

Investment Monitoring and Reporting Tool

As noted above, based on interview evidence, concerns were raised in relation to the Investment Monitoring and Reporting Tool (IMRT), which is a key tool designed to support RBM at the investment level. While considerable progress has been made to date in implementing it across the Agency and although extensive guidance and training has been provided, the use and applicability of the tool were identified as areas for improvement. Data quality and analytical ability are at risk due to the high degree of manual entry, the limited requirements for the use of standardized indicators, and limited controls over data entry. Finally, the inability to link investment results in IMRT to higher-level results, specific countries, or themes works against data mining, analysis, or efficient multidimensional reporting. Management has stated it is aware of these IMRT limitations and will work with Information Management and Technology Branch to improve the tool accordingly.

Recommendation 1

The Vice-President of SPPB should ensure that the PAA and corporate PMF are used as the single, common frame for internal planning, budgeting, and reporting and external parliamentary reporting of results. Work being done on the Agency-level thematic (e.g. headline) results and indicators should be appropriately reflected in the PAA and corporate PMF.

Recommendation 2

The Vice-President of MGPB, in collaboration with the Vice-President of SPPB, should modify the branch's existing management instruments to enhance the identification, measurement, and reporting of results.

3.3 Use of Results Information

In addition to examining the existence and alignment of results information and instruments, the audit also sought to determine whether the information produced was used in support of comprehensive, efficient, and effective reporting and decision making.

Efficiency of Data Collection and Reporting

Requests for performance information, whether ad hoc or formal, come from various sources, including Communications Branch, SPPB, the Minister's Office, the general public, and parliamentarians. The audit found that efficiency of data collection and reporting could be improved. Specifically, the Agency's existing repository of performance information is not used consistently across branches. As a result, information to respond to these requests must be retrieved (often manually) from Management Summary Reports, Annual Country Reports, annual reports, the Project Browser, or partners.

There is no classification of key performance indicators to help standardize or efficiently develop PMFs at the investment or intermediate levels. At the same time, at the investment level, file review indicated there was a high number of indicators for which data is collected: some projects had more than fifty performance indicators outlined in their investment-level PMFs. While unique indicators may be needed for certain investments, many commonalities exist, and as a result, opportunities exist for a convergence of indicators. At the present time, more than 8,000 indicators exist in the IMRT, including a vast array of similar indicators, named differently. Management has stated that efforts underway to identify headline results and indicators and to improve the MRT will also allow a standardization of indicators. However, until such time as this frame is deployed, the risk of inefficiency in data collection and reporting remains high.

In GPB, Country Strategies are used to guide the Agency's programming and project development. Those related to the 20 countries of focus are approved by the Minister. These include a suite of results aligned with CIDA's thematic priorities. The Annual Country Reports developed by GPB serve as the mechanism against which the expected results outlined in the Country Strategies are reported. Prior to the development of Country Strategies, the Agency used Country Development Performance Frameworks to guide programming and project development. Although the Country Development Performance Framework includes a detailed logic model and program-level PMF, there was limited evidence that they are currently used to support project development and approval or program reporting and accountability. Rather, as noted above, Country Strategies are used for this purpose. As such, developing and maintaining Country Development Performance Frameworks may not be an efficient utilization of Agency resources.

Integrated and Holistic Analysis

The audit also sought to determine whether mechanisms are in place to provide management with all facets of performance information, including risk and financial information. The audit found that although some instruments are designed for integrated reporting, in practice, this is not being done.

At the investment level, one of the key tools intended for the purposes of integrated reporting and analysis is the IMRT. It has been designed to consolidate all facets of performance information. Evidence of consolidated performance information was found to be most evident in GPB, where IMRT is more widely deployed. However, in some files examined, it was noted that risk information, identified in the narrative part of the IMRT's Management Summary Report, was not reflected in the risk register section of the same report. In addition to the risk register not being up to date, in some cases, it was also incomplete.

At the intermediate level, integration of all facets of performance was found to be lacking. Because PWCB and MGPB do not produce performance reports at the intermediate level, the audit focused only on the Annual Country Report, which is the primary intermediate-level performance report in GPB. The audit found that although the Annual Country Reports provide a good mechanism for the analysis and aggregation of performance results information at the country level, limited financial and risk information was included.

At the Agency level, commencing in 2011–2012, branches were required to submit a monitoring report to CFOB as part of the midyear review process. The audit examined the tools used for this purpose to determine whether the monitoring process compels the consideration of all aspects of performance. The audit found that information provided in the midyear report is focused more on the types of performance monitoring activities in use in branches than actual performance.

Using Results to Plan and Course Correct

The audit found that opportunity exists to strengthen linkages between Agency-level performance reporting and planning. Specifically, no documented evidence exists demonstrating that results information is formally or systematically used to inform Agency planning, resource allocation, or the midyear review. It is important, however, to note that interview evidence indicated that results are discussed as part of these deliberations, although these are not documented. As well, the audit noted that performance information was integrated into Agency decisions related to the Government of Canada's Deficit Reduction Action Plan exercise.

In addition, the audit noted that performance-related information is used to support decision making and course corrections at the investment level across all three branches:

• GPB uses intermediate- and investment-level results information to support its decision making at all levels and to inform Agency reporting.
• In MGPB, although RBM instruments should be modified, results information is used, for example, in investment decision making and the development of Institutional Strategies.
• PWCB investment-level results information is useful for partners and Development Officers in monitoring investments.

Reporting Against Results

Although results information informs reporting and oversight at the investment level, these results are not systematically being rolled up into Agency reporting. This is particularly the case for PWCB and MGPB, where because of the lack of intermediate frames, there are no direct channels through which information is gathered, aggregated, and reported upon. MGPB and PWCB are able to select and provide performance information that supports Agency reporting. However, there remains a risk that Agency-level reports may not be fully or systematically informed by investment-level results information.

There are clear channels for systematic, upward reporting in GPB: Management Summary Reports feed Annual Country Reports, which in turn feed the Departmental Performance Report and Development for Results Report. At the branch level, although branches prepare Integrated Business Plans, there is no requirement to report against them.

More positively, the information that is aggregated upward is used and cross-verified to feed multiple Agency-level reports, including the Departmental Performance Report, the Development for Results Report, and the Report to Parliament on the Government of Canada's Official Development Assistance.

Recommendation 3

To support efficient and effective responses to requests for performance information, the Vice-Presidents of SPPB, GPB, PWCB, and MGPB should ensure that a single repository for performance information is used that meets the needs of program branches.

Recommendation 4

The Chief Financial Officer should ensure that all facets of performance (i.e. risk, financial, and results) are integrated in the Agency's midyear review process.

Appendix A: List of Recommendations and Management Action Plan

Appendix B: List of Results-Based Management Instruments

Appendix C: Results-Based Management Terminology at the Investment, Intermediate, and Agency Levels