Results-Based Management for International Assistance Programming at Global Affairs Canada: A How-to Guide

How to develop an outputs and activities matrix

Please refer to section 3.3, Step 3 d) for a detailed explanation of how to develop an outputs and activities matrix.

2.4 The Theory of Change Narrative

The theory of change narrative is a crucial complement to the logic model and the outputs and activities matrix. It describes the project’s theory of change and focuses on what is not explicit in the logic model and outputs and activities matrix, such as the logical links between project outcomes and the key assumptions that underpin these links. It also justifies these links, assumptions and other project-design choices with evidence and lessons learned from other initiatives or practitioners. The narrative should also address any major risks to the achievement of outcomes and describe the measures that have been—or will be—implemented to respond to them. If there are any changes to the logic model and outputs and activities matrix, the theory of change narrative may need to be updated.

The theory of change narrative can be a helpful tool for anyone new to the project to more fully understand its logic. More specifically, it can communicate the details and considerations that were raised during the situation analysis and logic-modelling process, and that cannot be communicated using the logic model’s structure. It is the only part of the project documentation that explicitly discusses assumptions, which are just as crucial to understanding the logic of the project as the expected results. A well written theory of change narrative can also serve as a project description.

How to draft a theory of change narrative

Please refer to section 3.3, Step 3 g) for a detailed explanation of how to draft a theory of change narrative.

2.5 Indicators

Indicators are the core component of the performance measurement framework.

It is important that the stakeholders agree beforehand on the indicators that will be used to measure the performance of the project.

Quantitative indicators

• Quantitative indicators are used to measure quantities or amounts.

Qualitative indicators

• Qualitative indicators capture experiential information, such as the quality of something, or beneficiaries’ perception of their situation. They can help measure the presence or absence of specific conditions, or an individual or group’s perception of how a service compares with established standards.
• Qualitative indicators can capture contextual information about situations, events and practices.

Note: There has been much debate regarding the value of quantitative data and that of qualitative information and whether quantitative measures (or indicators) are better than qualitative ones. This debate is now almost settled in the evaluation field with the growing usage of mixed methods. Practitioners have abandoned the idea that these sources of information are irreconcilable: both types of information are necessary. In fact, all quantitative measures are based on qualitative judgments and all qualitative measures can be coded and analyzed quantitatively.

To adequately assess the achievement of results, an officer/manager needs both quantitative and qualitative measures. For example, ‎it is not enough to know how many women are participating in an activity. The quality of their participation and experience is also important to capture to have a full picture.

Because it is difficult to organize qualitative data for comparison or analysis, qualitative indicators should be quantified wherever possible. This can be done by using a scale, for example, “level of confidence (1-4 scale) of farmers (f/m) in the security of roads leading to local market”.

Structure of a performance indicator

Performance indicators are composed of three elements: a unit of measure, a unit of analysis and a context.

The unit of measure is the first element of the indicator: number, percentage, level, ratio, etc. It is important to include in the unit of measure the notion of proportionality, by ensuring that it contains both a numerator and a denominator. This is often expressed by stating the unit of measure as number out of total (#/total) or percentage out of total (%/total).

The unit of analysis is who or what will be observed: individuals, institutions, social artifacts or social groups. The type of unit of analysis will determine whether the data will need to be disaggregated by sex, age, ethnicity, rural/urban setting, socio-economic status, ownership or any other category relevant to the project or program. This disaggregation is vitally important to the usefulness of the data collected. For example, it is impossible to measure changes in women’s access to basic services if the data collected during project monitoring does not disaggregate by sex. Similarly, a project that aims to improve the health of a specific marginalized ethnic group through rehabilitating and staffing remote regional health centres would need those centres to collect patient information in a way that allows disaggregation by ethnicity.

Table 5 - Unit of Analysis by Type

The context is the set of circumstances that specify the particular aspect of the output or outcome that the indicator is intended to measure. For example, if the expected outcome is "Improved access to government-funded primary schools for girls and boys of province X in country Y", and it has been determined that one way to measure progress is to see how many children live within a certain distance from a publically-funded school, then the context could be “living within a one kilometre walk of a provincially-funded primary school.”

Table 6 - Illustration of the Structure of a Performance Indicator

Leading, lagging and coincident indicators

We generally use indicators to measure progress on outcomes in the logic model. Sometimes, however, you may also want to measure the assumptions articulated in the theory of change narrative represented by the arrows in your logic model. In this case you can use “leading” indicators to measure things preceding the change or “lagging” indicators to measure things that follow the change. Data on these indicators can validate these assumptions. As explained above, at each level in the logic model, we are making assumptions. Leading and lagging indicators allow us to track those assumptions by measuring a little lower or a little higher than the actual outcome itself, without actually measuring the next level in the logic model.

Ideally, indicators would always measure things that directly coincide with the changes described in the expected outcomes of your logic model. In some cases it may be difficult or impossible to find such “coincident” indicators. In these cases, you can also use "leading" or "lagging" indicators.

The concept of leading, lagging, and coincident indicators is borrowed from the business cycle in economics. The following example of a traffic light is helpful to further explain the concept.

Types of changes measured by indicators

Each indicator can be classified according to what level it measures in the logic model: outputs, immediate outcomes, intermediate outcomes and ultimate outcomes.

• Output Indicators:
  Measure the products and services stemming from project activities and delivered by the implementer to the intermediaries or beneficiaries. For example, # of training sessions delivered, or the level of satisfaction (1-4 scale) of female and male training participants with the relevance of the training to their tasks.

• Immediate Outcome Indicators:
  Measure the changes in capacity, such as skills, knowledge, abilities, or awareness, and sometimes access, of the intermediaries or beneficiaries as a consequence of the outputs. For example, #/total community health workers (f/m) who pass certification exams.

• Intermediate Outcome Indicators:
  Measure the changes in behaviour, practice or performance of intermediaries or beneficiaries as a consequence of the immediate outcomes. For example, %/total health institutions (public/private) providing gender sensitive services to ethnic populations in their language of choice.

• Ultimate Outcome Indicators:
  Measure the sustainable changes in the lives of beneficiaries as a consequence of the intermediate outcomes. For example, average annual incomes of smallholder farmers (f/m) from all sources.

Criteria of a strong performance indicator

1. Validity: Does it measure what it is intended to measure?

• Will this performance indicator really measure the outcome and output? Remember that different indicators are meant to measure the different levels of the logic model.

2. Reliability: Will it be consistent over time?

• Does this performance indicator enable you to measure the outcome and output over time?
• Will it consistently produce the same data if it is applied repeatedly to the same situation over time?
• If a different researcher collects the data, will it be consistent?

3. Sensitivity: Will it measure changes as they happen?

• When the change described by the outcome statement starts to happen, will the indicator be sensitive enough to pick up on that change?
• Will the performance indicator measure both improvements and deterioration in the situation (i.e. is it neutral)?

4. Simplicity: How easy will it be to collect the data?

• Are the sources of information easily accessible?
• Are the equipment and/or expertise needed to track the performance indicator readily available?
• Does this performance indicator enable a relatively easy analysis of the result?
• Is it clear and direct enough to be understood by all stakeholders?

5. Usefulness: Will the information collected be useful for decision-making?

• Does the information provided by the performance indicator meet the needs of its audience?
• Does knowing this information help you or other stakeholders to do things better or more effectively?
• Is the performance indicator expressed in a way that will resonate with the intended audience?
• Does the performance indicator make it easy to communicate the status of the result?
• Will this performance indicator provide the information in a timeframe that allows it to be useful?

6. Affordability: Do you have the resources to collect data?

• Can you afford to collect data on this performance indicator?
• Is the potential cost worth the information you will get?

Other Considerations

As part of the Paris, Accra and Busan high level forums on aid effectiveness, Canada has committed to making increased use of existing country systems for monitoring. For this reason, Global Affairs Canada encourages project officers and project partners to use monitoring systems or indicators that may already be in place in the partner country.

Selecting an indicator that respects each of the criteria above can be challenging. Time, resources and other restrictions often mean settling for what is realistic rather than ideal. Choose performance indicators that provide the best possible measurement of the outcomes achieved within the budget available and wherever possible use existing data sources and collection methods. Look for a balance between rigour and realism. In the end, the most important indicator criterion is that you actually collect data for it.

2.6 The Performance Measurement Framework

At Global Affairs Canada, the performance measurement framework is the Results-Based Management tool used to systematically plan the collection of relevant indicator data over the lifetime of the program/portfolio and project, in order to assess and demonstrate progress made in achieving expected results. The performance measurement framework is the “skeleton” of the monitoring plan: it documents the major elements of the monitoring system in order to ensure regular collection of actual data on the indicators identified in the performance measurement framework. The performance measurement framework contains all of the indicators used to measure progress on or toward the achievement of the program/portfolio’s and project’s expected outcomes and outputs. In addition, it specifies who is responsible for collecting data on the indicator, from what source, at what frequency and with what method. It also includes the baseline data and target for each indicator.

As with the logic model, the performance measurement framework should be developed and/or assessed in a participatory fashion with the inclusion of local partners, intermediaries (duty bearers / responsibility holders), beneficiaries (rights holders) and other stakeholders, and relevant Global Affairs Canada staff.

Using the Performance Measurement Framework for Management

The performance measurement framework facilitates the “management for results” during program/portfolio and project implementation. It provides a plan for the collection of data during implementation. The actual data collected on indicators identified in the performance measurement framework, and the program/portfolio’s and project team’s analysis of this data, allows the team to assess progress, and detect issues that may interfere with the achievement of expected outcomes early enough to take corrective action/make adjustments. An operationalized performance measurement framework is thus necessary for evidence-based program/portfolio and project management decision-making. Of course, this can only be done if there is a basis for comparison. For this reason, it is always necessary to capture baseline data and it is always necessary to set targets in the performance measurement framework. Remember: without knowing where you started and where you want to go, it is impossible to properly assess progress.

In sum, the performance measurement framework will help you:

• plan for the systematic collection of relevant data over the lifetime of the program/portfolio and project;
• document the major elements of the monitoring system; and
• ensure regular collection of actual data for every indicator in the performance measurement framework.

The data collected on the performance measurement framework indicators will help you:

• measure and assess progress on or toward the expected outcomes;
• demonstrate to all stakeholders the progress made in achieving outcomes; and
• make evidence-based decisions to keep the program/portfolio and project on track to achieve the expected outcomes.

Standard template – performance measurement framework

Global Affairs Canada has a standard template for a performance measurement framework.

Content of the performance measurement framework

The performance measurement framework is divided into eight columns: expected results, indicators, baseline data, targets, data sources, data collection methods, frequency, and responsibility. To complete a performance measurement framework, you will need to fill in each of the columns accurately.

Expected results (first column)

This column of the performance measurement framework simply reflects the outputs and outcomes of the logic model. It is critical that any changes made in one document be reflected in the other, so that the outputs and outcomes identified in the performance measurement framework and logic model match at all times during the life of the program/portfolio and project. See section 2.1 and section 2.2.

Indicators (second column)

Indicators must be identified for each outcome and output of the logic model. See section 2.5.

How many outcome indicators?

For each outcome, select two to three indicators (you may include more if needed). Include at least:

• one qualitative indicator
• one quantitative indicator

Indicator(s) should measure specific dimensions of an outcome, such as access or quality, including gender inequalities or environmental sustainability, as applicable. Whether additional indicators should be qualitative or quantitative depends on the specific dimension of an outcome you want to measure. Note that wherever qualifiers such as transparent, participatory, effective, equitable or sustainable are added to an outcome statement, they need to be measured. In other words, ensure indicators selected measure each element of the outcome statement. This may mean that more than three indicators may be required.

Additional indicators could also contribute to triangulation, which is the process of gathering information on the same issue from multiple sources. Multiple lines of evidence increase the reliability of data. For example, you will have a more complete picture of the quality of services if you ask service users, talk to service providers, and check service records.

This helps to better validate outcomes, while keeping the overall number of indicators manageable. Remember that it is the cumulative evidence of data collected on a cluster of indicators that managers examine to see if their projects and programs are making progress. No outcome should be measured by just one indicator.

How many output indicators?

For each output, select one to two indicators (you may include more if needed).

An output indicator can measure different aspects of a product or service, for example:

• Quantity – the existence or amount of a product or service (e.g. # of gender equality materials produced [by type], or # of health staff [f/m] trained in gender equality situation analysis)
• Timeliness – the timeliness of its delivery
• Quality – the quality of the product or service (including its adherence to specific standards) or the quality of the process through which it was produced or provided. For example, for an output such as “gender sensitive needs assessment carried out”, an indicator could measure women’s perception of their level of equitable participation (on a five-point scale) in the consultations carried out in the needs assessment process.

Baseline data (third column)

Baseline data provides a specific value for an indicator at the outset of a project or program. Baseline data is collected at one point in time, and is used as a point of reference against which progress on the achievement of outcomes will be measured or assessed.

It is required in order to establish realistic, achievable targets. Baseline data is needed for each performance indicator in the performance measurement framework, and should be disaggregated by sex, ethnicity, age, socio-economic status or any other category relevant to the indicator.

When should it be collected?

Baseline data should be collected before project implementation. Ideally, this would be undertaken during project design. However, if this is not possible, baseline data must be collected as part of the inception stage of project implementation in order to ensure that the data collected corresponds to the situation at the start of the project, not later. The inception stage is the period immediately following the signature of the agreement, and before the submission of the Project Implementation Plan (or equivalent).

Targets (fourth column)

A target specifies a particular value, or range of values, that you would like to see in relation to one performance indicator by a specific date in the future. Together, the targets established for the various indicators of a specific expected outcome will help you determine the level of achievement of that outcome. Targets should be set in light of baseline data to ensure that they, in fact, are a good measure of achievement. Without this information, there is a risk of setting unrealistic targets or even of setting targets that are too easily, or already, achieved. In the performance measurement framework, the target column must show end of project targets, but annual targets can also be included. Ideally, annual targets should continue to be updated through the annual work plan process.

Targets provide tangible and meaningful points of discussion with intermediaries, beneficiaries, and other stakeholders. If key targets are missed, it is a signal for stakeholders and managers to collectively analyze how and why plans or strategies have gone off track, how they could be brought back on track, and then take corrective measures in constructive and mutually supportive ways so that outcomes are achieved.

Targets should never be embedded in expected outcome statements

Targets should only appear in the performance measurement framework and not be included in the expected outcomes statements themselves. At the planning stage, targets are often indicative until a project implementation plan or a first annual Work plan has been approved. They can also be adjusted within reason as part of sound management for results during the life of the project. This is one of the reasons that targets should not be embedded in the expected outcome statements.

Moreover, excluding targets from the expected outcome statements allows the theory of change to stand alone. The logic model does not need to be adjusted if a target is adjusted, and can even be replicated across similar programming and subsequent phases where the same theory of change could apply. In these cases, context-specific indicators and targets can be established separately.

What to keep in mind when developing targets

• Targets should have the same unit of measure and unit of analysis as the indicator for which they are set.
• If indicators are disaggregated by sex, ethnicity, age, socio-economic status or any other relevant categories, targets should be disaggregated in the same way.
• Targets must be developed using an established baseline.
• Targets must be developed for the end of the project, specifying the expected achievement date.
• Additional timelines for targets can also be set and vary from short to long term (e.g. monthly, annual, mid-term, etc.).
• Targets must be realistic and reviewed regularly.
• Beneficiaries, intermediaries and other stakeholders should be involved in establishing and reviewing targets. This will ensure local ownership and help establish targets that are achievable.

Data sources (fifth column)

Data sources are the individuals, organizations or documents from which data about your indicators will be obtained. The implementer will need to identify data sources for indicators. Data sources can be primary or secondary.

• Primary data is collected directly by the implementer at the source.
• Secondary data is data that have been collected and recorded by another person or organization, sometimes for altogether different purposes.

Primary data will always be project specific. Using secondary data, when relevant to your indicators and outcomes, can help the project save funds and generate synergies with partner country systems, other projects, or between donors/organizations.

Table 7 - Examples of Data Sources

Data collection methods (sixth column)

Data collection methods^{Footnote 35} represent how data on indicators are collected. Choosing a data collection method depends on the type of indicator and the purpose of the information being gathered. Data collection methods can be informal and less structured, or more formal and more structured. Different methods involve “trade-offs with respect to cost, precision, credibility and timeliness.”^{Footnote 36}

When choosing data collection methods, it is important to ensure that those who will be using the performance information, including Global Affairs Canada, are comfortable with the trade-offs that stem from the collection methods chosen, and thus the type of performance information they will be receiving.^{Footnote 37} Data sources and collection methods should be established by implementers in collaboration with stakeholders and with support from monitoring/evaluation specialists.

The figure below illustrates some possible data collection methods. “The more structured and formal methods for collecting data generally tend to be more precise, costly and time consuming.”^{Footnote 38} If your indicators are disaggregated (by age, sex, ethnicity, etc.), it is necessary to ensure that the related data collection methods can indeed enable the collection of disaggregated data.

Figure 5 – Data Collection Methods

Texte version

This diagram illustrates Data Collection Methods that can range from informal and less-structured methods, such as:

• conversation with concerned individuals;
• community interviews;
• field visits;
• reviews of official records (management information system and administrative data);
• key informant interviews;
• participant observation; and
• focus group interviews.

To formal and more-structured examples, such as:

• direct observation;
• questionnaires;
• one-time survey;
• panel surveys;
• census; and
• field experiments.

Source: World Bank Book, entitled Ten Steps to a Results-Based Monitoring and Evaluation System, p. 85.

Choosing a data collection method depends on:

• a project or an organization’s resources, access, needs, constraints, etc.;
• the type of indicator;
• how the information collected will be used; and
• how often this information will be gathered.

Data collection methods should not be chosen in an ad hoc manner. They should be carefully selected as part of the indicator development process, while recognizing associated costs and limitations. In fact, the identification of data collection methods and data sources can help with the selection and validation of realistic and affordable performance indicators.

Selecting appropriate data collection methods and sources

• Determine which data collection methods best suit the indicators in question.
• If you are using primary data collection, keep in mind age and gender differences and cultural context. For example, women may need to be interviewed by women; if focus groups are used, it may be necessary to have separate focus groups for women and men, or adults and children.
• Whether using a quantitative or a qualitative indicator, wherever samples are used, they should be representative. If this cannot be achieved, you must identify the limitations to representativeness.^{Footnote 39}
• Use multiple lines of evidence. For example, look at both government and World Health Organization statistics when collecting data on morbidity rates in a country.
• Weigh the pros and cons of each data collection method (accuracy, difficulty, reliability, cost, time).

Table 8 - Additional Data Collection Methods

Frequency (seventh column)

Frequency looks at the timing of data collection: how often will information for each indicator be collected or validated? Will information for a performance indicator be collected regularly (quarterly or annually) as part of ongoing management for results and reporting, or at specific times during the project cycle, such as at midterm or end of project?

Considerations for deciding how frequently to collect data for a performance indicator include:

• the level of the logic model (output and immediate outcome indicators tend to be collected more frequently than intermediate or ultimate outcome indicators);
• the level of risk related to a particular expected outcome; and
• the difficulty or cost involved in collecting the data.

Responsibility (eighth column)

Responsibility refers to who is responsible for collecting the data for indicators in the performance measurement framework. It is important to be specific when identifying the responsible actors in the performance measurement framework. Use a title or role rather than the name of an individual (for example, field officer, gender expert, project manager, etc.).

How to develop a performance measurement framework

Please see refer to section 3.4 for a detailed explanation of how to develop a performance measurement framework.

2.7 The Results-Based Monitoring and Evaluation Plan

A results-based monitoring and evaluation plan is a detailed plan that expands on the performance measurement framework and specifies the logistics, budgets and other operational details of data collection and analysis. It is important to note that the performance measurement framework, while being the “skeleton” of the plan for the systematic collection of data, does not contain enough information to guide the implementation of a monitoring system. A preliminary results-based monitoring and evaluation plan should be developed before the project is submitted for approval, so that required resources are taken into consideration during the budgeting process. The monitoring and evaluation plan can be finalized by the implementer as part of the project implementation plan or equivalent.

Monitoring

The results-based monitoring and evaluation plan should establish specific monitoring activities, responsibilities (for collection, analysis and storage of the data) and timelines. It should provide a detailed explanation of the data collection tools identified in the data collection methods column of the performance measurement framework. This includes describing how samples will be selected and how data will be analysed, captured, stored and used. It should highlight any expected challenges related to the collection and analysis of data (including baseline data) and target setting, as well as outline strategies for addressing these challenges. Finally, it should also commit specific financial and human resources to these results-based monitoring activities, which should be reflected in the project budget. This may involve the hiring of a project monitor, the allocation of dedicated project staff and financial resources to monitoring, and the establishment of a monitoring system to collect data on the output and outcome indicators in the performance measurement framework.

Evaluation

The results-based monitoring and evaluation plan should specify any evaluations to be undertaken and ensure that sufficient project resources are set aside. Evaluations may be commissioned by Global Affairs Canada, the implementer, or jointly with the implementer or other stakeholders. An evaluability assessment may also be included in the plan. See section 3.4, Step 4 f) for more details.

Synergy between monitoring and evaluation

There are significant opportunities for synergy between monitoring and evaluation, which can translate into significant savings in data collection at midterm and at the end of the project. See section 3.4, Step 4 f) for more details.

Part Three:
Step-by-Step Instructions

3.0 Introduction

There are four main steps to results-based project planning and design.

Texte version

Step 1: Identify design team and stakeholders

Step 2: Conduct situation analysis

Step 3: Develop theory of change, including LM, OAM and narrative

Step 4: Develop PMF and M&E plan

Part three of this guide presents steps to help project teams understand the processes or techniques used to develop Global Affairs Canada’s Results-Based Management tools.

Step 1: Identify design team and stakeholders.

The composition of the design team can have a significant impact on the quality of project design. As outlined in section 1.4, a gender equitable, participatory approach to project planning and design can yield tremendous benefits.

Step 2: Conduct situation analysis.

In its description of what Results-Based Management entails, the Results-based Management Policy Statement starts with “Results-Based Management means: defining realistic expected results based on appropriate analyses […]” (emphasis added). Situation analysis is therefore a fundamental step in results-based project planning and design.

Step 3: Develop theory of change, including logic model (LM), outputs and activities matrix (OAM) and narrative.

This step focuses on how to determine a project's expected outcomes and the means to achieve them, and how to document the assumptions that are being made and the external factors and risks that may influence the achievement of the outcomes.

Step 4: Develop a performance measurement framework (PMF) and a results-based monitoring and evaluation (M&E) plan.

The final step of results-based project planning and design is the development of tools that will enable the gathering and analysis of the information needed for proper Results-Based Management of the project throughout its implementation.

3.1 Step 1: Identify Design Team and Stakeholders

Get the right people on your design team

Identify the team to be involved in the project design. Ensure your design team includes local stakeholders, if possible. This will help the team avoid incorrect (and often unconscious) assumptions about the local context that could lead to poor project design and negatively influence the achievement of expected outcomes. The composition of your design team may vary depending on the type of programming but should always include:

• subject-matter experts, such as sectoral specialists and gender equality, environmental and governance specialists; and
• a performance-management or Results-Based Management advisor, or a monitoring and evaluation expert.

Make sure they’re available

Once you have identified your team, check that they are available and willing to participate in all four steps of the project-design process. Consider holding this process where the project will be implemented to facilitate the participation of local team-members and stakeholders.

Identify stakeholders and keep them involved

Since you will need to design the project in a participatory way, the design team^{Footnote 40} should always identify key stakeholders, including local intermediaries and beneficiaries, and ensure that they are involved and consulted regularly during the design process.

3.2 Step 2: Conduct a Situation Analysis

Situation analysis is a structured exercise that helps the design team: a) identify the issues they plan to address; and b) understand the complex context (national, regional, political, cultural, social, gender, environmental, etc.) in which those issues exist. This should be done through research, consultation, analysis and discussion. As such, a situation analysis is a fundamental part of results-based project planning and design. It provides a critical part of the evidence behind the theory of change.

The situation analysis will help you and other members of the design team:

• Develop a shared understanding of the problem and its various elements, including the gender equality dimensions
• Develop a shared understanding of the key players, and their roles, responsibilities and capacities
• Identify project specific information gaps that will need to be addressed through further research and consultation
• Choose which elements, among the many elements of the situation, the project will attempt to address
• Identify potential links to program-level planning, including complementary projects
• Compare the broad problem and its various elements to other planned or operational projects being undertaken by other donors and by partner governments and organizations to identify potential overlap, synergies and opportunities
• Identify potential expected outcomes
• Start to develop a theory of change for the project

How to conduct a situation analysis?

The first step of a situation analysis is to pinpoint the issue or need to be addressed. Common sources of ideas include:

• Partner country priorities and plans
• Existing needs assessments
• Reports from existing or completed projects (gaps or needs outstanding)
• Programsectoral and gender equality, environmental and governance analyses and strategies
• Global Affairs Canada corporate, thematic, sectoral, program or project evaluations
• Evaluations from other donors or partners, including joint evaluations

If none of these are available or useful, consider undertaking needs assessments, analyses or evaluations, as required.

Understanding the context

Once you have identified an idea or issue to be addressed, the next step is to understand the context in which this issue takes place (cultural, socio-political, gender equality, economic, and environmental), the roles played by stakeholders, and the issue's different impacts on the lives of women, men, boys and girls. You can use data and information from a number of different sources as the basis for this analysis.

For example:

• National development plan or poverty reduction strategy, and other relevant national strategies and commitments, such as those involving gender equality, environment, etc.
• On-site socio-cultural, gender equality, economic or political studies
• Scoping study that integrates gender-based, environmental and governance analyses of the problem
• Previously conducted analyses of the issue
• Interviews/meetings/focus groups with potential stakeholders, intermediaries and beneficiaries
• Evaluations of previous projects addressing the same or a similar issue
• Interviews/meetings with subject matter experts and gender equality, environmental and governance experts
• The theory of change for similar projects

Using the information

Once preliminary data has been gathered, there are many ways of using this information to establish a picture of the context and narrow the focus of the project. Common tools for this stage include:

• Problem tree analysis
• Gender analysis (required for all initiatives)
• Human rights and child rights-based analysis
• Stakeholder analysis
• Stakeholder mapping
• Capacity-gap analysis
• Conflict analysis
• Political economy analysis

Situation analysis tools used at Global Affairs Canada

Problem tree analysis

The problem tree is one of the methods used most frequently at Global Affairs Canada—although staff and partners may choose to use others. This is a visual situation analysis tool that enables its users to break down a very complex issue into its components, and then to examine and explore the cause-and-effect relationships between these components. It enables users to identify potential reach (intermediaries and beneficiaries), activities, outputs and outcomes for a project and gives users an idea of other key stakeholders and how they relate to and experience the issues. As such, it is particularly well suited to supporting the articulation of a theory of change and the development of a logic model.

Its key steps are:

1. Identify the core problem(s).
2. Identify the causes and effects.
3. Note the relationships.
4. Review the problem tree.
5. Create a solution tree.

Figure 6 - The Problem Tree

Texte version

In a problem tree, the trunk represents the core problem(s), the roots represent the causes of the core problem and the branches represent the effects.

Stakeholder mapping

Stakeholder mapping is another tool used during the situation analysis stage. Stakeholder mapping enables the design team to identify key stakeholders—including intermediaries and beneficiaries—their relationships to each other, and their level of interest in, and influence over, the issues at hand.

Stakeholder mapping can be done as a separate exercise or as part of the problem tree exercise. Key questions to ask for every issue explored are:

• Who owns this?
• Who controls this?
• Who decides this?
• Who is responsible for this?
• Who has the power to change this?

3.3 Step 3: Develop the Project’s Theory of Change, including Logic Model, Output and Activity Matrix, and Narrative

The logic-modelling process

Once the situation analysis is complete, you should be ready to develop the project’s theory of change and its logic model. This will involve determining the outcomes and outputs of the project, the activities best suited to producing the outputs, as well as identifying assumptions and evidence to explain how one change is expected to lead to another. This process is also known as logic modelling; see the “Logic modelling” heading under section 2.2 for more details.

There are different ways of undertaking logic modelling. A commonly used approach is to bring together the key stakeholders in one room and use sticky notes to brainstorm on the theory of change. Because sticky notes can be moved and re-ordered, this is a helpful and accessible way to engage in logic modelling in the early stages of the process; draft outcome and output statements (one per note) and then organize them to depict your project's theory of change.

Reminders

Use participatory methodologies to ensure equitable and valuable participation from relevant stakeholders throughout the entire process, from brainstorming together to completing a final draft. This will help meet the requirements of Canada’s Official Development Assistance Accountability Act.

When developing your theory of change, always keep in mind the country context and priorities, as well as Global Affairs Canada program, branch and corporate priorities. Also consider potential limiting factors such as duration and budget.

The best way to develop a theory of change is to start with the ultimate outcome before determining the intermediate and immediate outcome and deciding what programmatic approaches are needed.

When developing your outcomes and outputs, consider the level of gender-equality integration in your project in order to determine at which level of the logic model gender-equality outcomes should be included and how.

Step 3 a) Identify the expected ultimate outcome

You should work backwards from the ultimate outcome, as described in the steps below. Starting with the outcomes (ultimate, intermediate and immediate) will ensure that the outputs and activities selected are those that are required to lead to the changes described.

Working together as a team:

• Starting with the issue you identified and building on the situation analysis, identify the ultimate beneficiaries of the project, and the desired change in their well-being or in the state, or conditions of their lives.
• Draft a proposed outcome statement for your ultimate outcome, following the syntax outlined in section 2.1. Write it on a sticky note.
• Make sure you identify any assumptions inherent in the ultimate outcome as well as any risks that may impact the achievement of the ultimate outcome. Write them on sticky notes.
• Ensure the type of change described in your statement is a sustainable change of state, conditions or well-being of the ultimate beneficiaries (and not surrounding circumstances). For example, “improved economic prosperity of youth (f/m) working in the tourism sector of country X,” not “improved economic performance of the tourism sector in country X.” Refer to section 1.2 for examples of changes at the different levels of the results chain.
• Post your issue and ultimate outcome sticky note on the working surface (e.g. a wall) and place any assumptions and key risks nearby.

Reminders

The ultimate outcome is the “why” of the project. It should describe a sustainable positive change in state, conditions or well-being of the beneficiaries (the rights holders).

Although the ultimate outcome usually takes place after the end of the project, it is important to measure it during the life of the project. This is to assess whether the project is:

• beginning to contribute to the expected change in the lives of the beneficiaries
• still relevant (i.e. has there been a change in the circumstances for the beneficiaries, for better or for worse, which would require an adjustment to the project?)

If your project is specific to gender equality (i.e. the project was designed specifically to address gender inequalities or women's empowerment and would not otherwise be undertaken), you should have gender-equality results at all levels of the logic model, starting at the ultimate outcome level.

The ultimate outcome has to be realistically grounded in the project’s theory of change. For example, if the project is working in a village Y in country X to improve the health of single mothers, then the ultimate outcome cannot be “improved health of all men and women in country X.” It should reflect the reality of the project: “improved health of single mothers in village Y of country X.”

Step 3 b) Identify expected intermediate outcomes

Once you have identified the ultimate outcome, continue brainstorming as a team to develop the intermediate outcomes.

• Ask yourselves what changes in behaviour, practice or performance are required to lead to the change described in the ultimate outcome.
• Another way to look at this is to think about who (intermediaries, beneficiaries) needs to change their behaviour, practice or performance, and in what way, in order for the ultimate outcome to take place.
• Make sure you identify the assumptions you make about why these changes would lead to the ultimate outcome, as well as the risks that may prevent this from happening. Write them on sticky notes.
• Draft proposed outcome statements for your intermediate outcomes, following the syntax outlined in section 2.1. Write them on sticky notes.
• Post your intermediate outcome sticky notes on the working surface (e.g. a wall) below the ultimate outcome statement, and then place the assumptions and key risks near the outcomes to which they apply.

If your project fully integrates gender equality, gender equality results should be included at the intermediate outcome level and below.

For projects with moderate or high environmental relevance, environmental considerations should be integrated into the outcome statements. Ideally, this should be done at both the immediate and intermediate levels and, at a minimum, at the immediate level.

Step 3 c) Identify expected immediate outcomes

Once you have identified your intermediate outcomes, brainstorm the immediate outcomes making sure to identify everything required to allow each intermediate outcome to occur.

• Ask yourselves what changes in capacity (such as skills, awareness, and knowledge, and sometimes access), on the part of whom (intermediaries, beneficiaries), are required to allow the changes in behaviour, practice or performance described at the intermediate outcome level to occur.
• Make sure you identify the assumptions you make about why these changes would lead to the intermediate outcomes, as well as the risks that may prevent this from happening. Write them on sticky notes.
• Draft proposed outcome statements for your immediate outcomes, following the syntax outlined in section 2.1. Write them on sticky notes.
• Post each immediate outcome note on the working surface (e.g. a wall) below the intermediate outcome statement to which it most logically contributes, and then place the assumptions or key risks near the outcomes to which they apply.

Immediate outcomes will lead or contribute to the intermediate outcomes and represent the changes that are directly linked to the existence of outputs (products and services).

If your project has limited gender-equality integration, gender-equality results should be included at the immediate outcome level and below.

For projects with moderate or high environmental relevance, environmental considerations should be integrated into the outcome statements. Ideally, this should be done at both the immediate and intermediate levels and, at a minimum, at the immediate level.

Step 3 d) Identify main expected outputs and planned activities

• Continue brainstorming with your design team to develop the main outputs and associated activities for the project, making sure to identify everything required to allow each immediate outcome to take place. Write each output and its associated activities on sticky notes. Refer to section 2.1 and section 2.3.
• Make sure you identify the assumptions you make about why these outputs would lead to the immediate outcomes, as well as the risks that may prevent this from happening. Write them on sticky notes.
• Post each output and its associated activities on the working surface (e.g. a wall) below the immediate outcome statement to which it most logically contributes, and then place the assumptions or key risks near the outputs to which they apply.

Remember, the outputs represent completed products or services stemming from the activities of an implementer. Activities represent the separate components required to complete those products or services. Another way to think about activities is in terms of the work breakdown structure: activities are the next level of breakdown under the outputs. Refer to section 2.3.

Step 3 e) Validate the theory of change

Arrange all of the sticky notes with your outcome and output/activity statements in the pyramid shape of the logic model. Check back and forth through the levels (from ultimate outcome to activities and from activities to ultimate outcome) to make sure everything flows in a logical manner and that the theory of change is sound and evidence-based, incorporates sectoral best practices and lessons learned, and integrates  gender equality, environmental sustainability and governance in international assistance programming. Make sure that each outcome is well supported by the level below. Make sure that all activities and outputs contribute directly to the immediate outcome for which they were identified. Make any adjustments required, such as moving or adding outcomes or outputs and activities.

Validate any assumptions and risks and make sure you document them so you can use them when you write your theory of change narrative (see section 3.3 Step 3 g)).

Check that your outcomes statements are robust and meet the criteria identified in section 2.1. One way to do this is to brainstorm potential indicators for each outcome. This helps you ensure that you will be able to measure the achievement of your outcomes. It also helps ensure you have not identified an indicator and tried to formulate it as an outcome (e.g. reduced maternal mortality rate) rather than use a proper outcome (e.g. improved maternal health ....). Early identification of indicators is a useful technique for refining outcome statements and the overall theory of change of a project.

Global Affairs Canada has developed the Logic Model Checklist to help staff and partners assess the soundness of the project’s design/theory of change as reflected in the logic model and outputs and activities matrix, or in other results framework tools.

Step 3 f) Pull it all together

Fill out the logic model template using the outcome and output statements you’ve developed during your brainstorming sessions.

Figure 7 - Completed Logic Model

Texte version

This is an example of a completed Logic Model. Each level of the results chain stems from the level below.

Ultimate outcome: 1000 Improved health of women, men, girls and boys living in region Y of country X.

Intermediate outcomes:
1100 Increased equitable use of clean drinking water by women, men, girls and boys in region Y.
1200 Improved provision of front line gender-responsive health services to women, men, girls and boys in region Y

Immediate outcomes:
1110 Improved equitable access to clean drinking water for women, men, girls and boys in region Y.
1120 Increased ability to maintain wells among female and male members of community water collectives in region Y
1210 Increased equitable access to health facilities for women, men, girls and boys in region Y.
1220 Improved skills of local health centre male and female staff in gender-sensitive triage, diagnosis, and primary health care in region Y.

Outputs:
1111 Wells built using gender equitable participatory approaches in region Y.
1112 Existing wells of region Y rehabilitated using gender equitable participatory approaches.
1121 Training on well maintenance developed and delivered to female and male members of community water collectives in region Y.
1122 Technical assistance provided to community water collectives for the sourcing of parts from local and regional suppliers.
1211 Regional health centres in region Y rehabilitated and equipped.
1212 Gender-sensitive awareness campaign on the availability of health services in newly rehabilitated regional health centres conducted.
1221 Gender-sensitive materials for skills development programs and on-the-job coaching on triage, diagnosis and primary health care developed.
1222 Gender-sensitive skills development programs and on-the-job coaching on triage, diagnosis and primary health care provided to male and female staff in regional health centres.

*Note: In the context of this project, gender sensitive is defined as: gender sensitive awareness campaign, training materials, and programs that are designed based on gender analysis to promote equal roles for women and men in healthcare (e.g. women and men as doctors and women and men as care providers); to challenge gender stereotypes and biases that lead to discrimination and harmful practices (e.g. boy preference, sexual abuse/harassment, gender-based violence); to support the rights of women and girls in health decision-making, particularly in sexual and reproductive rights; and to promote equal participation of, and benefit to, women and men (girls and boys).

Fill out the outputs and activities matrix, copying the immediate outcomes and outputs from the Logic Model and listing the activities for each output.

Figure 8 - Completed Outputs and Activities Matrix

Step 3 g) Write a narrative description of the theory of change

Every logic model should be accompanied by a narrative that describes the theory of change for the project. This narrative should be developed iteratively with the logic model. It should focus on what is not explicit in the logic model, and how the expected outcomes of the project will unfold. It should explain the linkages between each level, including assumptions and risks, and provide reference to the evidence and best practices that justify the design choices made. These linkages and assumptions should include the roles and contributions of other actors not directly involved in the project but on whom the achievement of project outcomes also depends. For example, reference should be made to any recipient-country government commitments, policies and programs important to achieving project outcomes. See section 1.2 and section 2.4.

One to three pages long

Ideally, the narrative should be one to three pages long. However, you may find that large projects will require more pages to cover the breadth of the initiatives.

Structure it by outcome

Start the narrative with a section discussing how the intermediate outcomes (end-of-project results) will contribute to the ultimate outcome. Focus your attention on the relationships between each intermediate outcome and the ultimate outcome, explaining the theory, best practice, assumptions and risks underlying your choice of intermediate outcomes.

Create separate sections/paragraphs for each intermediate outcome, where you explain how the project’s outputs will lead to the immediate outcomes, and how the immediate outcomes will lead to that intermediate outcome. Focus your explanation on the relationships between the outputs and the immediate outcome, and the immediate outcomes and intermediate outcomes, explaining the theory, best practice, assumptions and risks underlying your choices. Where applicable, include how environmental sustainability, gender equality and governance are integrated throughout the logic model to the intermediate outcome level.

Focus on assumptions

Describe the most important assumptions made with each step of the logic model (i.e. the ones without which the next level of outcomes could not be achieved). You should use references, quotes and evidence from your socio-economic, cultural, political, environmental, and gender analyses and consultations to justify the assumptions made at each step of the logic model.

Identify risks and response strategies

Include a brief mention of any key risks and contextual factors that could influence the achievement of outcomes. Identify any risk response strategies that you are undertaking or are planning to undertake.  

Refer to the work of other actors

Refer to the work other organizations are doing in the area and describe how their outcomes may relate to the project’s theory of change. In some cases, the work of others may explain the choices made in project design (e.g. choosing not to undertake an activity because it is being done by another donor).

Describe how participation will be encouraged

Describe the methods that you will use to foster participation of a broad range of stakeholders (including intermediaries and beneficiaries) throughout the project’s life cycle.

Step 3 h) Assessing the logic model and the outputs and activities matrix

Once a project has been designed, it must be reviewed, both for quality control and as part of the decision-making and approval process before moving to implementation. Whether reviewing your own design or that of a proposed project from another organization, the same guidance applies.

Global Affairs Canada staff should use the Logic Model Checklist when reviewing the logic model submitted by applicants with their proposals. Note that although many of the questions in the checklists are specific to Global Affairs Canada’s terminology and tools, the Logic Model Checklist includes questions related to the use of non-Global Affairs Canada Results-Based Management tools. The kinds of questions asked can inform a review of proposals where Global Affairs Canada has agreed to use partners’ own templates.

The assessment of the logic model and outputs and activities matrix should also be done in a participatory manner. Share the draft logic model, outputs and activities matrix and narrative with colleagues, thematic, sectoral and gender equality, environmental and governance  specialists, stakeholders, including beneficiaries, etc. This will help ensure that the final version reflects the shared understanding of the project’s theory of change, including the expected outcomes, assumptions and design.

3.4 Step 4: Develop a Performance Measurement Framework and a Results-Based Monitoring and Evaluation Plan

Key considerations

The development of a performance measurement framework starts at the design phase as an iterative process that goes hand-in-hand with logic-modelling. You should ensure that the content for the performance measurement framework is developed in a gender-balanced, participatory fashion. Include key local stakeholders, partners, beneficiaries, and appropriate specialists (sectoral, gender equality, environmental and governance) in the process.

As you develop your performance measurement framework, consider the following factors:

• local stakeholders’ experience and level of capacity to monitor project performance
• existing indicators tracked by host governments or other stakeholders that could be used to measure the project’s expected outcomes
• the reliability and validity of the data collected on such existing indicators
• opportunities to use and strengthen government systems
• current data gaps that the project could fill

Step 4 a) Copy expected outcome and output statements from the logic model to the performance measurement framework

Copy and paste the most recent outcomes and outputs from your logic model into the boxes of the first column in the performance measurement framework template.

Step 4 b) Formulate indicators

Establish performance indicators for all of your expected outcomes and outputs, following the guidance outlined in section 2.5. You will normally have started thinking about appropriate indicators during the process of validating your logic model and theory of change.

The process of identifying and formulating indicators may lead you to adjust your outcome and output statements. Ensure any changes made to these statements in the performance measurement framework are reflected in the logic model. If the logic model has already been approved by Global Affairs Canada and stakeholders, keep in mind that any changes in scope, scale or intent of the project need to be approved by the original approval authority at Global Affairs Canada. See section 4.2 for more details on this subject.

Validate and check the quality of your performance indicators, using the Global Affairs Canada’s Performance Measurement Framework Checklist. For example, are the indicators valid, reliable, sensitive, simple, useful and affordable? Are they gender sensitive? Where appropriate, do they include proportionality? If they deal with people, are they disaggregated by sex and any other categories of concern to the project?

Step 4 c) Determine data sources, and collection methods, frequency and responsibility

Determine the data sources and data collection methods for your chosen performance indicators. Look to include multiple lines of evidence wherever possible to increase the reliability of the data you will collect on your indicators. Consider sampling strategies. If you collect data from beneficiaries, will you collect it at the household, community or other level? For example, if your data source is school-aged children, will you collect data in the classroom or, assuming not all children are in school, within households? At first glance, the data collection at the household level may seem too costly, but with a statistically valid sampling methodology, it might well be within your reach. Sampling can help reduce cost while maintaining data reliability and validity.

Your examination of available data sources and relevant data collection methods may lead you to adjust your choice of indicators.

The frequency of data collection and the responsibility for gathering it are often a function of the data source. For instance, the data source might be an annual government report. In this case, collection frequency would be annual, in line with the frequency of the report, while the responsibility would likely rest with the implementer to collect it from the government source.

Determine the frequency and responsibility (for data collection and analysis) for each performance indicator. This would also be a good time to assess the cost of data collection.

Step 4 d) Enter baseline data

Use the data collected during the project baseline study to complete the baseline data column in the performance measurement framework.

If the baseline data study was not conducted during project design, but will be conducted during the inception stage of the project:

• Indicate this in the performance measurement framework with a statement such as: “Baseline data to be collected during project inception between XX/XX/XX – XX/XX/XX” or “Data to be established by XX/XX/XX.” Set the date by which this will be completed. Ideally, this should be done no later than 90 days after the agreement with Global Affairs Canada is signed.
• Some estimated baseline data may need to be provided for key indicators for design and planning purposes and for the setting of targets. In this case, indicate what the estimated value is and when the baseline will be confirmed (“Estimated baseline: X. To be confirmed during project inception between XX/XX/XX – XX/XX/XX” or “Estimated baseline: X. Data to be confirmed by XX/XX/XX.”). For project implementation, estimated values must be replaced by actual baseline data collected during project inception; the baseline column in the performance measurement framework must be updated. In other words, there should not be estimated baseline data in the performance measurement framework submitted with the project implementation plan (or equivalent).

Step 4 e) Define targets

Select your targets and determine expected achievement date

Your targets set the expectations for performance by the end of a fixed period of time, usually the duration of the project. This will help you determine realistic budgeting, allocation of resources and end-of-project scope and reach. You are also encouraged to set annual targets through the annual work plan, which will help you to better monitor progress over time.

If a baseline study has been conducted:

• Establish realistic targets for each indicator in relation to the baseline data you have identified.

If the baseline data will be collected later:

• You must, at minimum, establish targets for the key outputs and outcomes that will help you determine realistic allocation of resources and end-of-project scope and reach. This requires you to estimate some baseline data using, for example, reliable historical data/trends on your performance indicators (e.g. government data, information from a previous phase of the project, or information gathered during a needs analysis or situation analysis).
• Once all the baseline data have been collected, remember to provide targets for all of your indicators (and validate preliminary targets set for key indicators).
• A completed performance measurement framework, including any adjustments to baseline, targets or even indicators, must be submitted to Global Affairs Canada as part of the project implementation plan or equivalent.

Step 4 f) Draft a preliminary results-based monitoring and evaluation plan

Monitoring

• Review all indicators to determine whether, based on their data source or data collection method, they require specific arrangements to be made or instruments to be developed, such as interview guides, questionnaires, forms or ranking mechanisms.
• Make plans to test and adjust any data collection instruments prior to their use.
• Plan to provide training in data collection methods and the use of monitoring instruments to staff members, stakeholders or any others who will take part in data collection.

Resources for the development and testing of data-collection instruments, as well as for training staff and stakeholders, need to be allocated in the project budget.

By the time the monitoring and evaluation plan is finalized, the implementer should be able to answer questions related to the cost of data collection, sampling methodologies, sample sizes, statistical analyses to be used, data-capture templates and data-storage systems.

Evaluation

The evaluation component of your monitoring and evaluation plan should specify the following:

• Rationale and purpose: Why is the evaluation being undertaken? Why at this particular point in time? For whom is it being undertaken? Will it be used for learning, accountability or some other purpose?
• Specific objectives: What is the evaluation trying to find out?
• Tentative key questions: At the design stage of a new project, it may already be possible to identify key evaluation questions of interest to the stakeholders. For example, if the project is implementing an innovative approach, what are the elements one would like to assess and when? This informs both monitoring and evaluation data needs and ensures timely data collection.
• Scope: What is being evaluated? Is it a specific project component, activities taking place in a particular geographic area or something else?
• Timing: When will evaluations take place?
• Responsibility: Who will manage the evaluation? How will it be governed?
• Budget: How much will it cost to manage this evaluation or participate in it?
• Aside from a fully completed performance measurement framework, what should be put in place now in order to evaluate performance in a few years?
• Previous evaluations: Are there previous evaluations of similar projects, especially earlier phases that can help you plan this project and its evaluations better?
• Evaluability: Is an evaluability assessment necessary?

Evaluability Assessments

An evaluability assessment goes far beyond providing information regarding whether or not an initiative or a project can be evaluated. It also:

• reviews program design, and logic model and/or theory of change;
• assesses program performance measurement frameworks and monitoring systems;
• critically assesses the validity, reliability and usefulness of the baseline, monitoring and other available data sets;
• validates planned evaluation needs from the standpoint of stakeholders and users; and
• informs evaluation design to maximize the quality and utility of the planned evaluation.

Synergy between Monitoring and Evaluation

Answer the following questions during the development of the monitoring and evaluation plan to strengthen the synergy between monitoring and evaluation:

• Has consideration been given to involve an evaluator to assess and validate the theory of change, performance measurement framework and data collection instruments at the inception of the project and during its lifecycle?
• How will the performance measurement framework and the data collected on its indicators, including baseline data, facilitate or inform the evaluator’s work?
• Is there any data outside the performance measurement framework that could help to inform an evaluation?
• In the case of formative evaluation, can any of the data collected by the evaluator be used for the indicators in the performance measurement framework?
• How will roles and responsibilities be distributed between the implementer, other stakeholders, any external monitor, Global Affairs Canada staff and the evaluation teams?

Finalize the Monitoring and Evaluation Plan as part of the Project Implementation Plan

The results-based monitoring and evaluation plan should be finalized and submitted to Global Affairs Canada for approval as part of the project implementation plan or equivalent.

Step 4 g) Assess the performance measurement framework

Whether reviewing your own design or that of a proposed project from another organization, the same guidance applies.

Global Affairs Canada staff should use the Performance Measurement Framework when reviewing the performance measurement framework submitted by potential implementers with their proposals. Note that although many of the questions in the checklist are specific to Global Affairs Canada’s terminology and tools, the Performance Measurement Framework Checklist includes questions related to the use of non-Global Affairs Canada tools. The kinds of questions asked can inform a review of proposals where Global Affairs Canada has agreed to use partners’ own templates.

Implementers and Global Affairs Canada staff can also use this checklist during the iterative process of developing their performance measurement framework in order to validate and improve it.

The assessment of the performance measurement framework should be done in a participatory manner with the relevant stakeholders and subject matter experts.

Part Four: Managing for Results during Implementation

4.0 Introduction

Managing for Results during program/portfolio and project implementation entails collecting data on indicators that measure both the output and outcomes, using this and other information to compare expected outcomes with actual outcomes, and adjusting/adapting operations during implementation in order to optimize the achievement of the expected outcomes.

This cycle of measurement, evidence-based assessment of progress on or toward the expeted outcomes, learning and adjustment during implementation (as illustrated by the diagram below) is what makes Results-Based Management / Managing for Results an adaptive management approach as opposed to just an indicator data collection or reporting exercise.

Texte version

Managing for results during implementation. What do we do during implementation to manage for results to achieve expected outcomes?

• We collect data on indicators (qualitative and qualitative) that measure outputs and especially outcomes.
• We analyze the collected indicator data and other information.
• We use the analysis for an assessment of progress on or toward the expected outcomes.
• This assessment, supported by indicator data, is used in a purposeful way for:
  • Learning what works and what does not, and integrating lessons learned into evidence-based decision making.
  • Adjusting and adapting programming as needed to achieve our expected outcomes.
  • Evidence-based narrative results reporting and communication.

Lessons learned should be shared and integrated into the decision-making process for other similar current, as well as future programming.

4.1 Monitoring and Indicator Data Collection

Throughout implementation, Global Affairs Canada staff and the implementer monitor the project in different ways, according to their roles and responsibilities. The implementer has primary responsibility for collecting and analyzing data on all the indicators of the performance measurement framework, according to the frequency and data collection method indicated. More detailed information on data collection, including schedules and tools such as questionnaires, forms, etc., are normally set out in the results-based monitoring and evaluation plan. Monitoring by Global Affairs Canada staff varies according to the type of project or investment. It always entails reviewing reports, but can also include site visits, cross-referencing with other stakeholders, or the hiring of external monitors.

As discussed in section 1.3, collecting data on the project’s indicators on a regular basis empowers managers and stakeholders with real-time information on progress on or toward the achievement of outcomes. This helps identify strengths, weaknesses, and problems as they occur, and enables project managers to take timely corrective action during project implementation. This in turn increases the likelihood of achieving the expected outcomes.

Data collected during implementation is also a crucial foundation for evaluations. As mentioned earlier, the cost of an evaluation can be greatly reduced by diligent monitoring and documenting of the achievement of results. Moreover, evaluators come on board at a particular point in time. Even when they devote significant efforts to data gathering at the time of evaluation, the resulting data set can never replace the wealth of information generated through continuous results-based monitoring and performance measurement. For example, even the best evaluation-recall techniques cannot replace missing monitoring data needed to properly analyze trends for the project or program/portfolio being evaluated.. Lack of data can limit the quality of an evaluation or in some cases make evaluation too expensive to conduct. In short, monitoring indicator data is essential.

4.2 Making Adjustments to the Logic Model and Performance Measurement Framework of an Operational Project

The logic model and performance measurement framework are developed in the planning and design stage, but, as discussed in section 2.0 and section 2.6, they are not static. They are iterative tools that can and should be adjusted as required during implementation as part of ongoing management for results. The logic model and the performance measurement framework should be validated during the development of a project implementation plan or equivalent. As project circumstances evolve or as the analysis of the data collected on the indicators suggest adjustments are required to achieve the expected outcomes, additional changes to these tools may be required.

An advisable time to make these changes is during the submission of the project implementation plan and thereafter during the submission of the annual work plan or during project steering committee meetings. Regardless of timing, any such changes must be discussed and agreed amongst all project stakeholders, including Global Affairs Canada. In addition, such adjustments must be justified, documented and not change the scope, scale and intent of the project.

In particular, any change to intermediate or ultimate outcomes, or to targets at any level of the logic model, should be discussed with Global Affairs Canada to assess whether they constitute a change in scope. Examples of changes in scope include changing the geographic scope of a project, changing the project reach (i.e., number and type of beneficiaries) or removing, adding or significantly altering an outcome or targets at the outcome and output levels. If Global Affairs Canada determines that the changes constitute a change in scope and\or imply significant increases to the resources/funding required, this will trigger an amendment to the financial instrument used by the project and will need to be approved by the original approval authority at Global Affairs Canada.

Changes to the logic model and performance measurement framework that trigger the Canadian Environmental Assessment Act (2012) will require that steps be taken in compliance with the Act.

4.3 Reporting on Outcomes

Reporting is an important part of an organization’s ongoing operations and decision-making. Reporting helps to promote a continuous feedback loop in which reports on activities, outputs, and outcomes provide information and analysis for decision-making over the life of a project.

What is reporting on outcomes?

Results-based performance reporting is the process of reporting on progress on or toward the achievement of the expected outcomes: comparing what you expected to achieve with what you have actually achieved, and explaining any variation between the two. To report on outcomes, implementers must assess actual outcomes based on actual data collected during implementation on the qualitative and quantitative indicators identified in the performance measurement framework.

Why report on outcomes?

Reporting on outcomes, and not only on outputs, supports decision-making, ensures accountability to Global Affairs Canada, local stakeholders and Canadians, and provides a basis for citizen engagement in Canada and partner countries.

Reporting is thus more than a vehicle for meeting accountability requirements. Reports are important management tools that allow implementers, key stakeholders and Global Affairs Canada staff to:

• reflect strategically about the project, and the theory of change that informs it, in an ongoing way
• identify challenges and issues influencing the project’s ability to deliver expected outcomes (results)
• use performance information to make timely, evidence-based adjustments to the project
• draw lessons for improving development/programming effectiveness during the life of the project and beyond
• communicate about the project’s overall performance and outcomes achieved

Reporting includes a systematic analysis of the progress the project is making on or toward its expected outcomes, which supports a rigorous results-based approach to project management. It also provides a basis for assessing and communicating a project's contribution to broader programming.

Reporting guidance

The following guidance offers general advice on how to report on actual outputs and outcomes (i.e., outputs and outcomes achieved).

A results-based report (quarterly, midyear, annual or final) is a performance story about actual outcomes (substantiated by data collected on the indicators identified in the performance measurement framework) as compared to expected outcomes from the logic model. Any variance between the two should be explained, and include an assessment of their significance and impact on the project. The performance story should be contextualized, including a discussion of any risks that occurred and how they were addressed.

For every indicator

• Provide actual data as per the collection frequency identified in your performance measurement framework.
• Analyze the actual data (comparing it to corresponding baseline data and targets), and then use these data and this analysis as the basis of your narrative assessment of progress on or toward expected outputs and outcomes.

Describing progress

When describing progress made on or toward achieving outcomes and outputs, implementers should provide an evidence-based narrative that uses the actual data collected on the indicators (qualitative and quantitative) identified in the performance measurement framework. In other words, actual data provides the evidence that supports the assessment and assertion made by the implementer about the status of progress on the expected outcome or the outcomes achieved.

For each output and immediate outcome

• Describe progress made during the reporting period.
• Discuss the cumulative progress from project inception to date.
• Explain any variances as well as any unexpected outcome (negative or positive).

For each intermediate outcome and ultimate outcome

• Describe the cumulative progress from project inception to date.
• Explain any variances as well as any unexpected outcome (negative or positive).

Other considerations

Ensure that all narrative text on outcomes not only describes the change that has taken place, but also provides sufficient context and gives a sense of proportionality, for example:

• Where did the outcome occur (region and distribution in the region)?
• Who, and how numerous, were the beneficiaries or intermediaries who experienced the change (women, men, girls and boys, specific groups, organizations)?
• Where relevant, examples should include evidence that illustrates progress related to gender equality, environmental sustainability, and governance.
• Any factors or elements that could provide an explanation of performance (changes in political context, disaster, etc.) should be mentioned.

Ensure that all of your explanations are clear and concise. If unexpected outcomes occur, report on these as well.

Illustrative example - Reporting on outcomes^{Footnote 41}

Expected Intermediate Outcome: 1100 Increased environmentally sustainable use of potable drinking water by households in Region X

Progress from Project Inception to Date (Cumulative):

There has been a modest increase in the use of potable water by households in Region X since the start of this project in early 2008, from 24% of 250 community A households and 10% of 100 community B households, to at least 40% and 15% respectively as of 2009 (last reporting period). Although a household survey is not being conducted this year, evidence gathered through observation and conversations with stakeholders in the community, including the Women’s Water Collective (which has members from both communities) indicates that more households of both communities are using the wells this year.

The female head of a small farming household (community A) on the outskirts of Region X, said, “I got water from the well this year, instead of the river like I used to. Last year my children got sick often and my daughter did not have time to go to school. Now they seem healthier and she can go to classes almost every morning.”

The completion of several outputs to date, including the construction of 5/6 wells, two training sessions on well maintenance and 10 community awareness-raising sessions on the use of safe drinking water have led to an increase in access to safe drinking water and understanding of its importance by the members of the two communities (see actual data on immediate outcomes #1110 and 1120), both of which have contributed to this increase in use.

Furthermore, this increased use of safe drinking water is demonstrated by the fact that fewer women from both communities are using the river as their source of water. At the start of the project, 80% of 520 women (community A) and 95% of 260 women (community B) used the river daily. As of January 2010, 65% of 520 (community A) and 90% of 260 (community B) were using the river. As trends for this indicator show, the desired changes are not being experienced equally by both communities. As discussed in reporting on output 1111, construction of the last of the three wells planned for community B neighbourhoods (the one for the most populous neighbourhood) had to be postponed because of a risk of contamination from latrines located too close to the planned site. The construction of the remaining well next year should correct this imbalance.

60% of women (community A) and 23% of women (community B) now feel they use clean water most or all of the time. This is an increase from 35% and 15.4%, respectively, since the start of the project. A discrepancy between the percentages of women who feel they use clean water and those who probably use safe water (considering the numbers still using water from the river) indicates the need for more community awareness-raising.

Members of the Women’s Water Collective are continuing to inspect the wells to ensure their use doesn’t lead to water pooling and that water isn’t being wasted because of leaks. The percentage of well inspections passed has decreased from 90% (9/10) during year one to 70% (7/10) in year two. This decrease is a sign that users need to be encouraged to report pooling and leaks to the Women’s Water Collective right away, so parts can be sourced and repairs done in a timely manner.

Variance and Unexpected Outcomes:

Daily use of the river among women of both communities remains higher than expected. In the case of community A, there was a 15% variance between this year’s target for this indicator (50%/520 [260 women]—see annual work plan) and the actual usage (65%/520 [338 women]). An informal survey of members of the Women’s Water Collective is being carried out to find out why women who already have access to the wells are still going to the river, and adjustments to the awareness-raising sessions will be made based on the findings.

In the case of community B, the 40% variance between the annual target of 55%/260 (143) women set in this year’s annual work plan and the actual usage of 90%/260 (234 women) is largely due to the construction delays described above. New sites have been proposed and an environmental analysis of these sites supported their selection by the community. In the meantime, since observation and anecdotal evidence indicated that many women of community B did not feel comfortable using wells located in primarily community A neighbourhoods on their own, we are supporting the Women’s Water Collective’s project to organize inter-community water collections.

For comprehensive guidance on reporting, please see Global Affairs Canada’s International Assistance Results Reporting Guide for Partners, and its companions, RBM Check List 5.1 - Reviewing Results Reports from Global Affairs Canada Implementers and Results-Based Management Tip Sheet No. 3.2 – Outcomes, Indicators, Baseline, Targets and Actual Data: What’s the Difference?

Conclusion

Remember

The main purpose of Results-Based Management is to optimize and improve the achievement of the expected outcomes that a program/portfolio and project sets out to achieve.

This means managing the program/portfolio and project for results from start to finish and ensuring a continuous focus on the achievement of outcomes by:

• Ensuring a sound program/portfolio and project design based on a thorough analysis of the issue and the context in which it exists and developing an evidence-based realistic solution, i.e. the project’s theory of change
• Ensuring that program/portfolio and project planning and implementation documents (the logic model, outputs and activities matrix (in the case of projects), theory of change narrative, as well as the budget and risk management tools) reflect the theory of change
• Ensuring the program/portfolio and project planning and implementation documents include a monitoring framework (the performance measurement framework), which includes indicators, baseline data and targets and a results-based monitoring and evaluation plan
• Involving key stakeholders, including intermediaries and beneficiaries, during design, planning and implementation
• Taking into consideration gender equality, environmental sustainability and governance in all aspects of results-based program/portfolio and project planning, design and implementation
• Keeping in mind that intermediate and ultimate outcomes are not within the sole control of a single organization, program/portfolio or project, but that an organization, program or a project contributes to and influences the achievement of these outcomes
• Undertaking monitoring and evaluation by collecting and analyzing data on output and outcome indicators to measure progress on, or toward, the expected outcomes
• Using indicator data collected and assessed to compare expected outcomes with actual outcomes, and adjusting operations throughout program/portfolio and project implementation in order to maximize the achievement of results
• Identifying, monitoring and managing risks throughout the life of the program/portfolio and project
• Learning and sharing lessons, and integrating them into decisions during implementation and into other similar current, as well as future programming
• Preparing evidence-based (using output and outcome-indicator data and analysis), reports on progress on or toward the expected outcomes

The guide will be updated periodically as required. Enquiries concerning this guide should be directed to gar.rbm@international.gc.ca

References

Bradley, D. and H. Schneider. Participatory Approaches: A Facilitator’s Guide. Part 1. Kingston upon Thames, UK: Voluntary Service Overseas, 2004.

Canada. Global Affairs Canada. Policy on Gender Equality. Ottawa: Author, 2010.

Canada. Canadian International Development Agency. Results-Based Management in CIDA: An Introductory Guide to the Concepts and Principles. Ottawa: Author, 1999.

Canada. Canadian International Development Agency (since 2015 Global Affairs Canada). Results-based Management Policy Statement 2008. Ottawa: Author, 2008.

Canada. Global Affairs Canada. 2008 Results-based Management Policy Statement: Amended Terms and Definitions. Ottawa: Author, 2008.

Canada. Justice Canada. Canadian Environmental Assessment Act. Statutes of Canada 2012, c. 19, s. 52.

Canada. Justice Canada. Official Development Assistance Accountability Act. Statutes of Canada 2008, c. 17, s. 4.

Canada. Treasury Board of Canada Secretariat. Results for Canadians: A Management Framework for the Government of Canada. Ottawa: Author, 2000.

Canada. Treasury Board of Canada Secretariat. Supporting Effective Evaluations: A Guide to Developing Performance Measurement Strategies. Ottawa: Author, 2010.

Canada. Treasury Board of Canada Secretariat. Theory-Based Approaches to Evaluation: Concepts and Practices. Ottawa: Author, 2012.

Crossman, Ashley, Units of Analysis.

Drucker, Peter F. The Practice of Management. New York: Harper & Row, 1954.

Funnell, Sue, C., and Patricia J. Rogers, Purposeful Program Theory, Effective use of Theories of Change and Logic Models, John Wiley and Sons, Inc., Copyright 2011.

International Committee of the Red Cross. Programme/project management: The results-based approach, May 2008.

Investopedia. “What are leading, lagging and coincident indicators? What are they for?”

Morra Imas, Linda G. and Ray C. Rist. The Road to Results: Designing and Conducting Effective Development Evaluations. Washington, DC: The World Bank, 2009. License: CC BY 3.0 IGO.

Organisation for Economic Co-operation and Development. Quality Standards for Development Evaluation. Development Assistance Committee (DAC) Guidelines and Reference Series. Paris: Author, 2010.

Organisation for Economic Co-operation and Development. Glossary of Key Terms in Evaluation and Results Based Management, 2010.

Organisation for Economic Co-operation and Development. Management for Development Results Information Sheet. Development Assistance Committee (DAC).

Project Management Body of Knowledge (PMBOK) – Work Breakdown Structure

Rietbergen-McCracken, Jennifer, ed. “Participation in practice: the experience of the World Bank and other stakeholders.” World Bank discussion papers, no. WDP 333. Washington, DC: The World Bank, 1996.

Vogel, Isabel. Review of the use of ‘Theory of Change’ in international development. London: UK Department for International Development, April 2012.

United Nations Development Group, Results-based Management.

United Nations Development Group, Results-Based Management Handbook: Harmonizing RBM concepts and approaches for improved development results at country level, October 2011.

University of Wisconsin-Extension, Program Development and Evaluation.

Zall Kusek, Jody and Ray C. Rist. Ten Steps to a Results-Based Monitoring and Evaluation System. Washington, DC: The World Bank, 2004. License: CC BY 3.0 IGO.