Audit of IT Application Portfolio Management

Final report
Office of the Chief Audit Executive
November 2022

Table of Contents

• Acronyms
• Symbols
• Executive Summary
• Context
• Findings and Recommendations
  • 1. APM Roles and Responsibilities
  • 2. Leveraging APM Data for Decision-Making
  • 3. APM Data Quality and Integrity
• Conclusion
• Appendix A: About the Audit
• Appendix B: APM Maturity Model
• Appendix C: Aging IT and T.I.M.E. Assessments
• Appendix D: Portfolio of Applications
• Appendix E: OCIO Business Application Definition
• Appendix F: Management Response and Action Plan

Acronyms

APM

Application Portfolio Management

APHI

Application Portfolio Health Index

DPSD

Department Plan on Service and Digital

EA

Enterprise Architecture

GAC

Global Affairs Canada

GC

Government of Canada

IT

Information Technology

OCIO

Office of the Chief Information Officer (for the Government of Canada)

SSC

Shared Services Canada

TB

Treasury Board of Canada

TBS

Treasury Board of Canada Secretariat

T.I.M.E.

Tolerate, Innovate, Migrate, Eliminate

Symbols

SID

Information Management and Technology Bureau

SIS

Information Technology Client Support

Executive Summary

In accordance with Global Affairs Canada’s approved 2021-2022 Risk-Based Audit Plan, the Office of the Chief Audit Executive conducted an Audit of IT Application Portfolio Management.

What is Application Portfolio Management and why is it important?

Treasure Board Policy on Service and Digital requires departments to manage information, data, technology, cybersecurity and services in an integrated manner to enable digital service delivery. Application Portfolio Management covers IT applications, the most significant technology component that departments are responsible for managing

Applications are defined as a subclass of software that employs the capabilities of an electronic device directly and thoroughly for a task that the user wishes to perform.  Application Portfolio Management (APM) evolves around the governance of Information Technology (IT) applications throughout their entire lifecycle in support of maximizing the business value delivered. APM helps the department inventory applications and assess their technical and business value so that they can determine which applications to keep, modernize or eliminate. It is a proven methodology that is used to successfully manage an organization’s IT applications.

Background

Global Affairs Canada (GAC, the department) has a complex IT landscape, as it provides services at headquarters, regional offices throughout Canada, as well as abroad in 178 missions in 110 countries. The department supports a global mission network, which hosts 41 other government departments, agencies, crown corporations, provincial governments, and foreign diplomatic partners, to achieve the government’s international goals.

The department utilizes over 600 applications and software solutions to enable its services, which are managed both centrally and at the business and mission level. These applications may be hosted at the department’s legacy data centres managed by Shared Services Canada (SSC), in the cloud or locally at missions.

The department’s Application Modernization Program is detailed in the 2021-2024 Departmental Plan on Service and Digital (DPSD). The plan includes the intention to either migrate applications to the Cloud or to newer SSC Enterprise Data Centres, or to retire older applications that provide minimum value.

Regardless of where they are located, IT applications need to be managed throughout their lifecycle to ensure they provide programs and business lines with the functionality and value required to meet the department’s requirements in a secure and cost-effective manner.

Objective and Scope

The objective of the audit was to assess the adequacy and effectiveness of the management framework and processes in place to manage the department’s portfolio of IT applications throughout their lifecycle.

The scope of the audit includes the portfolio of IT applications managed by the department centrally, as well as those managed by missions (for additional details regarding the audit approach, see Appendix A).

Conclusion

Although some positive practices were identified, overall the department has not implemented an effective management framework and processes to manage the department’s portfolio of IT applications throughout their lifecycle. Improvements are needed to:

• coordinate, communicate, define and document, accountabilities, roles and responsibilities for managing APM; 
• report periodically the application portfolio health to senior management; implement a quality assurance process and controls to ensure higher quality APM data; and
• further integrate APM data into IM/IT operational activities such as enterprise architecture and IT incident management, business and IT continuity, strategic and investment decision-making.

Statement of Conformance

The audit was conducted in conformance with the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing and with the Treasury Board Policy and Directive on Internal Audit, as supported by the results of the quality assurance and improvement program.

Context

What is Application Portfolio Management and why is it important?

Applications are defined as a subclass of software that employs the capabilities of an electronic device directly and thoroughly for a task that the user wishes to perform^{Footnote 1} .  Application Portfolio Management (APM) evolves around the governance of Information Technology (IT) applications throughout their entire lifecycle in support of maximizing the business value delivered. APM helps the department inventory applications and assess their technical and business value so that they can determine which applications to keep, modernize or eliminate. It is a proven methodology that is used to successfully manage an organization’s IT applications.

The key components of an APM framework* are illustrated in the figure below. They include:

• the governance and oversight required to set priorities and make strategic decisions on the portfolio of applications
• the operational processes required to plan and build/acquire new applications, maintain them throughout their lifecycle, and decommission them once no longer required
• the monitoring and reporting of the health and performance of the portfolio of applications to inform decision making

*The APM framework was developed by the audit team based on expected practices for application portfolio management within an organization and informed by the CoBIT 2019 Framework.

Text version:

The figure illustrates the key components of an APM framework. The APM framework was developed by the audit team based on expected practices for applications portfolio management within and organization and informed by the CoBIT 2019 framework.

Requirements and Drivers:

• TB Policy on Service and Digital
• GC & GAC Application Modernization
• Leading IT practices (COBT)

Governance and Oversight:
Accountability and the overall framework relate to managing the portfolio of applications.
Management of the Portfolio of Applications:
Managing the portfolio of applications throughout their lifecycle:

• Plan/Build - Run/Operate - Decommission

Monitoring and Reporting:
Collection and analysis of APM data. Reporting on the portfolio to TBS and GAC stakeholders (e.g., health of applications, costs, risks). 
Monitoring and reporting is used to inform the Governance and Oversight component to inform decision making on the portfolio of applications.

The benefits of APM include having a departmental and Government of Canada (GC) wide view on the state of all IT applications and their risks throughout their lifecycle to allow for digital strategic decisions on the portfolio of applications. This includes:

• Identifying existing GAC or GC applications to re-use to avoid investing in duplicate applications;
• Decommissioning legacy applications to reduce costs and security risks;
• Quickly identifying applications that may be vulnerable to a new security threat;
• Migrating applications to a more efficient and secure environment;
• Upgrading applications that are costly to maintain, outdated, and/or insecure; and,
• Improved service delivery for Canadians.

In the broader context, the GC’s Digital Ambition^{Footnote 2}is the strategic plan that sets government‑wide priorities and lists key actions that departments and agencies need to transition to a more digital government and to meet the requirements of the Treasure Board (TB) Policy on Service and Digital. As depicted in the diagram below, the Policy on Service and Digital requires departments to manage information, data, technology, cybersecurity and services in an integrated manner to enable digital service delivery. APM covers IT applications, the most significant technology component that departments are responsible for managing.

As such, an effective APM framework is a key contributor to the effective management of technology within departments, which in turn allows for the department to provide the applications needed to enable digital service delivery to Canadians in a secure and cost-effective manner. Leveraging APM data also allows departments to make informed decisions about the portfolio of applications in order to meet the mandate of the department and maximize digital service delivery.

Text version:

The diagram depicts the TB Policy on Service and Digital and how Application Portfolio Management is integrated in the Policy. The Policy on Service and Digital includes four areas:

• Cyber Security
• Service Delivery
• Information and Data
• Information Technology

The area on Information Technology includes the following components: Infrastructure (that is managed by Shared Services Canada), Applications and End Points.
The Application of Portfolio Management is included as part of the component on Applications and is used to inform decision-making through the four Policy on Service Digital areas.

The GC Office of the Chief Information Officer (OCIO) developed guidance for departments related to APM. Amongst others, the OCIO APM User Guide establishes an APM maturity model that provides a roadmap for departments to set-up and improve their APM program according to TBS guidance (refer to Appendix B).

Application Portfolio Management at Global Affairs Canada

GAC has a complex IT landscape as it provides services at headquarters, regional offices, as well as abroad in 178 missions in 110 countries. The department supports a global mission network, which hosts 41 other government departments, agencies, crown corporations, provincial governments and foreign diplomatic partners, to achieve the government’s international goals.

As detailed in Appendix D, the department utilizes over 600 applications and software solutions to enable its services, which are managed both centrally and at the business and mission level. These applications may be hosted at the department’s legacy data centres managed by Shared Services Canada (SSC), in the cloud or locally at missions.

In 2018-19, budgets were made available by the GC to help departments and agencies migrate applications from older data centres into more secure and modern data centres or cloud solutions. This migration exercise is one of four pillars of the GC Workload Migration & Cloud Enablement strategy. As a result, the department has implemented an Application Modernization Program that is in alignment with the government’s initiative.

The department’s Application Modernization Program is detailed in the 2021-2024 Departmental Plan on Service and Digital (DPSD). The plan includes the intention to migrate applications to the Cloud or to newer SSC Enterprise Data Centres, or to retire older applications that provide minimum value. This is determined through the conduct of Aging IT and T.I.M.E. (Tolerate, Innovate, Migrate or Eliminate) assessments of applications, as further detailed in Appendix C.

 Application Portfolio Management Roles and Responsibilities

There are various groups within the department involved in APM activities. The Director General and Chief Information Officer of the Information Management and Technology Bureau (SID) is responsible for delivering the full range of IM/IT services and remains an integral part of the department’s international platform. The mandate of SID is to develop, operate and maintain the information management and technology systems, including the corporate information systems and IT applications to support the department’s activities and initiatives. SID’s following bureaus have roles and responsibilities related to APM:

• The Technology-Enabled Business Solutions Innovation & Integration Bureau (SIA) is responsible for collecting data on the portfolio of applications from the business and technical owners of the applications and manage the data in the Clarity system that is used to track and report on Application Portfolio data.  The APM team within the Technology-Enabled Business Solutions Innovation & Integration Bureau has also the responsibility to provide senior management and TBS with timely application-related information to ensure that identified risks are mitigated and that the applications are relevant and sustainable.
• The Digital Transformation Division (SIP) under SID is responsible for Enterprise Architecture (EA) and is currently working on developing the EA within the Department. The TB Policy on Service and Digital, defines EA as a conceptual blueprint that defines the structure and operation of an organization considering and aligning business, information, data, application, technology, security and privacy domains to support strategic outcomes. As it matures, the EA function is expected to be a key consumer of information on the application portfolio and influence how the portfolio of application evolves.
• The Information Technology Client Support (SIS) under SID is responsible for implementing the Application Modernization Programme, which oversees the migration of applications out of the legacy data centers that are scheduled to be closed. In this role, the Program is a key consumer of application portfolio information, and the decisions made in regard to application modernization may impact GAC’s portfolio of applications.

Findings and Recommendations

The following strengths were noted in conducting the audit:

• GAC performs an annual callout process to engage all application and technical owners in providing or updating departmental APM data regarding their respective applications.
• GAC has implemented an IT investment prioritization process that is in part supported by APM data.
• The department has recently improved the Applications Portfolio Health Index (APHI)^{Footnote 3} from 11% to over 19%; the APHI is a key APM performance measure calculated by TBS based on APM information provided by departments.
• The department has developed a Plan on Service and Digital which integrates technology, information, data, service and cybersecurity as required by the new TB Policy on Service and Digital

This section sets out the key findings. It is divided into three areas: APM Roles and Responsibilities, APM Data for Decision-Making, and APM Data Quality and Integrity.

1. APM Roles and Responsibilities

The success of APM is dependent on having an effective governance and management framework in place to provide strategic direction in support of the ongoing management of the portfolio of applications. An integral component to a mature APM management framework is accountabilities, roles and responsibilities that have been clearly defined, communicated and coordinated.  This is important for APM to ensure that accountabilities are well established, and roles and responsibilities are known to department officials who have obligations to fulfill to manage the department’s application portfolio activities, and especially considering that APM activities are performed by various groups within the department.

As per the TB Directive on Service and Digital, the Chief Information Officer is accountable for the APM program in the department.  This includes producing expenditure and status reports for APM activities. The audit team found that, except for the Chief Information Officer’s overall accountability, roles and responsibilities for department officials involved in APM activities are not clear.

The audit team found that the department has developed user guides to assist business and technical owners for reporting to the TBS concerning the health of business applications.  In addition, some guidance on APM activities is provided by the OCIO through the GCPedia website.

However, interviews with department officials and a review of documentation indicated that accountability, roles and responsibilities for APM activities have not been defined, documented and formally assigned within the department. Furthermore, there is no single group responsible for coordinating APM activities in the department (see table 2 below).^{Footnote 4} Some groups work in silos which creates gaps and overlap leading to a lack of coordination for managing applications throughout their lifecycle. 

Table 2: APM Lifecycle Roles and Responsibilities

The audit team also conducted interviews with other government departments who had a more mature APM function. It was noted that a key success factor for an effective APM function is to integrate APM activities into a single function, such as Enterprise Architecture, thereby allowing better communication and coordination for those officials accountable and responsible for APM activities. For these departments, their APHI score exceeded the average (37%) for all federal departments.

Without clearly defined roles and responsibilities and a coordinating function, there is an increased risk of gaps within APM activities and a lack of consistency and full lifecycle-view regarding the management of the portfolio of applications.

In conclusion, the audit team found that the department has not formally defined, documented and assigned accountability, roles and responsibilities and a coordination function for APM, resulting in APM being managed in a more siloed and uncoordinated fashion.

Recommendation 1

The Assistant Deputy Minister of Corporate Planning, Finance and Information Technology should formalize and communicate accountability, roles and responsibilities and identify a coordination function for managing applications throughout their lifecycle.

2. Leveraging APM Data for Decision-Making

The TB Policy on Service and Digital provides departments with direction to transition from an IT plan to the Plan on Service and Digital, with a lens on focussing on the linkages between IT, service, information, data, and cyber security.  The Policy on Service and Digital emphasizes the importance of integrated planning and decision-making and its impact on service, information, data, IT and cybersecurity for each function and ensuring these elements are considered throughout the development of new IM/IT initiatives. APM data can provide significant value to senior management as a key input into service and digital strategic and operational planning for the department. Furthermore, an integrated approach to APM can provide a more holistic lens into planning, which allows key interdependencies to be identified, including identifying systems that have limited business value and opportunities to reallocate investments into areas that directly support digital service delivery and improving services to Canadians.

The department has implemented an IT investment prioritization process for its Investment Plan and Department Plan on Service and Digital (DPSD). Although not the main source of information for decision-making, the group responsible for IT planning at GAC indicates that it uses APM data as one of its sources of information. However, APM data is not used to identify existing applications that could be re-used/leveraged to avoid duplication when planning for new applications and investment decisions. This could lead to continued investments in unused, costly and insecure applications, and the failure to allocate department resources to IM/IT initiatives that are more deserving of these resources. 

Although the department has transitioned from the IT Plan to the DPSD, the audit team found that the new department plan does not provide the same level of APM detail of the previous departmental IT Plan that identified a list of applications to be retired or innovated. In addition, APM data was not specifically used to develop the DPSD for 2021-22 and inform the priorities and actions of the DPSD.

Furthermore, the audit team found that action plans are not being developed to address applications identified as “requiring attention” by Aging IT and T.I.M.E. assessments in APM. Without the formalization and execution of action plans to deal with business applications identified as requiring attention, the department will face the burden of an increasing number of unused applications, making the IT infrastructure more difficult and costly to sustain, which will negatively impact digital services to Canadians.

It should be noted that in January 2022, the department was grappling with the effects of a cyberattack that disrupted its internet-based services.  The incident management team expected that APM data could provide assistance in order to identify applications affected by the security vulnerability; however, GAC was unable to do so due to poor data quality and integrity.  A lessons learned exercise was completed once the services resumed and it was noted that there was a need to refresh the inventory of department applications and to conduct a through review of APM processes. Had APM data been of better quality and integrated into Business and IT continuity planning, it could have assisted in reducing the downtime of key departmental internet services.

Finally, the department does not periodically monitor and report on the health of its portfolio of application to senior department executives even though this information is readily available and can be produced as a dashboard providing valuable information on the health, costs and risks of the portfolio of applications. Without periodic monitoring and reporting of APM activities, it will be difficult for senior management to measure progress in achieving its IM/IT strategic goals and to make decisions that are more informed on matters pertaining to the portfolio of applications and digital service delivery. For example, TBS uses APM data obtained from departments to calculate an Application Portfolio Health Index (APHI) serves to measure the health of applications. While GAC’s APHI has recently increased from 11% to over 19%, it still lags behind the GC average of 37%. Presenting the APHI to senior management on a regular basis could help drive discussions and action items on how to improve the department’s APHI level to close the gap with the rest of the GC.

In conclusion, the audit team found that processes were generally not in place for APM data to be used for IT decision-making such as IT strategic and investment planning, enterprise architecture’s review of existing application capabilities to prevent application duplication, inventory of applications for IT incident management, costing information for IT investment cost-benefit analysis and identification of candidates for decommissioning.

Recommendation 2

The Assistant Deputy Minister of Corporate Planning, Finance and Information Technology should develop and implement processes to improve the health of APM and leverage APM data for decision making in digital strategic and investment planning, and in operational activities related to APM such as Enterprise Architecture, business and IT continuity management, and IT Incident Management.

3. APM Data Quality and Integrity

The Government of Canada Office of the Chief Information Officer (OCIO)/TBS developed guidance for departments to report annually to TBS on the health of their application portfolio, including high-level IT expenditure data. Having accurate and comparative APM data is important for both the GC and departments to assess the achievement of digital strategic and operational objectives.

Although the department reports APM data annually to TBS, interviews with some department representatives indicated that they have little confidence in the quality and integrity of the data.  The audit team performed data integrity testing on key fields used in the GAC annual callout process to update APM data and used by TBS to determine the APHI, and noted that out of 641 applications in the GAC inventory of applications:

• 47 (7%) applications did not have an assigned business owner;
• 473 (74%) applications had not been assessed for Aging IT;
• 471 (73%) applications had not been assessed for T.I.M.E.;
• 271 (42%) applications did not have an assigned Information Security Category.

GAC has put in place an annual process, managed by the Architecture Solution’s group, to obtain from technical and application owners an update on their respective applications on key data elements of the APM. This process is aligned with the annual TBS reporting. However, the audit team found that Architecture Solution’s group does not review or perform quality control of the data that is in APM and updated annually and limited guidance is provided to application owners on how to provide quality data. The data quality and integrity of APM data is therefore left to the technical and application owners, who are mostly unaware of this accountability and the impacts of low quality APM data on decision-making.

In addition to providing the health of the department’s portfolio of applications, the department is required to provide Total Support Costs for each application.  The OCIO provides some guidance to departments on how to determine the costs that are to be included in the annual reporting of APM data to TBS. The audit team found that the Total Support Cost per application is being tracked and updated annually in APM; however, there is insufficient internal guidance being provided to application owners on how to ensure that the Total Support Cost is determined accurately and consistently across the department. Moreover, the department does not review the data quality and integrity of APM data including Total Support Cost. The audit team reviewed the Total Support Cost and identified the following:

• Only 289 (45%) applications of the 641 applications have identified a Total Support Cost; and,
• Of these, 85 (29%) applications have a Total Support Cost of $13 or less.

This information is important as Total Support Cost is used in the T.I.M.E. assessment, which should serve to guide the department’s IM/IT investment planning decisions. Without effective processes, guidance and controls for the collection and update of APM data, there is an increased risk that the quality and integrity of APM data will be diminished, resulting in a negative impact on the quality of decision-making, most notably IM/IT investment decisions.

In its guidance for the management of APM, the OCIO provides clear direction regarding what applications need to be reported to TBS. The audit team determined that the department does not report to TBS all applications that meet the OCIO definition of “Business Applications” (refer to Appendix E for details). GAC reports approximately 210 applications out of the inventory of 641 applications. Of the applications not reported to TBS, the audit team identified many applications that meet the business application definition of the OCIO.

Finally, the audit team interviewed staff from some of the missions who indicated that they had received no engagement from Architecture Solution’s group to identify or report on their applications. Due to the lack of comprehensive process and/or scanning capabilities to validate which applications are used by the department across its IT infrastructure, the audit team determined that GAC could not have appropriate assurance that the inventory of applications is complete. Therefore, it is likely that the inventory of applications being reported to TBS is neither complete nor accurate.

Without an APM data quality assurance process and comprehensive reporting to TBS of all GAC applications, the department is at risk of non-compliance with TBS requirements. The reporting of a sub-set of the inventory of applications also limits GAC’s oversight of most of its APM since assessments are not performed on applications not reported to TBS.

In conclusion, the audit team found that processes and controls were not in place to ensure the quality and integrity of APM data is appropriate for strategic decision-making.

Recommendation 3

The Assistant Deputy Minister of Corporate Planning, Finance and Information Technology should develop and implement processes and controls to ensure the data quality and integrity of APM data is appropriate and that all applications are identified for strategic decision-making, and to ensure that applications are reported to TBS in a manner that complies with expectations.

Conclusion

Although some positive practices were identified, overall the department has not implemented an effective management framework and processes to manage the department’s portfolio of IT applications throughout their lifecycle. Improvements are needed to:

• coordinate, communicate, define and document, accountabilities, roles and responsibilities for managing APM; 
• report periodically the application portfolio health to senior management; implement a quality assurance process and controls to ensure higher quality APM data; and
• further integrate APM data into IM/IT operational activities such as enterprise architecture and IT incident management, business and IT continuity, strategic and investment decision-making.

Appendix A: About the Audit

Objective

The objective of the audit was to assess the adequacy and effectiveness of the management framework and processes in place to manage the department’s portfolio of IT applications throughout their lifecycle. Specifically, the audit focused on the following key areas:

• Governance and oversight;
• Management of the Portfolio of Applications; and,
• Monitoring and Reporting.

Scope

The audit assessed the portfolio of IT applications managed by the department centrally, as well as those managed by missions. The scope was focusing on the governance and oversight of the portfolio of applications, the on-going management of these applications and on the tracking and reporting on the portfolio of applications.

Approach and Methodology

To achieve the audit objective, the following methods were used to gather audit evidence:

• Policy requirements analysis;
• Interviews with departmental officials and other government departments;
• Data analysis;
• Review of APM processes;
• Information analysis and consolidation;
• Other tests as deemed necessary.

Criteria

The criteria were developed following the completion of the risk assessment and considered the audit criteria related to the Treasury Board policy, guidance and directive as well as COBIT guidance (control objectives for IT) developed by the Information Systems Audit and Control Association to effectively manage a portfolio of applications.

Appendix B: APM Maturity Model

The OCIO has defined a maturity scale for APM. The audit team assessed that the Department is at maturity level 1.

*Source: Office of the Chief Information Officer for the Government of Canada

Text version: APM Maturity Model

The image explains the OCIO’s definition of maturity scale for Application Portfolio Management. The source is from the Office of the Chief Information Officer for the Government of Canada. The model provides a roadmap, which includes three levels of maturity, for departments to set-up and improve their APM program according to TBS guidance. It also includes criteria for determining at what level an organization is at and what it needs to move to the next level.

Appendix C: Aging IT and T.I.M.E. Assessments

The tools and guidance from Treasury Board Secretariat provide standardized lifecycle management and aging IT assessment methodologies which departments and agencies are to use to analyze the application inventory to help improve stewardship of applications, particularly those supporting critical departmental services.

Applications can be assessed across a number of factors, including the business value of the activities the application supports, the technical condition (quality) of the application, and the support costs of the application. Determining these factors for an application allows for an Aging IT assessment and a T.I.M.E (Tolerate, Innovate, Migrate and Eliminate) assessment to be completed which helps guide portfolio decision making and modernizing the inventory of applications. The assessments allow applications to be categorized and determine the extent to which attention is required on the application.

Text version:

The diagram illustrates applications’ Aging IT Assessment and T.I.M.E Assessment.

Aging IT Assessment

• Immediate Attention Required
• Attention Required
• Minimal Attention Required
• No Attention Required
• Not assessed

T.I.M.E Assessment

• Tolerate: Good business value and Acceptable use of IT resources
• Innovate: High business value and  Efficient use of resources
• Migrate: High business value and Inefficiently deployed
• Eliminate: Low business value and Inefficiently deployed

Appendix D: Portfolio of Applications

The table below provides detailed information in terms of ownership and some costs of the department’s IT applications.

   Source: List of applications received from Solution Architecture representing the yearly maintenance cost.

*Costs are unknown for some applications.

Appendix E: OCIO Business Application Definition

Business Applications must have all the following characteristics:

Appendix F: Management Response and Action Plan