IT SECURITY METRICS
This paper provides basic information on what metrics are and why IT security performance should be measured. Additionally, this section defines types of metrics that can be used to measure IT security controls, discusses the key aspects of making a metrics program successful, and identifies the uses of metrics for management, reporting, and decision making.
IT security metrics can be obtained at different levels within an organisation. Detailed metrics, collected at the system level, can be aggregated and rolled up to progressively higher levels, depending on the sise and complexity of an organisation. While a case can be made for using different terms for more detailed and aggregated items, such as “metrics” and “measures,” this document uses these terms interchangeably.
IT security metrics must be based on IT security performance goals and objectives. IT security performance goals state the desired results of a system security program implementation, such as “All employees should receive adequate security awareness training.” IT security performance objectives enable accomplishment of goals by identifying practices defined by security policies and procedures that direct consistent implementation of security controls across the organisation. Examples of IT security performance objectives, corresponding to the example goal cited above are “All new employees receive new employee training,” “Employee training includes a summary of the Acceptable Usage Policy,” and “Employee training includes a summary and a reference to the organisation’s security policies and procedures.” IT security metrics monitor the accomplishment of the goals and objectives by quantifying implementation of the security controls and the effectiveness and efficiency of the controls, analysing the adequacy of security activities, and identifying possible improvement actions.
IT security metrics must yield quantifiable information for comparison purposes, apply formulas for analysis, and track changes using the same points of reference. Percentages or averages are most common, and absolute numbers are sometimes useful, depending on the activity that is being measured. To be useful for tracking performance and directing resources, metrics need to provide relevant performance trends over time and point to improvement actions that can be applied to problem areas. Management should use metrics to assess performance by reviewing metrics trends, identifying and prioritising corrective actions, and directing the application of those corrective actions based on risk mitigation factors and available resources.
Benefits of Using Metrics
A security metrics program provides a number of organisational and financial benefits. Organisations can improve accountability for security by deploying IT security metrics. Departments and agencies can demonstrate compliance with applicable laws, rules, and regulations by implementing and maintaining an IT security metrics program Fiscal constraints and market conditions compel government and industry to operate on reduced budgets. In such an environment, it is difficult to justify broad investments in the IT security infrastructure. Historically, arguments for investing in specific areas of IT security lack detail and specificity, and fail to adequately mitigate specific system risk. Use of IT security metrics will allow organisations to measure successes and failures of past and current security investments and should provide quantifiable data that will support allocation of resources for future investments.
The types of metrics (implementation, efficiency and effectiveness, and impact) that can realistically be obtained and that can also be useful for performance improvement depend on the maturity of the security control implementation. Examples of implementation metrics that are applied at this level of maturity are the percentage of systems with approved security plans and the percentage of systems with password policies configured as required. As security controls are documented and implemented, the ability to reliably collect the outcome of their implementation improves. As an organisation’s IT security program evolves and performance data becomes more readily available, metrics will focus on program efficiency— timeliness of security service delivery and effectiveness—operational results of security control implementation. Measuring effectiveness and efficiency of implemented security controls and the impact of these controls on the organisation’s mission. These metrics concentrate on the evidence and results of testing and integration. Instead of measuring the percentage of approved security plans, these metrics concentrate on validating whether security controls, described in the security plans, are effective in protecting the organisation’s assets. For example, computing the percentage of crackable passwords within a predefined time threshold will validate the effectiveness of an organisation’s password policy by measuring the length of time required to break policy-compliant passwords. The impact metrics would quantify incidents by type (e.g., root compromise, password compromise, malicious code, denial of service) and correlate the incident data to the percentage of trained users and system administrators to measure the impact of training on security.
1. System stakeholders must be included in the IT security metrics development and program implementation.
2. A very important success factor is manageability of the metrics program. Results of many security activities can be quantified and used for performance measurement; however, since resources are limited and the majority of resources should be applied to correcting performance gaps, organisations should prioritise measurement requirements to ensure that a limited number of metrics are gathered.
3. Ascertain the quality and validity of data, data collection methods and data repositories used for metrics data collection and reporting, either directly or as data sources, should be standardised.
4. Any data collection, specifically for the purpose of IT security metrics, must be as nonintrusive as possible The establishment of a metrics program will require a significant investment to ensure that the program is properly implemented to maximise its benefits. The resources required for maintaining the program are not expected to be as significant.
Metrics Development Process
The IT security metrics development process consists of two major activities: 1. Identification and definition of the current IT security program; and 2. Development and selection of specific metrics to measure implementation, efficiency, effectiveness, and the impact of the security controls.
Stakeholder Interest Identification
The primary IT security stakeholders are:
• Chief Executive/SRO/Managing Director
• Chief Information Officer (CIO)
• Security Program Manager/Information Security Officer (ISO)
• Program Manager/System Owner
• System Security Officer
• System Administrator/Network Administrator
• IT Support Personnel
Secondary security stakeholders include:
• Finance Director
• Training Organisation
• Human Resources
The interests of each stakeholder will differ, depending on the security aspects of their role and on their position within the organisational hierarchy. Each stakeholder may require an additional set of customised metrics that provides a view of the organisation’s IT security performance within their area of responsibility. It is recommended that fewer metrics per stakeholder be used when an organisation is establishing a security program; the number of metrics per stakeholder will increase gradually with the maturity of the IT security program and of the metrics program. Stakeholders should be involved in each step of security metrics development to ensure organisational buy-in to the concept of measuring security performance.
Metrics Development and Selection Organisations may decide to use a weighting scale to differentiate importance of selected metrics and to ensure that the results accurately reflect existing security program priorities. This would involve assigning values to each metric based on the importance of a metric in the context of the overall security program. Metrics weighting should be based on the overall risk mitigation goals and is likely to reflect higher criticality of department-level initiatives versus smaller scale initiatives and is a useful tool that facilitates integration of IT security metrics into the departmental capital planning process.
Establishing Performance Targets After applicable metrics are identified and described, performance targets should be identified in the indicator line of the metric form. Performance targets establish a goal by which success is measured. The degree of success is based on the metric result’s proximity to the stated performance target. The mechanics of establishing performance targets differ for implementation metrics and the other three types of metrics (effectiveness, efficiency, and impact). For implementation metrics, targets are set to 100 percent completion of specific tasks. Setting performance targets for efficiency, effectiveness, and impact metrics is more complex, because these aspects of security operation do not assume a specific level of performance. Management will need to apply qualitative and subjective reasoning to determine appropriate levels of security effectiveness and efficiency and to use these levels as targets of performance for applicable metrics. Once the baseline is obtained and corrective actions identified, appropriate measurement targets and implementation milestones can be defined that are realistic for a specific system environment. If performance targets cannot be established after the baseline has been obtained, management should evaluate whether the measured activities and corresponding metrics are providing expected value for the organisation.
METRICS PROGRAM IMPLEMENTATION Prepare for Data Collection After the metrics have been identified, specific implementation steps should be defined on how to collect, analyse, and report the metrics. These steps should be documented in the Metrics Program Implementation Plan. Collect Data and Analyse Results This phase includes the following activities: • Collect metrics data, according to the processes defined in the Metrics Program Implementation Plan • Consolidate collected data and store in a format conducive to data analysis and reporting, for example, in a database or a spreadsheet • Conduct gap analysis – compare collected measurements with targets, if defined, and identify gaps between actual and desired performance • Identify causes of poor performance • Identify areas requiring improvement.