Metrics and Feedback

Benefit

Identification of quick wins derived from available defect information

Activity

Once per defined period of time (typically at least once per year), go over your both resolved and still open recorded security defects in every team and extract basic metrics from the available data. These might include:

• The total number of defects versus total number of verification activities. This could give you an idea whether you’re looking for defects with an adequate intensity and quality.
• The software components the defects reside in. This is indicative of where attention might be most required, and where security flaws might be more likely to appear in the future again.
• The type or category of the defect, which suggests areas where the development team need further training.
• The severity of the defect, which can help the team understand the software’s risk exposure.

Identify and carry out sensible quick win activities which you can derive from the newly acquired knowledge. These might include things like a knowledge sharing session about one particular vulnerability type or carrying out / automating a security scan.

Question

Do you use basic metrics about recorded security defects to carry out quick win improvement activities?

Quality criteria

Answers

Benefit

Improved learning from security defects in your organization

Activity

Define, collect and calculate unified metrics across the whole organization. These might include:

• Total amount of verification activites and identified defects.
• Types and severities of identified defects.
• Time to detect and time to resolve defects.
• Windows of exposure of defects being present on live systems.
• Number of regressions / reopened vulnerabilities.
• Coverage of verification activities for particular software components.
• Amount of accepted risk.
• Ratio of security incidents caused due to unknown or undocumented security defects.

Generate a regular (e.g. monthly) report for a suitable audience. This would typically reach audience like managers and security officer and engineers. Use the information in the report as an input for your security strategy, e.g. improving trainings or security verification activities.

Share the most prominent or interesting technical details about security defects including the fixing strategy to other teams once these defects are fixed, e.g. in a regular knowledge sharing meeting. This will help scale the learning effect from defects to the whole organization and limit their occurence in the future.

Question

Do you improve your security assurance program upon standardized metrics?

Quality criteria

Answers

Benefit

Optimized security strategy based on defect information

Activity

Regularly (at least once per year) revisit the defect management metrics you’re collecting and compare the effort needed to collect and track these to the expected outcomes. Make knowledgeable decision about removing metrics which don’t deliver the overall expected value. Wherever possible, include and automate verification activities for the quality of the collected data and ensure sustainable improvement if any differences are detected.

Aggregate the data with your threat intelligence and incident management metrics and use the results as input for other initiatives over the whole organization, such as:

• Planning security trainings for various personnel
• Improvement of security verification activities for both internally and externally develeoped collected
• Supply chain management, e.g. carrying out security audits of partner organizations
• Monitoring of attacks against your infrastructure and applications
• Investing in security infrastructure or compensating controls
• Staffing your security team and setting up the security budget

Question

Do you regularly evaluate the effectiveness of your security metrics so that its input helps drive your security strategy?

Quality criteria

Answers