Outcome-Based Metrics
Written by Scott Charbo
MIT 2009 Volume: 13 Issue: 8 (September)
It is vital that the government develop processes
by which to measure performance and outcomes
associated with its cybersecurity efforts.
While there has been significant speculation about what the czar would do, to whom he or she would report, and what level of authority the position would hold, there’s been little discussion about the effects of such a position on the structure and operations of the federal government’s security apparatus.
To be effective, comprehensive cyber-initiatives will require wholesale change management efforts across the federal government and transformation at virtually every level of the various agencies and departments. The government must assess and address such components as security metrics, service delivery models, changes to procurement for trusted supply chain support, governance models and a host of other operational processes to ensure that they advance and support cybersecurity efforts.
Perhaps most important, though, it’s vital that the government develop processes by which to measure performance and outcomes associated with its cybersecurity efforts. Organizations must ask what is important for risk mitigation and programmatic success in combating threats to the enterprise. This goes beyond the mere tracking of dollars spent to include measuring the effectiveness of those dollars in meeting defined outcomes for security operations.
For instance, counting the number of firewalls installed and the funds to purchase them does not reflect the true effectiveness in preventing cyberterrorists from entering the network. Agencies should envision the key measurement outcomes they desire and “reverse plan” those events, milestones and details that will lead them to achieving cybersecurity success.
Establishing metrics that weigh performance and outcomes isn’t just about counting things. For example, the level of penetration of a cybersecurity event is important, but level assignments tell a more meaningful story. Categorizing incidents by their depth of impact to the organization’s infrastructure and domain can help yield policy changes, show where investments are necessary, and uncover opportunities for training.
MORE MEANINGFUL ANALYSIS
Metrics within the security operations center or computer incident response communities that could help provide more meaningful analysis and enhanced cybersecurity include:
• Measuring the effectiveness of security monitoring policy. Data flow and the network behaviors become increasingly important as we look for indicators to populate the needed metrics. In simplest terms, for instance, this means reconciling the average volume of traffic through the trusted Internet connection, or TIC, divided by the average volume of traffic through other agency gateways. The result is a more comprehensive view of the cyber landscape and can serve as an indicator both for program success and adversaries’ interest.
• Measuring the effectiveness and efficiency of security monitoring services delivery. This metric speaks mostly to establishing a percentage of false positives and could be accomplished via analysis of false positives alerts issued to an agency divided by the total number of agency alerts created over the last 30 days. It’s also important to track the percentage of non-actionable alerts—or the number of alerts issued to an agency that the agency cannot verify, divided by the total number of alerts created over the last 30 days.
• Other key components of this metrics category include tracking the percentage of targeted incidents reported to the U.S. Computer Emergency Response Team (US-CERT) and weighing them against the number that US-CERT’s Einstein program did not detect, and recording sensors’ uptime percentage.
• Measuring business or mission impact of security monitoring activities and events. This metric considers total confirmed incident reports and alerts discovered or reported over the last 30 days that are not false positives and are actionable. Further categorizing these alerts in stages helps address them in a more timely and effective manner. For example, “stage one” alerts would be phishing e-mails and/or users visiting compromised Websites. Stage two would address Trojan or malware downloading after an initial infection. And stage three would cover command and control traffic.
While there are still many unknown details about Obama’s cybersecurity plans, it is encouraging that his administration is committed to protecting our national computer systems. But to be truly effective, the new cyberczar must move beyond traditional performance measurements and embrace a paradigm shift toward outcomes-based metrics. This, combined with an understanding of cybergovernance in a global borderless context, will go a long way toward meeting this administration’s strategic vision for cybersecurity. ♦
Scott Charbo, Accenture U.S. Federal’s director of cybersecurity, is former deputy undersecretary of the National Protection and Programs Directorate, where he managed the Cyber Security Initiative at the Department of Homeland Security, and former chief information officer at DHS.
