Good Reads: How to Measure Anything in Cybersecurity Risk

All Models are wrong, but some are useful. (George Box)
…and some are measurably better than others. (Douglas Hubbard).

Such is the premise of the book How to Measure Anything in Cybersecurity Risk, in which Douglas Hubbard and Richard Seiersen take a critical look at conventional methods of assessing Cybersecurity risk, and offer an alternative. A continuation of Hubbard’s series on business statistics and quantitative decision analysis, this book dives deep into the problem of how to inform business decisions in complex situations when data is scarce. While business statistics may not be everyone’s favourite topic, it is a remarkably engaging overview, and it can equally serve as a desk reference for anyone whose work involves helping organizations make informed decisions about how to reduce risk associated with cyber security.

What’s in the book?
How to Measure Anything n Cyber Security Risk is divided into three main sections. The first five chapters are a basic introduction to quantitative decision analysis. There is some overlap with Hubbard’s earlier work on quantitative business analysis, but the main focus is its application to measurement problems in Cybersecurity risk. Chapters four and five are an in-depth critique of the qualitative risk assessment methods that are central to most mainstream Cyber risk management methodology (e.g., ISO 27001, NSIT 800-53). Their systematic review of research in decision theory is more than enough to raise suspicions about the value of qualitative risk matrices as decision support tools, largely because of inherent rounding errors and their capacity to perpetuate errors and judgement biases.

Moving into Part II, the book presents an alternative approach that expresses risk in monetary terms, positing the 90 percent confidence interval for expected loss as a replacement for the low/medium/high risk ratings that come from a qualitative risk matrix. Part III sets out a maturity model for cyber analytics and an implementation plan for using quantitative risk analysis in an enterprise Cybersecurity program. Finally, a series of appendices and a companion web site give additional details about the underlying statistics and some sample Excel workbooks that provide a starting point for developing tools to apply the methods in modelling real situations.

What I liked
The book is very readable and practical, not things to take for granted in a book like this. The authors didn’t make it a cookbook. Instead of just breezing through the mechanics of using quantitative methods, they make a serious attempt, with a good measure of academic rigour, to address the business context and to help risk practitioners do a better job of informing decisions about cyber risk.

The systematic dissection of qualitative risk management methodologies makes a strong argument against using risk matrices. The case for the alternative, risk estimation based on Monte Carlo simulation, is equally compelling. And then they follow through with a step-by-step treatment of how to get the most of expert estimates, making Monte Carlo simulation accessible as a tool for turning 3-point estimates into 90 percent confidence intervals.

The Excel workbooks found on the companion web site make a good starting point for doing the simulation using native Excel functions. For those of us who haven’t thought about Bayesian probabilities since high school, chapters 9 and 10 are a good refresher on how to handle probabilities when multiple factors are at play, something that is all too common when analyzing risk in Cybersecurity.

What Could be Better
The section on implementation strategy is a bit thin. The authors set out the ambitious goal of changing the way organizations use threat and risk assessments to inform management decisions. They position quantitative methods as an improvement over risk matrices for evaluating how much to spend on improving cyber defense capability, and they set out a Cyber Analytics Maturity Model to help organization assess current state. But they could have spent more time talking about adoption strategy and change management. The chapter on implementation strategy doesn’t fully acknowledge the amount of organizational change involved.

The logical argument that quantitative methods are inherently better overlooks the reality of changing the way decision makers view risk assessments. Weaknesses aside, the fact is that risk matrices are deeply embedded in the way many people think and work. When presented for the first time with a confidence interval for Annualized Loss Exposure range instead of a Low/Medium/High risk rating, it would be quite reasonable for a decision maker be skeptical. The didn’t get to be decision makers without a healthy ability to challenge new things, especially things that touch on decisions for which they will ultimately be held accountable.

Conclusion
Adopting quantitative risk estimating is a major shift in the way people, think and work: it means giving up something that is easy and familiar and taking on something that involves more work. To be fair, the book does acknowledge that mature adoption of quantitative methods is a journey, and we get a picture of what a mature, data-driven cyber security management program looks like. That, unfortunately, is a long way away from the world that most of us live in. As with any innovation, adopting quantitative methods needs strong executive champions but, on the ground, it still starts with the efforts of pioneers and early adopters, people who can lead the way, show success and pave a way for the rest of the organization to follow.