Book Review: Security Metrics, Replacing Fear, Uncertainty, and Doubt

Man, I've been working on this book forever. I had to go out of town for some training last week. As always I brought a book along with me, and as always I spent my off time surfing the net and drinking with co-workers. So even though I have been one chapter away from finishing this book for like two weeks now, I just wasn't getting it done. Until today.

Security Metrics: Replacing Fear, Uncertainty, and Doubt
Andrew Jaquith
ISBN-10: 0-32-134998-9
299 pages.

So I should start by saying that I am a big believer that metric are badly needed in the Information Security field. I have long been looking for ways to measure my performance and the security posture of my organization. However, Andrew takes it a step further by explaining that it isn't enough for me to have metrics that are meaningful to me. We really need to have metrics that are meaningful to the entire industry, as is the case with accounting, finance, supply chain management etc. Without consistent, industry wide methods of measuring security, we're back to using hands or steps to measure distance. It is the technical equivalent of holding up your hands and saying "about this long."

Like most books that I read, Chapter 1 starts out by explaining the problem. He describes the way vendors sell products and provide no way of measuring the benefit of the product. Use a product, identify problem, fix problem, repeat. He calls it the Hamster Wheel of Pain.

Chapter two is where he starts to define security metrics. This chapter explains what qualities a good metric and a bad metric will have. For example, good metrics are cheap to gather and can be gathered consistently. Bad metrics are not cheap or consistent. He definitely rails against subjective measurement, and has no love in his heart for Annual Loss Expectancy. In fact, later in the book he refers to ALE as the stuttering, one-eyed, web-footed cousin of ROI and almost made me wet my pants laughing. In chapter two he also makes some of the same points as were made in The New School of Information Technology. Namely, that we need to have greater information sharing in our industry if we're going to develop industry wide standards.

Chapter three is the chapter I was waiting for. Show me the metrics. He starts off with a case study of using security metrics to prove or disprove a hypothesis (again a very New School thing to do). After that, he came out with a long list of metrics relating to perimeter security, control of systems, availability, and application security. To be honest, I was a little overwhelmed and unamused at the same time. Some of the metrics are pretty easy to gather, like number of email messages received per day. Others are almost impossible to gather such as the number of spam messages that were not detected by your spam filter. Seriously, how would you gather that? I am sure that it wouldnt' be cheap or consistent. Also, if your security operations aren't very mature, you may find that even some of the good metrics are difficult to measure. For example, we do not have centralized logging in place for all of our systems, so it would be hard to measure the number of viruses detected on our systems.

There is another problem that I have with a lot of security metrics. In many cases the statistic is just a meaningless data point. For example, how much spam was blocked by our mail gateway? If that number moved up or down does it mean that our security efforts were more fruitful or does it mean that there was more or less junk mail on the Internet that month? I don't think you can use this metric to measure your security posture, but it is a nice piece of trivia.

Chapter four presents metrics related to risk management, policy development, employee training, and other items that measure the effectiveness of the security program. The author takes great pains to stick with metrics that can be counted or measured. For example, the number of critical applications residing on servers that are compliant with the organizations security policies.

If you like numbers, counting, and statistics as much as myself and this guy do, then Chapter five is where it's at! In my junior year of college I took a class on business statistics and it was mostly great. This chapter was a review of the descriptive statistics that we learned about in the first few weeks: mean, median, mode, standard deviation, quartiles, etc. If you're already familiar with these concepts then you can probably skim this chapter. There was also some discussion about normal curves and their unique properties. The point of the chapter is taking the raw numbers that you've gathered from chapters two and three and turning them into insight.

Chapter six was my favorite chapter of the book because it dealt with how we present our metrics to senior management. I consider myself to be a creative person, but not in the area of visual art. So I was very pleased to see examples of how I can present complex data with more than just pie charts and line graphs. Even if those are the best choice for the data, the chapter still has excellent advise on how to make your graphs better (hint: less is more). Unfortunately, some of the coolest ideas just aren't possible with Excel, which is the only thing I have to work with. I think it would take some time and practice to learn how to draw the graphs in this chapter and where best to use each. However, you may find that the time is well spent.

Chapter seven was probably the least useful chapter for me. Even less useful than chapter five because at least chapter five made for some great review. Chapter seven is all about automating your metrics program. That seems like a pretty good idea, but I guess he didn't think it was terribly obvious because half of the chapter is devoted to convincing me that I should find ways to automate the collection of metrics. I guess since I got my start in IT on UNIX systems, it seemed like everyone would know that you want to automate everything that you can. Another 1/3 of the chapter goes into describing what wont work for automating your metrics, leaving you with just 1/6 of a chapter devoted to what does work. I guess I don't feel like I pulled much out of this.

Chapter eight, on the other hand, was back into good information territory. Here the author discusses some ways that the graphs and stuff that we created in chapter six can be consolidated into dashboards, scorecards, and grading systems. He discusses a few that have been tried, and then makes the case for creating a security related version of the Balance Scorecard. I thought this was pretty good stuff, especially because Balanced Scorecard is flexible enough that I can tailor it to my organization which has pretty weird security requirements.

So overall, I have to say that I'm glad I read the book. I'm not going to use all of the metrics that were presented in the book, but I don't think the author was trying to create the all powerful catalog of metrics that everyone should use. I also liked that the author made me feel good about not wanting to gather all of these metrics and present them all to management overnight. In fact, the author makes a point to mention that you should beware of metrics overkill or gathering metrics for the sake of metrics. Don't measure everything under the sun, only what is useful to your organization. I for one plan to start small and see if the insight I gain leads me to more things that I should be measuring.