Book Review - The Failure of Risk Management

Today I finished reading The Failure of Risk Management by Douglas Hubbard (ISBN:978-0-470-38795-5). The book comes in at 259 pages plus an appendix. Overall I found it to be an excellent read.

I should start by saying that I have been a disciple of Hubbard since reading his other book, How to Measure Anything (ISBN: 978-0470110126). In that book Hubbard talks about the variety of things that we just can't measure and then talks about how to measure them. There are a couple of themes that need to be taken away from that book, most important of which is that a measurement is ANYTHING that reduces your uncertainty about something. When you measure length with a ruler you learn that the length of an item is really close to 4 and 3/16 inches, but you could always be more precise. Usually you don't need to. The problem that most of us face when measuring immeasurables is that we can't wrap our heads around the idea that we don't need the same level of precision as we get from a ruler or thermometer.

The Failure of Risk Management takes many of the concepts from How to Measure Anything and applies them to Risk Management. One nice aspect of the book is that it doesn't focus on financial risk or information security risk, or product failure risk. It's just risk, across the board. There is a lot of repeated information in the two books, but I think that How to Measure Anything is more of a practical guide while Failure of Risk Management is more of an explanation of why we should do the stuff he talks about in How to Measure Anything.

The two books are very good companions to each other and I would recommend that security managers should read them both if you really want to see our profession become more than soothsaying and water witching. I think if you're dealing with someone who does not yet believe that Risk Management needs to be quantitative and backed up by experiments and scientific skepticism then they should start with the Failure of Risk Management. On the other hand, if you think we should become more quantitative but think it's too hard then you should start with How to Measure Anything. That will make the challenge seem less difficult. Follow up with the Failure of Risk Management to learn why the way we're currently doing things (heat maps, low/med/high charts) are flawed.

Just like the other book, the Failure of Risk Management throws just enough math at you to be interesting without making you feel like you're sinking. I'll be honest, if you try to read this in two or three long sittings you'll probably become fatigued by some of the math. Take it slowly, especially near the end of the book unless you're already pretty strong with statistics. Having said that, anyone that can follow college algebra should be able to keep up with the most difficult parts of the book. Don't be afraid, jump in.

I enjoyed the book, I give it five stars.

Book Review: Security Metrics, Replacing Fear, Uncertainty, and Doubt

Man, I've been working on this book forever. I had to go out of town for some training last week. As always I brought a book along with me, and as always I spent my off time surfing the net and drinking with co-workers. So even though I have been one chapter away from finishing this book for like two weeks now, I just wasn't getting it done. Until today.

Security Metrics: Replacing Fear, Uncertainty, and Doubt
Andrew Jaquith
ISBN-10: 0-32-134998-9
299 pages.

So I should start by saying that I am a big believer that metric are badly needed in the Information Security field. I have long been looking for ways to measure my performance and the security posture of my organization. However, Andrew takes it a step further by explaining that it isn't enough for me to have metrics that are meaningful to me. We really need to have metrics that are meaningful to the entire industry, as is the case with accounting, finance, supply chain management etc. Without consistent, industry wide methods of measuring security, we're back to using hands or steps to measure distance. It is the technical equivalent of holding up your hands and saying "about this long."

Like most books that I read, Chapter 1 starts out by explaining the problem. He describes the way vendors sell products and provide no way of measuring the benefit of the product. Use a product, identify problem, fix problem, repeat. He calls it the Hamster Wheel of Pain.

Chapter two is where he starts to define security metrics. This chapter explains what qualities a good metric and a bad metric will have. For example, good metrics are cheap to gather and can be gathered consistently. Bad metrics are not cheap or consistent. He definitely rails against subjective measurement, and has no love in his heart for Annual Loss Expectancy. In fact, later in the book he refers to ALE as the stuttering, one-eyed, web-footed cousin of ROI and almost made me wet my pants laughing. In chapter two he also makes some of the same points as were made in The New School of Information Technology. Namely, that we need to have greater information sharing in our industry if we're going to develop industry wide standards.

Chapter three is the chapter I was waiting for. Show me the metrics. He starts off with a case study of using security metrics to prove or disprove a hypothesis (again a very New School thing to do). After that, he came out with a long list of metrics relating to perimeter security, control of systems, availability, and application security. To be honest, I was a little overwhelmed and unamused at the same time. Some of the metrics are pretty easy to gather, like number of email messages received per day. Others are almost impossible to gather such as the number of spam messages that were not detected by your spam filter. Seriously, how would you gather that? I am sure that it wouldnt' be cheap or consistent. Also, if your security operations aren't very mature, you may find that even some of the good metrics are difficult to measure. For example, we do not have centralized logging in place for all of our systems, so it would be hard to measure the number of viruses detected on our systems.

There is another problem that I have with a lot of security metrics. In many cases the statistic is just a meaningless data point. For example, how much spam was blocked by our mail gateway? If that number moved up or down does it mean that our security efforts were more fruitful or does it mean that there was more or less junk mail on the Internet that month? I don't think you can use this metric to measure your security posture, but it is a nice piece of trivia.

Chapter four presents metrics related to risk management, policy development, employee training, and other items that measure the effectiveness of the security program. The author takes great pains to stick with metrics that can be counted or measured. For example, the number of critical applications residing on servers that are compliant with the organizations security policies.

If you like numbers, counting, and statistics as much as myself and this guy do, then Chapter five is where it's at! In my junior year of college I took a class on business statistics and it was mostly great. This chapter was a review of the descriptive statistics that we learned about in the first few weeks: mean, median, mode, standard deviation, quartiles, etc. If you're already familiar with these concepts then you can probably skim this chapter. There was also some discussion about normal curves and their unique properties. The point of the chapter is taking the raw numbers that you've gathered from chapters two and three and turning them into insight.

Chapter six was my favorite chapter of the book because it dealt with how we present our metrics to senior management. I consider myself to be a creative person, but not in the area of visual art. So I was very pleased to see examples of how I can present complex data with more than just pie charts and line graphs. Even if those are the best choice for the data, the chapter still has excellent advise on how to make your graphs better (hint: less is more). Unfortunately, some of the coolest ideas just aren't possible with Excel, which is the only thing I have to work with. I think it would take some time and practice to learn how to draw the graphs in this chapter and where best to use each. However, you may find that the time is well spent.

Chapter seven was probably the least useful chapter for me. Even less useful than chapter five because at least chapter five made for some great review. Chapter seven is all about automating your metrics program. That seems like a pretty good idea, but I guess he didn't think it was terribly obvious because half of the chapter is devoted to convincing me that I should find ways to automate the collection of metrics. I guess since I got my start in IT on UNIX systems, it seemed like everyone would know that you want to automate everything that you can. Another 1/3 of the chapter goes into describing what wont work for automating your metrics, leaving you with just 1/6 of a chapter devoted to what does work. I guess I don't feel like I pulled much out of this.

Chapter eight, on the other hand, was back into good information territory. Here the author discusses some ways that the graphs and stuff that we created in chapter six can be consolidated into dashboards, scorecards, and grading systems. He discusses a few that have been tried, and then makes the case for creating a security related version of the Balance Scorecard. I thought this was pretty good stuff, especially because Balanced Scorecard is flexible enough that I can tailor it to my organization which has pretty weird security requirements.

So overall, I have to say that I'm glad I read the book. I'm not going to use all of the metrics that were presented in the book, but I don't think the author was trying to create the all powerful catalog of metrics that everyone should use. I also liked that the author made me feel good about not wanting to gather all of these metrics and present them all to management overnight. In fact, the author makes a point to mention that you should beware of metrics overkill or gathering metrics for the sake of metrics. Don't measure everything under the sun, only what is useful to your organization. I for one plan to start small and see if the insight I gain leads me to more things that I should be measuring.

Book Review: The New School of Information Security

The New School of Information Security
Adam Shostack & Andrew Stewart
2008 Pearson Education Inc, ISBN: 0-321-50278-7
160 Pages plus 51 pages of end notes.

Overview
The New School of Information Security is a call to action for Information Security professionals to change the way that we think about information security. For many of us, our approach to information security is dominated by principles such as defense in depth and separation of duties. While this book does not discount those principles, it urges security professionals to start looking to other sciences to help answer questions such as “How much defense in depth is appropriate?” The book employs principles from economics, psychology, and sociology to help explain some of the problems that the Information Security industry faces. Much of the information in this book is interesting and though provoking, but you’ll find that I criticize it for being short on details frequently.

One criticism that I’ve seen in other reviews that I believe is accurate is that the information in this book could have been presented in a pamphlet. I think that it is safe to say that 85% of the meat of this book is in chapter 4. The other chapters simply reinforce the beliefs presented in chapter 4. I also hate the way the endnotes are presented in the book. When the authors make a bold claim, they do not put a number after the sentence so you can easily find their source. Instead you have to flip to the back of the book, find the chapter that you’re reading, then read through all of the quotes until you find the one that you want. My biggest criticism of the book is that it doesn’t really tell me what I should do right now to improve the state of information security. The book tells me to think lofty thoughts but in the meantime my users still give their password to every phisher that writes to them. I think that the book is very interesting, and I think that it is a view into the future of our craft, but don’t expect to come away knowing what your next steps are.

Chapter Description
Chapter 1 is largely an overview of the security state of the Internet. Nothing in this chapter will be particularly educational for a seasoned information security professional, but I don’t believe that the author meant for it to be. This chapter is largely useful for setting the tone of the next two chapters.

Chapter 2 examines the way that we currently respond to the problems presented in chapter 1. It examines the faults of the information security industry and the motivations of all the players: vendors, analysts, hackers, crackers, etc. While this information is interesting, I don’t believe that it is always accurate. For example, on page 31 the author mentions that the CISSP certification “…employs a syllabus that it refers to as ‘the common body of knowledge.’ It amounts to a statement by the certification body of what a security professional should think about. Because of what is left out, it is also an implicit statement about what should not be thought about.” I do not believe that the intent of the CBK is to serve as a compendium of all the relevant knowledge in the information security field. Rather, I believe that the CBK is a collection of the knowledge that (ISC)2 believes all security professionals should know. I believe that the CBK allows that one could specialize in any of the ten domains and gather more detailed information beyond what is present in the CBK. The chapter rightly mentions that most of our problems can be solved by doing a few things really well, but is short on examples. The industry, however, has focused on selling us products to fix problems in other products. We aren’t trying to solve problems at their root, and we often use fear and group think to decide which actions to take.

Chapter 3 talks about the sources of information that are used to create the groupthink and fear discussed in chapter 2. Evidence comes in the form of surveys, vulnerabilities, trade press, and companies each of which has major flaws (such as sampling bias in the case of surveys). They are particularly hard on statistics, especially those well known statistics that everyone can cite but nobody can prove. For example, everyone knows that in our lifetime we are going to swallow a number of spiders while we are sleeping, but nobody can point to a single scientific study where a researcher has watched people sleeping and counted the number of spiders that entered their mouths. In the book the authors point to the well known fact that 50 to 70 percent of breaches are caused by insiders.

Chapter 4 mainly focuses on breach notification and how that information, if shared properly, can be of tremendous value to the security profession. The chapter introduces the concept of Prisoners Dillema from Economics: a situation where two people acting in their own self interest bring about an outcome that is worse for both of them. The authors present evidence that in the security industry, the practice of withholding breach information is acting in our own self interest and deprives the community of valuable information that can be used to create scientifically-tested findings. They talk about the reasons why companies avoid sharing this information and what scientists could do if a large body of this information were available. Another interesting idea that the authors point out is the concept of semi-strong efficiency in the stock market, although they do not use that particular term. Semi-strong efficiency is the belief that the current stock price of a company reflects all of the publicly known information about the company. The authors use this theory and some event analysis of the stock market to support their belief that releasing breach information does not result in significant customer flight.

In chapter 5 the authors introduce other concepts from economics, psychology, and sociology that should be considered when evaluating security problems. Ideas like the Nash Equilibrium, free-rider problems, externalities, risk homeostasis, and agency problems. I found this information to be interesting, but the authors didn’t do a great job of tying these concepts to information security. An example of an externality would be that people who do not patch their computers do not get more spam than the people that do patch. The authors didn’t point that out, instead talking about the pollution generated by SUVs. Externality: people that drive SUVs do not suffer more smog than people who bike to work even though they are a larger contributing factor.

Chapter 6 is all about spending. What we spend money on, what we should spend money on, why we spend money, and how much money we should be spending. This is the first chapter to suggest that we should incorporate concepts like Net Present Value into our security spending. The chapter challenges some of the core beliefs of the information security profession, but is often light on details.

Chapter 7 describes what life will be like in the security field when we’ve all started sharing data and using scientific studies to prove or disprove the effectiveness of our practices. Essentially, life will not be perfect, but it will be better and our industry will be more respected by upper management.

Chapter 8, the last chapter, is a call to action for the information security field to start thinking differently. The authors invite us to start sharing our breach data and using scientific thinking to guide our decision making process. This chapter is pretty short.