I've started putting together a presentation for a conference that I'm writing a proposal for about how you can use security metrics and statistics to make Annualized Loss Expectancy work a little better. The idea is to get a compromise between the ease of qualitative risk analysis and the accuracy of quantitative risk analysis. I started writing up an example of what I mean, and it turned out pretty good, so I thought I would share it here.
How much should we spend on hard disk encryption for our sales force?
Let's start with a simplifying assumption. How much do you spend on each record that you lose? Well that answer might be based on the type of data that is lost, but you can probably come up with a range of numbers that is reasonably accurate. For example, you know that you're probably going to pay for a stamp for each letter, so you know that it has to be more than 42 cents per record. You probably have to figure that you're going to put two hours into accounting for the records on each breach. You should assume that the employee who lost the laptop is also going to lose some time while he is being interviewed by you. You can expect that your legal department is going to lose a couple hours of time while they draft up a letter to everyone involved. So at an absolute minimum, you know that it is going to cost five or six hours of employee time plus 42 cents per record. A quick look on the web shows that companies that sell expensive software want you to believe that the average cost is $197 per record. That's probably a reasonable high end of the range.
Next it would be nice to know how many records are on the average sales laptop. That's really not too hard to do. Get a list of all the sales people and find out how many you would need to sample to be 95% certain of a 3% confidence interval. Let's say you've got 50 sales people. A quick Google search finds me a sample size calculator. When I plug in my numbers it says that I need to sample 48 of my 50 sales people if I want that kind of accuracy. Crap, I'm not going to do that. But I know from the rule of five from statistics that if I randomly sample 5 of them, I am 93% certain that the true median falls between the top and bottom of that range. So let's randomly sample five of them and check out what is on their laptops right now. In this hypothetical scenario, I found that the laptops had 250, 128, 64, 0 and 0 records. I don't want to run into a divide by zero error, so I'll set a floor of one record. So I can set a range of 1 to 250 records on each laptop.
Let's assume that HR comes back and tells us that the average employee salary is $54/hour with a standard deviation of $14/hour. We decided that each laptop was going to require at least 6 hours of staff time regardless of how many records are in place. So if there is one record on the laptop, then have a cost per record of ($54 * 6) + 42 cents. That's $324.42 per record. If there are 250 records then the cost is $1.72 per record. That $324 per record doesn't withstand my smell test, so I'm going to throw it out in favor of of the $197 per record cited above. That means that when a laptop is stolen from our sales force, we should expect that it will cost between $1.72 per record and $197 per record and that there will be between 1 and 250 records exposed. That gives us a cost range of $1.72 to $49,250 for a single loss.
There are ways that we could get tighter numbers if we really needed them. We could actually sit down and survey 48 of our 50 sales people so that we would be more confident about the number of records on each sales person's laptop computer. We could also run our numbers through a Monte Carlo simulation to see if we get a tighter distribution of costs. All that would be unnecessary, however, if we found that even based on our broad estimation of costs, the software was too expensive to implement. Based on the parameters that we established above, I ran a small simulation and came up with an average of $6.67 per record with a standard deviation of $11.96. We know that there is a 66% chance that the true average is within one standard deviation of the mean, but we should also expect that we still wont have a scenario where the cost dips below $1.72 per record. So now we can say with a fairly high degree of certainty that the average cost per record will be between $1.72 and $18.63. And that gives us a single loss expectancy between $1.72 and $4,657.
Now we're getting numbers that don't seem unreasonable, and you can really tell your managers that you didn't just make something up. There is real math behind using real numbers behind this. What if you were trying to justify spending $5000 on disk encryption for your sales team and you expect the software to be useful for five years? You would only need to have two laptop thefts in the next five years to make this a good decision. You can repeat the same processes above to get a strong estimate of the number of laptop thefts that you're likely to suffer.
Also, remember to bring all of your costs back to today's dollars using Net Present Value. The total cost of your software today should include the discounted cost of employee time next year and software maintenance for next year. Decide how many years you're going to assume for the useful life of the product. Same goes for your losses. If you assume a loss of $4,657 per year, and a discount rate of 4% then the present value of those losses is $20,372.14. Subtract the present value of the cost of your software project and you've got NPV. If NPV is positive, then you've got a good project. If not, then you should look at other ways you can improve the situation for less money.
Showing posts with label NPV. Show all posts
Showing posts with label NPV. Show all posts
Wednesday, January 21, 2009
Friday, August 22, 2008
New School: How much should I spend to mitigate phishing?
I just finished reading The New School of Information Security by Adam Shostack and Andrew Stewart. I plan on writing a formal book review now that I've made it through the book, but I also wanted to get these thoughts down before they leave my head. In this entry I'm going to attempt to use New School thinking to analyze an event that happened just this week at my organization.
All summer long we've had phising emails sent to our campus and they have pretty much bounced off our users with little effect. However as the school year starts up, we have more people on campus and a phishing message that was sent out this week was able to gather some credentials and those credentials were used to send out spam from our servers.
I responded to the event in a very non-New-School way. I did what my gut said I should do and I stepped up user education efforts. Of course we blocked the affected account, and made some people reset their passwords, but I also had about 300 fliers printed up and distributed around campus reminding people that we don't ask for passwords over email. I've also initiated plans to make some updates to our web pages and automated emails that come out from our department.
Looking back on the whole thing (about two days later) I started trying to take a New School approach to the problem. One of the major ideas expressed in the book that I really agree with is that we need to be more open about events like this and share that information readily instead of keeping it to myself. So I typed up a detailed report and sent it off to my security contacts at other schools in the state.
But now I'd like to address how much money we should spend on mitigating phising attacks in the future. This is where that objective data comes in that we don't have. I don't know how many phishing emails have been sent to universities in the state in the last year. I don't know how many people have responded to the phishing emails, and I don't know how much cost this has imposed on our universities. So I'm not going to be able to answer the question. However, we can look at how the New School would suggest that we answer the question once enough data is present.
This week my school has received about 400 phishing email messages which resulted in about an hour of work for IT staff, and the people that had to reset their passwords. I capitalize an hour of employee time at $50/hour, so each one of those phishing messages cost us about 12.5 cents. However, as I mentioned above, all summer long we were receiving these messages and only had one event. I think there is merit behind the idea that a university is more vulnerable when class is in session than when it is in break. During the summer break I estimate that we had about 2000 phish messages sent to our campus, and that resulted in another $50 worth of loss. So during the breaks a phish message costs us 2.5 cents. With that data we can come up with a Weighted Single Loss Expentancy. (.25)(2.5)+(.75)(12.5) = 10 cents. We can assume that throughout the year, each phish message that we get is going to cost us 10 cents. Therefore, if we receive 10,000 phish messages each year our Annual Loss Expectancy is $1000.
So far this isn't very New School. This is right out of the CISSP Common Body of Knowledge. Here is where things diverge a little bit. One of the things mentioned in the book is a paper by Lawrence Gordon and Martin Loeb that describes how much we should be spending on Information Security. Their paper finds that we should spend somewhere between 25 and 37% of the expected loss on mitigation. So we should spend somewhere between $250 and $370 each year to reduce phishing. This runs counter to the conventional wisdom that says we should spend some amount of money less than $1000 to reduce phishing. One of the principles of Economics is that rational people think at the margin, and Gordon and Loeb point out that after 37% you have hit diminishing returns. In other words, an extra dollar spent on mitigation reduces loss by some number less than one dollar. In fact, if the resource being protected is not very vulnerable, then 37% is far too high. So we could say that it is worth $250/year to reduce phishing.
Now let's bring in my favorite finance concept, Net Present Value. Let's say that I put together a five year project to reduce phishing on campus. I want to spend $50 every year printing fliers. I also want to invest $500 of developer time right now into making changes to our web pages and automated email messages. Is that a good project? The PV of $250 paid each year for five years (5% discount rate) is $1,082.37. So if the PV of my project is less than that, then we're good. $550 in year one, plus $50 each in years 2 through 5 at 5% is $692.66. So it looks like my project is a good idea financially...Maybe.
There are a couple of things that we don't know. Sure we know how much I should spend, but we don't know what I should spend it on. My user education plan might be 10% effective compared to some unknown solution that is 80% effective. Obviously we would put money into the more effective form of mitigation. Another problem to consider (and this is one for the economists) is that the loss expectancy was based on what happened at my university. Well we have a spam firewall and we've already put money into user education, and some of the money spent on that is preventing phishing attacks from working. That money should come out of the $250 a year that we spend...unless an economist would consider that to be sunk cost. I'm not sure. Here is what we would need to answer the question of how much to spend on mitigating phishing positively...
All summer long we've had phising emails sent to our campus and they have pretty much bounced off our users with little effect. However as the school year starts up, we have more people on campus and a phishing message that was sent out this week was able to gather some credentials and those credentials were used to send out spam from our servers.
I responded to the event in a very non-New-School way. I did what my gut said I should do and I stepped up user education efforts. Of course we blocked the affected account, and made some people reset their passwords, but I also had about 300 fliers printed up and distributed around campus reminding people that we don't ask for passwords over email. I've also initiated plans to make some updates to our web pages and automated emails that come out from our department.
Looking back on the whole thing (about two days later) I started trying to take a New School approach to the problem. One of the major ideas expressed in the book that I really agree with is that we need to be more open about events like this and share that information readily instead of keeping it to myself. So I typed up a detailed report and sent it off to my security contacts at other schools in the state.
But now I'd like to address how much money we should spend on mitigating phising attacks in the future. This is where that objective data comes in that we don't have. I don't know how many phishing emails have been sent to universities in the state in the last year. I don't know how many people have responded to the phishing emails, and I don't know how much cost this has imposed on our universities. So I'm not going to be able to answer the question. However, we can look at how the New School would suggest that we answer the question once enough data is present.
This week my school has received about 400 phishing email messages which resulted in about an hour of work for IT staff, and the people that had to reset their passwords. I capitalize an hour of employee time at $50/hour, so each one of those phishing messages cost us about 12.5 cents. However, as I mentioned above, all summer long we were receiving these messages and only had one event. I think there is merit behind the idea that a university is more vulnerable when class is in session than when it is in break. During the summer break I estimate that we had about 2000 phish messages sent to our campus, and that resulted in another $50 worth of loss. So during the breaks a phish message costs us 2.5 cents. With that data we can come up with a Weighted Single Loss Expentancy. (.25)(2.5)+(.75)(12.5) = 10 cents. We can assume that throughout the year, each phish message that we get is going to cost us 10 cents. Therefore, if we receive 10,000 phish messages each year our Annual Loss Expectancy is $1000.
So far this isn't very New School. This is right out of the CISSP Common Body of Knowledge. Here is where things diverge a little bit. One of the things mentioned in the book is a paper by Lawrence Gordon and Martin Loeb that describes how much we should be spending on Information Security. Their paper finds that we should spend somewhere between 25 and 37% of the expected loss on mitigation. So we should spend somewhere between $250 and $370 each year to reduce phishing. This runs counter to the conventional wisdom that says we should spend some amount of money less than $1000 to reduce phishing. One of the principles of Economics is that rational people think at the margin, and Gordon and Loeb point out that after 37% you have hit diminishing returns. In other words, an extra dollar spent on mitigation reduces loss by some number less than one dollar. In fact, if the resource being protected is not very vulnerable, then 37% is far too high. So we could say that it is worth $250/year to reduce phishing.
Now let's bring in my favorite finance concept, Net Present Value. Let's say that I put together a five year project to reduce phishing on campus. I want to spend $50 every year printing fliers. I also want to invest $500 of developer time right now into making changes to our web pages and automated email messages. Is that a good project? The PV of $250 paid each year for five years (5% discount rate) is $1,082.37. So if the PV of my project is less than that, then we're good. $550 in year one, plus $50 each in years 2 through 5 at 5% is $692.66. So it looks like my project is a good idea financially...Maybe.
There are a couple of things that we don't know. Sure we know how much I should spend, but we don't know what I should spend it on. My user education plan might be 10% effective compared to some unknown solution that is 80% effective. Obviously we would put money into the more effective form of mitigation. Another problem to consider (and this is one for the economists) is that the loss expectancy was based on what happened at my university. Well we have a spam firewall and we've already put money into user education, and some of the money spent on that is preventing phishing attacks from working. That money should come out of the $250 a year that we spend...unless an economist would consider that to be sunk cost. I'm not sure. Here is what we would need to answer the question of how much to spend on mitigating phishing positively...
- How many phishing emails were sent to undergraduate Universities in Minnesota in the last year?
- Is an undergraduate University in Minnesota statistically less vulnerable to phishing when it is in break or is it simply because there are fewer people on campus? It is important to note the qualifiers here. A graduate university may have a more educated population that is less resistant to phishing. Also the population in Minnesota may be more trusting of email than the population in New York.
- What are the actual losses suffered by these universities because of phishing?
I'm sure there are other variables to consider, but just these three above can give you a good idea of how far we are from where the authors would like us to be. It would take tremendous information sharing to find out the answer to numbers 1 & 3. It would take years to study to answer question 2. I have to tell you, I get so bogged down in the numbers and variables that I'm not even positive that my analysis above is any good. I'm sure that an expert in economics or finance might read this and say "what a dumbass!" I'm just hoping to be 85% right. I have a lot of learning yet to do.
Saturday, August 9, 2008
Net Present Value of Best Practices
Yesterday I was reading a book review of "The New School of Information Security" by Adam Shostack and Andrew Stewart. The review was written by Richard Bejtlich of taosecurity.com. There is a great quote in his in review: "...if you think anti-virus and a firewall are required simply because they are "best practices," you need to read The New School of Information Security (TNSOIS)."
This led me to question what a "best practice" is. It seems to me that a best practice is something that you do because it is so painfully obvious that everyone else is doing it. It seems that the argument that is made in TNSOIS is that you can put a financial value on having anti-virus and firewall in place.
I've already documented my opinions on the value of anti-virus software. To sum up, I believe that it is poor. However, I've never tried to run the numbers because I've only recently become aware of concepts like Net Present Value. I do believe in keeping my systems patched though. So I thought to myself, if keeping your systems patched is a best practice, meaning that it is obvious to anyone that it should be done, then I bet it has a high NPV. After all, if it had a negative NPV nobody would do it, and as the NPV approaches zero it becomes less and less attractive to put money into that project. If everyone is doing it then the NPV must be really high. I would bet that the same could be said for running a firewall. So I started to think about how to put a value on my firewall.
So I had to start thinking about what exactly my firewall does for me. In a nutshell, it prevents computers on the Internet from connecting to computers on my Internal network, except for those connections that I allow. This provides me the benefit of preventing a large amount of malware from infecting my computers. So I thought that one way to measure the value of a firewall is to look at how long it takes a computer to become infected with malware when there is no firewall in place. The thing is, patching also does some of this for me.
The SANs institute recently came out with a statistic that an unpatched Windows machine will be infected with malware 5 to 20 minutes after it is attached to the Internet. However, I wasn't entirely pleased with their methodology. They took the position that if a computer is scanned by some worm that is trying to propagate, then it would be immediately infected. So they took the average amount of time that exists between scans of a computer on the Internet. That seems alright, but I prefer the methodology used by the Honeynet Project. They actually placed simulated unpatched machines on the Internet and timed how long the machine stood up before it downloaded a binary. They found that it is about 16 hours. I'm going to use that number, and say that if I don't have a firewall in place, and if I don't patch my machines, I can expect that each computer is going to have to be reimaged every day.
Now that I know how often I'm going to have to reimage a computer, I need to know how much it costs to reimage a computer. I am going to estimate that it takes two hours of a technicians time to boot from the network and apply our image. During that time the employee that uses the machine is going to be unproductive, so the organization is out 4 hours of production. I'm also going to estimate that the total hourly expense of these employees is $50/hour (wages, vacation time, sick time, retirement contributions, social security contributions, staff to support them etc). So that means that it costs $200 to reimage a computer, and I'm going to reimage the computer every day.
So what is that worth to an organization that has 200 computers and considers a computer to have a five year useful life? About 66 million dollars! Damn! So the Net Present Value is going to be the Present Value (66 million) minus the initial investment. Do you believe that you can put in a firewall and patch your systems for less than 66 million dollars? I think even if you bought an awesome firewall like the Sidewinder, and you spent thousands of dollars putting in more bandwidth so all your computers could run Windows update regularly and you put in a Systems Management Server that helps keep everything else patched you would still spend less than the $863,724 that comes after the 66 million. So your Net Present Value is about $66 million.
I suggested that if something really is a "best practice" then it will have a very high NPV, and I think this example goes to prove that. The Present Value that I've calculated shows that a company should be willing to spend up to $66 million to avoid having to reimage their computers that often. Notice that I bolded the words "up to." I'm not saying that a company should spend that much money if they are able to put a firewall in place and keep their systems patched for less than that. You would hopefully present to management several options for patching and firewall purchases and work out the NPV of each option and management would select the one with the highest NPV.
This example that I gave is simple, but it is not perfect. For one thing, it doesn't take into consideration that even with a firewall in place and a good patching program some computers are still going to be infected because they slipped through the cracks. You might want to calculate the yearly savings from having a firewall and patching and reduce it by 10% to account for the residual risk. The other thing that I've found it that it is really difficult to separate the two. For example, I know that just having a firewall is not going to be as effective as having well patched machines and a firewall, but I don't know how much less effective. Having only a firewall would tremendously drop the number of remote exploits that get run on your machines, but it wont help you against web sites that your users visit that have exploit code running on them. Patching would help with the latter.
One methodology that comes to mind is to repeat the experiment done by Honeynet project from inside your firewall. If you were to put several unpatched machines inside your network how long would it take for them to download a binary. It would be more helpful if you have computers that are in use by employees so that you can get the effect of them visiting web pages and engaging in unsafe behavior as users are known to do. Let's say that the average time until infected jumps from 16 hours to 40 hours. Calculate the new present value and the difference will give you some idea of what your firewall is worth. It this perfect? No, but I think that it is better than saying "we need to have a firewall in place because it is a best practice."
This led me to question what a "best practice" is. It seems to me that a best practice is something that you do because it is so painfully obvious that everyone else is doing it. It seems that the argument that is made in TNSOIS is that you can put a financial value on having anti-virus and firewall in place.
I've already documented my opinions on the value of anti-virus software. To sum up, I believe that it is poor. However, I've never tried to run the numbers because I've only recently become aware of concepts like Net Present Value. I do believe in keeping my systems patched though. So I thought to myself, if keeping your systems patched is a best practice, meaning that it is obvious to anyone that it should be done, then I bet it has a high NPV. After all, if it had a negative NPV nobody would do it, and as the NPV approaches zero it becomes less and less attractive to put money into that project. If everyone is doing it then the NPV must be really high. I would bet that the same could be said for running a firewall. So I started to think about how to put a value on my firewall.
So I had to start thinking about what exactly my firewall does for me. In a nutshell, it prevents computers on the Internet from connecting to computers on my Internal network, except for those connections that I allow. This provides me the benefit of preventing a large amount of malware from infecting my computers. So I thought that one way to measure the value of a firewall is to look at how long it takes a computer to become infected with malware when there is no firewall in place. The thing is, patching also does some of this for me.
The SANs institute recently came out with a statistic that an unpatched Windows machine will be infected with malware 5 to 20 minutes after it is attached to the Internet. However, I wasn't entirely pleased with their methodology. They took the position that if a computer is scanned by some worm that is trying to propagate, then it would be immediately infected. So they took the average amount of time that exists between scans of a computer on the Internet. That seems alright, but I prefer the methodology used by the Honeynet Project. They actually placed simulated unpatched machines on the Internet and timed how long the machine stood up before it downloaded a binary. They found that it is about 16 hours. I'm going to use that number, and say that if I don't have a firewall in place, and if I don't patch my machines, I can expect that each computer is going to have to be reimaged every day.
Now that I know how often I'm going to have to reimage a computer, I need to know how much it costs to reimage a computer. I am going to estimate that it takes two hours of a technicians time to boot from the network and apply our image. During that time the employee that uses the machine is going to be unproductive, so the organization is out 4 hours of production. I'm also going to estimate that the total hourly expense of these employees is $50/hour (wages, vacation time, sick time, retirement contributions, social security contributions, staff to support them etc). So that means that it costs $200 to reimage a computer, and I'm going to reimage the computer every day.
So what is that worth to an organization that has 200 computers and considers a computer to have a five year useful life? About 66 million dollars! Damn! So the Net Present Value is going to be the Present Value (66 million) minus the initial investment. Do you believe that you can put in a firewall and patch your systems for less than 66 million dollars? I think even if you bought an awesome firewall like the Sidewinder, and you spent thousands of dollars putting in more bandwidth so all your computers could run Windows update regularly and you put in a Systems Management Server that helps keep everything else patched you would still spend less than the $863,724 that comes after the 66 million. So your Net Present Value is about $66 million.I suggested that if something really is a "best practice" then it will have a very high NPV, and I think this example goes to prove that. The Present Value that I've calculated shows that a company should be willing to spend up to $66 million to avoid having to reimage their computers that often. Notice that I bolded the words "up to." I'm not saying that a company should spend that much money if they are able to put a firewall in place and keep their systems patched for less than that. You would hopefully present to management several options for patching and firewall purchases and work out the NPV of each option and management would select the one with the highest NPV.
This example that I gave is simple, but it is not perfect. For one thing, it doesn't take into consideration that even with a firewall in place and a good patching program some computers are still going to be infected because they slipped through the cracks. You might want to calculate the yearly savings from having a firewall and patching and reduce it by 10% to account for the residual risk. The other thing that I've found it that it is really difficult to separate the two. For example, I know that just having a firewall is not going to be as effective as having well patched machines and a firewall, but I don't know how much less effective. Having only a firewall would tremendously drop the number of remote exploits that get run on your machines, but it wont help you against web sites that your users visit that have exploit code running on them. Patching would help with the latter.
One methodology that comes to mind is to repeat the experiment done by Honeynet project from inside your firewall. If you were to put several unpatched machines inside your network how long would it take for them to download a binary. It would be more helpful if you have computers that are in use by employees so that you can get the effect of them visiting web pages and engaging in unsafe behavior as users are known to do. Let's say that the average time until infected jumps from 16 hours to 40 hours. Calculate the new present value and the difference will give you some idea of what your firewall is worth. It this perfect? No, but I think that it is better than saying "we need to have a firewall in place because it is a best practice."
Subscribe to:
Posts (Atom)