I Love Boobies and Information Security

If you work in Information Security, or any form of security for that matter, you're probably used to noticing things. Maybe we just pay a little more attention to the details around us. And if you're working on a college campus, maybe you've noticed the number of people wearing bracelets that say "I love boobies" or "I heart boobies."

Turns out that it is part of a breast cancer awareness campaign and it seems to be quite effective. How effective? So much so that one day while working up in my office I saw three customers in a row come in with the bracelet on. It seemed so rare to me that I asked one of them why there were so many people with the bracelet on. Interestingly, she didn't know of any organized campaign to get people to wear the bracelets and didn't realize that so many people were.

So I decided I should take a moment to figure out how many of our female students on campus are wearing these bracelets. Male students are kind of irrelevant because breast cancer isn't a major concern for them and they probably love boobies for reasons not associated with cancer. So I wanted to know what percentage of female students on campus are wearing this particular kind of bracelet.

This is where reading Douglas Hubbard's book on How to Measure Anything comes in handy. There are some people who would instantly tell me that I wont know unless I take a census of the female students on campus or at least survey about 1000 of them. But since I've read Hubbard's book I know that I don't need as much information or precision as my gut first tells me. I also know that the best way to go measure something is to go out and do it.

So I walked out the door of my building and counted 30 female students at random as I walked from one building to another. If I was able to get close enough to a woman to observe both of her wrists then she was counted, otherwise not. Out of 30, I saw one girl wearing such a bracelet. This very simple observation is enough to tell me that I can be 90% confident that the percentage of female students on campus wearing such a bracelet is between 8% and one one-thousanth of a percent (one observation divided by 7817 female students). I actually decided that I wanted to have more precision so I made a few more observations whenever I had to walk from one building to another.
So there you have it. With a few really simple observations my uncertainty about the number of female students with this bracelet on has been reduced and I can express the measurement as a number. If someone were to ask me, I could say that between one and five percent of the female students on my campus are wearing the bracelet. If the people behind the bracelet were hoping to have ten percent of college girls wearing them then without spending any money or any tremendous amount of time I could tell them that it is unlikely that they met their goal. If the goal was 3% I could tell them that they are close but that additional study is necessary to get a better answer.

So what does this have to with Information Security? Mostly it's just a demonstration that it isn't hard to measure things when you deconstruct the problem and measure it. If we wanted to measure the effectiveness of the I heart boobies bracelets, we have to deconstruct it to find out what the observable characteristics are. In this case, number of students wearing the bracelet. So what if you wanted to measure the effectiveness of your information security awareness program?

First, you have to deconstruct it down to the observable characteristics. If you want to know whether it worked or not, what might you see? One idea that jumps into my head is the number of unattended workstations left unlocked might go down. Sweet. I can observe that, and using the same technique that I used to measure boobie lovers on campus I can get an idea of what percentage of office computers are left unlocked. Or you could send a phishing email to several randomly selected people and count how many answer it. Take before and after measurements and see if there is a noticeable improvement in the numbers.

So what can boobies tell us about Information Security? You can have a lot of fun looking at and measuring things you cant touch.

Testing out the rule of five

A while back I read the book "How to Measure Anything" by Douglas Hubbaard.  In a nutshell, I thought the book was great, and it has a lot of simplifying assumptions in it that you can use when you're trying to measure something intangible, like Information Security.

There is one thing that I have had a little trouble accepting though, and that is the rule of five that he describes in one of the chapters.  It says that if you were to randomly sample five people in a population for some value (such as how many hours of sleep you got last night) there is about a 93% chance that the median value for the whole population will fall between the largest number you get from your sample and the smallest that you get from your sample.  

If I remember correctly, it all starts with the premise that if you sample two people there is a 50% chance that the range of their two numbers will not include the median.  Add a third person and there is another 50% chance.  .5 x .5 = .25 so now there is only a 25% chance that the median does not fall in that range.  A fourth person means we multiply .25 by .5 and get .125.  Finally the fifth person brings us to a probability of .0625 that the median is not included in our range.  So I've always had a little trouble with the first statement, that there is a 50% chance that the first two numbers will include the median.  I looked around the Internet and I haven't been able to find any other confirmation of the rule of five, except for other people citing Hubbard.  So I decided I would try a couple simple tests to see if this would work for me in theory.

The first test was to see if I could reproduce the 50% chance of picking two numbers that include the median.  I opened up my spreadsheet program and in the A column I put in this formula: =RANDBETWEEN(1,1000).  I copied that down 1000 rows to get 1000 random numbers between 1 and 1000.  This was my reference column.  I copied the values and pasted them into column B and then deleted column A.  That way the values wont keep changing every time I do some math on the page - if you're keeping track at home that means that a list of static numbers is now in column A.  Then I put that same formula into columns B and C.  This simulates the process of picking two numbers from the whole population.

If you're really paying attention, you will notice that I didn't actually chose two values from the sample, I generated two more random numbers.  So this isn't exactly the same, but I'm just trying to do a "back of the envelope" test here, and the values are probably close enough to some other random number.  In other words, I recognize that this isn't perfect, but it is close enough for my purposes.  In column D I just put in one formula: =MEDIAN(a1:a1000).  And it gave me the median value of my list of random numbers.  In column E I put in this formula: =MAX(b1,c1).  In column F I put in this formula: =MIN(b1,c1).  So now I know that column E has the upper bound of my range and column F has the lower bound.

In column G I put in this formula: =IF(E1>=D$1,"1","0"), and in column H I put in =IF(D$1>=F1,"1","0").  So if the range includes the median, I will have a 1 in column G and H.  In column I put =G1+H1.  Copy these formulas all the way down and column I will have a 2 in it every time the range includes the median.  BTW, if there is an easier way to do this I would love to hear about it.  The last step was in cell J1 where I put =COUNTIF(I1:I1000,"=2").  If there is really a 50% chance then this should be pretty close to 500.  What was my final number?  I have to admit I was surprised get 520.  Not bad.  Not proof, mind you, but definitely something to lend credence to the rule of five.

For my next experiment, I decided to get a little more fancy.  I whipped up the following python script.  In a nutshell, it creates a population of 1000 random numbers.  Then on it creates 500 independent random samples and checks to see if the median of the population falls within each sample and prints out the percentage of successes.  I ran this bad boy and got 96.2%.  So after all of this, I have to say I'm feeling pretty good about the rule of five, even if I can't find any independent verification of it.

from __future__ import division
import random

def intherange(median, sample):
  sample.sort()
  if sample[4] >= median:
    if sample[0] <= median:
      return 1
  return 0

population = []
sample = []

# create a list of 1001 random numbers
for i in range(1,1002):
  population.append(random.randint(1,1000))

# sort the sample and get the median
# dont forget to offset by one or you'll get one number above
# the median.
population.sort()
median = population[500]

# Let's take 500 samples and see what we get each time.
sum = 0
for i in range(1,501):
  sample = random.sample(population,5)
  sum += intherange(median, sample)

print sum/500

Beta, it's not just for fraternity names

Last week at Secure360 I gave a talk on using monte carlo simulations to deal with unknowns in the calculation of Annualized Loss Expectancy (ALE).  For those of you that need a refresher, the idea behind ALE is that you figure out how much an asset is worth (Asset Value or AV), and you figure out how badly an event would hurt that asset (Exposure Factor or EF).  Then you multiply those by how often it happens (Annual Rate of Occurrence or ARO) to get the Annualized Loss Expectancy (ALE).

In my talk I mentioned using Monte Carlo simulations and why I like them.  You see, I am of the belief that you can't nail any of those numbers (AV, EF, ARO) down precisely so you need to work out a reasonable range or even a slightly unreasonable range as long as you err towards inclusion.  By erring toward inclusion, I mean your range would be unrealistically wide rather than narrow.  Once you have your ranges you can whip up an excel spreadsheet that picks a random number between each of your ranges and spits out the ALE.  Repeat this over 5,000 to 10,000 rows and you've got your simulation.  But a good Monte Carlo simulation is more nuanced than that.

If you just pick a random number between each of your ranges, and there is no skewing of the numbers then after you've done this about 10,000 times you're going to get an average that is shockingly close to what you get if you just take the middle number of each range and multiply.  That is the law of large numbers in action.  What really makes your monte carlo simulations more accurate is that they also take into consideration the shape of your variables.  I'd like to talk about a couple of shapes, and my new favorite formula to use in monte carlo simulations.

For the most part, I have always stuck with two basic shapes, the uniform distribution and the normal distribution.  Uniform distribution is where there is an equally likely chance of any number in the range being the "true" value of the real thing being simulated.  I typically use this on asset value by utilizing the RANDBETWEEN() function.  I know that the asset value falls between x and y and there is an equal chance of any one of those numbers being accurate.  I typically use the normal distribution in cases where I have an average and a reasonable guess about the standard deviation.  For example, if I know that 75% of my users have experienced some phenomena give or take 8% then I will use a normal curve that would spike at 75% and taper off dramatically so that there are almost no values below 67% or above 83%.

But as I have continued to refine my practice of monte carlo simulations, it occurred to me that I need more shapes.  There are variables that don't fit neatly into one of these two shapes, and that is where the beta distribution (http://en.wikipedia.org/wiki/Beta_distribution) comes in.  Beta is able to reproduce a wide variety of shapes that may be more appropriate for your variables.  Let me give you an example.

Let's say one of the threats to your asset is power outage.  One thing you need to know is how often you're going to deal with a power outage in your data center.  Going back over the historical statistics that you've kept, and talking to your server and network people you've all agreed that there will probably be 3 power outages in the data center this year because your maintenance people suck and they are always making changes without telling anyone.  Everyone also agrees that there could be more power outages, but that the odds of having more outages go down quickly as the number increases.  This isn't something that is shaped like a normal curve, this is more of a straight line that is high on the left side and moves down as you go right.  Sure you can punish your data and force it into a normal curve, but instead lets try out our new beta distribution and see if we like that shape better.  In my spreadsheet, under annual rate of occurrence for power outage, I put in =INT(BETAINV(RAND(),1,5,3,8)) and copied that down 500 rows.  Out of the 500 rows, it returned 3 outages per year 347 times, 4 outages 127 times, and 5 outages just 31 times.

I'll let you look at wikipedia to see how the first two numbers (a and b) affect the shape of the distribution.  In a nutshell, if a is bigger than b then the distribution trends upwards.  If b is bigger than a, then the distribution trends downwards.  The difference between a and b is how dramatic that trend is.  If b is much larger than a you get a very L shaped graph where the numbers drop off quickly.  If a=1 and b=2 you get a straight line that trends downward.  The last two numbers in the formula are a bottom and top boundary to put on the distribution.  In my formula above, I say that there will always be at least 3 and never 8 or more.  So if I wanted a straight line reflecting outages of 3 to 8 times per year, then I could use this formula =INT(BETAINV(RAND(),1,2,3,9)).  When I ran that 500 times I got 3 outages 163 times, 4 outages 137 times, 5 = 95, 6 = 66, 7 = 38, and 8 outages 12 times.  In other words, there is only a 1.6% chance that we'll have 8 power outages in one year, but there is a 60% chance that we'll have 3 or 4 outages.  

Play around with some of the other shapes that you can make with your beta distribution.  I just created a spreadsheet where I could play with the a, b, xlow, and xhigh numbers and see how it charts.  I will be honest and tell you that I don't yet know how to calculate what a and b should be in my beta distribution, but I am still really happy because if I can make a distribution that more closely approximates what I expect to see in the real world then my simulations will return better data.  This is one of those areas where we can make our range estimate tighter without spending additional money on research, so even if it isn't exact, it is still good news.  I hope you are able to find value from this as well.

Real World Phishing Statistics

Over the last two days I have been dealing with the fallout of a big phishing message that hit our organization.  We had two messages come out in two days and one of them was really well done.  It had no grammar mistakes and it used the proper name of our organization and our services.  The spoofed reply to address even matched our domain, which has not been the case in previous phishing messages.  This one was head and shoulders above the rest.

We had to field a lot of calls from people asking about this message.  We get phishing messages every day so I have to admit that I am a little bit jaded about them now.  But after we got a few calls I ran a query to find out how many of these had come in and when we saw that it was a rather large number (465 users) we decided that we needed to take some action.  We queried our outgoing mail and locked the user account of anyone that responded to the message in any way.  The next day the second message came in and it went to about 840 users.  This time we hijacked the return address so that anyone who sent a response went into a mailbox that we control instead of the phisher.  This was helpful in preventing misuse of our systems, but it also allowed us to gather some statistics that I'd like to share with you.  Keep in mind that these are statistics for one single incident at a college in the Midwest United States.  It would be foolish to think these numbers will hold true for any organization in any part of the world.

The two messages went to a total of 853 unique users (there was some overlap in the two messages).  Of those 853 users, 29 responded to the message.  We have about 14,000 user accounts so that means that we can say with 95% confidence that 3.34% (+/- 1.17%) of our users will provide *SOME* response to a well-crafted phishing message.

Of the 29 total responses, we were able to intercept and examine 13 of them.  Of the 13 we found that 11 of them had responded with a username and password.  The other people told the phisher to "piss off" or asked if the request was real.  Based on these numbers we have a sample size of 29 out of a population of 853 and 84.61 provided their password.  So we can say with 95% certainty that 84.61% (+/- 14.83%) of responses to phishing messages contain the username and password.

There are a couple interesting demographic things you should know about our organization and my methodology for coming up with these numbers.  First of all, I came up with the statistics using the sample size calculator here: http://www.surveysystem.com/sscalc.htm

Next, our organization has about 14,000 users most of whom are from the Midwest and most of whom are between the ages of 18 and 22.  10% of the population has been through a computer-based computer awareness training program, and the organization sends email reminders about phishing about twice a year.

One thing that no human or machine could count is the number of calls, emails and shouts in the hallway asking if the message was real or not.  That actually consumed more of my time than dealing with the 29 people that responded to the messages.  So if you're ever working through how much a phishing attack might cost your organization, make sure you add in the value of people's time dealing with the same question about 6 million times.  6 million times in an organization with about 14,000 users.  That's a good starting metric, but you may get more.  

Applying Unknowns to Annualized Loss Expectancy

I've started putting together a presentation for a conference that I'm writing a proposal for about how you can use security metrics and statistics to make Annualized Loss Expectancy work a little better. The idea is to get a compromise between the ease of qualitative risk analysis and the accuracy of quantitative risk analysis. I started writing up an example of what I mean, and it turned out pretty good, so I thought I would share it here.

How much should we spend on hard disk encryption for our sales force?
Let's start with a simplifying assumption. How much do you spend on each record that you lose? Well that answer might be based on the type of data that is lost, but you can probably come up with a range of numbers that is reasonably accurate. For example, you know that you're probably going to pay for a stamp for each letter, so you know that it has to be more than 42 cents per record. You probably have to figure that you're going to put two hours into accounting for the records on each breach. You should assume that the employee who lost the laptop is also going to lose some time while he is being interviewed by you. You can expect that your legal department is going to lose a couple hours of time while they draft up a letter to everyone involved. So at an absolute minimum, you know that it is going to cost five or six hours of employee time plus 42 cents per record. A quick look on the web shows that companies that sell expensive software want you to believe that the average cost is $197 per record. That's probably a reasonable high end of the range.

Next it would be nice to know how many records are on the average sales laptop. That's really not too hard to do. Get a list of all the sales people and find out how many you would need to sample to be 95% certain of a 3% confidence interval. Let's say you've got 50 sales people. A quick Google search finds me a sample size calculator. When I plug in my numbers it says that I need to sample 48 of my 50 sales people if I want that kind of accuracy. Crap, I'm not going to do that. But I know from the rule of five from statistics that if I randomly sample 5 of them, I am 93% certain that the true median falls between the top and bottom of that range. So let's randomly sample five of them and check out what is on their laptops right now. In this hypothetical scenario, I found that the laptops had 250, 128, 64, 0 and 0 records. I don't want to run into a divide by zero error, so I'll set a floor of one record. So I can set a range of 1 to 250 records on each laptop.

Let's assume that HR comes back and tells us that the average employee salary is $54/hour with a standard deviation of $14/hour. We decided that each laptop was going to require at least 6 hours of staff time regardless of how many records are in place. So if there is one record on the laptop, then have a cost per record of ($54 * 6) + 42 cents. That's $324.42 per record. If there are 250 records then the cost is $1.72 per record. That $324 per record doesn't withstand my smell test, so I'm going to throw it out in favor of of the $197 per record cited above. That means that when a laptop is stolen from our sales force, we should expect that it will cost between $1.72 per record and $197 per record and that there will be between 1 and 250 records exposed. That gives us a cost range of $1.72 to $49,250 for a single loss.

There are ways that we could get tighter numbers if we really needed them. We could actually sit down and survey 48 of our 50 sales people so that we would be more confident about the number of records on each sales person's laptop computer. We could also run our numbers through a Monte Carlo simulation to see if we get a tighter distribution of costs. All that would be unnecessary, however, if we found that even based on our broad estimation of costs, the software was too expensive to implement. Based on the parameters that we established above, I ran a small simulation and came up with an average of $6.67 per record with a standard deviation of $11.96. We know that there is a 66% chance that the true average is within one standard deviation of the mean, but we should also expect that we still wont have a scenario where the cost dips below $1.72 per record. So now we can say with a fairly high degree of certainty that the average cost per record will be between $1.72 and $18.63. And that gives us a single loss expectancy between $1.72 and $4,657.

Now we're getting numbers that don't seem unreasonable, and you can really tell your managers that you didn't just make something up. There is real math behind using real numbers behind this. What if you were trying to justify spending $5000 on disk encryption for your sales team and you expect the software to be useful for five years? You would only need to have two laptop thefts in the next five years to make this a good decision. You can repeat the same processes above to get a strong estimate of the number of laptop thefts that you're likely to suffer.

Also, remember to bring all of your costs back to today's dollars using Net Present Value. The total cost of your software today should include the discounted cost of employee time next year and software maintenance for next year. Decide how many years you're going to assume for the useful life of the product. Same goes for your losses. If you assume a loss of $4,657 per year, and a discount rate of 4% then the present value of those losses is $20,372.14. Subtract the present value of the cost of your software project and you've got NPV. If NPV is positive, then you've got a good project. If not, then you should look at other ways you can improve the situation for less money.

More password cracking statistics

In my last post I talked about how we anonymized our password database, and took a random sample of hashes to attack with a dictionary. In this post I'd like to talk about the results that we got when we took the same random sample and attacked it with rainbow tables.

I have to say that even I was shocked at this statistics: 84.07% of the passwords in the sample were broken by the rainbow table attack. Incidentally we used Ophcrack with the fast windows xp table.

The average length of a cracked password was 7.84 characters with a standard deviation of 1.70 characters. I still see a lot of password policies written that prescribe passwords to be at least 8 characters long. Statistically speaking, almost half of the broken passwords were at least 8 characters long.

Of the broken passwords, only 2 had four character types. Not 2%, but 2. As a percentage it was 0.22%. 12.15% had three character types, 40.69% had three character types, and 44.82% had only one character type.

As I mentioned in my previous post, I plan to use this data after we've implemented our new password policy to measure its success. It would be interesting to know if other organizations audit their passwords like this and what kind of statistics they have found.