Monday, February 22, 2010

The Google Suckage Ratio

Sometimes we security people find ourselves in the uncomfortable position of having to buy software, services, or hardware to enhance our security. And in many cases we really don't know a whole lot about the vendors that we have to purchase from. Let's say, for example, that we need to purchase some vulnerability management software. There are some big names out there, like Nessus, Qualys, and nCircle but how do you choose which one to purchase.

I'm not going to go through the whole purchasing decision process. I would hope that part of your selection process involves developing a matrix of qualifications, assigning a weight to those qualifications, and assigning a score to each vendor. You should also put in some guidelines for each qualification so that multiple reviewers will give come up with roughly the same answers for the same product (inter-rater reliability).

One of the things that you might want to consider when you're making the decision is how favorably the company is viewed by others. After all, if the general consensus is that a company sucks, then you probably don't want to invest a big hunk of money with them. But how do you go about doing that? You can talk to a few friends of yours, but you're taking an awfully small sample to base your opinion on. You can also talk to references provided by the company, but there is going to be a high degree of selection bias in that approach. The approach I would like to consider is the Google Suckage Ratio.

It's dead simple the calculate the Google Suckage Ratio. Simply go to Google and type in a company name, like nCircle and look at how many results you get back. Now type the company name followed by "sucks." Take the total number of company sucks results and divide by the total number of company results and you get a Suckage ratio. You may be tempted to put the whole thing in double quotes, but I have found that you can get a staggeringly low number of hits. Instead we're going to look for the company name surrounded by a negative adjective.

CompanyCompany SucksCompany mentionsSuckage
There are a couple more considerations we should make so that our measurement is more reliable. When we do these searches, we should exclude the names of the other vendors that we're considering. After all, it may be a discussion board where people agree that nCircle is awesome and that Nessus sucks. So I ran the searches again with exclusion. Here is a sample search string: Nessus -Qualys -nCircle. I have also found that if you reverse the order of the excluded terms you can get a different number of hits.

One question that was brought up when I was discussing this with some co-workers was whether or not we need to consider every negative adjective out there. Do I also need to search for 'Nessus Blows' and 'Nessus really sucks'? I decided that the answer is no because I'm really trying to compare these companies to each other using the same measurement tool. In other words, I'm not saying that 9% of Qualys users are unhappy, I'm saying that the ratio of bad press to total press is higher for Nessus, about 50% higher (9 divided by 17). In theory, if we added in company name blows then we would still see about the same ratio between Qualys and Nessus.

Now I'm not saying that I endorse this method yet, only that I am intrigued by the idea. I would love to know if this is pure crap that I am suggesting here. I would also like to point out that I am not the first person to suggest doing this, but I don't know who is. One of the things that gives it credibility is the fact that Google has indexed so much of the Internet. When you dip into Google you're taking a very wide sample of the Internet.


Michael Janke said...

I use queries like that to try to get a feel for a vendor or product, but I've never attempted to quantify 'suckage'.

How would you handle old suckage verses recent suckage, or how would you quantify a small amount of suckage that was widely reported verses a lot of suckage that didn't get much internet traction?

Black Fist said...

@Mike Janke: Excellent questions, and thanks for commenting. In the advanced Google settings you can specify that you want to find documents older than a specific date. I think if you play with the Google hacking HOWTOs out there you can find ways to limit your search to a date range. As for specific suckage, you might try excluding some keywords from your search.

However, I worry that diving that deep into the search results might be an attempt to make this measurement more accurate than possible. I feel this is more of a numerical finger in the air, and I wouldn't put a large weight on this metric in your decision matrix.