The Google Suckage Ratio

Sometimes we security people find ourselves in the uncomfortable position of having to buy software, services, or hardware to enhance our security. And in many cases we really don't know a whole lot about the vendors that we have to purchase from. Let's say, for example, that we need to purchase some vulnerability management software. There are some big names out there, like Nessus, Qualys, and nCircle but how do you choose which one to purchase.

I'm not going to go through the whole purchasing decision process. I would hope that part of your selection process involves developing a matrix of qualifications, assigning a weight to those qualifications, and assigning a score to each vendor. You should also put in some guidelines for each qualification so that multiple reviewers will give come up with roughly the same answers for the same product (inter-rater reliability).

One of the things that you might want to consider when you're making the decision is how favorably the company is viewed by others. After all, if the general consensus is that a company sucks, then you probably don't want to invest a big hunk of money with them. But how do you go about doing that? You can talk to a few friends of yours, but you're taking an awfully small sample to base your opinion on. You can also talk to references provided by the company, but there is going to be a high degree of selection bias in that approach. The approach I would like to consider is the Google Suckage Ratio.

It's dead simple the calculate the Google Suckage Ratio. Simply go to Google and type in a company name, like nCircle and look at how many results you get back. Now type the company name followed by "sucks." Take the total number of company sucks results and divide by the total number of company results and you get a Suckage ratio. You may be tempted to put the whole thing in double quotes, but I have found that you can get a staggeringly low number of hits. Instead we're going to look for the company name surrounded by a negative adjective.

Company	Company Sucks	Company mentions	Suckage
nCirlce	26,700	189,000	14%
Nessus	156,000	921000	17%
Qualys	20,100	226,000	9%

There are a couple more considerations we should make so that our measurement is more reliable. When we do these searches, we should exclude the names of the other vendors that we're considering. After all, it may be a discussion board where people agree that nCircle is awesome and that Nessus sucks. So I ran the searches again with exclusion. Here is a sample search string: Nessus -Qualys -nCircle. I have also found that if you reverse the order of the excluded terms you can get a different number of hits.

One question that was brought up when I was discussing this with some co-workers was whether or not we need to consider every negative adjective out there. Do I also need to search for 'Nessus Blows' and 'Nessus really sucks'? I decided that the answer is no because I'm really trying to compare these companies to each other using the same measurement tool. In other words, I'm not saying that 9% of Qualys users are unhappy, I'm saying that the ratio of bad press to total press is higher for Nessus, about 50% higher (9 divided by 17). In theory, if we added in company name blows then we would still see about the same ratio between Qualys and Nessus.

Now I'm not saying that I endorse this method yet, only that I am intrigued by the idea. I would love to know if this is pure crap that I am suggesting here. I would also like to point out that I am not the first person to suggest doing this, but I don't know who is. One of the things that gives it credibility is the fact that Google has indexed so much of the Internet. When you dip into Google you're taking a very wide sample of the Internet.