Thwarting phishing on the cheap

If you've been following my blog then you know that phishing has been a real problem at my organization. Users are falling for phishing messages no matter how poorly written they are and we've suffered way too many intrusions because of it. Even though I fear that it is largely impractical, I feel like I have to do something to stem the tide. Since this is unquestionably the largest source of intrusions into our network, even an inefficient method of catching these might be better than nothing.

So the first question is: how can I prevent the phishing messages from getting to my users? That would be the most effective way to combat this. But we've got two Barracuda Spam firewalls already working on this problem. It is unlikely that I'm going to do a better job of blocking the phish messages on the way in. So I decided to focus on messages on the way out.

Here is what I'm trying out right now. This is far from a perfect solution and it is also not the finished poduct. This is my first toe in the water to using Snort to try and detect the responses to phishing messages.

First, I made a new ruletype called phishinghole. This is in my /etc/snort/snort.conf file.

ruletype phishinghole
# The phishinghole rule type gathers up any alerts that could
# be responses to phishing messages and keeps the tcpdumps
# in one easy to read file
{
  type alert
  output alert_full: alert
  output alert_syslog: LOG_LOCAL4 LOG_ALERT
  output log_tcpdump: phishinghole.pcap
}

So when I write up a rule that uses this alert type, an event will be written to the normal alert file, it will also go to our syslog servers, and the packet itself will be written to a file called phishinghole.pcap. That way, when I want to inspect today's catch, I don't have to go through a huge packet capture file looking for just the packets that interest me.

I also want to be able to classify these events properly, so I added this one line to my classification.config file:

config classification: phishing-response,Possible Response to phishing message,5

Next up, I need to write some rules. I'm going to look for any traffic coming from my network and going to some other network on port 25. That's email. I'm going to use my new phishinghole ruletype, and I want to log the message that this might be the response to a phishing email:

phishinghole tcp $HOME_NET any -> !$HOME_NET 25 (msg:"Possible Phishing Response";

Next we need to define the content that we're going to look for. When I first started doing this, I came up with three regular expressions to look for possible permutations of password, username, and email. However, I decided that I didn't want to take the performance hit of running three regular expression searches against every single packet that leaves the organization. I decided instead to look for the word password and if that matches, then run the other two regular expressions. That should trim down the number of packets I have to look at.

content:"password"; nocase;

Now the magic is in the regular expressions. This is the Perl Compatible Regular Expression I'm using to search for username. This will match regardless of case and whether the word is broken in two with a space or a dash "user name" or "user-name" or "User Name" etc. The second regular expression looks for the word email in a similar fashion.

pcre:"/user[\-|\s]?name/i"; pcre:"/e?[\-|\s]?mail/i";

I still need to review these messages manually to see if someone really did respond to a phishing message or just told the guy to go to hell. So I want to gather a few extra packets to get as much context as reasonably possible. This next part of the rule tells snort to record three additional packets in the conversation.

tag:session,3,packets;

I also want to make sure my rules are classified properly. I want these events to bubble up to the top of my priority stack, so I put this into the rule:

classtype:phishing-response;

and I ended the rule with a sid (todays date and a revision number)

sid:20090525; rev:1;)

I repeated the process for three more reasonable permutations of the word password. I figure that a phisher cant get too crazy with the spelling or capitalization or the message will lose credibility. Here are the finished rules that I came up with.

blockquote>
phishinghole tcp $HOME_NET any -> !$HOME_NET 25 (msg:"Possible phishing response"; content:"password"; pcre:"/user[\-|\s]?name/i"; pcre:"/e?[\-|\s]?mail/i"; nocase; tag:session,3,packets; classtype:phishing-response; sid:20090525; rev:1;)

phishinghole tcp $HOME_NET any -> !$HOME_NET 25 (msg:"Possible phishing response"; content:"pass word"; pcre:"/user[\-|\s]?name/i"; pcre:"/e?[\-|\s]?mail/i"; nocase; tag:session,3,packets; classtype:phishing-response; sid:20090525; rev:2;)

phishinghole tcp $HOME_NET any -> !$HOME_NET 25 (msg:"Possible phishing response"; content:"pass-word"; pcre:"/user[\-|\s]?name/i"; pcre:"/e?[\-|\s]?mail/i"; nocase; tag:session,3,packets; classtype:phishing-response; sid:20090525; rev:3;)

Now for the analysis
Since these are .pcap files, it's tempting to open up wireshark and get started at peeking through them. That's what I did and it works just fine.  However, after a couple of days I realized that I was getting way too many false positives, and I needed a new way to separate the wheat from the chaff.  Since the only thing in the .pcaps are email snippets, that means that all of the data I need to sift through is going to be in plain text.  So I ran the strings command against the pcaps just to make sure that I would get a dump of all text in the file.  Then I talked to our system administrators to find out if I could get a plain text feed of active user accounts on the domain.  Now I can do something like this to find out if I need to look at the file in more detail:

strings phishinghole.pcap | grep -f listofusernames

If I get any hits then I know I have an email that has the word username, password, email and a valid user name on our network in it. Even if that is a false positive, that is worth investigating. It's also pretty easy to find the username in the .pcap file when you know what username you're looking for so then you can see the message in context.

I'm going to let this run for a few more days and see if I'm satisfied with it. If I think I've got a winner here, I'll automate the process further.

Improving your Google hacking with Python

UPDATE: I made some major improvements to this code which also made it quite lengthy. You can find the full script here. The linked version adds support for a whitelist file so you don't get bothered by false positives. The improved script will also attempt to download the links that it gets from Google and make sure that it doesn't report any dead links to you.

One of the major information security problems that I have on my campus is that we have a tendency to improperly release data that is supposed to be secret. The primary vehicle for doing that is faculty members posting grades on the Internet. Every semester we have a couple hundred student records that are posted on the Internet, which has created problems for me semester after semester.

Now before I go on, I should point out that there is nothing wrong with posting student grades on the Internet, as long as the data has been sufficiently anonymized. If a professor was to assign random numbers to each of his students then the grades could be posted online using the random numbers. However, most of the time (in violation of campus policies and Department of Education regulations) the grades are posted by Student ID. That's a no no.

But each of our faculty members have their own personal web space where they can put stuff, and I don't have the ability to go through each of them every day and find where grades have been improperly posted, so I use my best friend Google to do it. I might do a search against google for any Excel spreadsheet that contains techids and grades that are on my domain:
"techid grade ext:xls site:mnsu.edu"

That's been my primary way of finding the leaks. But I wanted something better. I want to be able to run a script every day and just get the list of links that I need to check out. So I went to my favorite interpreted programming language, Python.

Below I've pasted a little script that I've put together to do the searching for me. I put my search strings into a file called "searchstrings" and this program runs them. It then pulls out all of the liks, and then removes the links that are not on my domain. Feel free to use this yourself if you want. I have a few changes that I would like to make. I plan to add in support for whitelisting links, and specifying which searchstring file you want to process. But this should show you the basic process that I'm using. I also stole some of this code from several places around the Internet, so please check out the references that I've put at the top.

#!/usr/bin/env python

# This script will connect to google, pull down some search results,
# remove the bullshit and only show me what I want to see.

# reference: http://docs.python.org/library/urllib.html
# reference: http://cis.poly.edu/cs912/parsing.txt
# reference: http://mail.python.org/pipermail/python-list/
#       2006-December/419591.html
# reference: http://www.velocityreviews.com/forums/
#       t326690-urllib-urllib2-what-is-the-difference-.html

# This string holds the site domain that you're looking for.  You should specify
# the domain in your Google queries to get tight results.  This string is only
# used as a filter to make sure that you're getting links from your domain
mydomain = "someschool.edu"

# The sleepiness variable sets how long the program should wait after each Google
# query.  If this number is too low then Google may block your IP.  Generally the
# more queries you're going to run the higher this should be.  A higher number
# leads to slower performance though.
sleepiness = 3

# The first thing we want to do is open the file searchstrings and import all of
# the queries we want to run into a list.
searchstringfile = open('searchstrings','r')
searchstrings = searchstringfile.readlines()
searchstringfile.close()

# Here we set the browser agent string that we're going to send to Google.
# We can't use Python's default since Google doesn't allow that.
UserAgentString = 'Mozilla/5.0 '
UserAgentString += "(Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.5)"
UserAgentString += "Gecko/2008120121 Firefox/3.0.5"

# We are going to use urllib2 for this job.  Urllib2 has many (not all) of
# the same features as urllib, but it also allows us to spoof our agent string
# which is necessary to grab data from Google.
import urllib2

# Now we have to build a request object.  Urllib2 will allow us to just send
# a string to google which would be a very simple request.  Since we need to spoof
# the agent string, we need to build a more complex request object to pass to
# urllib2.  It is also important to note that the search string must be in the
# request object since Google wants GET requests.  If we were to use post then we
# would trim the search query off of the url and uncomment the req.add_data line.
# I also added some code so that I can have comments in the searchstrings file.
requests = []
for eachstring in searchstrings:
if eachstring.find('#') == 0:
  continue
if eachstring.find('http') == -1:
  continue
req = urllib2.Request(eachstring)
req.add_header('User-Agent',UserAgentString)
# req.add_data('q=lolcat')
requests.append(req)

# This code was stolen from one of the references above  It uses the htmllib and a
# null formatter to extract all of the <a> tags from the result and dumps
# them into a list.  When the HTMLParser encounters a tag it runs the tart_<tag>
# function.  In this class we overload the start_a function.  Check if there are
# more than zero arguments in the <a> tag, and then extract just the
# href argument.  Append that argument onto the classes list of links.
# FYI, the htmllib.HTMLParser that is passed into the first line of the class means
# that this class is inherited from htmllib.HTMLParser. That is good to know in case
# you're wondering where the code for htmlparser.feed() is.

import htmllib, formatter
class LinksExtractor(htmllib.HTMLParser):
def __init__(self, formatter):
  htmllib.HTMLParser.__init__(self, formatter)
  self.links = []
def start_a(self, attrs):
  if len(attrs) > 0:
    for attr in attrs:
      if attr[0] == "href":
        self.links.append(attr[1])
def get_links(self):
  return self.links

# Now we can create a null formatter and an instance of our class
format = formatter.NullFormatter()
htmlparser = LinksExtractor(format)

# Here we use urllib2 to send our request to google.  The results are stored in
# a file-like variable called data. I also have the script sleep for a few seconds
# after every request so that the google doesn't think it is under attack.
import time
for eachreq in requests:
data = urllib2.urlopen(eachreq)
htmlparser.feed(data.read())
time.sleep(sleepiness)

links = htmlparser.get_links()
msulinks = []

# Now the variable links contains a list of all of the links found on the page.
# Let's go through and remove any of the stuff we're not interested in.
for link in links:
if link.find(mydomain) == -1: #The link doesn't contain mnsu.edu
  continue
if link.find('http') == -1  and link.find('https') == -1:
  continue
if link.find('/search?') > -1:
  continue
if link.find('.google.com') > -1:
  continue
if link.find('www.youtube.com') > -1:
  continue
msulinks.append(link)

for link in msulinks:
print link