Snort Email Alerts

I have a situation going on where I needed to watch for certain traffic matching a particular set of rules and of course the first thing that came to mind was to use snort. Snort allows me to write up very customizable rules so I knew I would have no problem detecting any of the problem traffic. In this particular case, I wanted to know if any SYN packets were being sent to a workstation. So the rule was pretty easy to write.

alert tcp any any -> $HOME_NET (msg:"Connection established to monitored machine"; flags:S; sid:20091029; rev:1;)

But I really didn't want to have to search through the logs looking for this kind of thing. After all, it might be months before I see the particular traffic that I'm looking for, so I decided that I wanted snort to email me if it say anything interesting. I don't, however, want for snort to email me about every alert that it sees because it tends to be a bit chatty at my site.

The way we get snort to treat alerts differently is to create a ruletype for the special rules. To do this, I went into my snort.conf file (usually located in /etc/snort/) and added the following lines:
ruletype emailalert

{
type alert
output alert_csv: csvlog default
}

Then I changed my rule above so that instead of being an alert, it was an emailalert:

emailalert tcp any any -> $HOME_NET (msg:"Connection established to monitored machine"; flags:S; sid:20091029; rev:1;)

So what we've done is told snort that if it sees any traffic matching my rule it should write something in csv form to a file called csvlog in whatever logging folder you're using. The reason I chose csv form is because everything gets written to a single line and I wont have to figure out down the road how to guess how many lines are in an alert message. Also, you can break the message down and only email the parts of the alert that you're interested in.

So what we could do is write a program that will read the file output and keep checking every few minutes to see if anything new has been added and then send an email alert. That would work, but you wouldn't really get real-time alerts. You would also have to put in code to find the end of the file and back off one line and keep track of what is new in the file. It all sounds messy. Instead, I used a named pipe. Snort is going to write to a file in its log directory called csvlog, so I change directory to /var/log/snort and used the following command to create a named pipe: mkfifo csvlog. Now all I have to do is write a program that will connect to the pipe and read a line from it. I don't have to deal with any of the other crap that I talked about above, and I can easily get real-time alerts. As usual, I went to my favorite programming language, python.

#! /usr/bin/env python

import smtplib
from email.mime.text import MIMEText

###################
# Here is the SMTP setup stuff
###################
smtpServer = "smtp.server.com"
smtpFrom = "nospam@blackfistsecurity.com"
smtpTo = "nospam@blackfistsecurity.com"

###################
# Here is a function for sending alert emails
###################
def SendAlertEmail(inMessage):
   msg = 'Snort has detected an emailalert\n'
   msg += inMessage
   msg = MIMEText(msg)
   msg['Subject'] = 'Emailalert from snort'
   msg['From'] = smtpFrom
   msg['To'] = smtpTo

   s =smtplib.SMTP(smtpServer)
   s.sendmail(smtpFrom,smtpTo,msg.as_string())
   s.quit


###################
# Main program loop.  Watch the named pipe and if
# anything shows up, email it using the function above
##################
infile = open('/var/log/snort/csvlog','r')
while True:
   data = infile.readline()
   if data:
       SendAlertEmail(data)

The easy way to test this is to telnet to a port that you know is open on one of the monitored machines. If an email message pops up in your inbox, then you know you're golden. But what if you're looking at a machine that normally doesn't have any open ports? What if you're specifically looking for an instance where a rogue listener opens up? The rule I wrote will detect that, but how do you test it?

Easy. Just change the rule we wrote so that the flags we're looking for are just SYN packets rather than SYN-ACK packets. (flags:SA; becomes flags:S;). Send a kill -1 to the process that is running snort so that you're rule set will reload. Then telnet to any port on the machine and you should see an email message get sent. Do not nmap the machine (like I did) or you will suffer a flood of email messages that will make you want to cry! Don't forget to set the rule back to what it was and send another kill -1 to the snort process either.

Improving your Google hacking with Python

UPDATE: I made some major improvements to this code which also made it quite lengthy. You can find the full script here. The linked version adds support for a whitelist file so you don't get bothered by false positives. The improved script will also attempt to download the links that it gets from Google and make sure that it doesn't report any dead links to you.

One of the major information security problems that I have on my campus is that we have a tendency to improperly release data that is supposed to be secret. The primary vehicle for doing that is faculty members posting grades on the Internet. Every semester we have a couple hundred student records that are posted on the Internet, which has created problems for me semester after semester.

Now before I go on, I should point out that there is nothing wrong with posting student grades on the Internet, as long as the data has been sufficiently anonymized. If a professor was to assign random numbers to each of his students then the grades could be posted online using the random numbers. However, most of the time (in violation of campus policies and Department of Education regulations) the grades are posted by Student ID. That's a no no.

But each of our faculty members have their own personal web space where they can put stuff, and I don't have the ability to go through each of them every day and find where grades have been improperly posted, so I use my best friend Google to do it. I might do a search against google for any Excel spreadsheet that contains techids and grades that are on my domain:
"techid grade ext:xls site:mnsu.edu"

That's been my primary way of finding the leaks. But I wanted something better. I want to be able to run a script every day and just get the list of links that I need to check out. So I went to my favorite interpreted programming language, Python.

Below I've pasted a little script that I've put together to do the searching for me. I put my search strings into a file called "searchstrings" and this program runs them. It then pulls out all of the liks, and then removes the links that are not on my domain. Feel free to use this yourself if you want. I have a few changes that I would like to make. I plan to add in support for whitelisting links, and specifying which searchstring file you want to process. But this should show you the basic process that I'm using. I also stole some of this code from several places around the Internet, so please check out the references that I've put at the top.

#!/usr/bin/env python

# This script will connect to google, pull down some search results,
# remove the bullshit and only show me what I want to see.

# reference: http://docs.python.org/library/urllib.html
# reference: http://cis.poly.edu/cs912/parsing.txt
# reference: http://mail.python.org/pipermail/python-list/
#       2006-December/419591.html
# reference: http://www.velocityreviews.com/forums/
#       t326690-urllib-urllib2-what-is-the-difference-.html

# This string holds the site domain that you're looking for.  You should specify
# the domain in your Google queries to get tight results.  This string is only
# used as a filter to make sure that you're getting links from your domain
mydomain = "someschool.edu"

# The sleepiness variable sets how long the program should wait after each Google
# query.  If this number is too low then Google may block your IP.  Generally the
# more queries you're going to run the higher this should be.  A higher number
# leads to slower performance though.
sleepiness = 3

# The first thing we want to do is open the file searchstrings and import all of
# the queries we want to run into a list.
searchstringfile = open('searchstrings','r')
searchstrings = searchstringfile.readlines()
searchstringfile.close()

# Here we set the browser agent string that we're going to send to Google.
# We can't use Python's default since Google doesn't allow that.
UserAgentString = 'Mozilla/5.0 '
UserAgentString += "(Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.5)"
UserAgentString += "Gecko/2008120121 Firefox/3.0.5"

# We are going to use urllib2 for this job.  Urllib2 has many (not all) of
# the same features as urllib, but it also allows us to spoof our agent string
# which is necessary to grab data from Google.
import urllib2

# Now we have to build a request object.  Urllib2 will allow us to just send
# a string to google which would be a very simple request.  Since we need to spoof
# the agent string, we need to build a more complex request object to pass to
# urllib2.  It is also important to note that the search string must be in the
# request object since Google wants GET requests.  If we were to use post then we
# would trim the search query off of the url and uncomment the req.add_data line.
# I also added some code so that I can have comments in the searchstrings file.
requests = []
for eachstring in searchstrings:
if eachstring.find('#') == 0:
  continue
if eachstring.find('http') == -1:
  continue
req = urllib2.Request(eachstring)
req.add_header('User-Agent',UserAgentString)
# req.add_data('q=lolcat')
requests.append(req)

# This code was stolen from one of the references above  It uses the htmllib and a
# null formatter to extract all of the <a> tags from the result and dumps
# them into a list.  When the HTMLParser encounters a tag it runs the tart_<tag>
# function.  In this class we overload the start_a function.  Check if there are
# more than zero arguments in the <a> tag, and then extract just the
# href argument.  Append that argument onto the classes list of links.
# FYI, the htmllib.HTMLParser that is passed into the first line of the class means
# that this class is inherited from htmllib.HTMLParser. That is good to know in case
# you're wondering where the code for htmlparser.feed() is.

import htmllib, formatter
class LinksExtractor(htmllib.HTMLParser):
def __init__(self, formatter):
  htmllib.HTMLParser.__init__(self, formatter)
  self.links = []
def start_a(self, attrs):
  if len(attrs) > 0:
    for attr in attrs:
      if attr[0] == "href":
        self.links.append(attr[1])
def get_links(self):
  return self.links

# Now we can create a null formatter and an instance of our class
format = formatter.NullFormatter()
htmlparser = LinksExtractor(format)

# Here we use urllib2 to send our request to google.  The results are stored in
# a file-like variable called data. I also have the script sleep for a few seconds
# after every request so that the google doesn't think it is under attack.
import time
for eachreq in requests:
data = urllib2.urlopen(eachreq)
htmlparser.feed(data.read())
time.sleep(sleepiness)

links = htmlparser.get_links()
msulinks = []

# Now the variable links contains a list of all of the links found on the page.
# Let's go through and remove any of the stuff we're not interested in.
for link in links:
if link.find(mydomain) == -1: #The link doesn't contain mnsu.edu
  continue
if link.find('http') == -1  and link.find('https') == -1:
  continue
if link.find('/search?') > -1:
  continue
if link.find('.google.com') > -1:
  continue
if link.find('www.youtube.com') > -1:
  continue
msulinks.append(link)

for link in msulinks:
print link