Time-based alerts for snort

Snort is a very powerful intrusion detection system, and I've learned that when you combine it with a little kung fu you can do amazing things with it.  You can write incredibly granular alerts that will catch almost anything while minimizing false positives.  However, there is one thing missing in the alerting functionality and that is the ability to specify that some alerts only fire at certain time.

For example, let's say that you maintain a customer portal for your supply customers to order new shipments of some product.  You can expect that you will have customers hitting this page and trying to log during normal business hours so you probably wouldn't want to alert on that.  However, the same kind of traffic happening at 11:30 at night might seem suspicious.  At that time, events that might not be terribly interesting (like five failed logon attempts) suddenly become much hotter.

So how can we get time based alerts in snort?  Well there are two ways that I can think of and hopefully someone can think of a few more.  The first is the less-than-elegant solution.  You can write a script that adds a few rule lines to your local.rules file at certain times of the day and sends a kill -1 to your snort process so that it will reprocess it's rules files.  The advantage of using this process is that it is the simplest way of doing it. One way of doing it would be to have two different versions of your local.rules file. Then write a cron job that switches the two files and sends the kill -1 to the snort process. The disadvantage to this process is that it isn't very flexible. Let's say you want to ignore web activity between 8am and 5pm. Well that is pretty easy using this approach. But let's also say that on a different system you want to ignore activity from a block of IP addresses during a different time period. Now you would need to have three and possibly four versions of your local.rules file and your cron job would start to get pretty complicated. You would also need to make sure that you keep your script updated with the process ID that snort is running under. If you restart the service for some reason you're going to need to update your scripts. So I didn't like this approach.

Here is what I ended up doing to get these time-based alerts. I'll describe it in a nutshell, and then go into the detailed process. First, I set up a named pipe in the log directory that snort writes to. Then I created another type of rule called OddHours and told snort to write the alert for these rules into the named pipe I created. Then I wrote a python script that monitors the named pipe and takes an action whenever a new alert is found. It checks the day of the week and the time of day and then makes a decision to either send me an email or do nothing. Last step was to write the snort rules that I was looking for.

OK, so first is the named pipe. I have a lot of difficulty explaining what a named pipe is, but basically it is a way of sending the output from one program into another. For you command line linux types you probably do this all the time, like when you pipe the output of ps into grep to find your snort process ID.

ps aux | grep snort

That | symbol is doing the same thing as our named pipe, but it doesn't have a name and it isn't persistent on the file system the way our named pipe is. So let's create it:

mkfifo oddhours.csv

Now the custom alerts. In the snort.conf file that comes with snort, you can do a search for redalert and find an example of doing this. The custom alert is just a way of telling snort that for some alerts you want to react differently that you usually do. So I added this in underneath the redalert example:
ruletype oddhours

{
type alert
output alert_csv: oddhours.csv timestamp,sig_id,msg,proto,src,dst,dstport
}

So now if snort sees traffic that matches a rule that is written to use this custom ruletype it will send the time of thealert, the signature ID, the message, protocol, source and destination information into our named pipe.

This would be a bad time to restart snort if you're following along with me. The thing about named pipes is that you must have both ends of the pipe wired up for them to work. In the next step we'll create a script that "listens" to the pipe but if you don't have that script running when you start snort, then snort will sit there and wait for the script to start.

OK, let's dig into that python script. This is pretty easy. Just read a line of text from the named pipe and check what time it is. Right now the script is just writing to a log file, but I've put in the logic to send the email message that I ultimately want to receive.

import time
import smtplib
from email.mime.text import MIMEText
import logging

#######################
# Here is the logging setup stuff
#######################
LOG_FILENAME = '/home/kevin/oddoutput.txt'
logging.basicConfig(filename=LOG_FILENAME, level=logging.INFO)
#######################
# Here is the SMTP setup stuff
#######################
smtpServer = 'smtp.domain.edu'
smtpFrom = 'email.address@domain.edu'
smtpTo = 'email.address@domain.edu'

#######################
# Here i a function for sending alert emails
#######################
def SendAlertEmail(inMessage):
    logging.debug('\tSending Email alert')
    msg = 'Snort has detected activity at an unusual hour.\n'
    msg += inMessage
    msg = MIMEText(msg)
    msg['Subject'] = 'Activity at an unusual time'
    msg['From'] = smtpFrom
    msg['To'] = smtpTo

    s = smtplib.SMTP(smtpServer)
    s.sendmail(smtpFrom, smtpTo, msg.as_string())
    s.quit
    logging.debug('\tMessage sent.')


infile = open('/home/kevin/snort/logs/oddhours.csv','r')
while True:
    data = infile.readline()
    if data:
        logging.info('Line of data received.')
        logging.debug('\tChecking if it is Saturday.')
        if time.strftime( '%a' ) == 'Sat':
            logging.critical('\t\tIt\'s Saturday!  ALERT ALERT')
            logging.critical('\t\t'+data)
            #SendAlertEmail(data)
            continue
        else:
            logging.debug('\t\tNope. It\'s '+time.strftime('%a'))
        logging.debug('\tChecking if it is Sunday.')
        if time.strftime( '%a' ) == 'Sun':
            logging.critical('\t\tIt\'s Sunday!  ALERT ALERT')
            logging.critical('\t\t'+data)
            #SendAlertEmail(data)
            continue
        else:
            logging.debug('\t\tNope.It\'s '+time.strftime('%a'))
        logging.debug('\tChecking if it is early in the morning.')
        if int(time.strftime( '%H' )) <= 7:
            logging.critical('\t\tIt\'s Early.  ALERT ALERT')
            logging.critical('\t\t'+data)
            #SendAlertEmail(data)
            continue
        logging.debug('\tChecking if it is late in the day.')
        if int(time.strftime( '%H' )) >= 17:
            logging.critical('\t\tIt\'s Late.  ALERT ALERT')
            logging.critical('\t\t'+data)
            #SendAlertEmail(data)
            continue
        logging.debug('\tPassed all tests.')
        logging.debug('\t'+data)

OK. Last thing we need to do is write some rules that make use of our custom ruletype. In this example, I want to see web surfing that happens on Saturday or Sunday or outside of the normal work day. So my rule is going to look for anything on my home_net that is going to port 80 or 443 with the Syn flag set. Keep in mind that I wouldn't do this on a production system unless you want a lot of email or you're sure that this doesn't happen. I'm using this example because it fires a lot of alerts and lets you check right away if your rule is working.

oddhours tcp $HOME_NET any -> any 80 (msg:"Surfing at an odd hour."; flags:S; sid:2009122301; rev:1;)
oddhours tcp $HOME_NET any -> any 443 (msg:"Surfing at an odd hour."; flags:S; sid:2009122302; rev:1;)

OK. Now we're ready to fire the whole thing off. First, start up the python script and set it to run in the background: python oddhours.py &

Then fire up snort and start watching the output file. You can change the logging level so that you can get more or less information from the script if you want to do some troubleshooting.

Later on down the road, you can create functions for each sid that you want to monitor. When a line of data comes in you can pass control over to the function that you've written for that sid. That way you can have different time-based rules for each sid. It would also be pretty easy to expand the script so that you could send email alerts to different people based on the sid.

And hey, if you know a better way to do this then please hook me up with the answer. I'll gladly post it on my blog and give you all the credit.

Snort alert_unixsock

In my last post, I talked about a program and named pipe I put together so that I could get email alerts when snort detected certain events. I am a little surprised that this functionality isn't built into the software already and I'm a little surprised that there isn't a built in way to send alert output to a program. In other words, not having to have a process running all the time watching a named pipe, but that gets run by snort when certain alerts fire.

Anyway, I wanted to talk about a failure I had with snort because I hope that someone will read this and either learn from my troubleshooting process or provide me with some info on what I did wrong. I want to talk about sending alerts to a unix socket.

When you run snort, you specify a log directory that you want snort to send logs to using the -l option at the command line. When you turn on alert_unixsock snort will send alerts to a socket called snort_alert in your log directory. You can turn on sockets by adding the line "output alert_unixsock" to your snort.conf file. I used it within a custom ruletype.

But before you do that, you need to create the socket. Snort will not do it for you. So using my favorite programming language, I whipped up something to create the socket, listen for connections, and print anything that it receives. I borrowed heavily from the python.org web site for this code.

#! /usr/bin/env python
import os
import socket

s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)

try:
    os.remove("/home/kevin/snort/logs/snort_alert")
except OSError:
    pass

s.bind("/home/kevin/snort/logs/snort_alert")
s.listen(3)
while True:
    conn, addr = s.accept()
    datain = conn.recv(1024)
    print(datain)
conn.close()

So I fired up this script which creates the listener, and then I fired up snort. I wrote up a rule that would fire almost constantly and I waited for something to be written to my screen by the socket reader. But I got nothing. Flabbergasted, I did a "tail -f /home/kevin/snort/logs/alert" and watched my constantly running alert fire off constantly. So I know that the rule I wrote is good, and I know that snort is doing something.

So the next logical thing to check was the program itself. I couldn't find anything that would act as a socket sniffer, which is what I would have liked to have done. So instead, I wrote up this script that would connect to the socket and send some data over.

#! /usr/bin/env python

import socket

s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect('/home/kevin/snort/logs/snort_alert')

s.send('Test message.')
s.close()

With my listener running, I fired up my sending program a couple times and sure enouh, my listener wrote "Test message" to standard output every time. So I know that my listener is good. It seems like snort isn't sending data to the socket. I checked, and double-checked, and triple-checked the syntax of the command in snort.conf, but there really isn't anything to it. I even delved into the murky world of strace to see what was going on.

access("/home/kevin/snort/logs/snort_alert", W_OK) = 0

Unless I'm mistaken, the zero at the end means that snort was able to write to the socket, in other words there were no permission problems. So I'm really at a loss. Does anyone have any ideas as to what I might be missing?

Snort Email Alerts

I have a situation going on where I needed to watch for certain traffic matching a particular set of rules and of course the first thing that came to mind was to use snort. Snort allows me to write up very customizable rules so I knew I would have no problem detecting any of the problem traffic. In this particular case, I wanted to know if any SYN packets were being sent to a workstation. So the rule was pretty easy to write.

alert tcp any any -> $HOME_NET (msg:"Connection established to monitored machine"; flags:S; sid:20091029; rev:1;)

But I really didn't want to have to search through the logs looking for this kind of thing. After all, it might be months before I see the particular traffic that I'm looking for, so I decided that I wanted snort to email me if it say anything interesting. I don't, however, want for snort to email me about every alert that it sees because it tends to be a bit chatty at my site.

The way we get snort to treat alerts differently is to create a ruletype for the special rules. To do this, I went into my snort.conf file (usually located in /etc/snort/) and added the following lines:
ruletype emailalert

{
type alert
output alert_csv: csvlog default
}

Then I changed my rule above so that instead of being an alert, it was an emailalert:

emailalert tcp any any -> $HOME_NET (msg:"Connection established to monitored machine"; flags:S; sid:20091029; rev:1;)

So what we've done is told snort that if it sees any traffic matching my rule it should write something in csv form to a file called csvlog in whatever logging folder you're using. The reason I chose csv form is because everything gets written to a single line and I wont have to figure out down the road how to guess how many lines are in an alert message. Also, you can break the message down and only email the parts of the alert that you're interested in.

So what we could do is write a program that will read the file output and keep checking every few minutes to see if anything new has been added and then send an email alert. That would work, but you wouldn't really get real-time alerts. You would also have to put in code to find the end of the file and back off one line and keep track of what is new in the file. It all sounds messy. Instead, I used a named pipe. Snort is going to write to a file in its log directory called csvlog, so I change directory to /var/log/snort and used the following command to create a named pipe: mkfifo csvlog. Now all I have to do is write a program that will connect to the pipe and read a line from it. I don't have to deal with any of the other crap that I talked about above, and I can easily get real-time alerts. As usual, I went to my favorite programming language, python.

#! /usr/bin/env python

import smtplib
from email.mime.text import MIMEText

###################
# Here is the SMTP setup stuff
###################
smtpServer = "smtp.server.com"
smtpFrom = "nospam@blackfistsecurity.com"
smtpTo = "nospam@blackfistsecurity.com"

###################
# Here is a function for sending alert emails
###################
def SendAlertEmail(inMessage):
   msg = 'Snort has detected an emailalert\n'
   msg += inMessage
   msg = MIMEText(msg)
   msg['Subject'] = 'Emailalert from snort'
   msg['From'] = smtpFrom
   msg['To'] = smtpTo

   s =smtplib.SMTP(smtpServer)
   s.sendmail(smtpFrom,smtpTo,msg.as_string())
   s.quit


###################
# Main program loop.  Watch the named pipe and if
# anything shows up, email it using the function above
##################
infile = open('/var/log/snort/csvlog','r')
while True:
   data = infile.readline()
   if data:
       SendAlertEmail(data)

The easy way to test this is to telnet to a port that you know is open on one of the monitored machines. If an email message pops up in your inbox, then you know you're golden. But what if you're looking at a machine that normally doesn't have any open ports? What if you're specifically looking for an instance where a rogue listener opens up? The rule I wrote will detect that, but how do you test it?

Easy. Just change the rule we wrote so that the flags we're looking for are just SYN packets rather than SYN-ACK packets. (flags:SA; becomes flags:S;). Send a kill -1 to the process that is running snort so that you're rule set will reload. Then telnet to any port on the machine and you should see an email message get sent. Do not nmap the machine (like I did) or you will suffer a flood of email messages that will make you want to cry! Don't forget to set the rule back to what it was and send another kill -1 to the snort process either.

Thwarting phishing on the cheap

If you've been following my blog then you know that phishing has been a real problem at my organization. Users are falling for phishing messages no matter how poorly written they are and we've suffered way too many intrusions because of it. Even though I fear that it is largely impractical, I feel like I have to do something to stem the tide. Since this is unquestionably the largest source of intrusions into our network, even an inefficient method of catching these might be better than nothing.

So the first question is: how can I prevent the phishing messages from getting to my users? That would be the most effective way to combat this. But we've got two Barracuda Spam firewalls already working on this problem. It is unlikely that I'm going to do a better job of blocking the phish messages on the way in. So I decided to focus on messages on the way out.

Here is what I'm trying out right now. This is far from a perfect solution and it is also not the finished poduct. This is my first toe in the water to using Snort to try and detect the responses to phishing messages.

First, I made a new ruletype called phishinghole. This is in my /etc/snort/snort.conf file.

ruletype phishinghole
# The phishinghole rule type gathers up any alerts that could
# be responses to phishing messages and keeps the tcpdumps
# in one easy to read file
{
  type alert
  output alert_full: alert
  output alert_syslog: LOG_LOCAL4 LOG_ALERT
  output log_tcpdump: phishinghole.pcap
}

So when I write up a rule that uses this alert type, an event will be written to the normal alert file, it will also go to our syslog servers, and the packet itself will be written to a file called phishinghole.pcap. That way, when I want to inspect today's catch, I don't have to go through a huge packet capture file looking for just the packets that interest me.

I also want to be able to classify these events properly, so I added this one line to my classification.config file:

config classification: phishing-response,Possible Response to phishing message,5

Next up, I need to write some rules. I'm going to look for any traffic coming from my network and going to some other network on port 25. That's email. I'm going to use my new phishinghole ruletype, and I want to log the message that this might be the response to a phishing email:

phishinghole tcp $HOME_NET any -> !$HOME_NET 25 (msg:"Possible Phishing Response";

Next we need to define the content that we're going to look for. When I first started doing this, I came up with three regular expressions to look for possible permutations of password, username, and email. However, I decided that I didn't want to take the performance hit of running three regular expression searches against every single packet that leaves the organization. I decided instead to look for the word password and if that matches, then run the other two regular expressions. That should trim down the number of packets I have to look at.

content:"password"; nocase;

Now the magic is in the regular expressions. This is the Perl Compatible Regular Expression I'm using to search for username. This will match regardless of case and whether the word is broken in two with a space or a dash "user name" or "user-name" or "User Name" etc. The second regular expression looks for the word email in a similar fashion.

pcre:"/user[\-|\s]?name/i"; pcre:"/e?[\-|\s]?mail/i";

I still need to review these messages manually to see if someone really did respond to a phishing message or just told the guy to go to hell. So I want to gather a few extra packets to get as much context as reasonably possible. This next part of the rule tells snort to record three additional packets in the conversation.

tag:session,3,packets;

I also want to make sure my rules are classified properly. I want these events to bubble up to the top of my priority stack, so I put this into the rule:

classtype:phishing-response;

and I ended the rule with a sid (todays date and a revision number)

sid:20090525; rev:1;)

I repeated the process for three more reasonable permutations of the word password. I figure that a phisher cant get too crazy with the spelling or capitalization or the message will lose credibility. Here are the finished rules that I came up with.

blockquote>
phishinghole tcp $HOME_NET any -> !$HOME_NET 25 (msg:"Possible phishing response"; content:"password"; pcre:"/user[\-|\s]?name/i"; pcre:"/e?[\-|\s]?mail/i"; nocase; tag:session,3,packets; classtype:phishing-response; sid:20090525; rev:1;)

phishinghole tcp $HOME_NET any -> !$HOME_NET 25 (msg:"Possible phishing response"; content:"pass word"; pcre:"/user[\-|\s]?name/i"; pcre:"/e?[\-|\s]?mail/i"; nocase; tag:session,3,packets; classtype:phishing-response; sid:20090525; rev:2;)

phishinghole tcp $HOME_NET any -> !$HOME_NET 25 (msg:"Possible phishing response"; content:"pass-word"; pcre:"/user[\-|\s]?name/i"; pcre:"/e?[\-|\s]?mail/i"; nocase; tag:session,3,packets; classtype:phishing-response; sid:20090525; rev:3;)

Now for the analysis
Since these are .pcap files, it's tempting to open up wireshark and get started at peeking through them. That's what I did and it works just fine.  However, after a couple of days I realized that I was getting way too many false positives, and I needed a new way to separate the wheat from the chaff.  Since the only thing in the .pcaps are email snippets, that means that all of the data I need to sift through is going to be in plain text.  So I ran the strings command against the pcaps just to make sure that I would get a dump of all text in the file.  Then I talked to our system administrators to find out if I could get a plain text feed of active user accounts on the domain.  Now I can do something like this to find out if I need to look at the file in more detail:

strings phishinghole.pcap | grep -f listofusernames

If I get any hits then I know I have an email that has the word username, password, email and a valid user name on our network in it. Even if that is a false positive, that is worth investigating. It's also pretty easy to find the username in the .pcap file when you know what username you're looking for so then you can see the message in context.

I'm going to let this run for a few more days and see if I'm satisfied with it. If I think I've got a winner here, I'll automate the process further.