Testing Secure Computing's TrustedSource

The other day I was applying some patches to my Sidewinder firewall and noticed a new feature that had been added to my rules. I now have the ability to apply TrustedSource checks on my policies. This is awesome!

So what is TrustedSource? It's some technology that Secure Computing has that assigns a reputation score to IP addresses on the Internet. When it was first launched it was only used to block SMTP connections from servers that had a bad spam reputation, but now that has been extended so that I can use it to block just about any connection from computers on the Internet that are up to no good. I enabled this on a test rule to make sure that it wasn't going to break anything. Then I got to thinking: how could I possibly test this to see if it is working properly? It turns out that it is really hard to do that, and I was not successful, but here is the process that I tried.

First, I went to our external firewall, which is a Cisco PIX, and searched through the logs for IP addresses that were probably bad. I started by grepping my logs for the words "Deny tcp src outside:". That would give me a list of every entry in the logs where some computer on the outside of my network tried to make a connection that was denied. From there I needed to get a list of IP addresses. So I used the cut command to give myself just the ip address and port, and then I cut it a little more to get just the ip address. I ran that through sort and uniq and finally got a list of ip addresses, with the largest offender on top. Here is the full command:

grep "Deny tcp src outside:" firewall-log-file.txt | cut -d" " -f9 | cut -d: -f2 | cut -d/ -f1 | sort | uniq -c | sort -rn | more

Here are the top six results. The first number is the number of hits in the log file.
25109 87.106.60.73
25036 118.168.201.136
24099 202.155.254.172
13813 41.196.181.243
12324 124.155.149.21
10026 219.143.125.162

So now we should check and see what TrustedSource thinks of these IP addresses. I went to www.trustedsource.org and looked up each IP address.

87.106.60.73 Neutral
118.168.201.136 Neutral
202.155.254.172 Neutral
41.196.181.243 Neutral (suspicious)
124.155.149.21 Neutral (suspicious)
219.143.125.162 Neutral

So most of the addresses have a web reputation of Neutral, but two of them have a Mail reputation of suspicious. That might not indicate any lack of quality on the part of TrustedSource. It is possible that the machine that hit my Cisco PIX only recently started doing that. Also, TrustedSource seems to create a reputation for both an IP address AND a service. So in that case, a machine might be trusted for making a port 80 connection to your server, but is considered suspicious when it makes a port 22 connection. I think the reasoning is that most of these attacks come from home user computers that are infected with a bot. Instead of completely knocking them off the Internet, we're just going to ignore their bot traffic. After looking around on the web page for a bit, I was able to find an ip address that had a web reputation of suspicious: 67.205.89.49

Now for some fun, we want to spoof some ip traffic to the sidewinder and see what makes it through. First, a control. I tried to nmap port 80 on a server that I know has port 80 open and run a packet capture on the inside of the Sidewinder. Low and behold, my Sidewinder is so awesome that I couldn't get useful results from even this simple test. When I run an nmap against a known host on the machine, I get back results that port 80 is open, but since I'm using a proxy rule, the handshake is with the Sidewinder, not the actual server, so I get no results in my packet capture. Time to try something else. This time I opened up a command prompt and telnetted to the IP address of the server. So far so good, I'm connected. Then I typed "get index.html" and my connection was immediately dropped. Looking in the log files I see that the Sidewinder didn't appreciate me playing fast and loose with the HTTP protocol and that's why my connection was dropped.

Jul 28 23:29:01 2008 CDT f_http_proxy a_proxy t_attack p_major
pid: 5407 ruid: 0 euid: 0 pgid: 5407 logid: 0 cmd: 'httpp'
domain: htpp edomain: htpp hostname: kevins.firewall category: protocol_violation
event: malformed request netsessid: 488e9c8d000d9351 srcip: 111.111.111.111
srcport: 50396 srcburb: external dst_local_port: 80 protocol: 6
src_local_port: 0 dstip: 222.222.222.222 dstport: 80 dstburb: internal
attackip: 111.111.111.111 attackburb: external acl_id: blackfist_rule
reason: Poorly formed request: (null)

Once again, my bad packets were not even seen by the server. If I'm going to get some packets over to the server I'm going to have to do something more creative. So I decided to try using HPING2 to craft some packets and send them off to the Sidewinder. My first attempt failed, however, because of defense in depth. The VLANs on my network are configured to drop traffic if the source IP address is not valid for that VLAN. The workaround is to assign my network port to the same VLAN as the external interface of the firewall. Now when I send a forged packet it wont go through any Cisco ACLs and I should be able to hit the Sidewinder with it. Once again I set up my packet capture on the external interface of the Sidewinder and I used this command to send a SYN packet to the Sidewinder on port 80 from an untrusted IP address.

hping2 -a 67.205.89.49 -c 1 -S 134.29.74.10 -p 80

And it worked! I see the SYN packet hit the Sidewinder and the Sidewinder sends an ACK. But then something strange happens, my computer seems to send a TCP Reset to the Sidewinder. Here is what the tcpdump looks like:

03:11:55.371558 IP 67.205.89.49.2369 > some.server.edu.http: S 215914139:215914139(0) win 512
03:11:55.371600 IP some.server.edu.http > 67.205.89.49.2369: S 3835073201:3835073201(0) ack 215914140 win 32768
03:11:55.371902 IP 67.205.89.49.2369 > some.server.edu.http: R 1:1(0) ack 1 win 1460

After thinking about it for a minute, I figured out what was going wrong. When I send the SYN, the Sidewinder sends the ACK. However, the ACK goes to the real machine on the Internet, which was not expecting an ACK from my Sidewinder. As a result, it sends back a RST which is received by my Sidewinder. Well this is frustrating! But at least I know it's not my machine that is sending the rogue RST. Now I have to set up some rules to block any traffic going to or coming from 67.205.89.49 so that I can hammer away on my Sidewinder without the real server getting involved. This is where I left off for the day. I'll try again some other time. A big part of me is screaming "There must be an easier way to do this!" On the other hand, I suppose that we Security professionals have worked hard to make it so that it isn't easy to do this. Grrr.