Open Source Application Layer Firewall part 5

Continuing with our discussion of setting up a home-grown application layer firewall. So far we have set up an insecure application that needs protection. Then we configured our server to act as a router. Then we set up stateful packet inspection using PF, and turned it into a proxy firewall using Apache. Now I'd like to talk about how you can really look deep into the application traffic to stop unauthorized activity at a much higher level than what is possible with a stateful inspection firewall.

Now that we have a proxy firewall, we want to set up Apache mod_security to look deep into our http requests and identify malicious traffic. We're going to use packages to install mod_security, so the first thing we should do is set up our package path. I'm going to use the main OpenBSD distribution site for this, but you should probably choose a mirror that is closer to you. First we want to set up the environment variable:

# export PKG_PATH=ftp://ftp.openbsd.org/pub/OpenBSD/4.4/packages/i386

You can also add this to your .profile file so that you will have that set up every time the machine boots.

PKG_PATH=ftp://ftp.openbsd.org/pub/OpenBSD/4.4/packages/i386
export PKG_PATH

Now we can search the package list for mod_security.

# pkg_info -Q modsecurity
modsecurity-apache-1.9.3p2 

And then install it:

# pkg_add modsecurity-apache-1.9.3p2
modsecurity-apache-1.9.3p2: complete                                   
--- modsecurity-apache-1.9.3p2 -------------------
To finish the install of modsecurity-apache-1.9.3p2, you need
to enable the module using the following command

/usr/local/sbin/mod_security-enable

The manual is found at /usr/local/share/doc/mod_security.

If you already have Apache running on your machine,
you should not use "apachectl restart" - instead,
you should fully stop and then start the server.
#
# /usr/local/sbin/mod_security-enable                                    
Enabling  module...
[activating module `security' in /var/www/conf/httpd.conf]
cp /usr/local/lib/mod_security.so /usr/lib/apache/modules/mod_security.so
chmod 755 /usr/lib/apache/modules/mod_security.so
cp /var/www/conf/httpd.conf /var/www/conf/httpd.conf.bak
cp /var/www/conf/httpd.conf.new /var/www/conf/httpd.conf
rm /var/www/conf/httpd.conf.new

Let's follow the directions and shut down the httpd server:

# apachectl stop
/usr/sbin/apachectl stop: httpd stopped

Now take a moment to examine our httpd.conf file. You will notice that one line has been added to the configuration by the mod_security-enable script

# diff httpd.conf httpd.conf.bak
274d273
LoadModule security_module    /usr/lib/apache/modules/mod_security.so # 

Now that we have mod_security set up in Apache, let's do something with it. I'm going to put together a basic set of options, and put in one single filter that will block any requests going to my server that have the word "kevin" in them. Open up /var/www/conf/httpd.conf and enter this into one of the VirtualHosts that we created earlier.

 SecFilterEngine on
   SecFilterCheckURLEncoding On
   SecFilterScanPOST On
   SecAuditEngine On
   SecAuditLog /var/www/logs/audit_security.log
   SecFilterDefaultAction "deny,log,status:500"
   SecFilter "kevin" 

The indenting is not necessary, but it makes it look good and helps to identify all of the mod_security related stuff in this virtual host. You can also put configuration directives like this in the global configuration rather than within a single VirtualHost. However, if you're going to have multiple servers behind this firewall, you may find that you need different rules for each one, so I'm going to do it this way. Restart httpd to make the changes take effect.

Now let's test it. I'm going to go to my web application and enter the following:
Firstname: Shamus
Lastname: McFinnigan
Card: some number
I hit submit and the application works as expected. Then I tried the same thing but with the first name kevin instead. Mod_security blocked the request, and gave me an http 500 error. Mod_security is working. Congratulations, you now have a firewall that is doing layer 7 inspection of your traffic before it hits the server. You have an application layer firewall.

The rule set that we've built in here doesn't really do much for us, unless we want to discriminate against the Kevin's of the world. But what if we were using this to protect some appliance and we didn't know whether or not the coders had put proper input validation into their form fields? Well one thing we could do is use our proxy server to look at the requests that go by and find out what variable names are in use, and then write filter rules. For example, I know that my crappy application should only accept First and Last names that are alphabetical characters and no more than 12 characters in length. I also know that credit card numbers should only contain digits and dashes and that there should be four digits followed by a dash. So I can take out my SecFilter Kevin line and replace it with this:

SecFilterSelective "ARG_Fname" !^([A-Z]|[a-z]){1,12}$
SecFilterSelective "ARG_Lname" !^([A-Z]|[a-z]){1,12}$
SecFilterSelective "ARG_Ccard" !^([0-9]{4}-){3}[0-9]{4}$

So from the first name variable we're going to filter anything that does not match our regular expression (!). The regular expression indicates that we want to start with the first character of the variable (^) and that it should be an uppercase or lowercase letter ([A-Z]|[a-z]). That should occur at least one time but no more than 12 times {1,12} and then there should be nothing else $.

So now, regardless of how crappy the application behind the firewall is, it is protected pretty well. Sure it's not going to be bullet proof, but adding in this kind of input validation will add a lot of strength to your application security. Try doing this with a Cisco ASA box.

Check out the rules at http://www.gotroot.com/downloads/ftp/mod_security/rules.conf for more ways to strengthen your box. I suggest adding these in one at a time and making sure that your apache process doesn't crap rather than putting in the whole wad. Also, ff making up your own rules seems difficult, try out this online rule creator! http://jcksn.com/tools/modsecurity/

What about the core rule set?
Yeah, that would be pretty cool, wouldn't it? Well it's not going to happen anytime soon, at least not the way I've set things up here. One of the things I've tried to do in this setup is leverage the work of other people that are smarter than me to make a good firewall. One of the big points is using the chrooted Apache software that comes with OpenBSD. In order to use th mod_security core rule set, you have to be running Apache2. The core rule set depends on a newer version of mod_security, and that version depends on Apache2. Installing Apache2 on OpenBSD is pretty easy, but it isn't chrooted which means you have to do all that work yourself. I'd like to start working on another version of this that uses a chrooted Apache2, but first I need to figure out how to do that and keep it secure.

Open Source Application Layer Firewall part 4

Picking up where we left off on this whole firewall thing, let's get into the Apache config. In part 1 we talked about why we need an application layer firewall and how it is more secure than a stateful inspection firewall. In part 2 we set up our server that needs protecting, and in part 3 we created a basic stateful inspection firewall to protect our application. The stateful firewall isn't allowing any traffic into our server though. Now we're going to open a hole to the server that needs protecting and use the proxy services built into OpenBSD to create a proxy firewall.

One of the most beautiful things about using OpenBSD for our firewall OS is that it comes already setup with Apache in a chroot jail. That means that if an attacker were somehow able to compromise the httpd process that person will not own the whole firewall. It can be very difficult to set up Apache in a chroot jail, and it can be difficult to work with Apache once it is in the chroot jail. The good news is that OpenBSD has done a lot of the work for us.

The first thing we need to do is set up our SSL keys for this server. I'm going to follow the same instructions I did in part 2 to create a self signed ssl certificate, and I'm going to name the certificate after the public IP address that I want clients to connect to.

Now we have to edit the configuration file for our chrooted apache installation, which is in /var/www/conf/httpd.conf. The file is broken into several sections:
Section 1 - Global environment.
Uncomment line 273: LoadModule proxy_module /usr/lib/apache/modules/libproxy.so
Section 2 - Main Server Configuration
Comment out line 310: #Port 80.

The first change was necessary so that our proxy server stuff would work. I commented out line 310 because we're going to use listen statements later in the configuration file to control which port the servers are running on. Most of the work is going to happen in section 3, Virtual Hosts.
Section 3 - Virtual Hosts
At the start of this section there are some comments. Immediatly after those comments I put in the two virtual hosts that I want to set up on this box. One will accept unencrypted communication on port 80 and proxy them to our server. The configuration for that looks like this:

<virtualhost 111.11.11.111:80>
   ProxyRequests off
   ProxyPass / http://192.168.1.10/
   proxyPassReverse / http://192.168.1.10/
</virtualhost>

The second virtual host listens on port 443 and proxies traffic to the same server on an encrypted channel

<virtualhost 111.11.11.111:443>
   SSLEngine on
         SSLCertificateFile /etc/ssl/111.11.11.111.crt
         SSLCertificateKeyFile /etc/ssl/private/111.11.11.111.key
         ProxyRequests off
         ProxyPass / https://192.168.1.10/
         ProxyPassReverse / https://192.168.1.10/
</virtualhost>

After I put those lines in, I deleted everything in the section on SSL Virtual Host Context (which started on line 1013), but kept the SSL Global Context Section (which started on line 954).

The SSL Global Context, and the stuff in Sections 1 and 2 give us pretty safe configuration templates for our server to run with. Then we removed the default virtual hosts that come in the file (the ones that listen on all ip addresses) and replaced them with two specific virtual hosts. Each one listens on a specific port on a specific IP address. That way if you need to have services for more than one back-end server, you can easily cut and paste the configuration that you have for this one. Now let's test and see if our configuration file passes the smell test.
# apachectl configtest.

If we don't get any errors, then we can start up our httpd server with SSL support. It's pretty easy to do.
# apachectl startssl (or you can use # httpd -DSSL)

Now if that started properly, we should set up our server so that it will start up every time it boots up. Go ahead and edit /etc/rc.conf and change the line that read httpd_flags=NO to read httpd_flags="-DSSL"

It isn't quite working yet, though. Sure we have a listening process on port 80 and 443 that will forward traffic to our back end server, but we're getting blocked at layers 3 and 4. We need to open up this traffic in pf. So add this to the bottom of your /etc/pf.conf file

# This rule allows http(s) in to our server
pass in log on $ext_if proto tcp from any to 111.11.11.111 port 80
pass in log on $ext_if proto tcp from any to 111.11.11.111 port 443 

Then reset your pf rules with this:
# pfctl -F rules -f /etc/pf.conf

Now direct your browser to the external address that you're listening on and see if you get the web page being served up by your back end server. If so, then congratulations, you now have a Proxy Firewall.

In the next entry we'll talk about mod_security and how you can turn your proxy firewall into an honest to goodness Application Layer Firewall.

Open Source Application Layer Firewall part 3

Now that the excitement of Secure360 has died down and we've shrugged off the effects of a phishing attack I finally have some time to work on the last few investigations that popped up recently and add another blog posting on my homebrew, open source application layer firewall.

In part 1 we talked about why an Application Layer firewall was necessary and why it was more secure than a simple Stateful Inspection Firewall. In part 2, we started setting up our lab environment by creating a web server with a simple (and insecure) php application that will need protecting. Here in part 3, we will begin creating our Application Layer firewall.

I'm not going to spend any time talking about the hardware that you want to use for something like this. It is largely dependent on your environment and what you're trying to do. It also isn't something that I am an expert on. There is a great write up here on selecting hardware for high network performance. I suggest you read that before you put something into production. What I will tell you is that for my example, I will be using a virtual machine with two NICs. One of them will be connected to a public network that people can access, and the other is connected to a private network and can only talk to the web server we created in part 2.

So the first step is to install OpenBSD on our firewall. Could I use linux? Of course I could, but OpenBSD has an excellent reputation for being proactively secure and auditing the crap out of their code. OpenBSD also comes with Apache already set up in a very secure chroot environment that we'll be taking advantage of. Even if you don't agree that OpenBSD is more secure than your linux flavor of choice, you will probably agree that running Apache in a chroot environment is more secure than not. And you will probably agree that setting up Aapche in a chroot environment is not trivial, so OpenBSD is the easiest choice as well. However, if I should find myself overflowing with some of that "time" stuff that I hear other people have, I might start documenting how to do this in linux as well.

I don't want to spend a great deal of time talking about the initial installation of the Operating System either. It is pretty easy to set up if you just follow the defaults. I use the whole disk for my operating system, and in this case I put everything in one partition. If I were doing this in production I might decide that I wanted /var to be in a separate partition so that I wouldn't have to worry about my machine crashing if log files fill up the whole hard drive. We wont have any users storing their crap on this server, so we don't really need to have /home or /usr on their own partitions. Also remember that in most cases you want to use the generic kernel, not the multiprocessor kernel.

Once the operating system is installed, follow these directions to get connected to an anonymous CVS server, and download the patch branch (# cd /usr; cvs checkout -P -rOPENBSD_4_4 src). Then follow these directions to compile and update your server. You are now patched and following stable.

Alright. You've got a patched server with two NICs, one of which is on your public network so clients can reach it, and the other is on the private network with our application server. Now we can start tuning this as a firewall.

For the rest of this HOWTO I'm going to try to work my way up the ISO model from layer to layer. We've just taken care of layers 1 and 2 by setting up our machines. Next we need to allow layer 3 routing. This is only marginally necessary. Since we are going to have proxy servers acting as go betweens, the operating system doesn't really need to route traffic. However, you never know when there is going to be some silly application that we wont be able to proxy and you'll have to route. So it's best to just do this now and never worry about it again.
Turn on IP forwarding:
# sysctl net.inet.ip.forwarding=1

Next, edit /etc/sysctl.conf and set net.inet.ip.forwarding=1 so that it will be set for you every time this machine boots. Congratulations, you now have a router!

Next up, let's work our way through some layer 3 and layer 4 technology, the pf firewall. Pf is a packet filter firewall that supports stateful packet inspection, NAT, and a whole host of other good stuff. It has built in support for anti-spoofing and scrubbing packets as they come in. It is every bit as capable as any other stateful inspection firewall. We're going to use PF to block most of the crap that will hit this firewall. I'm going to start with a very restrictive firewall rule set that only allows ping and ssh into the firewall. Edit /etc/pf.conf to read like this. Substitute your external interface for vic0.

ext_if="vic0"

# It is a good idea to not process stuff on loopback
set skip on lo

# Let's scrub our incoming traffic
scrub in

# The default deny rule
block in log on $ext_if
block out log on $ext_if

# This rule allows icmp echo in
pass in inet proto icmp all icmp-type echoreq

# This rule allows ssh in
pass in log on $ext_if proto tcp from any to 134.29.32.68 port 22

The rule set scrubs incoming traffic and blocks everything going in and out. I was once chastised on the OpenBSD mailing list for forgetting that pf processes rules from the top down and runs the last rule to match the traffic. That's why our default deny needs to come first. The other two rules, the ones that allow ssh and icmp have an implicit "keep state" on the end of them. That's why we don't need to create a rule that allows the return traffic out. Now we can turn on pf by typing
# pfctl -e
You will also want pf to turn on when the firewall is booted so make sure you edit /etc/rc.conf and set pf=YES and pf_rules=/etc/pf.conf.

Now use pfctl to check the rules that are running and make sure everything looks right.

# pfctl -s rules
scrub in all fragment reassemble
block drop in log on vic0 all
block drop out log on vic0 all
pass in inet proto icmp all icmp-type echoreq keep state
pass in log on vic0 inet proto tcp from any to 134.29.32.68 port = ssh flags S/SA keep state

Congratulations, you now have a stateful inspection firewall that doesn't allow anything to pass through to the private interface, but will let you ping and ssh into it.

If you're ready, you can move on to Part 4 now.

Open Source Application Layer Firewall part 2

Continuing my discussion on setting up an open-source application-layer firewall. In the last post we talked about the need for such a device and kind of layed the groundwork for what was going to come. In this post I'm going to talk about setting up our vulnerable application that requires protecting.

The first thing I set up was my back-end server. This is the machine that I am trying to protect. I didn't need to do anything fancy with this machine, so I just slapped a default installation of OpenBSD on a virtual machine. The virtual machine has one virtual NIC on it, and that was initially set up with a live IP address so that I could download the sets from the Internet. Once the installation was complete, I rebooted and changed the ip address to something static. Here is how that is done.
1. Edit hostname.em0. On OpenBSD, the interfaces are not named eth0, eth1, etc. Instead the name comes from the manufacturer or the NIC. You can run ifconfig to see which interface has an ip address. In my case, that interface is em0, so that cards configuration is in hostname.em0. The file was set up to allow DHCP, but I don't want that. So I'll set it up with a static ip address on a private network.

#vi /etc/hostname.em0
inet 192.168.1.10 255.255.255.0

I also need to set a default gateway which is in the file /etc/mygate

#vi /etc/mygate
192.168.1.1

Now I shut down the virtual machine and changed the virtual NIC to use a host-only network. When I booted up, I no longer had access to the Internet from this machine. Next up, we need to create an SSL certificate to be used when making https connections to this machine.

openssl genrsa -out /etc/ssl/private/192.168.1.10.key 1024
openssl req -new -key /etc/ssl/private/192.168.1.10.key /
-out /etc/ssl/private/192.168.1.10.csr
openssl x509 -req -days 365 -in /etc/ssl/private/192.168.1.10.csr /
-signkey /etc/ssl/private/192.168.1.10.key /
-out /etc/ssl/192.168.1.10.crt

We need to make a quick change to the configuration file too. The file /var/www/conf/httpd.conf instructs httpd to look for certificates at /etc/ssl/server.crt. Let's change line 1041 of the default httpd.conf file to point to the correct file name: /etc/ssl/192.168.1.10.crt. You will also have to make a similar change to line 1046. Now we can fire up our http server and test it out. This is really easy to do since we're using the httpd server that comes with OpenBSD. You just have to type httpd. I'm going to use the -DSSL option so that it will fire up with SSL support using the keys we just made.
httpd -DSSL

We can use lynx to test it. Type in lynx http://localhost and you should see a text version of the default Apache startup page. Press Q to quit and type lynx https://localhost to see if you've got working SSL. You should ignore the ssl warning because we're using a self-signed certificate. Note that you wont be able to pull anything up on a browser from another machine because we haven't opened up the http/https ports in pf. If lynx indicates that the https server is working right, then we can make it start up by default whenever we fire up this machine.

#vi /etc/rc.conf
# Find the httpd_flags line and change it to this:
httpd_flags="DSSL"

That is all the configuration that we need to do right now to test this thing out. However, if you're feeling a little saucy like I was, we can whip up a quick and simple php application so that we can test out the features of mod_security down the road a bit. Here is how I did that. I added another virtual NIC and configured it for dhcp so that I could get to the Internet.

Install php from packages and update httpd.conf to support php

# export PKG_PATH=ftp://ftp.openbsd.org/pub/OpenBSD/4.4/packages/i386/
# pkg_info -Q php more (look for one called php5-core-something)
# pkg_add php5-core-5.2.6

Edit the configuration file /var/www/conf/httpd.conf. Find the line that reads AddType application/x-httpd-php and change it to read: AddType application/x-httpd-php .php .php4 .php3 .htm .html

Now I edited the file /var/www/htdocs/index.html and added this to the file after the BODY tag.

<?php
$Fname = $_POST["Fname"];
$Lname = $_POST["Lname"];
$Ccard = $_POST["Ccard"];
echo "Hello, ".$Fname." ".$Lname;
echo "<br />";
echo "Credit card: ".$Ccard;
?>

<form method="post" action="<?php echo $PHP_SELF; ?>">
First Name<input type="text" size="12" maxlength="12" name="Fname"><br />
Last Name<input type="text" size="12" maxlength="36" name="Lname"><br />
Credit Card<input type="text" size="12" maxlength="36" name="Ccard"><br />
<input type="submit" value="submit" name="submit"><br /></form><br />

Now I have a basic and very insecure web application that runs on my web server. This is what we will use for testing out our Application Layer Firewall. Once we have everything set up, we should be able to establish an SSL connection to this server that is proxied through our firewall. Our firewall should also be able to make sure that we don't enter improper text into our form. I will now remove the virtual NIC that connects this to the Internet so that there is no access to this server except through our firewall.

Here is part 3.

Open Source Application Layer Firewall

This is going to be a really long posting if I go into the level of detail that I want. So I'm going to have to break this up over several postings. This will be the introduction and table of contents posting. As I add more postings on this topic, I'll update this posting accordingly. Once I have enough content, I'll probably add it to the Popular Topics link list on the right.

A while back we purchased a Sidewinder firewall as part of our strategy to achieve PCI compliance. The plan that we're enacting is to isolate our entire cardholder environment behind the Sidewinder. In most cases, a simple and cheap stateful inspection firewall would be sufficient for that, but we've also got some web applications that process credit card data and we need to comply with section 6.6 of the DSS as well. That calls for an Application Layer Firewall (or a source code audit, but we decided on Application Layer Firewall).

Now don't get me wrong, I love my Sidewinder. I have referred to it as the coolest device ever attached to the Internet, although it has lost at least 5% of its coolness since they changed the name to something dumb that I wont repeat. But there are some things about the Sidewinder that make me sad. For example, it's not open source so I don't really know how it works. As a security guy that makes me uncomfortable. I also don't like that the configuration files aren't plain text. I can't put them into a revision control system and quickly see what changes were made without my knowledge by some other firewall administrator. And of course there is the fact that Sidewinders are expensive. So I thought I would take some time and see if I couldn't build an open source alternative to the Sidewinder for people who aren't fortunate enough to have one. This isn't going to be as full featured as a Sidewinder, but it will be more secure than a simple Stateful Inspection Firewall and it will help meet the requirements of section 6.6 of the PCI DSS. Remember, though, that your acquiring bank is the referee as far as what meets the requirements and what doesn't. You may want to check with them and see if you can use this instead of some commercial product.

So I guess this is a good time to talk about what exactly an Application Layer Firewall is. In a nutshell, it's a firewall that operates at the Application layer of the OSI. More practically speaking, it's a firewall that acts as a proxy server and makes decisions about passing traffic based on the payload of the connections it proxies. When you make a request of a web server that is protected by a proxy firewall, your connection actually terminates at the firewall, and the firewall initiates a new connection to the web server. Before initiating the connection, the firewall might look at what resources you're asking for and make the decision to deny the connection. This model is inherently more secure than a stateful inspection firewall that only looks at the destination address and the destination port. The downside is that proxy firewalls are usually slower and there aren't proxies available for every service that runs out there. I know there are people that will argue with me about my belief that a proxy firewall is more secure by its very nature than a stateful inspection firewall, and I don't feel like getting into an argument about that. So I'll put it this way: PCI DSS section 6.6 calls for an Application Layer Firewall. If you're not using something that operates at the Application layer of the OSI model then you are not compliant. Again, keep in mind that your acquiring bank is the referee, not me. For a good explanation of why Application Layer Firewalls are necessary, check out this link: http://jeremiahgrossman.blogspot.com/2009/04/disagree-with-concept-or-implementation.html (make sure you read my comments in the comments thread because everything I say over any media is worth reading).

What is our open-source application layer firewall going to look like when it's done? Well basically we're going to be using OpenBSD for our operating system, and we're going to use PF to handle any stateful inspection or natting that we might need. We're going to use Apache with mod_proxy, mod_ssl, and mod_security to act as a proxy for our http/https/ftp traffic and provide application-layer security. Can your stateful inspection firewall look for Cross Site Scripting or SQL injection in the http part of your traffic? Can it perform input validation for your applications before the traffic gets to the servers? Mod_security can, and that is why it rules ass over simple stateful inspection. We're going to use OpenBSDs relayd to act as a generic proxy for ports where we don't have a full application proxy to work with.

I'm also going to make use of a back-end server that doesn't have much on it. It's a very simple setup that will be used to make sure that our firewall is passing services. I have all of this running on virtual machines. The backend machine has a host-only network that connects to an interface on the firewall. The firewall has the host-only network and another nic that is bridged to my real network. That's where the "public" ip addresss will be reached by the clients. So watch this space as I start to fill in the How To posts.

Here is part 2.

Here is part 3.
Here is part 4.

Here is part 5.

Fix it with *nix.

I recently had to deal with something that can only be described as an EPIC SECURITY FAIL on the part of a software vendor. I'm half tempted to name the vendor just to shame them, but I'm going to refrain on the off chance that what we're seeing is misconfiguration on the part of our system administrator. Unlikely, but I don't want to get called out if I'm wrong.

Anyway, we're using this help desk ticketing software that has a reputation for sucking. I don't know anyone that has used this in real life and didn't say "Oh man, this sucks." We recently upgraded and along with the upgrade we finally got the ability for users to interact with a web client rather than a Windows-only fat application. That made me happy until I saw that the authentication portion of the web application was completely unencrypted. What if we were using Active Directory to log into this thing? We aren't, but if we were we would have our AD passwords going out in the clear. Besides, I'm sure that some of us are using the same password for both systems. So I talked to the system administrator and found out just how deep the depths of sucking in this application are.

The web client is basically an application running on Apache Tomcat that is distributed as an all-in-one binary bundle of some kind. That means that we have no way of slapping an SSL certificate on this bad boy or making it run on a different port (like 443). In other words, we're completely screwed. Luckily, Black Fist is a wise and resourceful security manager. I devised a way to make this thing suck less (I cannot completely remove the suction) and make it acceptable to me in some way.

I decided to build up a reverse proxy in front of the crapplication that would provide SSL encryption and then take steps to make sure that the unencrypted traffic didn't get back on the wire. We are using a virtual server to provide the application, so the first thing I did was have a second virtual NIC added to the machine in a private network. Then I had another virtual machine built up with two NICs: one for the public and on in the same private network. Then I installed OpenBSD on my proxy server because OpenBSD is sweet. Yes, I could have used linux, but I chose OpenBSD.

Obviously I want to take advantage of the sweet firewall that comes with OpenBSD, so I set up pf as follows.

# Edit /etc/rc.conf to make pf firewall start automatically at system startup.
pf=YES
pf_rules=/etc/pf.conf

and then...

# Edit /etc/pf.conf with a basic setup that allows https traffic to my proxy front end
ext_if="vic0"
int_if="vic1"

# It is a good idea to disable filtering on the loopback if
set skip on lo

# It is also a good idea to scrub incoming traffic
scrub in

# This is our default deny rule. Deny traffic in, but let
# everything out.
block in log on $ext_if
pass out log all keep state

# This rule allows icmp echo in
pass in inet proto icmp all icmp-type echoreq keep state

# This rule allows ssh in
pass in log on $ext_if inet proto tcp from any to external_ip port 22 flags S/SA synproxy state

# This rule allows https in
pass in log on $ext_if inet proto tcp from any to external_ip port 443 flags S/SA synproxy state

Relayd is going to act as our proxy server because it is built in and simple.

# Edit /etc/rc.conf to turn on relayd at system startup
relayd_flags=""

And then...

# Edit /etc/relayd.conf to turn on our proxy server
ext_addr="external ip address"
helpdesk="192.168.1.10"
table { 192.168.1.10 }

#
# Global Options
#
interval 10
timeout 1000
prefork 5

# Relay and protocol for HTTP layer 7 loadbalancing and SSL acceleration
#
http protocol httpssl {
header append "$REMOTE_ADDR" to "X-Forwarded-For"
header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By"
header change "Connection" to "close"

# Various TCP performance options
tcp { nodelay, sack, socket buffer 65536, backlog 128 }

ssl { no sslv2, sslv3, tlsv1, ciphers HIGH }
ssl session cache disable
}

relay sendtohelpdesk {
# Run as a SSL accelerator
listen on $ext_addr port 443 ssl
protocol httpssl

# Forward to hosts in the webhosts table using a src/dst hash
forward to port 8180 mode loadbalance \
check http "/directory/file.html" code 200
}

A quick reboot just to make sure that everything starts up on boot. Sure enough, my firewall rules are in place

$ sudo pfctl -s rules
scrub in all fragment reassemble
block drop in log on vic0 all
pass out log all flags S/SA keep state
pass in inet proto icmp all icmp-type echoreq keep state
pass in log on vic0 inet proto tcp from any to external_ip port = ssh flags S/SA synproxy state
pass in log on vic0 inet proto tcp from any to external_ip port = https flags S/SA synproxy state

and on relayd...

$ relayctl show summary
Id Type Name Avlblty Status
1 relay sendtohelpdesk active
1 table helpdesk:8180 active (1 hosts up)
1 host 192.168.1.10 99.97% up

When I visit my front end ip address on port 443 with my browser, I am seamlessly proxied to the backend server. All the traffic crossing the wire is encrypted, and the traffic between the proxy server and the real server is in memory on the virtual machine server. It's not perfect, but a big improvement over sending unencrypted authentication data over the wire.

There are a couple other things I want to point out. First of all, I am not a master of pf. I know it well enough to make things work. I'm sure that my pf.conf file could use some work. If you have any suggestions, let me know. I would love to improve this. I also know that I didn't have to use a table in relayd.conf. I chose to do that in case we ever add another server like this. It's there for future-proofing, not because of necessity.

I hope this helps somebody out there, or at a minimum gives someone an idea on how to fix some problem they're having with a less conventional technique. Anyone that knows me knows that I am all about unconventional.