26 January 2013

NSM WIth Bro-IDS Part 3: Grabbing Packets

The lights are dimming, the curtains have been drawn and the crowd is going silent. Now it's time to start the show!

E-mail Alerts


I've been using bro at work and I really, really like the hourly emails it sends out with connection details. If you recall, I didn't install any type of MTA in part two. Before I jump too far into bro, I want to go ahead and install mailutils using:

sudo apt-get install mailutils

This results in the following:


apt/dpkg will go ahead and prompt for postfix configuration. Since I'm only delivering to my VM I'll choose "Local only"; for a machine that only sends mail (and doesn't receive), I would normally choose "Satellite system" or "Internet with smarthost":


To test whether you can send mail to local users, use the 'mail' command. So that I can put it into one line, I take the output of 'echo' and send it as the content of a mail message that is sent to demo@localhost:

echo "foo" | mail -s "test message" demo@localhost

'mail' is a pretty useful utility, you can also use it to read the current user's mail spool just by typing 'mail' and hitting enter. If you are presented with something similar to the following, congratulations, you can send mail to local users!


You can delete the message by typing 'd 1' (for 'd'elete message number '1') and then exit with 'quit'.

One Quick "gotcha" - Cron


I didn't realise it at first but, at least on Ubuntu, the bro installation process doesn't configure the system to start bro when the system boots and there are a couple of housekeeping items that bro needs to take care of on regular time deltas. Addressing this issues is as simple as

1) Adding a line to /etc/rc.local to start bro at boot via 'broctl start':


2) Adding a line to /etc/crontab to take care of housecleaning items every five minutes via 'broctl cron':


Bro Configuration Files


The bro configuration files can be found in /usr/local/bro/etc:


The first file I want to edit is node.cfg. There is an interface line in there that, by default, is set to eth0. To make sure I set the appropriate interface, I ran 'ifconfig' to see which interface didn't have an IP and then edited the file. The output of ifconfig and the modified node.cfg are below:



Note the "type=standalone" line. It is possible to run bro in a clustered mode, either with multiple hosts or multiple instances on the same host. Since I'm dealing with less than 5 Mb/s of traffic, it is pointless to configure multiple instances other than as an academic exercise. While I may address that configuration in another post, right now I want to keep the configuration simple so I'm running in standalone mode.

The next change is to networks.cfg. This file tells bro which networks are "local" and is used for determining incoming vs. outgoing connections. My psql_test network, the one using my FreeBSD VM as a router, uses the 10.10.10.0/24 IP space, so that's the only network I'm defining in networks.cfg:


The last file to edit is broctl.cfg.  In production I would want to make a change to the MailTo line, add a custom MailFrom line and set the path for sendmail. I don't mind having emails go to root@localhost in a test environment so the only change I'm going to make is to add the path for sendmail:


Installing the Configuration and Starting bro


All of my groundwork is ready to come together. Bro's configuration files are ready, my FreeBSD router is pushing data over the span port and I have a handful of virtual machines ready to generate some traffic. Now to start bro!

The broctl tool is used for basic administration tasks. You can use it to start/stop bro, update the configuration, get stats for running manager/worker instances, get basic stats on the current traffic rate being monitored and more. First, start broctl using 'sudo broctl':


Note the prompt changed to '[BroControl] >'. To exit the broctl program, type 'quit' and hit enter. Don't do that yet, though, there are a few things to do first!

Since this is a new installation, I need to 'install' the configuration files. This is done by typing 'install' at  the BroControl prompt and hitting 'enter'. I actually did that and hit Ctrl-L to clear the screen before I thought to take a screenshot so you'll get something *VERY SIMILAR* but possibly not *EXACTLY* like this:


Now use the 'start' command to start bro:


To get the status of the newly started process, use 'status':


At this point I switched over and started 'sudo apt-get update && sudo apt-get upgrade' on a couple of Ubuntu virtual machines and 'sudo portsnap fetch update' on a couple of FreeBSD virtual machines to generate some traffic, then I came back to bro and used 'capstats' to get some basic information on the capture interface:


 Note that I'm only pulling about .8 Mb/second at less than 1,000 packets per second. That's fine considering my uber-slow DSL connection. In production I have virtual machines monitoring a couple of hundred megabits of traffic per second. It's pretty cool.

Carving the Logs (or, Going for the Scoobies!)


At this point you're possibly (probably...) thinking, "um...Kev...this is cool and all but...where's the payoff? Why have I gone through all of this?

Bro stores its logs (by default) in /usr/local/bro/logs. In that directory is a series of directories created by timestamp. An example directory would be /usr/local/bro/logs/2013-01-24. There is also a "current" directory, where the log files for the current bro process are kept. This is a symbolic link to bro's spool directory and every hour bro archives those logs to the appropriate directory. For this post I'll work entirely in /usr/local/bro/logs/current:


If you have been monitoring an active network then you should be able to run 'ls' and see several log files present:


Each of these files is interesting but there are a few that are particularly relevant to me. Recently I've been responding to a LOT of problems with outdated Java versions being exploited by exploit packs like Blackhole and CoolEK. I had written a Snort rule to look for Java versions up to 1.7.0_10 but I'm already taxing my Snort machines pretty hard. It would be really wicked if I could look for Java versions or wacky user-agent strings - Internet Explorer from a FreeBSD server or curl from a Windows server, for example. They aren't necessarily impossible, just generally unlikely. As it turns out, bro identifies the software it sees making network connections and logs it in software.log:


You can see that it identified 'portsnap' from one of my FreeBSD virtual machines, wget from a Linux virtual machine and 'rekonq' (the new version of Konqueror) from my KUbuntu virtual machine. If I wanted to see only portsnap connections I could have used:

grep portsnap software.log

If I wanted to look for Java as the user-agent, I could have used:

grep -i java software.log

See where this is going? By logging in plain-text, you can search the bro logs using any common command-line tool. Very cool!

Another interesting file is conn.log. This file keeps a list of all connections it sees, with data equivalent to that gathered from netflow. It provides a timestamp, IPs, ports, bytes transferred, packets transferred, duration of the connection and more. A complete description of the fields can be found in the bro documentation. The sample from my test shows several connections from my FreeBSD virtual machine:


Bro assigns an UID per connection and it uses the same UID for the same connection whether it is in the connection log, the HTTP log or the DNS log, making it trivial to gather information about a particular network conversation. If I pick one of the unique IDs from that table, for example if I choose 'AHNlJWaph34', I can search for any occurrence of that UID in any log with a simple grep:


This lets me know I not only had a connection between 192.168.1.76 and 192.168.1.254 on port 53, I can verify that it was indeed a DNS lookup for update5.freebsd.org and that update5.freebsd.org resolves to the IP address 204.9.55.80.

Other Logs

While I won't go into samples, there are many more useful logs bro creates out-of-the-box. If your network has any FTP servers you can monitor those FTP connections -- you can see which user is getting or putting which files, provided it is pure FTP and not SFTP, via ftp.log.

Bro keeps track of any SSL certificates it sees and can do real-time validation of those certificates. The identifying information for these certificates is kept in known_certs.log.

As machines come on the network, bro logs their IP in known_hosts.log, so you can keep track of which machines are on at what time of the day. This is highly useful for trending and monitoring for deviations from the norm.

If your network has SMTP servers you can get much better logs in smtp_entities.log and smtp.log than what most MTAs provide out-of-the-box. IPs, recipients, senders, mail message IDs, subject lines, the names of any attachments and, optionally, a MD5 of the attachment, are all available.

One of the most frustrating things for me when doing incident response is trying to guess what file a machine downloaded just before Snort starts alerting me to botnet behaviour. Unless we are in an environment with mature IR and NSM programs, most of us have no reliable way of determining what domain the infected machine was trying to access at a given IP or, if HTTP, if the infected machine then tried to do a GET or POST and to which script at a given site. This can be particularly frustrating if the site is at a shared hosting provider like Amazon, Rackspace or one of a thousand Moldovan, Russian or Dutch web hosts. Bro addresses this huge problem via http.log, a wonderful resource for following the flow of web requests from a given host. As with dns.log, the same UID used in conn.log is used in http.log.

Getting Away From the Command Line

I live at the command line. At work I use plain-text email (mutt). I generally have ten to twenty tabs open with SSH sessions to various Linux or Unix servers at any one time. Tools like grep, awk, sed, vi, cat and cut are an integral part of my day. They are awesome for quick-and-dirty work but have I mentioned that my current favourite open source project is ELSA? As it turns out, the author is a professional incident handler/responder and he's a pretty big fan of bro so ... ELSA has support for the bro connection log. In part four of this series I'll cover tying two really awesome projects together by sending bro connection data to ELSA, giving you the ability to monitor connections on multiple networks from totally different buildings, data centres, countries or even continents via a single web-based tool.

21 January 2013

NSM With Bro-IDS Part 2: The Install

With the router setup in my last post, this post will cover setting up a virtual machine that monitors the span port from the router and installing bro from source.

Configuring the VM -- VirtualBox


For the bro virtual machine there were two changes I wanted to make to how I've configured my virtual machines up until now. For this installation I wanted to use the amd64 release of Ubuntu Server 12.04 LTS. In my specific virtual environment there is absolutely no benefit to using amd64 over x86, it's purely for the fun of it, but in a production environment it might make a lot of sense to have a 64-bit virtual machine. Additionally I wanted to use a second drive for the bro install. This would let me still setup a "template" 64-bit installation with fairly minimal disk usage and then add basic storage for the bro install that I can expand as necessary.

As with other times I created a "base" virtual machine that I then cloned. The actual VirtualBox virtual machine configuration looks like (note the virtual machine names - with my virtual environment growing I decided to rename most of the virtual machines to reflect their usage):


Configuring the VM -- Operating System


As I've noted, I chose Ubuntu Server 12.04 LTS for the operating system. The primary hard disk had 4 GB partitioned for swap and I devoted the rest to the root ("/") partition:


Since I planned to clone the virtual machine, this one doesn't have the additional 8 GB "bro" drive. 

The Guest VM -- Updates and Data Drive


With the OS installed, I updated it with

sudo apt-get update && sudo apt-get upgrade

Then I installed the openssh-server package:

sudo apt-get install openssh-server

To ease administration I logged out from the VirtualBox interface and used SSH to connect from my MacBook host.

I then connected via SSH and added the data drive (the screenshots are more useful from an SSH session than from the VirtualBox interface). First I used fdisk to partition it with one large primary partition:


Then I used "mkfs.ext3" and "mount" to create the new filesystem and mount it as "/usr/local/bro":


With the appropriate addition to "/etc/fstab", the new partition will get mounted automatically at boot:


The Guest OS -- Bro installation


I want to go ahead and say that this entire section is based on documentation at the project website at bro-ids.org. They have some really fantastic documentation and I relied on it heavily. I am NOT going to try to reproduce that, I am merely stating the exact process I followed. If you want to make ANY changes to the install process (even if you don't want to make changes!) you should read the documentation at the project site.

Even though I installed bro from a source tarball, first I needed to install several Ubuntu libraries. This was accomplished with:

sudo apt-get install cmake make gcc g++ flex bison libpcap-dev \
libssl-dev python-dev swig zlib1g-dev libmagic-dev \
libgoogle-perftools-dev libgeoip-dev

This installs a LOT of dependencies:


The source tarball for the 2.1 release can be downloaded via wget and extracted with tar:

wget http://www.bro-ids.org/downloads/release/bro-2.1.tar.gz
tar zxf bro-2.1.tar.gz

This creates a "bro-2.1" directory. From here it is very standard Unix software installation. Change directory to the bro-2.1 directory and run:

./configure
make
sudo make install

Note that the output of "make" and "make install" can be pretty substantial:




I'll cover a basic bro configuration in my next post but there was one more step I needed to take before I considered the general installation completed: the bro control program, broctl, is not in the sudo path on Ubuntu Server by default. To add /usr/local/bro/bin to the sudo path you have to edit the sudoers file; the appropriate way to do that is with "sudo visudo". It opens a standard nano session and the only modification that needs to be made is to the secure_path line: add ":/usr/local/bro/bin" to the end of the line and save the file/exit.

With the addition to sudoers saved, you should be able to verify a successful installation with "sudo broctl". You should see something very similar to:


If you do not, I would go back through the "make" and "make install" components to see why something failed to build or install properly. You can try running broctl with its full path:

sudo /usr/local/bro/bin/broctl

If you are mounting a separate partition at /usr/local/bro, and you have had to reboot since doing "make install", you should verify that the partition is actually mounted and that it was mounted when you did the "make install".

Otherwise, keep an eye out for part 3. It will cover a very basic bro configuration, basically just enough to verify you are monitoring traffic, and will allow you to ascertain that you are able to extract relevant information from bro.

20 January 2013

NSM With Bro-IDS Part 1: First, you ...


I have this really great co-worker named John. We may differ on some things but when it comes to food we are peas in a pod. Through some fateful twist we both have a deep appreciation for Cajun cooking, particularly gumbo and jambalaya. If you've read traditional Cajun recipes, there are five little words that you can pretty well assume are going to be in any recipe for anything fixed in a pot, skillet or dutch oven.

"First, you make a roux"

I really wanted to blog about network security monitoring and a couple of open-source intrusion detection systems. Specifically I wanted to cover
bro, a very cool project from ICSI that I've only recently started testing
suricata, a threaded "alternative" to Snort that I've only tested for about a month
OSSEC, a HIDS that most of my friends are wondering why I haven't already covered
Snort, the gold-standard of open source intrusion detection
OSSEC is pretty easy, it's one server and then some client installs, but I started thinking about the requirements for the others and realised I'm going to need a router with a span port and a network link for the bro/suricata/snort virtual machines to be able to see the span traffic (this effectively sets the router up as a tap). So, part one of my bro series isn't really about bro at all, it's about setting up the environment so I can *install* bro. To paraphrase all those really awesome recipes from southern Louisiana: first you build a router.

FreeBSD + pf


I already have the psql_test internal network with a bunch of hosts on it and I've established in previous posts that it works a *lot* like a real network. I also have enough virtual machines in production networks running FreeBSD, acting as routers and packet filters for other virtual machines on the virtual networks, to think that this should be pretty easily accomplished in VirtualBox.

Create the FreeBSD VM


This is the easy part. I use my fbsd_8_3_i386_101 VM and cloned it to fbsd_8_3_i386_254


"Adapter 1" was changed from NAT to the 'psql_test' network: 


I added "Adapter 2" as a bridged interface so I could access that IP from my home network (specifically I wanted SSH access):


The IDS VMs would need access to the network traffic so I added an "Adapter 3" as an internal network:


Keep in mind that this isn't the order they'll show up in FreeBSD so take note of the last segments of the MAC addresses (or you can just look at the adapter definition "Advanced" tab when you're setting up the IPs in the guest OS).

The Guest Operating System -- /etc/rc.conf


Since FreeBSD is already functional, I really only need to modify a couple of files to turn my potential server into a router. The first step is to identify which VirtualBox Adapter is associated with each em* device. Running "/sbin/ifconfig -a" will give you each interface and its MAC address (remember I said to note each Adapter's MAC?). In my case em0 is the interface on my home network, em1 is the bro_bridge interface and em2 is the psql_test interface, so I need to bridge em0 and em2 and then set em1 up as a span. For packets to actually forward I need to enable the gateway functionality. To be able to SSH into the machine from my home network I need to enable sshd and, finally, since this machine will NAT traffic between the psql_net network and my home network, I need to enable pf. All of these things can be done in /etc/rc.conf. Taking everything I've laid out into consideration, my new /etc/rc.conf is:


The Guest Operating System -- /etc/pf.conf


pf needs to be configured to NAT traffic and have any necessary filtering rules configured. In this case I'm going to specifically pass any inbound traffic coming into the router on the psql_net interface and specifically pass any outbound traffic leaving the router for the home network. pf will pass traffic on any interface unless there is an explicit block so the pass rules are extraneous but I like having them there for maintainability and readability (plus I usually have a block rule right after the NAT line). An *extremely* basic pf.conf to get the job done:


The Guest Operating System -- Reboot


At this point the fastest way for everything to take effect (pf rules, new IPs, gateway functionality) is to just do a reboot. After the reboot I like to do a quick ifconfig just to make sure the internal IP was assigned and the bridge was created:

To verify the router works, start a virtual machine and set the only active interface to be one on the psql_test network. Set the gateway as the IP of the psql_test interface on the router VM, set the DNS server in resolv.conf as a valid DNS server (I used 8.8.8.8 for Google's public DNS) and try to ping something like google.com (I use google.com because they respond to pings). 

Running 'tcpdump -i em1' on the router while the ping is active should let you know if the bridge and span are correctly configured. Probable errors here are incorrectly determining which em* device maps to which Adapter, the configuration of the bridge and typos in either /etc/rc.conf or /etc/pf.conf. Assuming everything matches the examples you *should* have a fully functional router with a way to mirror traffic to multiple virtual machines simultaneously, a useful tool for testing multiple intrusion detection engines with the same network traffic.

13 January 2013

Enterprise logging with ELSA


When I made the decision to start blogging or giving directions and tips regarding some of the open source tools I use every day, I did so with some of my favourite projects in mind. For anyone who knows me, they would expect software like FreeBSD and PostgreSQL, products a lot of us use every day without realising it, but I had one in mind above the rest - an enterprise-grade log aggregator called ELSA.

ELSA, short for Enterprise Log Search and Archive, is a log aggregator and search tool built on a collection of utilities common to most Unix and Linux system administrators: syslog-ng, MySQL, Sphinx, Apache and perl. The project page can be found here.

To say that it's useful is an understatement. In a professional capacity I have probably fifty servers and a couple of hundred switches, routers and appliances logging to ELSA nodes. If it speaks syslog, ELSA can take it. So what sets ELSA apart from all of the other utilities? Why should anyone use ELSA instead of, say, plain syslog? Why not GrayLog2? Why not go commercial and buy Splunk?

First, let me say that those are fantastic products. Syslog is the standard for logging in the Unix world. I used GrayLog2 for a couple of months when I started looking at syslog alternatives and it rocked until it hit relatively high load. The graphs and the user interface are sleek and can make managers drool. Once I started pushing more than about 1k events/second, though, Java started falling down and services started crashing. Not cool. I thought about really digging in and tweaking the devil out of it but I'd have had to learn the intricacies of both MongoDB and ElasticSearch and, at the time, I didn't have the time to devote to learning a bunch of new products. I wanted something based on tried and tested Unix tools that any Unix SA could come behind me and help troubleshoot if something went wrong.

I demo'd Splunk and I have absolutely nothing bad to say about the product. There are connectors for all sorts of devices and operating systems, it uses a multi-node indexing and querying configuration so you can have any <n> number of aggregators with any <n> number of query servers. This lets you add log collectors as your infrastructure requires and allows and, because they charge per GB you index, you can literally throw as many machines into that role as you need without affecting your licence.

That's the rub, though. Splunk is an awesome product but it carries an awesome price tag. My unit alone generates some 15 - 25 GB of data per day, depending on load, and I want to be able to search *all* of it from one place. My division generates two to three times that. Care to guess what a 50GB or 100 GB/day licence from Splunk costs?

Then I stumbled upon ELSA. The installation looked simple, the author provides a single script that you run and it takes care of everything. Maintenance looked fairly simple, it uses syslog, MySQL, Apache, perl and Sphinx, and I'm already intimately acquainted with four of those five tools. It could run on FreeBSD, yet another plus. The licence was acceptable (I prefer BSD but the GPLv2 works). It supports AD/LDAP and local database authentication with a robust permissions system. I got approval from my boss and started tossing hardware at it.

Like Splunk, ELSA uses a very distributed model. Each server forwards their logs via syslog to an ELSA node. You can setup any number of ELSA nodes.

As well as having any <n> number of nodes, you can have any <n> number of web front-ends, and each front-end can query any <n> number of nodes. Selecting which nodes you query with the front-end is handled through a configuration file on the front-end but the node has to be configured to allow those queries. This means you can have separate ELSA nodes for networking equipment, web servers, database servers, your ERP, AD/LDAP or any other departmental use and each department can have their own front-end that queries just their nodes -- but InfoSec can have a "master" front-end that can query *every* node.

Did I mention ELSA is my favourite open-source project at the moment?

So how complex is it to setup and start using? Well well well, let's go to our trusty virtualisation environment and start working!

EDIT** Please note that the following instructions are now out-dated; updated instructions can be found at  http://opensecgeek.blogspot.com/2013/07/enterprise-logging-with-elsa.html

There was some conversation on the ELSA users mailing list recently about problems with Debian 6 and the web front-end which the author, Martin, quickly addressed. To give him another datapoint I opted to run a node and the web front-end (hereafter referred to as the WFE) on Debian 6.

VM Setup


As always, I opted for a minimal installation and cloned it rather than doing two completely separate installations. Following my previous naming scheme I named them "Debian_6_i386_121" and "Debian_6_i386_122":


I wanted them to have network interfaces on the local "psql_test" network (as 10.10.10.121 and 10.10.10.122) but also to be able to get on the Internet so I gave them NAT interfaces on "Interface 2". Each VM was allocated 1GB of RAM since they have to build some of the utilities and I gave them each a 20 GB hard disk:

As with the Ubuntu VMs, I put everything on one ext3 partition and gave a swap partition roughly twice the size of the amount of RAM:


ELSA Installation


After they were finished installing I went ahead and installed the server packages for OpenSSH and MySQL with

    sudo apt-get install mysql-server openssh-server

To keep things simple I left the password for the MySQL "root" account empty. On a production deployment I would set it and then add a /etc/elsa_vars.sh file where I could put it for the ELSA installation but for this purpose, blank works just fine.

The ELSA node installation is two commands:

    wget "http://enterprise-log-search-and-archive.googlecode.com/svn/trunk/elsa/contrib/install.sh"
    sudo sh -c "sh install.sh node"

On the vm hosting the WFE, it's almost identical:

    wget "http://enterprise-log-search-and-archive.googlecode.com/svn/trunk/elsa/contrib/install.sh"
    sudo sh -c "sh install.sh web"

With both of those running, I stepped back for about thirty minutes. My Macbook is peppy but those are some bandwidth intensive installations and my DSL connection is most decidedly NOT peppy.

When I came back I was greeted with screens similar to the following. On the node installation:

And on the WFE:



Testing


Since I have a host of virtual machines already on the psql_test network, I went ahead and cranked up all three FreeBSD machines (from the PostgreSQL replication/DNS posts) and my KUbuntu VM. On the FreeBSD machines I edited /etc/syslog.conf and, just past the comments sections, added a line that directed syslog to send anything it received to 10.10.10.121. To do that, use:

    *.* @10.10.10.121

To see that in the config file:


Then I restarted with:

    sudo /etc/rc.d/syslogd restart

I then did that on FBSD_8_3_i386_103 and FBSD_8_3_i386_104, so now all three virtual machines are sending all of their syslog data to 10.10.10.121.

So that I could query the node from the WFE, I edited /etc/elsa_web.conf on the VM hosting the WFE, Debian_6_i386_122, and added the node information to the backend node section:


This was followed by an apache restart on the WFE:

    sudo /etc/init.d/apache2 restart

Interrogating ELSA


Having a rocking syslog configuration does nothing if you can't get the information back out of it. Now that my BSD machines were sending their logs to it, I needed to query it so I logged in on my KUbuntu virtual machine, opened a browser and pointed it at the ELSA WFE at 10.10.10.122:


The first time I saw that I thought, "you have to be kidding me...that much work and that's what we're given?" So I started looking. We have 10.10.10.102 sending logs there, let's see if we can get anything out of it:


You can ignore the "Warning:" in red. By default the WFE looks for a local node installation (the same machine can be your node and WFE) and I didn't comment that out of the elsa_web.conf file, but whoa whoa, that looks useful! We just got everything that matched "10.10.10.102". What about if I look for "syslogd"?


Note I got back data from two separate hosts, and that the "host=<ip>" area is clickable. Clicking on that adds it to the query bar, so if I click on the first one and click "Submit Query", I get:


Did I mention that ELSA is currently my favourite open source project?!

Some more reading


After deploying it on real hardware I realised my early comments were completely ludicrous. The more I used it, and the more data I started throwing at it, the more I realised this tool was *incredible*.  I started pointing all of my Windows (via SNARE), Linux (syslog, syslog-ng and rsyslog) and BSD (syslog, rsyslog) servers to it. I started pointing appliances, switches and routers at it and added a second node.

Once I started reading the author's blog at ossectools.blogspot.com I started to get a REAL look at what I could do with ELSA. I created a PostgreSQL and MySQL database and pointed ELSA at those as alternate datasources. I started investigating pushing my IDS logs at it and then combining attack data from the IDS alerts with data in the databases. It didn't take me long to realise Martin had written a log search and archival tool that was perfect for incident handlers and incident responders.

To wrap things up, I highly recommend giving this project a test drive. Martin has some great documentation, both at the official project page and at his blog, "Open-Source Security Tools", and the mailing list for the project is both highly supportive and very active.

09 January 2013

BIND part 3: Full DLZ-backed domain


Not too long ago I gave a presentation at a tech conference regarding using DNS blacklisting/blackholes/sinkholes to identify and mitigate the effects of malware. Specifically, it was that presentation that sparked my last blog post on configuring BIND to use DLZs. After that presentation I received a phone call from an entity wanting to use DLZs for their entire domain, not just for their DNS blackhole. Since my DNS servers already have to do at least one database lookup for every DNS query, and since I already have instructions on how to build a DNS blackhole using DLZs, I thought it fitting to go ahead and extend that to the logical ending -- using DLZs for an entire domain.

Database Changes


Since I already have a working PostGreSQL, BIND and DLZ deployment in a virtual environment, I'm going to use that as my workspace. Using either "\d" from the psql CLI or looking at the output of a "pg_dump -s dnsbh" from the system CLI shows the following existing columns for the dns_records table:
  • id serial
  • zone text
  • host text
  • ttl integer
  • type text
  • data text
Where zone is the zone being queried, host indicates whether it's a single host or a wildcard for all subdomains, type indicates A/AAAA/MX/NS/etc and data indicates the destination IP address for that record.

To return "proper" responses for all queries related to a domain, there are some columns that need to be added:
  • mx_priority, the priority for MX records
  • contact, the responsible contact for the zone
  • update_serial, the serial number for the zone update
  • refresh, the amount of time slaves or secondary servers wait between update requests
  • retry, the amount of time slaves or secondary servers wait between update requests after a failed request
  • expiration, the amount of time before a record in the cache is considered stale
  • minimum, the minimum amount of time to keep the item in the cache
  • primary_ns, the DNS servers for the zone
To add these columns to the dns_records table, I'll issue the following at the psql CLI on the PostGreSQL master:
ALTER TABLE dns_records
ADD COLUMN mx_priority INTEGER,
ADD COLUMN contact CHARACTER VARYING (255),
ADD COLUMN update_serial INTEGER,
ADD COLUMN refresh INTEGER,
ADD COLUMN retry INTEGER,
ADD COLUMN expiration INTEGER,
ADD COLUMN minimum INTEGER,
ADD COLUMN primary_ns CHARACTER VARYING (255); 

Supporting the DLZ driver functions


The PostGreSQL DLZ driver expects some value (even if that value is "NULL") to be returned for the MX priority in its lookup() function as the third value returned, between 'type' and 'data'. This function performs the query defined by the seconds "SELECT" statement in the BIND configuration file. Those two things combined means that I need to add the mx_priority column to the SELECT query. I will post the entire BIND configuration later so don't worry just yet about editing the file, just be cognisant that it will have to happen.

The PostGreSQL DLZ driver needs to be able to query for SOA and NS records and it does this via the authority() function. The query for this function should return values for ttl, type, mx_priority, data, contact, update_serial, refresh, retry, expiration and minimum. This will get added after the existing final "SELECT" statement in the BIND configuration. There is a caveat to this - the query used by the lookup() function *CAN* be written to return SOA and NS records; if this is the case then the query for the authority() function can be an "empty query" and written as "{}" (note there are no spaces between the braces). I am specifically NOT letting my lookup() function pull NS and SOA records so a query is necessary for the authority() function.

The next item added to the BIND configuration is a query that returns values for the DLZ driver's allnodes() function. It will return values identical to the authority() function but the query doesn't differentiate between NS/SOA and other records so it will be nearly identical to the query written for the authority() function.

Finally, it is possible to add a query that supports the allowzonexfr() function and returns the zone and IP addresses of clients allowed to perform zone transfers for that zone. Note that if you want to allow zone transfers then you *must* support both the authority() and allowzonexfr() functions. Since part of the point of database replication and backing BIND with a database is to allow instant access to updates, I will NOT add a query to support this functionality. It is recommended against by the original bind-dlz team, for what I believe to be very good reasons, and I believe the proper way to handle those updates is via database replication to all secondary DNS servers. Using this model effectively removes the "master/slave" relationship and allows any DNS server in the organisation to act as a primary DNS server without having to change anything in the BIND configuration.

Editing the BIND configuration


1) Adding mx_priority for the lookup() function

{
  select
    ttl,
    type,
    mx_priority,
    case when lower(type)='txt' then '\"' || data || '\"' else data end
  from
    dns_records
  where
    zone = '$zone$' and
    host = '$record$' and
    not (type = 'SOA' or type = 'NS')}
}

2) Adding a query for the authority() function

{
  select
    ttl,
    type,
    data,
    primary_ns,
    contact,
    update_serial,
    refresh,
    retry,
    expiration,
    minimum
  from
    dns_records
  where
    zone = '$zone$' and
    (type = 'SOA' or type='NS')
}
3) Adding a query for the allnodes() function
{
  select
    ttl,
    type,
    host,
    mx_priority,
    data,
    contact,
    update_serial,
    refresh,
    retry,
    expiration,
    minimum
  from
    dns_records
  where
    zone = '$zone$'
}

Put all the queries together


Taking all of the above modifications into consideration, the new dlz section of my BIND configuration now looks like this (on my production machines the select statements aren't so separated, I just do it here for clarity):

dlz "postgres" {
  database "postgres 4
  {host=localhost port=5432 dbname=dnsbh user=dnsbh password='password_you_used'}
  {
    select
      zone
    from
      dns_records
    where
      zone = '$zone$'
  }
  {
    select
      ttl,
      type,
      mx_priority,
      case when lower(type)='txt' then '\"' || data || '\"' else data end
    from
      dns_records
    where
      zone = '$zone$' and
      host = '$record$' and
      not (type = 'SOA' or type = 'NS')
  }
  {
    select
      ttl,
      type,
      data,
      primary_ns,
      contact,
      update_serial,
      refresh,
      retry,
      expiration,
      minimum
    from
      dns_records
    where
      zone = '$zone$' and
      (type = 'SOA' or type='NS')
  }
  {
    select
      ttl,
      type,
      host,
      mx_priority,
      data,
      contact,
      update_serial,
      refresh,
      retry,
      expiration,
      minimum
    from
      dns_records
    where
      zone = '$zone$'
  }";
};

Adding domain data to the database


I decided to test with a very simple domain. At the minimum I wanted a primary nameserver, a mail server, a random machine and something to justify a CNAME record so I could perform the following lookups:
  • SOA
  • NS
  • A
  • CNAME
  • MX
  • PTR
To do that, I used the following IP:name combinations:
  • 10.10.10.103 -- ns1.demo.local
  • 10.10.10.112 -- mail.demo.local
I then decided to use imap.demo.local as a CNAME to mail.demo.local.

First, I'll add the SOA records for the demo.local zone and for reverse lookups for the 10.10.10.0/24 network:
INSERT INTO dns_records (zone, host, ttl, type, data, mx_priority, contact, update_serial, refresh, retry, expiration, minimum, primary_ns) VALUES ('demo.local', '@', '86400', 'SOA', NULL, NULL, 'hostmaster.demo.local.', '2013010801', '3600', '1800', '604800', '86400', 'ns1.domain.local.');

INSERT INTO dns_records (zone, host, ttl, type, data, mx_priority, contact, update_serial, refresh, retry, expiration, minimum, primary_ns) VALUES ('10.10.10.in-addr.arpa', '@', '86400', 'SOA', NULL, NULL, 'hostmaster.demo.local.', '2013010801', '3600', '1800', '604800', '86400', 'ns1.domain.local.');
The NS records for my zone and IP range:
INSERT INTO dns_records (zone, host, ttl, type, data, mx_priority, contact, update_serial, refresh, retry, expiration, minimum, primary_ns) VALUES ('demo.local', '@', '86400', 'NS', 'ns1.demo.local.', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);

INSERT INTO dns_records (zone, host, ttl, type, data, mx_priority, contact, update_serial, refresh, retry, expiration, minimum, primary_ns) VALUES ('10.10.10.in-addr.arpa', '@', '86400', 'NS', 'ns1.demo.local.', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
The MX record with a priority of 10 for mail.demo.local:
INSERT INTO dns_records (zone, host, ttl, type, data, mx_priority, contact, update_serial, refresh, retry, expiration, minimum, primary_ns) VALUES ('demo.local', '@', '300', 'MX', 'mail.demo.local.', '10', NULL, NULL, '3600', '1800', '604800', '86400', NULL);
The A records for ns1.demo.local (10.10.10.103) and mail.demo.local (10.10.10.112):
INSERT INTO dns_records (zone, host, ttl, type, data, mx_priority, contact, update_serial, refresh, retry, expiration, minimum, primary_ns) VALUES ('demo.local', 'ns1', '86400', 'A', '10.10.10.103', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);

INSERT INTO dns_records (zone, host, ttl, type, data, mx_priority, contact, update_serial, refresh, retry, expiration, minimum, primary_ns) VALUES ('demo.local', 'mail', '86400', 'A', '10.10.10.112', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
The  PTR records so reverse lookups work:
INSERT INTO dns_records (zone, host, ttl, type, data, mx_priority, contact, update_serial, refresh, retry, expiration, minimum, primary_ns) VALUES ('10.10.10.in-addr.arpa', '103', '86400', 'PTR', 'ns1.demo.local.', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);

INSERT INTO dns_records (zone, host, ttl, type, data, mx_priority, contact, update_serial, refresh, retry, expiration, minimum, primary_ns) VALUES ('10.10.10.in-addr.arpa', '112', '86400', 'PTR', 'mail.demo.local.', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
Finally, the CNAME to point imap.demo.local to mail.demo.local:
INSERT INTO dns_records (zone, host, ttl, type, data, mx_priority, contact, update_serial, refresh, retry, expiration, minimum, primary_ns) VALUES ('demo.local', 'imap', '86400', 'CNAME', 'mail.demo.local.', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
At this point the following lookups should work when performed against the DNS server:
  • dig soa demo.local
  • dig ns ns1.demo.local
  • dig mx demo.local
  • dig mail.demo.local
  • dig imap.demo.local
From here it's a trivial matter to add as many A records or CNAMEs and corresponding PTR records as necessary. My authoritative DNS servers handle over 60k IPs and I make manual changes every day so if I were backing them with DLZ I would develop a web-based management interface but for a small network (a /24 or smaller), with a single DNS administrator, manually interacting with the database really isn't too terribly prohibitive.

A New Year, A New Lab -- libvirt and kvm

For years I have done the bulk of my personal projects with either virtualbox or VMWare Professional (all of the SANS courses use VMWare). R...