BIND part 2: DNS blackhole via DLZs

In my last post I detailed installing BIND with DLZ support -- in this post I'll actually USE that option.

Step One: Database setup

PostgreSQL, by default, now creates databases in UTF-8 encoding. The DLZ driver in BIND uses ASCII/LATIN1, which introduces some interesting peculiarities. Specifically, querying for 'zone="foo.com"' may not work. That's okay, you can still create LATIN1 databases using the template0 template. Since PostgreSQL replication is already configured, everything but the BIND configuration is done on the database master. First, add the new database user:

createuser -e -E -P dnsbh

Since this user won't have any extra permissions, just answer 'n' to the permissions questions.

Now add the new database, as latin1, with dnsbh as the owner:

createdb dnsbh -E 'latin1' -T template0 -O dnsbh

The database schema is pretty flexible; there are no required column names or types, as long as queries return the correct type of data. I like to use a schema that reflects the type of data the column holds so I'll use the following create statement:

create table dns_records (
    id serial,
    zone text,
    host text,
    ttl integer,
    type text,
    data text
);
create index zone_idx on dns_records(zone);
create index host_idx on dns_records(host);
create index type_idx on dns_records(type);
alter table dns_records owner to dnsbh; 

This can be modified, of course. The data, type, host and ttl fields can have size restrictions put in place and you can drop the id field altogether. The bind-dlz sourceforge page lists several more columns but those are only necessary for a full DLZ deployment, where the DNS server is authoritative for a domain, not for a purely recursive caching server.

Step Two: BIND setup

If you read the bind-dlz configuration page you'll find a set of database queries that get inserted into named.conf. You can modify these if you like but it's much easier to use what's provided. There are, at a minimum, four lines that need to be included. The first indicates the database type and the number of concurrent connections to keep open (4). The next is the database connection string (host, username, password). The third returns whether the database knows about a given zone and the fourth returns the actual IP for any lookup that isn't for types MX or SOA. Add the following to /etc/namedb/named.conf and restart BIND (I split the fourth "line" into multiple lines for readability - you can put it all on one line or leave it as below):

dlz "postgres" {
database "postgres 4
{host=localhost port=5432 dbname=dnsbh user=dnsbh password='password_you_used'}
{select zone from dns_records where zone = '$zone$'}
{select ttl, type,
  case when lower(type)='txt' then '\"' || data || '\"' else data end
  from dns_records
  where zone = '$zone$' and
  host = '$record$' and
  not (type = 'SOA' or type = 'NS')}";
}

At this point BIND should restart and it should continue to serve DNS as normal -- meaning it's time to test the blackhole. A quick and easy test is to dig for sf.net:

dig @127.0.0.1 sf.net

Then add sf.net to the dns_records table on the PostgreSQL master:

insert into dns_records(zone, type, host, data, ttl) values ('sf.net', 'A', '*', '127.0.0.1', '3600');

Dig for sf.net again:

dig @127.0.0.1 sf.net

Note the 3600 second TTL. If a downstream DNS server were querying our blackhole then the value of 127.0.0.1 would get cached for an hour. To remove the sf.net entry from the blackhole, go back to the PostgreSQL master:

delete from dns_records where zone = 'sf.net';

www.malwaredomains.com and projects like ZeusTracker keep a list of domains seen in the wild that are used to provide malware. The malwaredomains.com site has a great tutorial on how to do DNS blackholes via BIND configuration files. Since they provide a list of domains it's pretty trivial to script something to pull those domains out and jam them into the blackhole table.

BIND part 1: Recursive resolver

One of the most commonly used services on the Internet is DNS. We use it every time we type something into Google, check our email or stream a video from YouTube. It is THE address book for nearly everything we do on the Internet.

It is also used nefariously. Malware authors will use DNS instead of static IPs so that they can constantly switch malware servers, making IP-based firewall rules useless. Groups like malwaredomains.com track the domains used by malware and provide a readily-accessible list so that DNS operators can blackhole those lookups. This works great but any changes mean a BIND restart/reload and adding them manually can be a real issue if you have multiple DNS servers and you're not managing them via something like fabric/chef/puppet/etc.

I have taken it one step further. I use readily-available information from groups like ZeusTracker and Malware Domains, then I use the DLZ functionality in BIND, so that I only have to enter the domain into a database table and it immediately becomes active in BIND -- since BIND checks the database before checking its local zones and its cache, no restart or reload is necessary.

In THIS post, I'm detailing the first step: just setting up a recursive resolver for general use with some groundwork laid for the second step, using DLZs for blackholing lookups.

I'm going to use FBSD_8_3_i386_102 as the database master (see the previous post), FBSD_8_3_i386_103 as a recursive resolver with DLZ support so that I can blackhole domains and FBSD_8_3_i386_104 as a test client. Because of the way networking works in VirtualBox, some changes need to be made to the virtual machine that is acting as the BIND server so that it can access the Internet but still serve DNS to the internal network.  Under Settings -> Network, I'm going to click "Adapter 2" and add a NAT interface. This lets me use Adapter 1 to communicate with the other virtual machines in the psql_test internal network and Adapter 2 to communicate with the world.

Now fire up the three FreeBSD virtual machines. On FBSD_8_3_i386_103, edit /etc/rc.conf. Even though VirtualBox says "Adapter 2" is the NAT interface, that adapter is really the new em0 and the old em0 (with a static IP) is now em1. The two lines in /etc/rc.conf should look like:

ifconfig_em0="dhcp"
ifconfig_em1="10.10.10.103 netmask 255.255.255.0"

The netif script doesn't restart dhclient so reboot and make sure networking works.

BIND is included by FreeBSD in the default install but it doesn't have DLZ support. That means installing BIND from ports:

sudo portmaster -t -g dns/bind99

There are a couple of options to toggle at the configuration. I generally disable IPv6, enable DLZ, enable DLZ_POSTGRESQL and then have the ports BIND replace the system BIND:

There are some cons to replacing the system BIND with BIND from ports, namely that a freebsd-update affecting BIND will require a re-installation of the custom-built BIND.

The default BIND configuration is almost perfect for a general caching server so let's start with that. In /etc/namedb/named.conf, find the line that looks like:

listen-on        { 127.0.0.1; };

Comment out that line (lines starting with // are treated as comments).  This tells BIND to respond to DNS queries on any IP. I wouldn't use that configuration on a production resolver but it works fine for this scenario.

If your ISP requires you to use their DNS servers then you can set your BIND instance to "forward only". In that configuration BIND will still serve from its cache but it doesn't try to go out to the root and TLD servers - it just queries your ISPs DNS servers. To enable this configuration in /etc/namedb/named.conf, locate the following section:

/*
    forwarders {
        127.0.0.1;
    }
*/

Remove the /* and */ (those are block commenting delimiters) and change the IP to that of your ISPs DNS server. Then find and uncomment the line that looks like:

// forward only;

To enable and start BIND, edit /etc/rc.conf and add the line:

named_enable="YES"

Then run

sudo /etc/rc.d/named start

An easy test of whether it's answering requests is to run

dig @10.10.10.103 google.com 

Notice that it took a second or so to return anything. That was BIND loading the data into its cache. If you run that command again it should answer almost immediately. An authoritative resolver requires considerably more work but for a basic recursive resolver, that just serves DNS from its cache or makes look-ups on behalf on clients, the setup is fairly trivial.

Multi-slave PostgreSQL replication

I've recently had a rather large problem with an overloaded database server.  I had several machines with read-only access to a PostgreSQL instance but data was inserted via scripts on the local host and updated interactively via a web application. The scenario is fairly simple and a perfect use-case for multi-slave replication -- let the scripts and web application update the master, replicate to <n> slaves and have the servers with read-only access query the slaves.

This is incredibly easy to setup in a virtual environment, all the more so because the basic process applies regardless of the number of accounts that need access to the data, the number of replication slaves, the number of databases hosted on the master server, etc.

In the end I'll need at least two new virtual machines -- one to act as the database master and one to act as the slave. To keep it simple, I'll clone out the FreeBSD virtual machine. On Mac OS you can control-click the virtual machine and select "Clone":

When cloning, it's important to remember to reinitialise the MAC address of the NIC. This can always be done later but it's easiest to make sure that checkbox is selected:

A linked clone is a lot like deploying a new server but booting over the network from the hard drives of another server. It's best here to select "Full clone", this ensures a clean separation between the original and new virtual machines.

Once the new clone is ready you can spin it up and login (I used my 'demo' user from earlier). Since the master and slaves will need the same set of software installed, I can save a LOT of installation time by installing PostgreSQL and its dependencies *now* and then cloning that VM as the slaves. This can be done with

sudo portmaster -t -g databases/postgresql91-server

 Note this will prompt you to install the PostgreSQL-9.1 client, server and their dependencies:

Remember that portmaster will show you both what it plans on installing, as above, and then what it installed:

To set PostgreSQL to start at boot, edit /etc/rc.conf and add the line

postgresql_enable="YES" 

 This is also a convenient time to edit the hostname line and change it to reflect that this is a new virtual machine. I just changed the 101 to 102.

At this point I'm ready to shut down the virtual machine and make two clones that will be the slaves.  To isolate all of the PostgreSQL traffic I'm going to change the network type for the new VM to "internal network".  There is an option to set a network, I typed in "psql_test".

Note that now my virtual machine list has had FBSD_8_3_i386_102, FBSD_8_3_i386_103 and FBSD_8_3_i386_104 added to reflect the new master (102) and the two new slaves (103 and 104).

Start all three new virtual machines and login (again, I used my 'demo' user). Each VM will need an appropriate IP address added in /etc/rc.conf. The format of the entry is

ifconfig_<interface>="<ip_address> netmask <actual_netmask>"

In the case of FBSD_8_3_i386_102, the actual entry looks like this:

ifconfig_em0="10.10.10.102 net-mask 255.255.255.0"

For the other two virtual machines I used 10.10.10.103 and 10.10.10.104. The entire /etc/rc.conf for the master looks like:

Go ahead and reboot all three virtual machines at this point, just to make sure they'll come back up with their correct IP addresses and that each can ping the other two.

PostgreSQL does not create the necessary data directory and configurations at installation. It requires the user to run the PostgreSQL binary with the "initdb" option. To do this, use:

sudo /usr/local/etc/rc.d/postgresql oneinitdb

"oneinitdb" differs from "initdb" in that you don't need the line in /etc/rc.conf with "postgresql_enable" for "oneinitdb" to work. Here is it unnecessary but it is force of habit.

Now for the master configuration. Full PostgreSQL configuration is beyond the scope of this post, I just want to focus on the necessities to get replication working. First edit /usr/local/pgsql/data/postgresql.conf and add the following lines at the end of the file:

listen_addresses = '*'
wal_level = hot_standby
max_wal_senders = 2

This tells PostgreSQL to accept connections on any interface/IP and to allow up to two clients to pull the write-ahead log. max_wal_senders needs to reflect the number of slaves you are deploying.

Start and stop PostgreSQL so it creates the necessary WAL data -- this is a critical step or replication *will not work*. To do this, use:

sudo /usr/local/etc/rc.d/postgresql start

With PostgreSQL running, create a local admin user:

sudo -u pgsql createuser -e -E -P 

I named mine "demo". Test the new user and add a replication user, "rep_user", with a password of "reppass" and then exit the PostgreSQL client:

psql postgres
create user rep_user replication password 'reppass';
\q

To ease permissions for the next step, I'm going to set the password for the pgsql user to something I know. This can be done on each slave or, using the steps below, only on the master.

sudo passwd pgsql

Now stop PostgreSQL:

sudo /usr/local/etc/rc.d/postgresql stop

On each slave, copy the entire PostgreSQL data directory from the master. This allows PostgreSQL to start making updates on each slave using the streamed WAL (write-ahead log).

sudo -u pgsql scp -r pgsql@10.10.10.102:/usr/local/pgsql/data /usr/local/pgsql/

On each slave, edit /usr/local/pgsql/data/postgresql.conf and change the following lines:

listen_addresses = '*'
wal_level = hot_standby
max_wal_senders = 2

To:

wal_level = hot_standby
hot_standby = on

Then, on each slave, add /usr/local/pgsql/data/recovery.conf, with the following two lines (note: from "primary_conninfo" to "rep_pass'" is all one line, in case it wraps):

standby_mode = on
primary_conninfo = 'host=10.10.10.102 port=5432 user=rep_user password=rep_pass'

The master needs to be configured to allow the rep_user to connect from each slave. I added the following lines to /usr/local/pgsql/data/pg_hba.conf on the master:

host replication rep_user 10.10.10.103/32 md5
host replication rep_user 10.10.10.104/32 md5

Now, restart postgresql on the master and each slave with:

sudo /usr/local/etc/rc.d/postgresql start 

To test that replication is now active, on all three systems issue:

psql -l

Then, on the master, create a new database:

createdb demo

And verify on each slave that the new database was created:

psql -l

From here you can add databases, users, etc., as necessary. If you need to add more slaves (or rebuild a slave) after you already have a database server in production, no problem. Stop the PostgreSQL instance on the master, add the relevant IP to pg_hba.conf, configure the new slave as described above, start PostgreSQL on the slave and restart it on the master.

Setting up a test lab -- virtualisation is the key

One of the most common questions I'm asked is about what kind of lab I use to test.  Learning how to setup database replication, configure zones in BIND or the intricacies of tcpdump are all easier to do with multiple computers but, like many people, my desk, electric bill and wallet have no room for multiple computers so I rely *heavily* on virtual machine environments. At work I use my personal ESX licence from VMWare and a dedicated computer acting as the host for everything from Windows 7 and 8 to Debian, Ubuntu and FreeBSD virtual machines, but I don't have a piece of dedicated hardware for it at home. Instead, I rely on two products: VMWare Fusion and VirtualBox.

I love VMWare. I do. I'm guilty as charged. When I bought my new MacBook I bought three additional items: a Thule laptop sleeve, a Timbuk2 messenger bag and a "professional" licence for VMWare Fusion. The day it arrived I opened it up, took care of getting it on the network at work, re-ran the registration stuff for Apple...and then installed VMWare Fusion. The name of this blog isn't, "A peek in my Commercial Closed Source Toolbox", though, so that's about all I'm going to say about VMWare right now.

Enter the free competitor for running virtual machines on the desktop: VirtualBox. It's freely available and I run it on everything from FreeBSD to Gentoo Linux to Debian Linux to Windows 7.  When I needed virtual machines to give a six-hour introductory class to the FreeBSD command line, I created and exported the appliances using VirtualBox. When I wanted to make sure I could run the virtual machines for my SANS 503, 504 and 617 classes (I was a student, not an instructor) on my Debian Linux desktop, I converted them on a Windows 7 machine to VirtualBox virtual machines and then copied them to the Debian host.  So, for the purposes of this blog, my virtual environment is VirtualBox.

Which brings us back to the whole point of this post -- what does my lab look like? The honest answer is it changes. I generally rely on just a couple of different operating systems and then I clone out what I need and configure the individual environments, so let's run through creating a test lab from scratch. Since my current favourite environments are FreeBSD and Ubuntu Server, let's start there.

Step One: VirtualBox and the VMs

This one is simple enough.

http://www.virtualbox.org

They have a *fantastic* user manual with an introduction to virtualisation and how to create/interact with your virtual machines at

https://www.virtualbox.org/manual/UserManual.html 

They have put many, many hours into that documentation and it's orders of magnitudes better than anything I could come up with.

Just to keep it simple, I use the same configuration for nearly every virtual machine I produce:


o 1024 MB RAM
o 20.00 GB Hard Disk

o NAT networking by default (we'll change as needed)

o I name each VM based on the OS, the version, the architecture (i386 or amd64) and a number that I use for the last octet of static IPs

o I start building FreeBSD machines first and give them IPs starting at 101

o I start the Ubuntu or Debian VMs at 111

o If I'm including RHEL or CentOS I'll start those at 121

This post will NOT go in-depth on creating virtual machines. Instead, I'll touch a little on the key notions for each OS and leave the bulk of the installation process to the on-screen documentation, the VirtualBox documentation and the FreeBSD handbook. OS maintenance is beyond the scope of this post, I just want to give the reader an idea of where I'm starting and what they need to catch-up on before we get started in the really fun stuff (where I WILL go into deep detail on installation options, file edits, etc).

Step Two: FreeBSD

My current favourite server operating system is FreeBSD. Since I occasionally like to make virtual machines available and don't want to ever run the risk of running afoul of licence issues, the BSD licence suits me perfectly (do what you want with it, I'm not responsible for what you do, just leave the licence header wherever you see it).  I use it for routers, firewalls, web servers, database servers, intrusion detection sensors, you name it. The selection of software from "ports" is fantastic for my needs and a "base" install makes a solid DNS server with ZERO additional software.  I'm not going to step through creating the virtual machine, that takes the fun out of it, but I will give a couple of pointers in case you've never done a FreeBSD installation.

You can get information about how to download it via either BitTorrent or FTP at

http://www.freebsd.org/releases/8.3R/announce.html

9.0 is the current production release and 9.1 is coming out soon. I personally loathe the installer in 9.0 and love the installer in 8.3, plus everything I maintain is still on 8.3, so I'm sticking with it.

Gotchas:

1) FreeBSD has a notion of "slices" and "labels". These are revered and every FreeBSD admin from the last ten or more years has had to learn how they differ from "partitions" that are in the Windows and Linux worlds. Since this VM will be devoted entirely to FreeBSD, at the "FDISK Partition Editor" you can safely use the entire disk (option A, then Q to proceed)

2)  I like to use 2GB of the disk for swap, then the rest for the OS. On a real server we'd break out into multiple labels but for our purposes, this works just fine

3) For my "template" VM I always do a minimal install and then choose "custom" and add the man pages

4) When you add your "regular" user, make sure to add them to the "wheel" group. I added a user called "demo" with an empty default group, wheel as the additional group and /usr/home/demo as the home directory.

5) After the installation, it's important to setup a few necessary tools. Almost everything we'll install will be done via what they call "ports".  See the FreeBSD Handbook at

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/

In five years I haven't seen an issue about installation, or question about FreeBSD in general, that wasn't answered in the Handbook or added to it.

We'll need a tool to manage the installed ports. I prefer portmaster. After the installation completes and the system reboots, login as root and:

portsnap fetch extract
cd /usr/ports/ports-mgmt/portmaster
make install clean

Additionally, we're going to need "sudo"  so the user you created can run commands as root without having to actually use the root account. Now that portmaster is installed, we can use it to install sudo:

/usr/local/sbin/portmaster -t -g security/sudo

This command tells portmaster to install the sudo program from the "security" group of software in the ports tree, to recursively handle any dependencies and to create an installation package in case we want to install the same software on another machine without having to compile it from source.

The default options for everything it prompts are okay; you can continue through each selection by hitting enter.

Just before portmaster installs anything, it tells you what it's going to install:

Once it finishes compiling it shows you what it just installed:

Again, full installation instructions for ports are outside of the scope of this post, the FreeBSD Handbook truly is a fantastic reference.

Step Three: Ubuntu Server

Definitely use the "Long Term Support" (LTS) version available from the Ubuntu project at

http://www.ubuntu.com/download/server

I like stable, and 12.10 wasn't available when I started running Ubuntu Server, so I'm sticking with 12.04.

I've hated on Ubuntu for a long time. Recently I was working on deploying a piece of software called ELSA and is developed and tested on FreeBSD and Ubuntu. I was trying to standardise on Debian at the time and deploying ELSA on Debian just wasn't happening without modifying or augmenting the system perl. I moved to Ubuntu Server and haven't looked back.

There really are no gotchas for Ubuntu. Installing it is about as simple as it gets in the Unix or Linux world without buying it pre-installed. However, for the sake of consistency, a few necessary items.

1) I still use 2GB swap and the rest for /. I've not had good performance with ext4 on hardware and I'm comfortable with ext3 so that's how I setup the disks. To do that you have to use manual partitioning.

2) Since I'm going to need a stable environment, and I'm going to want to *always* know *exactly* what is installed on the system, I disable any form of automatic updating in favour of using the built-in command line tools.

3) I also like to start from a clean slate. Too many years of running certain RPM-based Linux distributions have taken their toll on trying to use "software groups", and since it's our template we want only the bare minimum necessary to build any type of server, so I don't install any additional software

Perhaps my favourite thing about Ubuntu over its parent distribution, Debian, is the message you get at login. You get all sorts of useful information: how many updates are available, disk usage, etc

Now we have two different operating system environments and we're set to start rolling out new virtual machines based on those templates. Again, check the documentation from the VirtualBox project. They cover cloning here. Generally, though, you're going to want two or three copies of a VM - one that you can use as a template and then a couple that can act as clients/servers/test beds. I'll make two copies of each of these so I'll have an entire template/client/server set for both FreeBSD and Ubuntu Server, and I'll almost certainly add a Kubuntu (Ubuntu with KDE) VM to mimic someone using Linux in a Desktop capacity. When I get to posts on ELSA, it's too handy having a graphical web browser to NOT spend the fifteen minutes installing Kubuntu in a VM.