Sunday, 26 February 2017

Farming ELK Part One: A Single VM

My road to ELK was long and winding. Let's just say I was very anti-ELK for a very long time but, over the last year, that opinion has changed *significantly*, and now I am one of the biggest ELK supporters I know. The unification work done by Elastic, the addition of granular permissions and authentication, the blazing query response time over large data sets and the insight provided by the beats framework have really won me over.

But don't take MY word for it.

The VM

To make things simple, I want to start my ELK series on an Ubuntu virtual machine. It's well-supported by Elastic and you can use apt for installation/updates, so it uses the native package manager.

This is a pretty basic VM, I've just cloned my UbuntuTemplate VM. 4 CPUs, 4GB of RAM and 10GB of disk. While it is NOT what I'd want to use in production, it is sufficient for tonight's example.

If you're saying, "but wait, I thought the UbuntuTemplate VM only had one CPU!" then yes, you're correct! After I cloned it into a new VM, UbuntuELK, I used VBoxManage to change that to four CPUs:

Next I needed to increase the RAM from 512MB to 4GB. Again, an easy change with VBoxManage!

Then I changed the network type from NAT (the default) to 'bridged', meaning it would have an IP on the same segment of the network as the physical system. Since the VM is running on another system, that lets me SSH to it instead of having to use RDP to interact with it. In the snippet below, "enp9s0" is the network interface on the physical host. If my system had used eth0/eth1/eth<x>, I would have used that instead.

The more I use the VirtualBox command line tools, the happier I am with VirtualBox!

I didn't install openssh as part of the installation process (even though it was an option) because I wasn't thinking about needing it so I DID have to RDP to the system and install/start openssh. That was done with:

sudo apt-get install ssh
sudo systemctl enable ssh
sudo systemctl start ssh

Then I verified it started successfully with:

journalctl -f --unit ssh

Dependencies and APT

As I said, you can install the entire Elastic stack -- logstash, elasticsearch and kibana -- using apt, but only elasticsearch is in the default Ubuntu repositories. To make sure everything gets installed at the current 5.x version, you have to add the Elastic repository to apt's sources.

First, though, logstash and elasticsearch require Java; most documentation I've seen use the webupd8team PPA so that's what all of my ELK systems use. The full directions can be found here:

but to keep things simple, just accept that you need to add their PPA, update apt and install java8.

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

Since Java 9 is currently available in pre-release, you'll get a notice when you add their repository. When you do the install you'll get a prompt to accept the Oracle licence. There is a lot that happens with the above three commands so don't be surprised when you see a lot of things happen after each one.

Now for the stack! You can go through the installation documentation for each product at:
First, add the Elastic GPG key (note piping to apt-key just saves the step of saving the key and then adding it):

wget -qO - | sudo apt-key add -

Now add the Elastic repository to apt (alternatively you can create a file of your choosing in /etc/apt/sources.list.d and add from 'deb' to 'main' but I like the approach from the Elastic documentation). This is all one command:

echo "deb stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

Update apt:

sudo apt-get update

Now install all three pieces at once:

sudo apt-get install logstash elasticsearch kibana

I had about 170 MB to download so it's not exactly a rapid install!

Now make sure everything starts at startup:

sudo systemctl enable logstash
sudo systemctl enable elasticsearch
sudo systemctl enable kibana

Make Sure It All Chats

By default, logstash doesn't read anything or write anywhere, elasticsearch listens to localhost traffic on port 9200, kibana tries to connect to localhost on port 9200 to read from elasticsearch and kibana is only available to a browser on localhost at port 5601. Let's address each of these separately.


Since elasticsearch is the storage location used by both logstash and kibana, I want to make sure it's running first. I can start it with:

sudo systemctl start elasticsearch

Then make sure it's running with:

journalctl -f --unit elasticsearch

The last line should say, "Started Elasticsearch.".


Kibana can read from elasticsearch, even if nothing is there. Since I need to be able to access kibana from another system, I want kibana to listen on the system's IP address. This is configurable (along with a lot of other things) in "/etc/kibana/kibana.yml" using the "" option. I just add a line at the bottom of the file with the IP of the system (in this case,

Then start it with:

sudo systemctl start kibana

You can check on the status with journalctl:

journalctl -f --unit kibana

I decided to point my browser at that host and see if kibana was listening as intended:

So it is! This page also indicates elasticsearch can be reached by kibana, otherwise I'd see something like this (again, this is BAD and means elasticsearch can't be reached!):


For testing, I want an easy-to-troubleshoot logstash configuration. The simplest I can think of is to read in /var/log/auth.log and push that to elasticsearch. To do that, I added a file called "basic.conf" to /etc/logstash/conf.d/ (you can name it anything you want, the default configuration reads any file put in that directory). In that file I have:

input { file { path => "/var/log/auth.log" } }
filter { }
output { elasticsearch { hosts => ["localhost:9200"] } }

This says to read the file "/var/log/auth.log", the default location for storing logins/logouts, and push it to the elasticsearch instance running on the local system.

On Ubuntu, /var/log/auth.log is owned by root and readable by the 'adm' group, but logstash can't read the file because it runs as the logstash user. To remedy this I add the logstash user to the adm group by editing /etc/group (you can also use usermod).

With the config and group change in place, I started logstash:

sudo systemctl start logstash

Then I checked kibana again:

Notice the bottom line changed from 'Unable to fetch mapping' to 'Create'! That means:

  • logstash could read /var/log/auth.log
  • logstash created the default index of logstash-<date> in elasticsearch
  • kibana can read the logstash-<date> index from elasticsearch

Now I know everything can chat to each other! If you click "Create", you can see the metadata for the index that was created (and you have to do this if you want to query that index):

The First Query

Now that I've gone through all of this effort, it would be nice to see what ELK made of my logfile. I technically told logstash to only read additions to the auth log so I need to make sure it gets something - in my terminal window I typed "sudo dmesg" so that sudo would log the command.

To do a search, click "Discover" on the left pane in kibana. That gives you the latest 500 entries in the index. I got this:

Notice this gives a time vs. occurrence graph that lets you know how many items were added to the index in the last 15 minutes (defaults). I'm going to cheat just a little bit and do a search for both the words "sudo" and "command" since I know sudo commands are logged with the word "command" so I replace '*' with 'sudo AND command' in the search bar (the 'AND' is important):

The top result is my 'dmesg'; let's click the arrow beside of that entry and take a better look at it:

The actual log entry is in the 'message' field and indeed the user 'demo' did execute 'sudo dmesg'! One of my favourite things about this entry is the link between the entry and the Table version -- this link can be shared with others so if you search for something specific and want to send someone else directly to the result, you can just send them that link! 

Wrapping It Up

This is a REALLY basic example of getting the entire ELK stack running on one system to read one log file. Yes, it seems like a lot of work, especially just to read ONE file, but remember - this is a starting point. I could have given logstash an expression, like '/var/log/*.log', and had it read everything - then query everything with one search.

With a little more effort I can teach logstash how to read the auth log so that I can do interesting things like searching for all of the unique sudo commands by the demo user. With a little MORE effort after that, I can have a graph that shows the number of times each command was issued by the demo user. The tools are all there, I just have to start tweaking them.

I do get it, though, it can seem like a lot of work the first few times to make everything work together. A friend of mine, Phil Hagen, is author for a SANS class on network forensics and he uses ELK *heavily* - so heavily that he has a virtual machine with a customised ELK installation that he uses in class. He's kind enough to provide that VM to the world!

He has lots of documentation there and full instructions for how to get log data into SOF-ELK. I've seen it run happily with hundreds of thousands of lines of logs fed to it and I've heard reports of millions of lines of logs processed with it.

Saturday, 18 February 2017

Headless VirtualBox Part Three: Bring on the Clones!

Today I wanted to work on a logstash grok statement and kibana dashboard for nmap output and, eventually, Qualys scans. The idea was that I wanted to have a dashboard that let me see any new hosts discovered on the network in the last twenty-four hours, new services that showed up on existing systems and new services on the network period.

That's great, I just need several VMs to scan and show up on the network. Since I recently made the FBSDTemplate VM, I can just clone it out. I did that on my headless Ubuntu system...but that's okay, because as it turns out, VirtualBox lets you clone VMs on the command line!

And, to no Unix or Linux user's surprise, it's faster than cloning via the GUI.


In the GUI you get a pretty pane that has a list of all your VMs. To see those at the command line is a simple command:

VBoxManage list vms

When you list your VMs, you get both the "name" of the VM and the ID of the VM. You can use either of these when using the VBoxManage command -- I like using the name because I don't have to copy/paste it.

Notice I have two VMs:

You can also show only your running VMs with:

VBoxManage list runningvms


In the GUI you'd need to select the VM you want to clone, use the hot-key or right-click and choose "Clone", then follow dialogues about whether you want to do a full or linked clone, whether you want to change the MAC address for network cards, etc.

Cloning at the command line, though, is a single command!

If I want to clone FBSDTemplate to a new DHCP and DNS server called FBSDNetworkServices, I can do that in one step. By default it will change the MAC address *and* create a full clone! When I created my clone I used the "--register" option so VirtualBox would be immediately aware of it.

VBoxManage clonevm FBSDTemplate --name FBSDNetworkServices --register

That's it, job done! All that's left is to start it.

Here is everything I did, start to finish:

The entire process, start to finish, took about thirty seconds *and I could do it over SSH*. I realise I'm coming late to the party, that tons of people have been using VBoxManage to do SSH-based management of headless VirtualBox servers for years. As with everything that's "old hat", there are always people being introduced.

That's where I am with VBoxManage and, on the whole, I'm chuffed to bits!

Friday, 17 February 2017

Headless VirtualBox Part Two: Scripting the Setup

In January I posted about using the VirtualBox command line tool "VBoxManage" to make a new virtual machine on a headless system ( Doing that for each one you want to make, though, gets tiring. It's a lot easier to script the process.

A few years ago I sat through a forensics class with a brilliant lad named Kevin. You can check him out at While our SANS instructor was telling a story about an experience he had with shadow copies, Kevin wrote a python script that searched a disk image for shadow copies and mounted each one into its own directory. By the end of the next break he'd added error checking and various other tidbits. That is not the depth of script I'm prepared to write.

Instead, I am content with a basic shell script that has some constants declared and then uses those to create a new VM.

Something Simple - Using What I Already Know

What I came up with was this:

In Plain Text

The text version (in case someone wants to cut/paste/edit) is:



HD_FILE="VirtualBox VMs/$VM_NAME/$VM_NAME.vdi" 


echo Creating VM
$VBOX_CMD createvm --name $VM_NAME --ostype $VM_TYPE --register

echo Creating HD
$VBOX_CMD createhd --filename "$HD_FILE" --size $HD_SIZE

echo Adding IDE Controller
$VBOX_CMD storagectl $VM_NAME --name "IDE Controller" --add ide --controller PIIX4

echo Attaching HD
$VBOX_CMD storageattach $VM_NAME --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium "$HD_FILE"

echo Attaching DVD
$VBOX_CMD storageattach $VM_NAME --storagectl "IDE Controller" --port 0 --device 1 --type dvddrive --medium $INST_FILE

echo Setting RDP Port
$VBOX_CMD modifyvm $VM_NAME --vrdeport $RDP_PORT

echo Enabling RDP
$VBOX_CMD modifyvm $VM_NAME --vrde on

echo Setting Memory Size
$VBOX_CMD modifyvm $VM_NAME --memory $MEM_SIZE 
echo Powering on VM
$VBOX_CMD startvm $VM_NAME --type headless
Basically, I just took the steps from my previous post about headless VirtualBox and replaced the VM info with constants. Note that this script creates a VM with a 10GB hard drive and 128MB of RAM. That is fine for FreeBSD but if you create an Ubuntu Server VM you want at least 512MB of RAM or the installer may fail. Guess how I know...

Now if I want to roll out an Ubuntu VM I can just make sure I have the install ISO, edit a few constants at the top, run the script and the new VM is ready for installation and listening for a VRDE connection on port 3389.

You can get the above script with:

git clone

Sample Output

When it runs, it looks a lot like this (notice I've changed the VM name from FBSDTemplate to OpenSecGeekScript):

Quick and easy!

In Closing

A more sophisticated script has its appeal - it would be nice to run a command and have it prompt for the VM name, a selection from a list of supported OS types, the amount of RAM, the hard disk size and even the ISO to use for installation. Perfect is the enemy of the good and, in this case, this is good enough for me. Well, it's good enough for a first run!

Sunday, 5 February 2017

Preparing a Drive With Secure Erase


American Football and The Super Bowl: Cleaning is More Interesting

Millions of people in the US today are getting ready to watch the Super Bowl. I don't particularly care about American Football, I'm much more of a rugby and, recently, hockey fan, and it looks like my region went straight from summer to spring so I thought I'd do a bit of cleaning today.

Which really means I started moving the Ubuntu system I have had on the desk in my living room and promptly got sidetracked.

In my last post I looked at setting up VirtualBox on that system so I could run some VMs without using the SSD in my laptops (despite my Windows 10 laptop having a separate 1TB spinning disk where I store the VMs...). Whilst moving the system today one of the side panels came off and I was reminded that I have two hard drives in that computer, one 250GB SSD and one 300GB with spinning platters, and I realised I've sort of gone about things all wrong.

I really like VMWare for virtualisation and they GIVE you their hypervisor; the Ubuntu system is currently headless and I use my laptops and table for everything, why not run their hypervisor on that system?

This is the problem with when I start to clean...I always get sidetracked on tech tangents. My train of thought went something like this:
I have a system where I want to install VMWare
That system has a spare drive I want to use
Drive re-use is pretty common
People make the mistake of thinking "deleted" means "gone"
I wonder how long a secure erase would take on that old drive


There are several tools that support the "SECURITY ERASE UNIT" command but this drive is already in a Linux system and hdparm is available -- and hdparm supports functions like placing a drive in High Security mode and secure erase. That's not to say it will always be successful - I've heard people talk about having it fail and secondhand reports of people saying THEY have "heard reports of it failing" - but I have yet to have a drive report it successful and then recover anything from the drive.

Best part - it is only two commands (but don't expect it to go quickly...)!


First, enable High Security mode by setting a password for the drive. In this case I'm going to use the password 'something' (my drive is /dev/sdb):

sudo hdparm --user-master u --security-set-pass something /dev/sdb

Note that once a password is set the drive can ONLY be used by entering the password. That means if you reboot the system YOU CAN NOT ACCESS THE DATA ON THAT DRIVE without entering the drive password.

Then kick off the secure erase:

sudo hdparm --user-master u --security-erase something /dev/sdb

When I did it on my Linux box, I also used "time" to see how long it took - since that was the question I really wanted to answer:

time sudo hdparm --user-master u --security-erase something /dev/sdb

On the actual system it looked like this:

So, about an hour and a half for it to run. Note this is a really old drive, a Maxtor 6V300F0, at least ten years old. On a modern SSD it can complete in seconds because it basically tells the drive to set everything to zero.

After finishing successfully, the password is automatically removed and the drive is in a clean, usable state.

Sum It Up

For years we have relied on "dd if=/dev/zero" to prepare a hard drive for reuse or when selling/donating a computer. It's not a 100% way to "wipe" the data from a hard drive but it's "good enough" in a lot of situations. In the same way, using something like BitLocker or FileVault2 to encrypt an entire drive, then reformatting the encrypted drive, can be "good enough".

It's not good enough, though, for drives that have sensitive data or for SSDs.

For those systems it's better to use something that can access all parts of the drive and that is designed, from the beginning, to actually erase the contents of a drive, regardless of filesystem, operating system or drive health (bad blocks, for instance). Commercial systems exist to do this but there is a viable alternative built into Linux. 

And, if you deal with PII or need to be absolutely certain, there are certified drive destruction companies out there that will turn your hard drive into confetti!

Sunday, 29 January 2017

Headless VirtualBox Part One: Creating a VM With SSH and RDP

For the duration of this blog I've used VirtualBox running locally. In 2012 and 2013 I ran it on a MacBook Pro. Now I run it on a combination of Windows 10 and that same MacBook Pro. That's well and good but having a lot of virtual machines can really fill a hard drive.

The organisation for which I work, and specifically the department in which I work, has some pretty serious funding problems. So problematic that in 2015 I bought two 27" monitors out of my own pocket to replace the two 19" monitors I had in the office. Later that year my monitor for my desktop at home died so I brought them home for a while. A few weeks ago I got tired of using the 19" again so I took the 27s back to work. That's left me with a desktop system at home that doesn't have any monitors and me wondering how best to use it.

Enter my comments about filling the hard drives in my laptops with virtual machine images! Since I have a machine with no monitors, I'm kind of curious about running VirtualBox "headless". It's certainly not as convenient as running the VMs locally but the hardware is available. I also have an entire shrimp, andouille and bleu cheese pizza, a few cans of Diet Coke and an otherwise free Saturday night so why not give it a shot?

Installing on Ubuntu 16.04.1 LTS

Since I use macOS at work and macOS and Windows 10 at home, I haven't installed VirtualBox on Linux in a while - in a very long while, as a matter of fact, since I've primarily used OS X (and now macOS Sierra) as my desktop OS since right after Snow Leopard came out.

VirtualBox has full download/installation instructions for Linux available at:

Since he box I have available is already running Ubuntu 16.04 LTS, I followed the instructions for Debian-based distributions.

Note: because the system doesn't have monitors, I'm going to do everything over either SSH or VRDE, the VirtualBox Remote Desktop Extension, which is basically RDP.

First, add the VirtualBox repository to apt:

echo 'deb xenial contrib' | sudo tee -a /etc/apt/sources.list.d/virtualbox.list

Download and add the Oracle public key for apt:

sudo apt-key add oracle_vbox_2016.asc

Then synch the package index files from the apt sources and install the latest version of VirtualBox:

sudo apt-get update
sudo apt-get install dkms virtualbox-5.1

NOTE!! You can't interact with VirtualBox over VRDE until after you've installed the extension pack *as root*. You can download it with (one line):


And then install it with (one line):

sudo VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-5.1.14-112924.vbox-extpack

Unless something has gone terribly wrong, it should be a fairly quick and painless process!

Creating My First VM

The first VM I want to add is FreeBSD - it's a light install and the only install I've done so far that's quicker is OpenBSD. First, download the ISO (one line):


Then create a new VM configuration with VBoxManage:

VBoxManage createvm --name "FBSD Test" --ostype FreeBSD_64 --register

I think a 5GB hard drive is much more than enough for a simple install but I'm going to add a 5GB disk, just in case:

VBoxManage createhd --filename "FBSD_Test.vdi" --size 5000

Live CDs and DVDs are usually treated as IDE devices so I'm going to add an IDE controller (one line):

VBoxManage storagectl "FBSD Test" --name "IDE Controller" --add ide --controller PIIX4

Attach the 5GB hard drive from above to the VM (one line):

VBoxManage storageattach "FBSD Test" --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium FBSD_Test.vdi

Attach the FreeBSD CD to the VM (one line):

VBoxManage storageattach "FBSD Test" --storagectl "IDE Controller" --port 0  --device 1 --type dvddrive --medium FreeBSD-11.0-RELEASE-amd64-disc1.iso

Make sure VRDE is enabled in the configuration:

VBoxManage modifyvm "FBSD Test" --vrde on

Then start the VM:

VBoxManage startvm "FBSD Test" --type headless

If everything has gone correctly, you should see a message similar to:

VM "FBSD Test" has been successfully started.

The entire process should look something like this. Notice I've split the long lines with " \ ", an operator in Unix and Linux that lets you split long commands across multiple lines - in this case, I've done it for visibility purposes so that each section is easier to read.

But do I have a proper running VM that's waiting to have FreeBSD installed from the ISO to a 5GB disk?

Remote Desktop

When I had a Linux (or FreeBSD) desktop, I used rdesktop to RDP to Windows hosts:

On OS X (and now macOS), I prefer the actual Microsoft Remote Desktop client available from iTunes:

The configuration for a new connection is fairly straight-forward, all you really need to provide are a connection name and PC name (this can be a FQDN, an IP address or an AD name). Note that VRDE uses the local PAM authentication on Linux so the username and password should be that of the user running VirtualBox:

Once I started the session I saw the FreeBSD install menu - at least a partial success!

I enabled scaling and then repositioned the window a little to get a better view:

To save you having to deal with screenshots up through the installation, I'll give a quick spoiler: it works!

NOTE: After the install completes, you may have to perform the equivalent of manually ejecting and removing a CD/DVD image. Despite rational convention, there is a storageattach function but no storagedetach - instead, you have to attach an empty drive. In this scenario I used "IDE Controller port 0 device 1" as my CD/DVD drive where I attached the FreeBSD ISO so now I need to detach it by attaching an "emptydrive":

VBoxManage storageattach "FBSD Test" --storagectl "IDE Controller" --port 0 --device 1 --type dvddrive --medium emptydrive


VirtualBox is really intended for personal use on the desktop and competes most closely with VMWare's Workstation/Fusion, Parallels and Microsoft's Hyper-V (don't be misled by that statement, Hyper-V certainly has a place in production virtualisation environments). If you need to run production virtual machines on a dedicated VM server there are multiple solutions available.

VMWare makes ESXi available for free:

If you're in a Linux environment you can base your VMs on KVM or Virtuozzo with OpenVZ:

Or, if you like Amazon's AWS, you can use Xen:

As mentioned, there's also Hyper-V:

That doesn't mean you HAVE to run VirtualBox locally, though - they have a complete command set available to do interesting things with virtual machine using VBoxManage that gives you a lot of power when coupled with access tools like SSH.

In truth I may never use it again - this is the first time in many years of using VirtualBox that I've done anything with a headless setup - but if nothing else it was an interesting way to spend a Saturday night!

Thursday, 26 January 2017

BIND and ELK: Or, How I Learnt to Stop Worrying and Trust in Grok

As I'm pretty sure I've said every time I discuss DNS, I like BIND. What I don't like, though, is BIND logging, and that caused a problem for me at work today when I wanted to import BIND query logs into ELK. Let's take a look at why...and then how to solve it!

Don't worry, I'm going to come back to doing basic BIND and ELK installs in other posts but right now I have an itch that needs to get scratched.

Example BIND Logs

First, let's take a look at what BIND's query log looks like. To do that, I've installed Ubuntu 16.04 Server in a VM and then installed BIND via apt. I made sure to enable query logging and did a few lookups with 'dig' and 'host', just to get a few different types of queries. Be warned that there are some gotchas around logging if you're going to try this at home - I'll address these when I do an actual post about BIND on Ubuntu later.

So what do they look like?

If you're familiar with syslog, that is definitely NOT a syslog message! They list no facility nor program name. Even if you decide to send the messages to syslog, you're left with mingling them with other messages (not that there is anything wrong with that, I just don't think it's very clean - I like have a single file that is JUST query logs). That's okay, though, right? After all, you can have syslog-ng or rsyslog pick the contents of the file up and ship it over to an actual syslog server and it's perfectly searchable with grep, perl, python, whatever. But if you want to ingest it in ELK, and be able to do interesting things with it, it's not that simple...

What Logstash Sees

To give you an idea of how logstash sees things, I've setup a VERY SIMPLE ELK installation. By "very simple", I mean I've installed everything on one VM running Ubuntu 16.04 Desktop and copied the query log to that system so I can feed it directly into logstash and have it display how it parses each line to standard out (to screen). Right now it's not trying to store anything in elasticsearch and I'm not trying to search for anything in Kibana, I just want to see how each line is parsed:

Notice how everything meaningful from our log entry is encapsulated in the "message" field - and that's not very useful if you want to do something interesting like determine how many unique IPs on your network have searched for a specific domain or which domains <this IP> looked up over <this time period>. To do that, we have to have the logs in a format elasticsearch can understand - and for that, I'm going to use grok.

Constructing a Grok Expression

Grok expressions can be daunting, especially in the beginning (I am very intimidated by them). While searching for something else I came across a great tool for building grok expressions:

I decided to use it to try to match a few lines from my query log:

If you want to try it yourself, here is what I pasted in:

25-Jan-2017 21:45:35.932 client ( query: IN A +E (
25-Jan-2017 21:46:05.665 client ( query: IN A +E (
25-Jan-2017 21:46:11.422 client ( query: IN A +E (
25-Jan-2017 21:46:35.125 client ( query: IN A +E (
25-Jan-2017 22:43:05.510 client ( query: IN A +E (

After a few iterations of choosing MONTHDAY, MONTH and YEAR, and then manually adding my on dashes in between, I had the timestamp for each line matched with the following expression:


And in the tool, it looked like this:

With a bit more work, I was able to work out an expression that matched all of the lookups I'd performed:

%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME} client %{IP}#%{NUMBER} \(%{HOSTNAME}\): query: %{HOSTNAME} IN %{WORD} \+%{WORD} \(%{IP}\)

That's a great first step! However, if I just use that in my logstash configuration, I still can't search by query or client IP because elasticsearch wont know about those. For that to work, I need to assign some names to the elements in my log entries (notice: this is done inside the grok filter):

%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME} client %{IP:clientIP}#%{NUMBER:port} \(%{HOSTNAME}\): query: %{HOSTNAME:query} IN %{WORD:query_type} \+%{WORD} \(%{IP}\)

Test Again

First I updated my very basic logstash configuration to have a "filter" section with a grok expression to match my DNS log examples:

And then restart logstash (note I'm calling the logstash binary directly so I can use stdin/stdout rather than having it read a file or write to elasticsearch):

Now logstash knows about things like "clientIP", "port" and "query"!


Okay, I know, it needs a little more work. For example, I need to work on a mutate filter that replaces the @timestamp field (notice that's the time the log entry was received by logstash, not when it was created in query.log) with an ISO8601 version of the time the log entry was created. I also need to add a custom field, something like "received_at", to make sure I capture the time the item was received by logstash. Both of those are exercises for another time and for anyone who may be following at home.

I also know the current filter is incomplete because I only tested it with A record lookups. Would the same filter work for PTR lookups? DNSSEC validation? All of my tested queries were recursive where EDNS was in use (the +E flag), will it fail on recursive queries with no flags? (Spoiler: yes, it will, that should be an optional flag field). Please do not copy/paste the above filters and expect them to be "production-ready"!!

I am certain there are other ways to solve this problem and there are almost certainly cleaner filters out there. I'm just starting to scratch the surface of what can be done with filters and with ELK in general, but knowing there are tools and methods available to ingest custom log formats (and normalise those formats into something elasticsearch knows how to index and search!) goes a long way towards having the confidence to build out an ELK deployment that your organisation can use.

Saturday, 21 January 2017

OSG 2.0 Setup: A Simple OpenBSD Router

The first VM for my lab is the OpenBSD router. OpenBSD has a phenomenal security track record, no additional software needs to be installed and you're guaranteed the latest and greatest pf. That last point is *especially* important to me because, IN MY PERSONAL OPINION, pf in FreeBSD sort of stagnated. That's not to say it's bad, it's just based on a fairly old version of OpenBSD's pf and doesn't support some of the things I really like (like the ability to dump the pfsync interface).

I'm going to skim over some very important things in this post. For example, this is not an introduction to learning the 'vi' editor, the basics of the Unix command line or an overview of using VirtualBox. There are tutorials and intros out there that are far better than I could cover the topics. My assumption is that you either know some Unix (or Linux) and networking basics or are interested enough to use your favourite search engine to figure out how to edit files, what certain commands do, etc.

With that, let's get started!

Getting OpenBSD

OpenBSD is available for download "for free". If you like OpenBSD, please consider either buying it or making a donation to the OpenBSD Project:

Disclaimer: I am not affiliated with the OpenBSD Project in any way other than as a happy user of their software.

If you choose to download OpenBSD, you can select your download location from their list of mirrors:

I grabbed my ISO from the MIT mirror. VirtualBox is 64-bit OS friendly so I always use the amd64 port (quick link to the latest as of writing:

For production use, I would absolutely recommend verifying the SHA hash for your download!

A New VirtualBox VM

The first step is to create a VM for the router. This is pretty straight-forward. VirtualBox is not just 64-bit friendly, if you choose BSD as your virtual machine type you can choose "OpenBSD (64-bit)" as your version!

For this scenario I'm going to accept all of the defaults. This means very meager resource allocations - but that's okay, it's enough for this purpose. This means a virtual machine with 64MB of RAM and 2GB of disk. Very meager indeed!


For almost any VM installation, you'll use the ISO file as a CD/DVD. This is done through the "Storage" section of the VM settings -- just choose the installation ISO as the optical drive and start the VM:

As soon as the VM boots you'll have the option to begin the installation:

The installation process is very quick and very straight-forward, especially if you are doing a "vanilla" installation. Advanced users will want to do interesting things like changing partition sizes but I will, once again, accept all of the defaults.

The one exception is that during the install process you may receive a message similar to, "Directory does not contain SHA256.sig. Continue without verification?". It is safe, in testing, to say "Yes".

When the installation completes you can send the VM the power off signal.

Making a Clone

I am a stickler for having a "reference" installation that I can keep updated, basically a template version of each operating system that I can use to make copies so I don't have to do a new install each time I want to try something out. Since I need to make some quick changes to this VM before booting and configuring the router anyway, this is a good time to make a full clone.

First, make sure to remove the OpenBSD install ISO as a storage device (Storage under the VM properties)!

To make the clone, just right-click the VM and choose "Clone", then select "Full Clone". You may not even see the progress meter since the VM is so small. I named mine "OpenBSD Router".

Two Networks

Since the clone VM is going to be a router, now is the time to add the second network interface. This is done through the Networking section of "Settings":

NB: by default, the "Name" field may say "intnet" - I clicked the box and named it "LabServerNet" since the first network I want to add is for my server VMs!

Make it a Router

The OpenBSD project has some fantastic documentation regarding configuring OpenBSD to be a router. That documentation can be found at:

It is not my goal to duplicate that. Instead, I'm going to do the bare minimum to have a basic router that meets my needs. To that end, everything I do below is stolen from that page so please please PLEASE read their documentation -- aside from fantastic software, the OpenBSD project has some of the best, if not THE best, documentation available. Use it!

First, you should have an em0 interface that has an IP address:

You should also have an em1 interface that does NOT have an IP address:

This naming scheme is perfect, in my opinion. It is easy for me to remember that em0 is the outside interface and em1 is the inside interface (0 == outside, 1 == inside).

Next I want to make sure the internal interface has an address. For some reason I'm partial to as my internal network so that's what I'll assign to that interface:

ifconfig em1 netmask up

Checking ifconfig for em1 should now be more interesting:

To make this persist through a reboot, you need to make sure there is a file in /etc that corresponds to the interface. This is specific to OpenBSD and makes configuring an interface very straightforward (certainly more maintainable, in my opinion, than something like /etc/networks/interfaces!). For em1, that means creating /etc/hostname.em1 and making sure the configuration for em1 is added. You can do this with a one-line command!

echo 'inet' > /etc/hostname.em1

This means, "assign the IPv4 address, with a netmask of and broadcast of, to the network interface labelled em1".

To make sure OpenBSD knows to forward IPv4 traffic between interfaces, enable IP forwarding in /etc/sysctl.conf. Again, this is a one-liner!

echo 'net.inet.ip.forwarding=1' >> /etc/sysctl.conf

This setting will take effect after you reboot the VM (but you don't need to do that yet!).

Note also that this time I used '>>' instead of '>'. If you're new to Unix, '>' will overwrite whatever is in the file (which is fine if you're creating a file) and '>>' will append if the file already exists (which is really what I find myself doing most of the time and is safer than using '>' if there is ANY question as to whether you need to keep what's already there).


By default, pf is enabled and running. You can verify this with

pfctl -s s

On a fresh installation, don't be surprised if you see connections to port 123 on a remote system. NTP, the Network Time Protocol, is active by default to make sure your VM keeps its clock up-to-date. Don't worry, this doesn't mean someone hacked your VM!

Configuring pf to NAT is a simple one-liner added to the end of /etc/pf.conf (note: I am assuming you know how to use vi to edit a file; if you don't, you want to learn the vi basics since it's found on almost all Unix installations and many Linux installations by default):

pass out on em0 inet from em1:network to any nat-to em0

What does that say?

Basically, that says, "let IPv4 traffic from any IP address in the same network as em1 go to anywhere and make it have a source address of em0". Again, please read the OpenBSD documentation for PF and NAT. The actual NAT FAQ for pf can be found here:


At this point you should be able to reboot and test the router. This means you'll need another VM on the LabServerNet network and configured to use as its gateway. A quick and easy way to do that is to clone the original OpenBSD VM to another clone just to use as a test VM. I cloned the VM to "OpenBSD Test", set the first interface to use LabServerNet and boot it up, then configured em0 to use the new router with:

ifconfig em0 netmask up
route add default

Then I tested everything by doing a DNS lookup for against Google's DNS servers:

dig @

The whole thing looked like this:

On the router itself, I logged in and checked the pf state table with "pfctl -s s". This returned:



There is a LOT of really good documentation from the OpenBSD project, please read it. They cover configuring pf and routers much, much better than I can. I just wanted to hit the highlights for a very, very basic router (the "quick and dirty" setup). As a Linux sys-admin I was very fond of ipchains, then of iptables/netfilter, but I have always been partial to pf on OpenBSD and FreeBSD.

With the router ready, I can start building out the rest of the lab!