Wednesday, 21 November 2012

Setting up a test lab -- virtualisation is the key

One of the most common questions I'm asked is about what kind of lab I use to test.  Learning how to setup database replication, configure zones in BIND or the intricacies of tcpdump are all easier to do with multiple computers but, like many people, my desk, electric bill and wallet have no room for multiple computers so I rely *heavily* on virtual machine environments. At work I use my personal ESX licence from VMWare and a dedicated computer acting as the host for everything from Windows 7 and 8 to Debian, Ubuntu and FreeBSD virtual machines, but I don't have a piece of dedicated hardware for it at home. Instead, I rely on two products: VMWare Fusion and VirtualBox.

I love VMWare. I do. I'm guilty as charged. When I bought my new MacBook I bought three additional items: a Thule laptop sleeve, a Timbuk2 messenger bag and a "professional" licence for VMWare Fusion. The day it arrived I opened it up, took care of getting it on the network at work, re-ran the registration stuff for Apple...and then installed VMWare Fusion. The name of this blog isn't, "A peek in my Commercial Closed Source Toolbox", though, so that's about all I'm going to say about VMWare right now.

Enter the free competitor for running virtual machines on the desktop: VirtualBox. It's freely available and I run it on everything from FreeBSD to Gentoo Linux to Debian Linux to Windows 7.  When I needed virtual machines to give a six-hour introductory class to the FreeBSD command line, I created and exported the appliances using VirtualBox. When I wanted to make sure I could run the virtual machines for my SANS 503, 504 and 617 classes (I was a student, not an instructor) on my Debian Linux desktop, I converted them on a Windows 7 machine to VirtualBox virtual machines and then copied them to the Debian host.  So, for the purposes of this blog, my virtual environment is VirtualBox.

Which brings us back to the whole point of this post -- what does my lab look like? The honest answer is it changes. I generally rely on just a couple of different operating systems and then I clone out what I need and configure the individual environments, so let's run through creating a test lab from scratch. Since my current favourite environments are FreeBSD and Ubuntu Server, let's start there.

Step One: VirtualBox and the VMs

This one is simple enough.

They have a *fantastic* user manual with an introduction to virtualisation and how to create/interact with your virtual machines at 

They have put many, many hours into that documentation and it's orders of magnitudes better than anything I could come up with.

Just to keep it simple, I use the same configuration for nearly every virtual machine I produce:

o 1024 MB RAM
o 20.00 GB Hard Disk
o NAT networking by default (we'll change as needed)
o I name each VM based on the OS, the version, the architecture (i386 or amd64) and a number that I use for the last octet of static IPs
o I start building FreeBSD machines first and give them IPs starting at 101
o I start the Ubuntu or Debian VMs at 111
o If I'm including RHEL or CentOS I'll start those at 121

This post will NOT go in-depth on creating virtual machines. Instead, I'll touch a little on the key notions for each OS and leave the bulk of the installation process to the on-screen documentation, the VirtualBox documentation and the FreeBSD handbook. OS maintenance is beyond the scope of this post, I just want to give the reader an idea of where I'm starting and what they need to catch-up on before we get started in the really fun stuff (where I WILL go into deep detail on installation options, file edits, etc).

Step Two: FreeBSD

My current favourite server operating system is FreeBSD. Since I occasionally like to make virtual machines available and don't want to ever run the risk of running afoul of licence issues, the BSD licence suits me perfectly (do what you want with it, I'm not responsible for what you do, just leave the licence header wherever you see it).  I use it for routers, firewalls, web servers, database servers, intrusion detection sensors, you name it. The selection of software from "ports" is fantastic for my needs and a "base" install makes a solid DNS server with ZERO additional software.  I'm not going to step through creating the virtual machine, that takes the fun out of it, but I will give a couple of pointers in case you've never done a FreeBSD installation.

You can get information about how to download it via either BitTorrent or FTP at

9.0 is the current production release and 9.1 is coming out soon. I personally loathe the installer in 9.0 and love the installer in 8.3, plus everything I maintain is still on 8.3, so I'm sticking with it.


1) FreeBSD has a notion of "slices" and "labels". These are revered and every FreeBSD admin from the last ten or more years has had to learn how they differ from "partitions" that are in the Windows and Linux worlds. Since this VM will be devoted entirely to FreeBSD, at the "FDISK Partition Editor" you can safely use the entire disk (option A, then Q to proceed)

2)  I like to use 2GB of the disk for swap, then the rest for the OS. On a real server we'd break out into multiple labels but for our purposes, this works just fine

3) For my "template" VM I always do a minimal install and then choose "custom" and add the man pages

4) When you add your "regular" user, make sure to add them to the "wheel" group. I added a user called "demo" with an empty default group, wheel as the additional group and /usr/home/demo as the home directory.

5) After the installation, it's important to setup a few necessary tools. Almost everything we'll install will be done via what they call "ports".  See the FreeBSD Handbook at

In five years I haven't seen an issue about installation, or question about FreeBSD in general, that wasn't answered in the Handbook or added to it.

We'll need a tool to manage the installed ports. I prefer portmaster. After the installation completes and the system reboots, login as root and:
portsnap fetch extract
cd /usr/ports/ports-mgmt/portmaster
make install clean
Additionally, we're going to need "sudo"  so the user you created can run commands as root without having to actually use the root account. Now that portmaster is installed, we can use it to install sudo:
/usr/local/sbin/portmaster -t -g security/sudo
This command tells portmaster to install the sudo program from the "security" group of software in the ports tree, to recursively handle any dependencies and to create an installation package in case we want to install the same software on another machine without having to compile it from source.

The default options for everything it prompts are okay; you can continue through each selection by hitting enter.

Just before portmaster installs anything, it tells you what it's going to install:
Once it finishes compiling it shows you what it just installed:

Again, full installation instructions for ports are outside of the scope of this post, the FreeBSD Handbook truly is a fantastic reference.

Step Three: Ubuntu Server

Definitely use the "Long Term Support" (LTS) version available from the Ubuntu project at

I like stable, and 12.10 wasn't available when I started running Ubuntu Server, so I'm sticking with 12.04.

I've hated on Ubuntu for a long time. Recently I was working on deploying a piece of software called ELSA and is developed and tested on FreeBSD and Ubuntu. I was trying to standardise on Debian at the time and deploying ELSA on Debian just wasn't happening without modifying or augmenting the system perl. I moved to Ubuntu Server and haven't looked back.

There really are no gotchas for Ubuntu. Installing it is about as simple as it gets in the Unix or Linux world without buying it pre-installed. However, for the sake of consistency, a few necessary items.

1) I still use 2GB swap and the rest for /. I've not had good performance with ext4 on hardware and I'm comfortable with ext3 so that's how I setup the disks. To do that you have to use manual partitioning.

2) Since I'm going to need a stable environment, and I'm going to want to *always* know *exactly* what is installed on the system, I disable any form of automatic updating in favour of using the built-in command line tools.

3) I also like to start from a clean slate. Too many years of running certain RPM-based Linux distributions have taken their toll on trying to use "software groups", and since it's our template we want only the bare minimum necessary to build any type of server, so I don't install any additional software

Perhaps my favourite thing about Ubuntu over its parent distribution, Debian, is the message you get at login. You get all sorts of useful information: how many updates are available, disk usage, etc

Now we have two different operating system environments and we're set to start rolling out new virtual machines based on those templates. Again, check the documentation from the VirtualBox project. They cover cloning here. Generally, though, you're going to want two or three copies of a VM - one that you can use as a template and then a couple that can act as clients/servers/test beds. I'll make two copies of each of these so I'll have an entire template/client/server set for both FreeBSD and Ubuntu Server, and I'll almost certainly add a Kubuntu (Ubuntu with KDE) VM to mimic someone using Linux in a Desktop capacity. When I get to posts on ELSA, it's too handy having a graphical web browser to NOT spend the fifteen minutes installing Kubuntu in a VM.

So...I thought I'd Jump Into Puppet (Puppet Part One)

A Quick Note A lot of folks like to write about things they know very well and about which they can answer questions. Generally speaking ...