NSM With Bro-IDS Part 5: In-house Modules to Leverage Outside Threat Intelligence

Unless you just do not follow the InfoSec world or main-stream media, the odds are pretty high that you've heard about the Mandiant APT1 report available at

http://intelreport.mandiant.com

There is some awesome information in that report and they've provided a ton of data in the form of Indicators of Compromise, or IOCs. It wasn't too long afterwards that Seth Hall released a script module for Bro IDS that incorporated some of those indicators. That module is available here:

https://github.com/sethhall/bro-apt1

Within a couple of days, Symantec released their own "Comment Crew: Indicators of Compromise" report. That report can be found here:

http://www.symantec.com/content/en/us/enterprise/media/security_response/whitepapers/comment_crew_indicators_of_compromise.pdf

To my knowledge, nobody has released a Bro module that incorporates any of that data and I wanted to use it -- so guess what I did tonight?

Good Programmers Borrow, Great Programmers Steal

Since I've never written a custom module for Bro, and because I had already started using Seth's module, I decided that imitation was the greatest form of flattery and decided to use his module as a template for my own. After all, this guy eats, sleeps and lives Bro. If you're going to follow anyone's lead on writing a module for it, whose code would be better for re-use?


I used the default location for my bro install, /usr/local/bro, so Seth's module is installed in /usr/local/bro/share/bro/site/apt1. That means any custom modules will go in /usr/local/bro/share/bro/site, so I went ahead and changed to that directory and copied Seth's module to a new directory called "sccrew" (Symantec Comment Crew):

cd /usr/local/bro/share/bro/site
sudo cp -prv apt1 sccrew
cd sccrew

I saved all of the domains in the Symantec report to a plain text file - one domain per line, each domain in quotes and all but the last with a comma at the end of the line. That means the file looks something like this:

"domain_0.com",
"domain_1.com",
"domain_n.com"

Some stuff you can toss, some stuff you keep...

The README.rst file and .git directory are both going to be useless for this purpose so you can go ahead and remove them, leave them, it doesn't matter, but it's good practice to go ahead and remove them.

The other files are important. data.bro contains the actual indicators. If you view it in vi, you'll see Seth's comments, the module name, a list of hashes, a list of domains, etc. I removed everything in the file EXCEPT the domain block, then deleted the existing domains and replaced them with the domains I saved from the Symantec report. Then I changed the module name from APT1 to SCCREW and saved the file.

__load__.bro basically "includes" the data.bro and main.bro files. Leave it untouched.

main.bro controls the actions to take. I modified the APT1::Domain_Hit to be a SCCREW::Domain_Hit and removed the other notice types. I changed the module name to SCCREW, removed the x509_certificate and http_message components and changed anything in the dns_request section from APT1 to SCCREW. A little tweaking of the message that accompanies alarms to indicate it's from the Symantec report, not Mandiant's APT1, and the file was ready to be saved.

From there it was trivial to edit my /usr/local/bro/share/bro/site/local.bro and add the magic line that calls the sccrew module:

@load "sccrew"

Update the Running Instance!

Then I told bro to install the new configuration and update its running configuration:

sudo broctl check
sudo broctl install
sudo broctl update

And with that, boom, I had a module loaded that would alert based off of the Symantec indicators.

You can view the actual module here:

https://github.com/kevinwilcox/bro-sccrew

Many, many thanks to Seth for providing a great template to use for in-house modules and to Symantec for releasing a fairly small dataset that served as a great tool for looking at how to create a custom module for DNS alarms!

MySQL and ELSA - When Your Storage Runs Out

There has been recent discussion on the ELSA users mailing list about variations on the following scenario:

o SysAdmin (SA) has a single large drive
o SA installs Linux or BSD, MySQL and ELSA
o Something consumes all free space
o SA realises they need to to move ELSA and MySQL to another drive

I didn't think a lot about it when it came up the first time. After it came up multiple times, though, I figured I would write up some instructions, starting with item three from above -- something fills up the hard disk.

The Environment

I already have a running ELSA installation on Ubuntu Server, and I configured it with a single drive, so all of the setup is already done. If you're following along with some of my earlier posts, the exact VM I'm going to work with is Debian_121_elsanode:

I know that I'm going to fill my drive, and I know I'm going to have to add a drive, so I'll go ahead and do that before I boot the VM. In this case I'll add an 8GB data drive called, "Debian_121_elsanode_mysql_data". The process is outlined in screenshots below.

In the VM settings, choose "storage" and then select the SATA controller. Click the icon for "Add Hard Disk":

I want to "Create new disk":

I like to use VMDK files if I think I might export the VM as an appliance:

I don't need to waste the time setting aside all of the space for the new drive, I won't have it long enough to use more than a couple of hundred MB -- and that's a stretch, odds are it's less than 100 MB:

Give the new drive an unique name inside of VirtualBox. "Create" will finally create the new drive:

When I booted the VM I verified Ubuntu picked up the two hard disks with:

dmesg | grep -e sda -e sdb

Note the drive sizes in the output - new drives are added in lexicographical order but it's always nice to verify you're working with the correct drive:

Prepping the New Drive

In reality, the new drive would get added *after* realising there is a storage issue. For the purposes of this demo, though, that's okay, it doesn't matter if the existing drive fills up and then I add the new one and move ELSA/MySQL or if I add the new drive, fill the old one and then move what I need.

So, let's go ahead and prep the new drive. I won't start moving data to it but I'll go ahead and partition and format it.

First, partition the drive with fdisk.

sudo /sbin/fdisk /dev/sdb

This starts fdisk. I can use 'p' inside of fdisk to show me the existing partitions for /dev/sdb:

To create a new partition that uses the entire drive, I'll hit 'n' for new, 'p' for primary, accept the default value of '1' for the partition number, accept the first and last cylinders, then use 'p' again to show the new partition:

Use 'w' to write the changes to the partition table and quit.

With the new partition ready, I need to format it before I can mount it. I prefer ext3, your mileage may vary. To format it in ext3, I'll use:

sudo /sbin/mkfs.ext3 /dev/sdb1

Note that when it runs, I get journal and superblock information. On physical drives, particularly large drives that aren't SSD, this can take a while to run.

One utility every Linux and Unix admin should use on a regular basis is 'df', for 'disk free'. I use the '-h' flag to get "human readable" output -- basically it just outputs all values in kilo-, mega- or gigabytes. It's  great for seeing, in short order, the amount of free space on a partition:

Fill The Old Drive

So I have about 16GB free on my root partition (where both ELSA and MySQL live). To quickly fill that up, I'm going to use a utility called 'dd'. My input will be /dev/zero (so I quickly get values - /dev/random and /dev/urandom can be considerably slower) and I'll output to a file called "consume_drive". Since I'm not giving 'dd' a count, it will run to completion - in this case, until the drive fills up.

Just to verify, I used the mysql command to connect to the locally running database instance, list the existing databases and try to create a new one (note it errors due to full disk):

Recovery Step One: Stop the Running Processes

At this point I would start receiving errors from Sphinx, MySQL and probably rsyslog saying I had disk issues. To kill all of them, I used killall with each process name and used the mysql init script to stop mysqld:

sudo killall -9 syslog-ng searchd perl
sudo /etc/init.d/mysql stop

Note that you DEFINITELY want to make sure searchd and elsa.pl aren't running, otherwise the system load can go through the roof when MySQL stops:

Recovery Step Two: Mount the New Drive, Move Data

With everything stopped, I mounted the new drive as /mnt using:

sudo mount /dev/sdb1 /mnt

I moved everything in /data to the new drive:

cd /data
sudo mv * /mnt/

Then I moved the MySQL database directory from /var/lib/mysql to the new drive:

sudo mv /var/lib/mysql /mnt/

Just for display purposes, the output of 'mount' and a directory listing of /mnt are included:

Recovery Step Three: Mount Point for the New Drive

Since I'm adding the new drive to house all of the MySQL and ELSA data, and I'd already decided to mount the drive as /data, I just need to add one line to /etc/fstab to reflect the new disk and mount point:

Recovery Step Four: MySQL Link

Since all of my MySQL data will live on /data, and I don't really want to fuss around with editing the MySQL configuration file to point to the new location, I'll create a symbolic link from /data/mysql to /var/lib/mysql (remember: hard links can't cross mountpoints, soft/symbolic links can) using:

sudo ln -sf /data/mysql /var/lib/mysql

Right now that location doesn't exist but it will on reboot, as long as the appropriate entry is in /etc/fstab and there are no filesystem issues.

EDIT -- 16 February 2013

I heard from Mike Miller at Miller Twin Racing that on machines with SELinux enabled, there are additional steps that must be taken. It turns out if you try to restart MySQL at this point then you get a failure, even though the symlink is in place:

If you look at the security contexts for /var/lib and /var/lib/mysql, you'll see that /var/lib has a context of var_lib_t and /var/lib/mysql has a context of mysqld_db_t:

There are multiple ways to solve this, the cleanest probably being to add a custom data_dir context that the mysql user can access/write. Since I am treating /data as an extension of /var/lib, a reasonable compromise for me was to give /data the same context as /var/lib and set the context for both /data/mysql* and the /var/lib/mysql symlink to that of the original /var/lib/mysql. This is accomplished with:

semanage fcontext -a -t var_lib_t /data
semanage fcontext -a -t mysqld_db_t "/data/mysql(/.?*)"
semanage fcontext -a -f -l -t mysqld_db_t /var/lib/mysql
restorecon -Rv /

Again, thanks to Mike Miller at Miller Twin Racing for that heads up and the correction!

Recovery Step Five: Reboot

Yes, really, reboot. You can remount /dev/sdb1 as /data to test but this is a virtual, non-production environment and I've followed the steps in this blog post a half-dozen times before actually writing it so I'm pretty confident of the outcome. On reboot /data gets mounted, the symlink for /var/lib/mysql is live and MySQL should be able to restart. You can verify all of this after reboot with a simple 'ps' and 'grep':

ps aux | grep -e perl -e syslog-ng -e sphinx -e mysql

If everything works, you should see output similar to:

Recovery Step Six: ... Profit

Okay, so maybe no profit, but you can rest comfortably knowing that you now have quite a bit of disk space available for your database and syslog needs, that you now can migrate services from a smaller disk to a larger disk with some basic Unix-fu and that you are almost certainly a better system administrator because of it!

NSM With Bro-IDS Part 4: Bro and ELSA, a Happy Couple

In part three of my Bro series I started pointing out how Bro can almost single-handedly transform the way you approach network security monitoring (and network monitoring in general). Gone are the days of wondering which of the 300 sites a potentially infected machine was trying to access on that Amazon virtual server, what jar file was requested by that piece of malware and whether Sys-Admin Jack has *really* enabled Remote Desktop connections from the world on one of the public-facing web servers under his care. To gather this data "out-of-the-box" requires at least a working knowledge of the Linux or Unix command line interface and a basic understanding of tools like grep, cut and awk. Bro can generate a mountain of data, kept in plaintext log files, in a very short amount of time, and searching through all of that text at the CLI can be a daunting task.

But wait a second...we have already solved this problem! Since we have a functional ELSA deployment already, and we have verified that we can send log data to it, it is trivial to configure a machine running bro to send its bro logs to an ELSA node via rsyslog.

rsyslog on Ubuntu uses the /etc/rsyslog.d/ directory for user-add and site-specific configurations. By default, the only items in that directory on the Ubuntu machine I've used for bro are for postfix, the "default" rsyslog configuration and a stub for ufw:

I added a file called "60-bro.conf" (the numbering is important, it reflects the order in which the files should be included) with the following contents:

##### Using local7 because that's where Martin put it
###### and it's a pretty standard usage
$ModLoad imfile #
$InputFileName /usr/local/bro/logs/current/ssl.log
$InputFileTag bro_ssl:
$InputFileStateFile stat-bro_ssl
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/smtp.log
$InputFileTag bro_smtp:
$InputFileStateFile stat-bro_smtp
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/smtp_entities.log
$InputFileTag bro_smtp_entities:
$InputFileStateFile stat-bro_smtp_entities
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/notice.log
$InputFileTag bro_notice:
$InputFileStateFile stat-bro_notice
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/ssh.log
$InputFileTag bro_ssh:
$InputFileStateFile stat-bro_ssh
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/ftp.log
$InputFileTag bro_ftp:
$InputFileStateFile stat-bro_ftp
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/conn.log
$InputFileTag bro_conn:
$InputFileStateFile stat-bro_conn
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/dns.log
$InputFileTag bro_dns:
$InputFileStateFile stat-bro_dns
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
# check for new lines every second
$InputFilePollingInterval 1
###### 10.10.10.121 is the IP of the ELSA node
local7.* @10.10.10.121

ELSA's author, Martin, has some pretty awesome documentation regarding setting up a bro instance, setting up a bro cluster and configuring bro to log to syslog. Yes, I lifted the above directly from there and added the section for bro_dns. Specifically, I recommend you take a look at:

http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html

The really big plus here is that this is not specific to bro - any process that writes a log file can have that log file sourced by rsyslog, which means any text log can be pulled into ELSA...but what practical application does that have for bro?

Why, I'm glad you asked! Let's contrive a situation based on my existing VM infrastructure. To review, I have the following in place:


o 10.10.10.115 -- bro
o 10.10.10.121 -- the ELSA node
o 10.10.10.122 -- the ELSA web front-end
o 10.10.10.150 -- a KUbuntu client (simulating <n> desktops)
o 10.10.10.254 -- the FreeBSD router

A Contrived Situation

It's no small understatement that network and InfoSec folks have to wear a lot of hats, and it's not uncommon for the two groups to be comprised mostly of the same individuals. The same person who gets called to handle the investigation of a server compromise may also have to help find a stolen or "misplaced" computer or smartphone, analyze traffic to find the machine spewing funky traffic into a VLAN or, my personal favourite, find out why in the world the bandwidth usage spiked at a certain time.

To set the stage, I used my KUbuntu VM and downloaded the latest stable Linux kernel from kernel.org. It's a pretty hefty download, just under 100MB, so it's great for demonstrating this.

Here's the scenario. You're the tech for an organisation with a big event coming up in two days and you have "all hands on deck". Ordinarily your Internet connections are sufficient for your business use but for the next few days everyone is in the office, basically working around the clock, and your Internet resources are stretched pretty thin. Because you are a fairly small organisation you don't have a networking group and a security group, you have...you.

A little after 2330 you get a phone call saying the network has gone to pieces. Everything inside the company network works fine, and now everything seems okay, but for a few minutes "the Internet" slowed to a crawl and you have to find and explanation because "they can't work when the Internet is that slow" and they're afraid it's going to happen again.

You fire up your VPN client, connect back to the office and, sure enough, everything is okay now. Speed tests show good numbers, the few servers you have look okay, everything looks good. With all the system and firewall logs showing nothing odd, and no alerts firing on the IDS, it's time to go to the network logs.

Step 1: Identify the connections

Bro has a fantastic log called conn.log. With bro now being pulled into ELSA (or parts of it, anyway), it's fairly trivial to search for all of the connections Bro saw between 2300 and 2330. Just set the appropriate start and end times, change the search type from "Index" to "Archive" and use the following in the search bar:

+class:bro_conn

So if the call came in at 2330 on 8 February 2013, your initial search might look something like this:

Note: if you do NOT change the search type to Archive, you MUST supply something to search for instead of just setting a filter. If you search in Archive mode, though, you can supply just a filter and no search string. Be careful, on a busy node you may get back far too much information to be useful -- the time selections or searching a specific node may really help.

In my case, the following was returned:

Unless you only get a few results back, this may not seem to hold much useful information, but look at the "Field Summary" - srcip, srcport, dstip, dstport and a host of others. In this case, I would want to know if I had any really "chatty" talkers or listeners, probably listeners since people were complaining about Internet speeds. It's possible that it's a talker but, in my experience, if speeds are going to crud then it's probably someone trying to archive the Internet and that means a large number of bytes in.

Step Two: Who's been downloading the big stuff?

To get a list of all of the bytes_in counters, just click on "bytes_in" in the field summary. ELSA will automatically present a chart giving the "bytes_in" values and how many times those values showed up in the given time period. For this scenario, I was presented with the following:

Here I get something hinting at an anomaly. Most of the values are in the tens or thousands of bytes but whoa, is that a value of over eighty-three million? That's certainly an outlier, let's take a look at it. Just click on the '83612841' and ELSA will use that as a required search item so your search gets restricted to items that a) are only in the bro_conn class and b) have a bytes_in value of 83612841:

That's fantastic! It looks like whoever is using the machine with IP 10.10.10.150 has downloaded a pretty big file at 2325. That alone is enough to probably explain the anomaly - everyone else's downloads or activity could certainly be impacted by the one person who thought they needed to grab a 100 MB file. So, was it legitimate? Well, you can certainly find out who's using it and ask them. If you have bro's http.log being sourced by rsyslog then that would actually have shown up on this search. I do not in this instance but it is a trivial addition.

Step Three: Using the raw log archive to see what was downloaded

If you don't have it sourced by rsyslog, and want to go back after-the-fact and check the log archive, you can do that pretty easily. Bro gives me the UID for that connection, it's the second value in the raw output of the log entry, in this case JRibhV6DJge, and if you remember from my part three, that UID stays consistent across all of the various log files. So, if I wanted to search for it in the archive, I could do the following:

cd /usr/local/bro/logs/2013-02-08
grep JRibhV6DJge http.23\:00\:00-00\:00\:00.log.gz

That gives me the following output:

And there's my answer - at 2325, someone downloaded the latest linux kernel tarball from kernel.org. In this instance a single  user can cause a problem with one big download. Even with a 1Gbps, a single user who gets really lucky can saturate the connection with a large download. If they're downloading a lot of small files it may be more sporadic but you can use some slightly different parameters to see which machines are downloading a lot of small files or which machines are making a lot of connections relative to the rest of the computers on the network. These are always pretty good places to start because they're basic steps, they don't seriously impact the network and they're easy to verify.

There are a lot of other things to do with ELSA and Bro, this is just scratching the surface. For more information I highly recommend the ELSA and Bro project pages at:

http://code.google.com/p/enterprise-log-search-and-archive/
http://www.bro-ids.org/