E-mail Alerts
I've been using bro at work and I really, really like the hourly emails it sends out with connection details. If you recall, I didn't install any type of MTA in part two. Before I jump too far into bro, I want to go ahead and install mailutils using:
sudo apt-get install mailutils
This results in the following:
apt/dpkg will go ahead and prompt for postfix configuration. Since I'm only delivering to my VM I'll choose "Local only"; for a machine that only sends mail (and doesn't receive), I would normally choose "Satellite system" or "Internet with smarthost":
To test whether you can send mail to local users, use the 'mail' command. So that I can put it into one line, I take the output of 'echo' and send it as the content of a mail message that is sent to demo@localhost:
echo "foo" | mail -s "test message" demo@localhost
'mail' is a pretty useful utility, you can also use it to read the current user's mail spool just by typing 'mail' and hitting enter. If you are presented with something similar to the following, congratulations, you can send mail to local users!
You can delete the message by typing 'd 1' (for 'd'elete message number '1') and then exit with 'quit'.
One Quick "gotcha" - Cron
I didn't realise it at first but, at least on Ubuntu, the bro installation process doesn't configure the system to start bro when the system boots and there are a couple of housekeeping items that bro needs to take care of on regular time deltas. Addressing this issues is as simple as
1) Adding a line to /etc/rc.local to start bro at boot via 'broctl start':
2) Adding a line to /etc/crontab to take care of housecleaning items every five minutes via 'broctl cron':
Bro Configuration Files
The bro configuration files can be found in /usr/local/bro/etc:
The first file I want to edit is node.cfg. There is an interface line in there that, by default, is set to eth0. To make sure I set the appropriate interface, I ran 'ifconfig' to see which interface didn't have an IP and then edited the file. The output of ifconfig and the modified node.cfg are below:
Note the "type=standalone" line. It is possible to run bro in a clustered mode, either with multiple hosts or multiple instances on the same host. Since I'm dealing with less than 5 Mb/s of traffic, it is pointless to configure multiple instances other than as an academic exercise. While I may address that configuration in another post, right now I want to keep the configuration simple so I'm running in standalone mode.
The next change is to networks.cfg. This file tells bro which networks are "local" and is used for determining incoming vs. outgoing connections. My psql_test network, the one using my FreeBSD VM as a router, uses the 10.10.10.0/24 IP space, so that's the only network I'm defining in networks.cfg:
The last file to edit is broctl.cfg. In production I would want to make a change to the MailTo line, add a custom MailFrom line and set the path for sendmail. I don't mind having emails go to root@localhost in a test environment so the only change I'm going to make is to add the path for sendmail:
Installing the Configuration and Starting bro
All of my groundwork is ready to come together. Bro's configuration files are ready, my FreeBSD router is pushing data over the span port and I have a handful of virtual machines ready to generate some traffic. Now to start bro!
The broctl tool is used for basic administration tasks. You can use it to start/stop bro, update the configuration, get stats for running manager/worker instances, get basic stats on the current traffic rate being monitored and more. First, start broctl using 'sudo broctl':
Note the prompt changed to '[BroControl] >'. To exit the broctl program, type 'quit' and hit enter. Don't do that yet, though, there are a few things to do first!
Since this is a new installation, I need to 'install' the configuration files. This is done by typing 'install' at the BroControl prompt and hitting 'enter'. I actually did that and hit Ctrl-L to clear the screen before I thought to take a screenshot so you'll get something *VERY SIMILAR* but possibly not *EXACTLY* like this:
Now use the 'start' command to start bro:
To get the status of the newly started process, use 'status':
At this point I switched over and started 'sudo apt-get update && sudo apt-get upgrade' on a couple of Ubuntu virtual machines and 'sudo portsnap fetch update' on a couple of FreeBSD virtual machines to generate some traffic, then I came back to bro and used 'capstats' to get some basic information on the capture interface:
Note that I'm only pulling about .8 Mb/second at less than 1,000 packets per second. That's fine considering my uber-slow DSL connection. In production I have virtual machines monitoring a couple of hundred megabits of traffic per second. It's pretty cool.
Carving the Logs (or, Going for the Scoobies!)
At this point you're possibly (probably...) thinking, "um...Kev...this is cool and all but...where's the payoff? Why have I gone through all of this?
Bro stores its logs (by default) in /usr/local/bro/logs. In that directory is a series of directories created by timestamp. An example directory would be /usr/local/bro/logs/2013-01-24. There is also a "current" directory, where the log files for the current bro process are kept. This is a symbolic link to bro's spool directory and every hour bro archives those logs to the appropriate directory. For this post I'll work entirely in /usr/local/bro/logs/current:
If you have been monitoring an active network then you should be able to run 'ls' and see several log files present:
Each of these files is interesting but there are a few that are particularly relevant to me. Recently I've been responding to a LOT of problems with outdated Java versions being exploited by exploit packs like Blackhole and CoolEK. I had written a Snort rule to look for Java versions up to 1.7.0_10 but I'm already taxing my Snort machines pretty hard. It would be really wicked if I could look for Java versions or wacky user-agent strings - Internet Explorer from a FreeBSD server or curl from a Windows server, for example. They aren't necessarily impossible, just generally unlikely. As it turns out, bro identifies the software it sees making network connections and logs it in software.log:
You can see that it identified 'portsnap' from one of my FreeBSD virtual machines, wget from a Linux virtual machine and 'rekonq' (the new version of Konqueror) from my KUbuntu virtual machine. If I wanted to see only portsnap connections I could have used:
grep portsnap software.log
If I wanted to look for Java as the user-agent, I could have used:
grep -i java software.log
See where this is going? By logging in plain-text, you can search the bro logs using any common command-line tool. Very cool!
Another interesting file is conn.log. This file keeps a list of all connections it sees, with data equivalent to that gathered from netflow. It provides a timestamp, IPs, ports, bytes transferred, packets transferred, duration of the connection and more. A complete description of the fields can be found in the bro documentation. The sample from my test shows several connections from my FreeBSD virtual machine:
Bro assigns an UID per connection and it uses the same UID for the same connection whether it is in the connection log, the HTTP log or the DNS log, making it trivial to gather information about a particular network conversation. If I pick one of the unique IDs from that table, for example if I choose 'AHNlJWaph34', I can search for any occurrence of that UID in any log with a simple grep:
This lets me know I not only had a connection between 192.168.1.76 and 192.168.1.254 on port 53, I can verify that it was indeed a DNS lookup for update5.freebsd.org and that update5.freebsd.org resolves to the IP address 204.9.55.80.
Other Logs
While I won't go into samples, there are many more useful logs bro creates out-of-the-box. If your network has any FTP servers you can monitor those FTP connections -- you can see which user is getting or putting which files, provided it is pure FTP and not SFTP, via ftp.log.Bro keeps track of any SSL certificates it sees and can do real-time validation of those certificates. The identifying information for these certificates is kept in known_certs.log.
As machines come on the network, bro logs their IP in known_hosts.log, so you can keep track of which machines are on at what time of the day. This is highly useful for trending and monitoring for deviations from the norm.
If your network has SMTP servers you can get much better logs in smtp_entities.log and smtp.log than what most MTAs provide out-of-the-box. IPs, recipients, senders, mail message IDs, subject lines, the names of any attachments and, optionally, a MD5 of the attachment, are all available.
One of the most frustrating things for me when doing incident response is trying to guess what file a machine downloaded just before Snort starts alerting me to botnet behaviour. Unless we are in an environment with mature IR and NSM programs, most of us have no reliable way of determining what domain the infected machine was trying to access at a given IP or, if HTTP, if the infected machine then tried to do a GET or POST and to which script at a given site. This can be particularly frustrating if the site is at a shared hosting provider like Amazon, Rackspace or one of a thousand Moldovan, Russian or Dutch web hosts. Bro addresses this huge problem via http.log, a wonderful resource for following the flow of web requests from a given host. As with dns.log, the same UID used in conn.log is used in http.log.
