Friday, 13 January 2017

OSG 2.0 Setup: What Does it Look Like?

I know that I want to look at several things in my test lab - FreeBSD servers, Ubuntu servers, Grouper, OpenLDAP, ELK + Kafka, SecurityOnion, CIF and others. To do this, each server needs to be able to communicate with the others. It also means each server may need Internet access to get the specific pieces of software I want to try.


There are a couple of options to accomplish this. I can have multiple network interfaces for each system, so that each server is multi-homed, but that makes it more difficult to look at things like SecurityOnion. The better way is to have each server connected to the same internal network and then let them connect to the Internet through a router. An even better way is to have segmented networks with desktops in one network, internal services in another and external services in another. The best way is to have multiple internal VLANs, similar to a properly segmented network, but I'm not going to go THAT far with least, I don't plan on it!

Instead, I'm going to start with a flat network - the type many small businesses will have. Servers and workstations will be in the same IP space, everything will go through an OpenBSD router and the router will send a copy of all Internet traffic to SecurityOnion. It will look like this...

*AHEM* A Small Clarification

One quick note on the router and SecurityOnion systems. The OpenBSD router would have three interfaces - one "internal", one "external" (to the Internet) and one that acts as a "span". The SecurityOnion system would have two interfaces - one for Internet access that sits on the "internal" network and a second to receive "span" traffic from the router. The "span" interface will NOT have an IP address and traffic will NOT be forwarded from one interface to the other.

But Is It Realistic?

This basic setup can scale for most small and medium-sized environments. You may have high availability routers, multiple Internet connections, Bro or Snort clusters and multiple VLANs behind the routers but conceptually the model is the same. Linux and BSD routers perform very well at multi-gig speeds and can perform NAT functions at hundreds of thousands of sessions per second for tens of thousands of devices. OpenBSD with pf and CARP is networking poetry in motion (and something I want to write about this time!). Do not be afraid to experiment!!

Thursday, 5 January 2017

Starting Over

When I left off a couple of years ago, I had a lot of stuff left unwritten. I had ideas about follow-ups to my DFIR post, I had ideas about follow-ups to my OSSEC post and I had ideas about a series on SecurityOnion.

Really, it was the DFIR follow-ups that put a halt on my writing. I have drafts that are three years old; I don't like their tone, I don't like their message and I don't like their content. Sometimes you have to know when to throw something away

So I've decided to scrap my drafts and, yes, start over with a clean lab.

The Environment 2.0

Over the last few years I have followed the trend that so many of my favourite projects have followed -- almost everything I have now (as far as security tools and servers) is running on Ubuntu Server LTS. It's a great place to start, I like it...but I really miss the BSDs. So, for OSG 2.0, I want to use OpenBSD and FreeBSD as often as possible. A lot has changed since I stopped using FreeBSD 8.3 (including a major overhaul to the package manager!) so this should be a good excuse to jump back in.

The other big change to the environment is that I've switched laptops! I loved my MacBook Pro and I still think they are brilliant machines -- I still opt to use one at work. Recently, though, I've converted to a Dell running Windows 10 Pro and I have to say I like it. That it comes with spill protection is a huge benefit - my MacBook Pro was dealt a mortal blow by a cup of diet grape juice one night and Apple does NOT offer spill protection.

Windows 10, VirtualBox and some BSDs...what could go wrong?

Wednesday, 9 April 2014

Heartbeat/HeartBleed - Resources Only

I'm not going to rehash everything that's been said about how the vulnerability works, why anyone should patch or what else they need to do - at least not yet.

This post is just a list of resources so I can keep my sanity.


The OpenSSL advisory:

An easy way to check for whether heartbeat extensions may be enabled:
openssl s_client -connect <server>:<port> -tlsextdebug | grep -i heartbeat

SANS Webcast covering the vulnerability (9 April 2014, 20.15 EDT):

SANS Webcast "part two" (10 April 2014):

SANS Webcast on *CLIENT* side exploitation:

Snort and Suricata rules:


An NMAP script to check if a server is vulnerable (note: originally only checked on of the three TLS versions):

A MetaSploit module:

Even better, a MetaSploit module for exploiting clients:

A scanner I've been really pleased with:

Another python-based scanner, checks all TLS versions:

*Another* python-based scanner, this time for checking your clients:

Another server scanner my co-workers have been really pleased with:

An excellent, fast server scanner called masscan now has heartbleed support:


Enter your server's <host>.<domain> to see if it is vulnerable:

This one, courtesy of Qualys, gives a really nice health check:

OS Advisories And Updates






Cygwin (okay, not a distribution, but they were second only behind Gentoo in patching)


Sunday, 12 January 2014

DFIR - Do We Or Don't We?

Before you start reading, be warned: this is a *very* long post.  It probably should have spanned multiple posts.

Last week I sat down with my CISO and had a chat about Google's incident response framework called Google Rapid Response, better known as "grr".  I explained that I had installed it on a couple of virtual machines, started testing out the web-based interface and pulling files from remote systems, and that so far it was REALLY cool but that I was having some problems writing a blog post about it.

When he asked why I told him because every time I tried I ended up with two paragraphs covering general digital forensics and incident response (DFIR) and that I didn't think I could jump into a good grr post without spending some time talking about DFIR.

"Have you considered doing a non-technical post?"

The obvious answer was no, I had not.  This blog started for me to document stuff I like to work on and so I have an easy-to-follow guide for potentially complicated installations or configurations.  I hadn't really thought about covering a "soft topic" for others.  I considered his question for about three seconds, tilted my head, grinned and thanked him for showing me how to jump into covering some DFIR tools.

It is simple, really.  You can't discuss DFIR tools without first discussing some of the DFIR basics.   Which tools do your DFIR team use?  How do you create your DFIR team?  Where do those tools and people fit into your DFIR process?  And, what I consider the most important of all, why do DFIR in the first place?

Each of these questions could be a chapter in a book - if you're an actual forensicator or responder you're probably ready to argue that they could each be their own book, and I agree with you! I won't spend quite that much time answering them but I do want to use this post to address the question that drives all others - to DFIR or not to DFIR?

DFIR - Do We Or Don't We?

Let's say you are a freelance IT consultant and you have several small businesses as clients.  Maybe you grew up and opted to stay in a small town so you do the IT work for friends that have become doctors, lawyers, dentists, dry-cleaners, fabricators, landscapers, plumbers, carpenters, publishers or any of the countless other types of small business owners out there.  Even though there are significant differences in each of these businesses, they all have some things in common, right?  At the very least:

  • They all have customers of their own
  • They all need to keep track of those customers
  • They all bill their customers for services rendered
  • They all buy business supplies
  • They all have computers in their businesses (or they wouldn't be your client!)
  • They all probably use the computers to track their customers and income/expenditures for taxes
  • They all may lose business if their business records are leaked to the public

On most days it is business as usual - maybe a broken printer, someone bought a new smartphone and they want to sync their email, any of the myriad of small fires that need to be put out.

Then one afternoon you get a phone call from Bob, the guy who runs the dry-cleaning service.  He explains that he was trying to open a video sent by an old classmate, Joe.  You and Bob went to school together and you remember Joe, so this sounds pretty legit.  Then Bob tells you that when he opened the video he had a warning that he needed to update his Flash player so he clicked "Accept".  After that, he saw a little black window pop-up then disappear and now he has a pop-up saying his anti-virus found 300 "critical infected files".

How do you respond?  Do you try to "clean up" the system and keep it in production?  Do you give him a temporary system while you spend days analysing a forensically-sound copy of the hard drive to determine what malware was downloaded and what it accessed on the system?  Do you have a network device in-line for his business that lets you analyse all of his network traffic for potential data exfiltration?  Do you advise Bob to file a report with local or regional law enforcement?

Is that advice the same as it would be for Elizabeth, the lawyer, if she had the same thing happen?  How does that advice change if the victim is the doctor?

If we're all being honest with ourselves then I think we can safely say that no, the advice will not be the same.  The doctor will have HIPAA concerns and may have intimately detailed customer records accessible from each system in his office -- the Federal fines could put him out of business if he can't show protected data wasn't exposed.  Bob may only have a first name and telephone number for each client - potentially unprotected information in his jurisdiction so he may not have to report a data breach even if his entire database were exposed.

Some clients will have different statutory and contractual requirements regarding investigation and reporting.  In some situations it may be completely acceptable to rely on anti-virus or anti-malware software to "clean" the system.  In some situations it may be acceptable to reformat and reinstall the operating system, what we call "nuke and pave."  Other situations may require intense, in-depth analysis by trained forensicators spanning several days, weeks or even months.  Larger organisations will have to take those same requirements into consideration when making the "do we or don't we" decision.

If You Do

Regardless of the scenario, you're almost certainly going to do something.  That "something" will be determined by the leadership of each organisation and those decisions will need to be well-informed.  It is easy to say, "yes, we're going to do forensics and/or incident response" but what if there is no funding for that decision?  What if there is nobody at the organisation trained in those areas?

At the very least, the people making the decision will need to understand:

  • Forensic hardware can be expensive
  • Forensic software can be expensive
  • This will be a long-term investment
  • This is about protecting the organisation's resources, reputation and clients
  • Results will not be immediate and can require days, weeks or months
  • DFIR is an evolving field, training is never-ending...and can be expensive

Once the decision is made to commit to DFIR, the practitioners should also have some basic understandings:

  • Forensic hardware will take training to use correctly
  • Forensic software will take training to use correctly
  • Both the hardware and software will require regular use to remain proficient
  • Processes will have to be documented, validated and then followed - every single time
  • DFIR is an evolving field, training is never-ending

It's not enough for an organisation to decide to do forensics and response, that organisation needs to understand at every level that this is a long-term decision.  It can be costly, it will require a human element, it will require training and it will require support.  A forensicator or incident responder can't perform those duties without the support of their employer.

Using DFIR To Enhance Security Operations

Pop quiz: what do SourceFire, Mandiant (pre-FireEye), FireEye and TippingPoint have in common?

Answer: they all use DFIR to enhance and improve their services.

In the same way they use DFIR to enhance the services they offer their clients, a big selling point for doing DFIR in-house is that it can be used enhance internal security operations - specifically, intrusion detection and analysis.

Consider the scenario where a responder, Alice, finds that a piece of malware sends a small UDP packet to a CnC server once every thirty-six hours.  Once Alice knows how that packet looks she can work with the intrusion detection folks to help them write a rule looking for any similar packets going across the network.  Alice can then work with the appropriate individuals to look for any previous occurrences of similar packets to help find other systems that may not have been noticed as compromised.

If Alice can determine specifics about the malware that landed on the system then she can work with the appropriate groups to determine if similar files may have been downloaded by other systems - either via analysis of network traffic or by starting an Enterprise-wide search for files with similar characteristics.  A group from Mandiant released a product called OpenIOC designed *specifically* so that responders could normalise these indicators of compromise and then search for them across multiple Windows-based systems in an Enterprise simultaneously.  Some software will allow you to search across multiple platforms (grr, for example, will search across Linux, OS X and Windows).

In these ways, the findings from DFIR are not only useful from determining whether a particular piece of data may have been accessed (or exfiltrated), the findings may be useful in determining how wide a given infection may spread or how deeply an organisation may be compromised.

Summing It Up

There are multiple factors that have to be taken into consideration regarding how to address digital forensics and incident response - hardware, software and training costs, time spent performing each analysis, time documenting and validating the response process, creating an agreed-upon incident response plan and more.  It can lead to better-trained employees with broader skills and it can be used to enhance other areas of the organisation's security operations but these benefits come at a cost.

Consider the following rant I have given a couple of times in the last few months:

Traditional InfoSec thought has been the trite castle and moat analogy, right? The barbarians are at the gates, screaming to be let in, but the bridge is up and at first they can't get across the moat.  Then they build a bridge across the moat but that can't breach the walls.  Then they push siege towers over the bridge to exit on top of the walls.  Then they fight wall-by-wall until they get into the heart of the castle.  We keep insisting firewalls (the moat) and VLANs (the walls) are enough to keep attackers out but we're delusional - the problem isn't the barbarians outside, it's the users inside.
What we really have to envision is a dark battlefield, dotted with foxholes.  We know there is a perimeter and we try to protect against anyone coming in on the ground with firewalls and IDS.  We try to monitor those parachuting in via authentication...but we're left with these thousands of foxholes all over the battlefield.  Each foxhole may be occupied by one of ours, one of theirs, both or neither, and the only way we can know is by gathering intelligence.  Without real-time intelligence from each host on the network we are essentially blind to that particular foxhole, meaning we don't know how that part of the battleground looks.
DFIR is a part of what gives defenders that "full view" of the battlefield.

Every organisation will have to weigh these benefits against the non-trivial costs and make its own decision.

Thursday, 19 December 2013

ELSA Authentication Using LDS

As a general rule, I'm not overly fond of using Active Directory or LDAP to authenticate for access to my logs. I take particular exception to having to rely on network connectivity, Active Directory availability and LDAP functionality to access logs that may help troubleshoot problems with network hardware, an AD cluster or LDAP problems. I would much, much rather my logging solution be self-contained.

That said, there has been a LOT of discussion the last few months on the ELSA users mailing list about how to configure ELSA for AD/LDAP authentication and I realised that I had absolutely no clue how to do it.

So I started envisioning a scenario...

The Scenario...

A not-so-long time ago, in a not-so-far-away land, four friends were sitting in a coffee shop discussing what they would do after graduation. They were all very anxious because their parents had given them horrid names: TSUser, MUser, SUser and ISUser.

Each of them had unique backgrounds.

TSUser had a strong technical support background.
MUser had a strong background in making videos and posters.
SUser had paid for his college degree by selling broken toys and rotten food to local stores.
ISUser had grown up reading Nancy Drew, Hardy Boys and John Grisham books. She always wondered how she could make things do interesting things that they weren't designed to do.

Over a particularly strong round of lattes, they decided they would forego the embarrassment of applying for jobs with their names and just start their own widget company. They wanted a company that would stand out above the rest, that would allow each of them to play to their unique strengths in technology, marketing, sales and security. They named their new company, "Test".

"Test" knew they would have computers and networking equipment and that that hardware would generate logs. To manage those logs, they decided they were going to deploy ELSA. "Test" knew they would need one group of administrators and everyone else would be a regular user.

"Test" has also decided that they want to have password policies and central authentication (TSUser and ISUser were pretty on the ball...) but they didn't want to have to worry about trying to manage group policy or domain stuff so they decided to use Lightweight Directory Services.

LDS Installation and Setup

I decided to start with a plain Windows Server 2012 install. You can test it out for six months at:

I'm going to assume you have a working 2012 installation and you can see the server Dashboard. Note my server is named "WIN12-LDS" and has an IP address of

Start the install with clicking "Manage" and "Add Roles and Features":

You can click-through the "Before you begin" page. You can also click the option on the bottom left to not show that page again:

LDS is a role-based service:

My server is named "WIN12-LDS" and does indeed have an IP address of "":

Choose "Active Directory Lightweight Directory Services":

As soon as you click it, it displays an option to install the management tools (you want to do this):

Clicking "Add Feature" will take you back to the "Server Roles" page - just click Next:

There is nothing on the next page that has any bearing on us so click through:

The next page is just an overview of LDS:

At the next page you can choose to restart the server after the installation finishes, if the reboot is necessary (for LDS it is not):

After the installation finishes you can either run the LDS setup wizard or close the wizard. Since you can run the wizard multiple times (you can have multiple LDS instances on different ports), I'm opting to close the installer:

From the Dashboard, you can now run the service configuration wizard:

The setup wizard gives another quick overview of LDS:

I went ahead and named this instance, "ELSA"

This is the first LDS instance on this server, it defaults to the "standard" LDAP port of 389:

To keep things simple, I went ahead and let LDS create the new "partition". Since the company is named "Test", I'm setting the Organisation as test (o=test):

Unless you're a power-user, you can go with the default location for the new data files. "ADAM" is the old name of LDS:

Since we're going to run LDS as a network service, I'm leaving it running as the network service account:

We get a quick warning because we're not using full-blown Active Directory to manage the local server. Yes, we want to continue:

I didn't create a separate user JUST for LDS management. You can certainly do so. I opted to keep it simple and let the Administrator user be the only administrator for LDS:

A Microsoft article I found on LDS deployment suggested importing the following three LDIF files:

The next page is a summary of the configuration wizard options:

And boom goes the dynamite, LDS is ready to have users and groups added to it:

Setting Up Our Organisation in An LDAP Structure

With LDS installed and configured, the next step is to make sure the "Test" organisation, its departments and their employees are setup. This is where we'll add the users, set their passwords and, for those with SOME LDAP experience, setup the various OUs.

Microsoft provides a great GUI tool for basic LDS management called the Active Directory Services Interfaces Editor, or "ADSI Edit". You can start it from the Dashboard, under the "Tools" heading:

When it starts, you'll need to specify an AD or LDS instance to manage:

Note that I chose, "Select or type a Distinguished Name or Naming Context". This lets us work directly with the "o=test" instance. Since we're not on an AD-joined system, we have to provide the server name (in this case, localhost):

"Test" should load. As you click through the ELSA/test instance, the base items created by the configuration wizard should appear. Initially I added the first Organisational Unit (OU), "InfoSec", directly under the "o=test" level:

When I thought about it a little, I decided I wanted to add an OU called "Company Domain". This let me put each department's OU under the "Company Domain" OU. I added "Company Domain" in the same way I added "InfoSec" and then moved "InfoSec" under that new OU:

I then added the next three OUs, one each for Marketing, Sales and Tech Support, directly under "Company Domain" using the same process as when I created "Company Domain" and "InfoSec". Notice how it looks with all of those OUs created:

Now that the OUs are in place, we need groups for the ELSA users and admins. I decided to put those groups under the "Company Domain" OU. First, select "OU=Company Domain" and right-click it, then choose "New -> Object":

We know we want to add a new group:

First, let's add "ELSA Admins":

You can follow that same process for any additional groups you want to create. I added another called, "ELSA Users."

With my OUs and groups created, now I needed users. Note that users are part of an OU - when adding my four scenario users, I'm going to add them under their respective departmental OU. I included the steps for InfoSec's "ISUser":

Users are added as disabled accounts with expired passwords and without any group membership. First, I'm going to reset the password for "ISUser." In this case I used a password of, "DNpass1". This will be the password I use for every user I create for this demo. Note that users and groups have a Common Name (CN), not OU:

With the password set, I can go ahead and enable the account and set some necessary values. All of those things are handled through the objects "Properties":

First, let's set an email address for the user:

Set the password to follow the expiration period (setting password policies is out of the scope of this document!):

Make sure a password is required for the user:

ELSA's authentication was originally written for full-blown AD, which has a sAMACCOUNTNAME attribute. LDAP and LDS don't have that attribute by default. LDS, though, has "userPrincipalName", and it needs to be set to the value for the user's username:

Make sure the account is enabled:

It's a little time consuming but it will need to be done for each user. Note that for large deployments you can import users with default values via various scripting methods (go go PowerShell!).

When you configure an application to authenticate via LDAP, you generally get a "bind" username and password. That username is used to connect to the directory (LDAP) and it needs read access to the directory tree. You'll use a different one for each service connecting to your LDAP. In this case I want it to have a descriptive name so I'm going to create a user called "elsadn". That user isn't going to be part of the company, it's purely utilitarian, so I'm adding it under "Roles":

Even though a person won't use this account, you'll need to set a password and the expiration/password attributes for this account just like the regular users.

Once you square that user away, you need to make sure it is part of the "Readers" group. Any account in the "Readers" group has read-access to the LDS instance and it's necessary for you to setup any application that requires a "bind dn" account:

The "member" attribute is used to add users to the group:

You can add by Distinguished Name (DN). For the "elsadn" user, that is


You'll add "ISUser" to the group "ELSA Admins" using the exact same process:

Note that the full DN for "ISUser" is

CN=ISUser,OU=InfoSec,OU=Company Domain,O=test

Follow the same process for adding each of the four users to the "ELSA Users" group. Once you've added the last one, the "member" attribute of "ELSA Users" should look like:

Congratulations, that finishes up the LDS component!

ELSA Configuration

The next step is configuring ELSA for authenticating against LDS/LDAP and identifying which groups are authorised to login (allowed_groups) and which are admins (admin_groups).

All configuration is handled through /etc/elsa_web.conf. The first item in that file that's relevant is "admin_groups". I went ahead and commented out the defaults because I want only the "ELSA Admins" group to have admin access.

The next item in that file is "allowed_groups". As with "admin_groups" I commented out the defaults and only allow the "ELSA Users" group to login.

With those two sections changed, they now look like:

Zooming in on just those two sections:

 The next section to edit is for authentication. Set the "auth" method to "ldap" (not shown). If you've been following my ELSA posts, and following those instructions, you'll notice it's set to use "db" for authentication. Change that to "ldap".

Next you need to setup the "ldap" section. Here's where some of that LDS configuration is really important.

I don't know if it's necessary but the "ldap" section template has "admin_groups" listed again. I went ahead and filled in that value again.

"base" signifies where the root of the directory starts. In this case I'm putting everything under the "Company Domain" OU.

"bindDN" and "bindpw" refer to the username (full Distinguished Name) and password for the "elsadn" user that we're going to use for our initial binds.

The default template has a "searchattrs" value of "sAMACCOUNTNAME". That doesn't exist in LDS. Instead, LDS has "userPrincipalName", so I'm using that instead.

"host" is the actual URL for connecting to the LDAP/LDS server. If you're using "secure LDAP" then that would be ldaps:// (setting up LDAPS is outside of this document).

For the lookup/authentication of users to work, you have to have a search filter. By default ELSA compares the username to sAMACCOUNTNAME. Martin has recently added the ability to add a custom filter so I'm adding one that looks for an object of either type organizationalPerson or user and with an attribute called 'userPrincipalName' that matches the username used for authentication.

With all of those changes made, the "ldap" section now looks like:

And for clarity, just that "ldap" section:

After you save the config file, make sure to restart Apache so the changes take effect! On Ubuntu (and most Linux installations) that is:

sudo /etc/init.d/apache restart

Logging In!

With ELSA restarted, it's time to login!

I wanted to show the difference between logging in as a regular user and logging in between an admin user, both in how it appears to the user and how it appears in the database.

First, I logged in with one of the regular users, "TSUser":

Assuming everything works, you should see the "standard" ELSA search page:

Notice there is only the one tab at the top ("ELSA").

I closed my browser and started a new session to connect again, this time as an admin user ("ISUser"):

ELSA respected group membership - notice the "Admin" tab! If you click that "Admin" tab then you should be rewarded with all the items you'd expect. If you choose "Manage Permissions" then you'll see ELSA has added new users and groups for each user/group that has authenticated. I would expect to see "ISUser", "TSUser" and each of the "ELSA Admins" and "ELSA Users" groups:

That is exactly what's presented. If you click "Add" under "Exceptions" (you have to add exceptions so users can perform queries), you should see the "Exception" types:

Looking pretty good!

If we go back and then login with another user in the "ELSA Users" group, e.g. the MUser, that user then shows up in the "Manage Permissions" drop-down:

If you drop down to the database, you can see where the users/groups/mappings have been added to the appropriate tables. A select from the users table show us all of the users who have logged in:

A select from the groups table shows each of the groups that have logged in (note the user objects are also considered groups for permissions purposes):

And some basic permissions mapping has been added to users_groups_map:

Wrapping It Up

If you've followed along, and seen the above, then congratulations, our "Test" company can send their logs to ELSA and they can authenticate against their LDS server!

This is a fairly portable approach - useful since most of us will not be both the admins of our ELSA boxes and our LDAP/LDS/Domain admins. Instead, we'll probably be given the DN information and some generic LDAP information - but as long as you understand what each option does you should be able to configure ELSA to use your directory services solution for authentication and group membership.