DFIR - Do We Or Don't We?

Before you start reading, be warned: this is a *very* long post.  It probably should have spanned multiple posts.

Last week I sat down with my CISO and had a chat about Google's incident response framework called Google Rapid Response, better known as "grr".  I explained that I had installed it on a couple of virtual machines, started testing out the web-based interface and pulling files from remote systems, and that so far it was REALLY cool but that I was having some problems writing a blog post about it.

When he asked why I told him because every time I tried I ended up with two paragraphs covering general digital forensics and incident response (DFIR) and that I didn't think I could jump into a good grr post without spending some time talking about DFIR.

"Have you considered doing a non-technical post?"

The obvious answer was no, I had not.  This blog started for me to document stuff I like to work on and so I have an easy-to-follow guide for potentially complicated installations or configurations.  I hadn't really thought about covering a "soft topic" for others.  I considered his question for about three seconds, tilted my head, grinned and thanked him for showing me how to jump into covering some DFIR tools.

It is simple, really.  You can't discuss DFIR tools without first discussing some of the DFIR basics.   Which tools do your DFIR team use?  How do you create your DFIR team?  Where do those tools and people fit into your DFIR process?  And, what I consider the most important of all, why do DFIR in the first place?

Each of these questions could be a chapter in a book - if you're an actual forensicator or responder you're probably ready to argue that they could each be their own book, and I agree with you! I won't spend quite that much time answering them but I do want to use this post to address the question that drives all others - to DFIR or not to DFIR?

DFIR - Do We Or Don't We?

Let's say you are a freelance IT consultant and you have several small businesses as clients.  Maybe you grew up and opted to stay in a small town so you do the IT work for friends that have become doctors, lawyers, dentists, dry-cleaners, fabricators, landscapers, plumbers, carpenters, publishers or any of the countless other types of small business owners out there.  Even though there are significant differences in each of these businesses, they all have some things in common, right?  At the very least:

• They all have customers of their own
• They all need to keep track of those customers
• They all bill their customers for services rendered
• They all buy business supplies
• They all have computers in their businesses (or they wouldn't be your client!)
• They all probably use the computers to track their customers and income/expenditures for taxes
• They all may lose business if their business records are leaked to the public

On most days it is business as usual - maybe a broken printer, someone bought a new smartphone and they want to sync their email, any of the myriad of small fires that need to be put out.

Then one afternoon you get a phone call from Bob, the guy who runs the dry-cleaning service.  He explains that he was trying to open a video sent by an old classmate, Joe.  You and Bob went to school together and you remember Joe, so this sounds pretty legit.  Then Bob tells you that when he opened the video he had a warning that he needed to update his Flash player so he clicked "Accept".  After that, he saw a little black window pop-up then disappear and now he has a pop-up saying his anti-virus found 300 "critical infected files".

How do you respond?  Do you try to "clean up" the system and keep it in production?  Do you give him a temporary system while you spend days analysing a forensically-sound copy of the hard drive to determine what malware was downloaded and what it accessed on the system?  Do you have a network device in-line for his business that lets you analyse all of his network traffic for potential data exfiltration?  Do you advise Bob to file a report with local or regional law enforcement?

Is that advice the same as it would be for Elizabeth, the lawyer, if she had the same thing happen?  How does that advice change if the victim is the doctor?

If we're all being honest with ourselves then I think we can safely say that no, the advice will not be the same.  The doctor will have HIPAA concerns and may have intimately detailed customer records accessible from each system in his office -- the Federal fines could put him out of business if he can't show protected data wasn't exposed.  Bob may only have a first name and telephone number for each client - potentially unprotected information in his jurisdiction so he may not have to report a data breach even if his entire database were exposed.

Some clients will have different statutory and contractual requirements regarding investigation and reporting.  In some situations it may be completely acceptable to rely on anti-virus or anti-malware software to "clean" the system.  In some situations it may be acceptable to reformat and reinstall the operating system, what we call "nuke and pave."  Other situations may require intense, in-depth analysis by trained forensicators spanning several days, weeks or even months.  Larger organisations will have to take those same requirements into consideration when making the "do we or don't we" decision.

If You Do

Regardless of the scenario, you're almost certainly going to do something.  That "something" will be determined by the leadership of each organisation and those decisions will need to be well-informed.  It is easy to say, "yes, we're going to do forensics and/or incident response" but what if there is no funding for that decision?  What if there is nobody at the organisation trained in those areas?

At the very least, the people making the decision will need to understand:

• Forensic hardware can be expensive
• Forensic software can be expensive
• This will be a long-term investment
• This is about protecting the organisation's resources, reputation and clients
• Results will not be immediate and can require days, weeks or months
• DFIR is an evolving field, training is never-ending...and can be expensive

Once the decision is made to commit to DFIR, the practitioners should also have some basic understandings:

• Forensic hardware will take training to use correctly
• Forensic software will take training to use correctly
• Both the hardware and software will require regular use to remain proficient
• Processes will have to be documented, validated and then followed - every single time
• DFIR is an evolving field, training is never-ending

It's not enough for an organisation to decide to do forensics and response, that organisation needs to understand at every level that this is a long-term decision.  It can be costly, it will require a human element, it will require training and it will require support.  A forensicator or incident responder can't perform those duties without the support of their employer.

Using DFIR To Enhance Security Operations

Pop quiz: what do SourceFire, Mandiant (pre-FireEye), FireEye and TippingPoint have in common?

Answer: they all use DFIR to enhance and improve their services.

In the same way they use DFIR to enhance the services they offer their clients, a big selling point for doing DFIR in-house is that it can be used enhance internal security operations - specifically, intrusion detection and analysis.

Consider the scenario where a responder, Alice, finds that a piece of malware sends a small UDP packet to a CnC server once every thirty-six hours.  Once Alice knows how that packet looks she can work with the intrusion detection folks to help them write a rule looking for any similar packets going across the network.  Alice can then work with the appropriate individuals to look for any previous occurrences of similar packets to help find other systems that may not have been noticed as compromised.

If Alice can determine specifics about the malware that landed on the system then she can work with the appropriate groups to determine if similar files may have been downloaded by other systems - either via analysis of network traffic or by starting an Enterprise-wide search for files with similar characteristics.  A group from Mandiant released a product called OpenIOC designed *specifically* so that responders could normalise these indicators of compromise and then search for them across multiple Windows-based systems in an Enterprise simultaneously.  Some software will allow you to search across multiple platforms (grr, for example, will search across Linux, OS X and Windows).

In these ways, the findings from DFIR are not only useful from determining whether a particular piece of data may have been accessed (or exfiltrated), the findings may be useful in determining how wide a given infection may spread or how deeply an organisation may be compromised.

Summing It Up

There are multiple factors that have to be taken into consideration regarding how to address digital forensics and incident response - hardware, software and training costs, time spent performing each analysis, time documenting and validating the response process, creating an agreed-upon incident response plan and more.  It can lead to better-trained employees with broader skills and it can be used to enhance other areas of the organisation's security operations but these benefits come at a cost.

Consider the following rant I have given a couple of times in the last few months:

Traditional InfoSec thought has been the trite castle and moat analogy, right? The barbarians are at the gates, screaming to be let in, but the bridge is up and at first they can't get across the moat.  Then they build a bridge across the moat but that can't breach the walls.  Then they push siege towers over the bridge to exit on top of the walls.  Then they fight wall-by-wall until they get into the heart of the castle.  We keep insisting firewalls (the moat) and VLANs (the walls) are enough to keep attackers out but we're delusional - the problem isn't the barbarians outside, it's the users inside.

What we really have to envision is a dark battlefield, dotted with foxholes.  We know there is a perimeter and we try to protect against anyone coming in on the ground with firewalls and IDS.  We try to monitor those parachuting in via authentication...but we're left with these thousands of foxholes all over the battlefield.  Each foxhole may be occupied by one of ours, one of theirs, both or neither, and the only way we can know is by gathering intelligence.  Without real-time intelligence from each host on the network we are essentially blind to that particular foxhole, meaning we don't know how that part of the battleground looks.

DFIR is a part of what gives defenders that "full view" of the battlefield.

Every organisation will have to weigh these benefits against the non-trivial costs and make its own decision.