Wednesday, April 2, 2014

The State Of The Blog

I get enough people asking me about the fate of the blog where I thought it would make more sense to just crank out a blog post. I’m still here, but my time continues to be so limited that I’ve had to continue to put the blog on hold. A couple years back I started a great new job building out a world class cyber investigations team and that continues to take up the majority of my time. I’m planning a few blog posts about what I have learned as a hiring manager to help those who are looking to break into the field and how best to approach things like resumes, cover letters, and interviews.

What really killed my ability to stay on top of the blog was starting an MBA program last fall. It turns out a full-time job coupled with being a full-time MBA student doesn’t leave much free time. Pretty much anything that doesn’t involve my family, job, or school will be on hold until I graduate in the spring of 2015 or the University of Florida punts me out of their MBA program. I’ve managed to survive one term so far and I’m cramming for finals for my second term this week.Three more terms to go after that.

I’m hoping to carve out some time to crank out a couple blog posts now and again before graduation and then get back to my usually blogging schedule of a blog post every two to four weeks. I have a couple blog posts that I really want to get out this year including some interviews.

Monday, September 23, 2013

Ever Get The Feeling You’ve Been Cheated? (Part 2)

I’m back in graduate school these days which is one of the reasons why I’m long overdue on this blog post. Returning to school has provided me with perspective of a student when thinking about the issue of digital forensics degrees. The more I think about it, the less I like the idea of the digital forensics academic programs compared to some alternatives.

The last blog post resulted in plentiful public and private feedback. A common question was what I expected from the graduate of digital forensics programs. I don’t expect someone with a digital forensics degree and no experience to “hit the ground running” where they are immediately cranking out competent digital forensics exams. What I do expect from undergraduate students is that they will be able to perform basic digital forensics exams with about six months of substantial training from my team. I also expect that they will be able to talk intelligently about file system forensics in the initial job interview. If a candidate doesn’t know digital forensics beyond the tools, they were cheated and they’re yet another digital forensics degree victim. I might as well just draw a chalk outline around the chair they sat in for the interview because it’s a crime scene.

If a candidate has a graduate degree in digital forensics, I have the same six month expectation of when they can start to perform acceptable digital forensics exams. Additionally, they had better be able to keep up in an advanced NTFS discussion during the interview. I won't go into the specifics here because I don't want to give away my hiring methods and questions, but I expect a working knowledge of NTFS from the undergraduate degree holders and much more out of the people with a graduate degree.  If you have that shiny new digital forensics graduate degree, you also better have something you are passionate about and skilled at when it comes to the digital forensics world.

So how do you get to the place where you can be successful in a job interview and land that first job? In general, forget about getting a digital forensics degree at an undergraduate level. You’re better off building a firm intellectual foundation for yourself by mastering the fundamentals of computer hardware and software by going through a program such as computer engineering, electrical engineering, or a similarly structured information technology program. Most digital forensics programs are just warmed over mediocre information technology programs with enough poorly taught digital forensics content so that the school can call it a digital forensics degree.

If you want to be excellent at digital forensics, you need a strong understanding of the fundamentals of the technology that you are going to be investigating. The medical profession figured this out a long time ago when it came to training doctors. Medical school is about teaching students about the fundamentals before they move onto their more specialized job roles. Specialties such radiology and pathology are specializations in the medical world that are roughly similar to what we do in the technical world. Both of those jobs require a rigorous general education in medical school before more highly specialized training through residency and fellowship educational processes.

If someone in high school were to come to me today and ask me what the best way to prepare for a digital forensics career, I would tell them to find the best value they can in a degree such as computer or electrical engineering and to supplement that education with some specialized digital forensics training. The specialized training could take the form of a strong digital forensics undergraduate minor, a graduate or undergraduate certificate program, or a full digital forensics graduate program. Some of the best programs in the digital forensics world aren’t actually full digital forensics programs. You do not have to get an degree in digital forensics to prepare for and begin a rewarding career in the field.

Value is important when it comes education which is why I caution students about taking on excessive student loans. Racking up $80,000 in loans for a mediocre digital forensics degree is senseless. I can understand higher student loans if someone is fortunate enough to get into certain top-tier schools such as Cal Tech, MIT, or Stanford, but the math just isn’t likely to work for an expensive degree in digital forensics from Burning Stump Junction University (BSJU). If you are here in the United States, you likely have very fine options that are being offered in your state schools at in-state tuition prices. You will likely be much better off getting that computer engineering degree from the University of Your State at in-state tuition prices than going into massive debt for digital forensics degree at BSJU.

Sunday, July 7, 2013

Gary “Doc” Welt (You Can Help Batman)

In the way of warning, this blog post has almost nothing to do with digital forensics and everything to do with something more important. One of the nice things about having my own blog is that I am my own editor and I don’t have to ask permission to write about something that has very little to do with the original purpose of the blog.

I originally set out to write a follow-up to my last post dealing with the deficiencies that I’m seeing in digital forensics education. That blog post generated quite a bit of interest and I’m grateful for all of the responses both in public and private. I’ll get back to that topic in my next blog post and, as an added “bonus”, I’ll even talk about the new CCFP “cyber forensics” certification being offered by ISC2.

But none of that seems all that important to me as I write this on the 4th of July weekend given how many people over the years have sacrificed everything they had to defend the United States of America and the rest of Western Civilization against a whole host of profoundly bad people. Even a cursory glance at world history shows that peace and prosperity is not the natural state of human affairs. Being able to sustain a place like the United States requires an incredible amount of continuous effort by many people with the brunt of the burden falling on the United States military.

By day, I am a mild mannered digital forensics geek who has the honor and privilege to lead a pack of world-class border collies. By night (and sometimes weekends), among other things, I’m a rookie competitive action shooter. I started doing this early this year and it’s been an amazing experience in large part because of the people involved in the sport. They tend to be some of the nicest and most generous people that I've encountered in many years and this generosity reminds me of the digital forensics community in many ways. My primary game is USPSA action shooting and my home club is the Wyoming Antelope Club in Clearwater, Florida.

It’s through the Wyoming Antelope Club that I became aware of a real live superhero by the name of Gary “Doc” Welt. “Doc” Welt spent around thirty years of his life as a United States Navy SEAL. You can read about Gary’s career here and you will also read why I’m writing this. Gary Welt has been diagnosed with Amyotrophic Lateral Sclerosis (ALS) which also known as Lou Gehrig’s disease. ALS is a very tough set of cards to be dealt. Gary provides a very clear explanation of what he’s up against in this YouTube video. The life expectancy of someone diagnosed with it tends to be two to five years. There is a small percentage of people who live beyond that time. This is the same disease that Stephen Hawking has and, as CNN explains...

Most people with ALS survive only two to five years after diagnosis. Hawking, on the other hand, has lived more than 40 years since he learned he had the disease, which is also known as Lou Gehrig's Disease in America and motor neuron disease, or MND, in the United Kingdom.

So if anyone has a shot at beating the odds in the face of ALS, it’s a  superhero like Gary Welt.  What is interesting is that Gary’s military service might be one of the things that increased his risk for getting ALS. The Mayo Clinic reports that:

Recent studies indicate that people who have served in the military are at higher risk of ALS. Exactly what about military service may trigger the development of ALS is uncertain, but it may include exposure to certain metals or chemicals, traumatic injuries, viral infections and intense exertion.

I call Gary Welt a superhero because he is one. Think about it. No one would deny that Batman is a superhero, but he’s a superhero who doesn’t have any intrinsic superpowers. He wasn’t bitten by a radioactive spider or exposed to gamma radiation which provided him special powers. He wasn’t born on Krypton and sent to Earth. Batman is superhero because he's an exceptionally trained, highly intelligent, and supremely well-conditioned human being with a vast equipment budget. That also describes the US Navy SEALS. Most people can’t even get into their training pipeline much less complete it because of the mental and physical toughness that is required. They do incredibly complicated and challenging work with some of the most sophisticated weapons systems in the world. So even if you are mentally and physically tough enough, you aren’t going to become a SEAL if you are a dullard.

What about equipment? We all know that Batman has all sorts fantastic equipment like the Batmobile, Batcopter, Batcycle, Batboat, and all the rest of his goodies. The SEALS have their own stuff that might as well be right out of a comic book. Check out the picture below.

050505-N-3093M-007<br />

That’s right. The SEALS have their own version of the Batsub. They just call it a SEAL Delivery Vehicle. Put some capes on those guys and give it a bit more of a snappy name and you’ve got a picture right out of a comic book.

Not enough to convince you? Fine. The SEALS have their own version of the BatBuggy. Look at this:


The SEALS just happen to call their BatBuggy a Desert Patrol Vehicle. Not the most creative name, but it can be equipped with a variety of weapons including a 40mm grenade launcher so it doesn’t need one. Good luck with that, Joker.

The only meaningful difference that I can see between a superhero like Batman and a superhero like Gary Welt is that Batman is fictional and “Doc” Welt and the rest of his SEAL brothers are real. “Doc” Welt is a superhero who has devoted his life to fighting bad guys and protecting the rest of his. Now we have an opportunity to try and return the favor by helping him out when he’s in a tough fight. How often do you get to say that you helped a real life superhero?

As the Red Circle Foundation webpage set for for him explains:

We are raising money to help Gary and his wife modify their home for his condition and for wheelchair access. The VA (Veterans Affairs) does a lot of good, but they are a slow moving bureaucracy and time is critical for the Welt family.

The primary way that you can help Gary is donating money via the Red Circle Foundation website. I think the current setup is that any money you give via that portal will result in 90 percent going to Gary and 10 percent to help pay for Red Circle Foundation costs. If you follow Gary’s progress at the HelpGaryWelt Facebook page you’ll see them discussing how that works.

I know the digital forensics community to be a very generous bunch with a culture of sharing and helping one another out. He’s an opportunity for us stand together to help someone who has done so much for others. How often can you say that you got the help Batman? Please consider giving anything you can to help a real live superhero like Gary “Doc” Welt.

Photo Credits and Captions

SDV Photo:

Atlantic Ocean (May 5, 2005) - Members of SEAL Delivery Vehicle Team Two (SDVT-2) prepare to launch one of the team's SEAL Delivery Vehicles (SDV) from the back of the Los Angeles-class attack submarine USS Philadelphia (SSN 690) on a training exercise. The SDVs are used to carry Navy SEALs from a submerged submarine to enemy targets while staying underwater and undetected. SDVT-2 is stationed at Naval Amphibious Base Little Creek, Va., and conducts operations throughout the Atlantic, Southern, and European command areas of responsibility. U.S. Navy photo by Chief Photographer's Mate Andrew McKaskle (RELEASED)

DPV Photo:

Camp Doha, Kuwait (Feb. 13, 2002) - U.S. Navy SEALs (SEa, Air, Land) operate Desert Patrol Vehicles (DPV) while preparing for an upcoming mission. Each Dune Buggy" is outfitted with complex communication and weapon systems designed for the harsh desert terrain. Special Operations units are characterized by the use of small units with unique ability to conduct military actions that are beyond the capability of conventional military forces. SEALs are superbly trained in all environments, and are the masters of maritime Special Operations. SEALs are required to utilize a combination of specialized training, equipment, and tactics in completion of Special Operation missions worldwide. Navy SEALs are currently forward deployed in support of Operation Enduring Freedom (OEF). U.S. Navy photo by Photographer's Mate 1st Class Arlo Abrahamson. (RELEASED)

Friday, May 17, 2013

Ever Get The Feeling You’ve Been Cheated?

The famous John Lydon quote strikes me as an appropriate title for a blog post on the state of digital forensics academic programs in the United States. I have been a hiring manager for high tech investigations teams since about 2007 and was involved in assessing candidates for the teams that I was before I became a leader. During the early years, it was rare to see applicants who had degrees in digital forensics, but I’m finding it increasingly common in recent years. One of the things that I have been struck by is how poorly most of these programs are doing in preparing students to enter the digital forensics fields.

It’s not just undergraduate programs that are failing to produce good candidates. I have encountered legions of people with masters degrees in digital forensics who are “unfit for purpose” for entry level positions much less for positions that require a senior skill level. The problem almost always isn’t with the students. They tend to be bright and eager people who just aren’t being served all that well. One of the core issues that I see with the programs that aren’t turning out prepared students are the people who are teaching them. It’s almost universal that programs who have professors who do not have a digital forensics background are turning out students who don’t understand digital forensics. This seems like an obvious and intuitive statement, but given how many digital forensics programs there are who are being lead and taught by unqualified people, it apparently isn’t obvious enough.

If you want to learn to be a good digital forensics examiner, you have to be taught be people who are good digital forensics examiners. If you are interested in learning digital forensics from an academic program, it is your responsibility to look beyond the promotional material and be an informed and educated consumer of your education. The last thing you want is a massive student loan and a degree that looks good on a resume, but then falls apart during a technical interview for that great entry level job that you had your heart set on. One of the best ways to make sure you don’t get burned is to carefully study the backgrounds of the professors who will actually be teaching your classes. We’re a bit too early in the development of the digital forensics field to see a host of full tenured professors with PhD’s in Digital Forensics, but that doesn’t mean you can’t screen out professors who have no earthly clue what they are teaching. Pay very close attention to the curriculum vitae of the people who are going to be teaching your classes. Does the CV show any actual interest in the field of digital forensics? I have seen many CV’s for people teaching digital forensics who don’t show any research or training in the digital forensics field. What it looks like is that we have quite a few institutions that have decided that the digital forensics field is hot right now and to capitalize on it, they press unqualified professors into teaching digital forensics classes just so they can lure paying students (and their tuition money) into their programs. Avoid these programs. Your future depends on it.

We are in a time where there are many fine academic programs available to aspiring digital forensics people who wish to learn digital forensics and launch successful careers. Unfortunately, there are more bad programs than good ones. It’s vital if you are going to spend the time and money getting an education that you don’t get cheated. It’s your life and your responsibility to look beyond the glossy promotional material and make sure you are trusting the right people to get you where you want to go.

Sunday, February 24, 2013

Microsoft Windows File System Tunneling

I’m way overdue on doing a proper blog post so I thought I’d swing back into action by showing everyone some really esoteric Microsoft Windows knowledge that I picked up by accident several years ago working on a research project. Around 2010 or so, I was working on Adobe Flash cookie research with Kristinn Gudjonsson of log2timeline fame. During that research, I discovered an odd occurrence dealing with Windows date and time stamps which led me into learning about file system tunneling.

So for purposes of this blog post, you don’t need to know anything about Adobe Flash cookies. I presented on them during the CEIC 2010 Conference and end of my presentation I went into the file system tunneling aspect of what I had found. This blog post covers that portion of the presentation. The only thing you need to know for purposes of this blog post is that Adobe Flash cookies, at least back when I as doing the research on them many years ago, can be deleted and then quickly replaced with a file of the exact same name and file location as the previous one.

For purposes of this demonstration, we’re using a Adobe Flash cookie file with the name “settings.sol”. I think I was using EnCase Version 6 for making this presentation in case anyone is curious about the tool being used.

Here is our baseline “settings.sol” file before any changes are made to it.


Here is what happens when a change has been made and the original file has been deleted and replaced with a file of the exact same file name and location. I don’t have the path listed in these screenshots, but the paths of the new files of the same name are being placed in the exact same location as the old files of the same name.

Here is the brand new file with the exact same name in the exact same location:


So we are seeing what we would expect for a brand new file that was created after the old one was deleted…except look at the file creation timestamp. It exactly matches the file created timestamp of the old file that was just deleted. That’s not accurate, is it? We know the file was actually created on 04/01/10 12:17:27PM, but it’s showing the created file time of the deleted file.

Here is another way of looking at it.


Here we see the old “settings.sol” file that was deleted and we also see the new “settings.sol” file that was created in the exact same location. As you can see, the new file has last accessed, last written, and entry modified time stamps that show when it was created, but it has the wrong file created time. It kept the file creation time of the old file of the exact same name that was in the exact same location.

What on earth is going on?

This almost made me go completely insane trying to figure this out. Like any good digital forensics person, I could absolutely not let this one go and had to run it down until I had an answer. I lost track of how many digital forensics gurus that I contacted to get help. At first, I was a little reluctant to do so in case it turned out to be something obvious that I missed. Who wants to humiliate themselves with a flagrant display of ignorance in front of some digital forensics icon? I was able to save face because the responses I received were all along the lines of telling me that I had an interesting problem on my hands and they didn’t have an answer for me.  The person who solved the problem in the end was Eoghan Casey. Eoghan listened to me describe what I was seeing and said he thought I might be observing file system tunneling. I did some further research and it turned out that Eoghan was exactly right.

You can find the official Microsoft write up on file system tunneling at their knowledge base 172190 article at The relevant text (and the spelling errors are theirs) from that article is:

The Microsoft Windowsproducts listed at the beginning of this artice contain file system tunneling capabilities to enable compatibility with programs that rely on file systems being able to hold onto file meta-info for a short period of time.

When a name is removed from a directory (rename or delete), its short/long name pair and creation time are saved in a cache, keyed by the name that was removed. When a name is added to a directory (rename or create), the cache is searched to see if there is information to restore. The cache is effective per instance of a directory. If a directory is deleted, the cache for it is removed.

I haven’t performed any recent research to update this information, but at the time I was doing this research in 2010, file tunneling impacted a broad range of Microsoft operating systems including 2k, XP (including 64bit), and NT. Microsoft hasn’t updated the article since 2007, but I’d be surprised if it wasn’t an issue for Windows 7 and potentially Windows 8.

The Microsoft 172190 article goes on to explain that file system tunneling is an issue for both FAT and NTFS file systems because:

The idea is to mimic the behavior MS-DOS programs expect when they use the safe save method. They copy the modified data to a temporary file, delete the original and rename the temporary to the original. This should seem to be the original file when complete. Windows performs tunneling on both FAT and NTFS file systems to ensure long/short file names are retained when 16-bit applications perform this safe save operation.

Microsoft explains how to disable file system tunneling in the same 172190 article. All you need to do is to go to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem setting in the registry and add a DWORD called MaximumTunnelEntries which you then set to 0.

So the Microsoft article talks about “file systems being able to hold onto file meta-info for a short period of time”. The short period of time is a default setting of fifteen seconds. However, if you wish to change that time, it’s yet another registry tweak.  You just have to head over to the same key as you would to disable file system tunneling, but this time you just need to add a new DWORD called “MaximumTunnelEntryAgeInSeconds” and then whatever time you would like. This is all explained in the Microsoft 172190 article.

So like any good forensic geek, I wanted to test what I had found because I didn’t want to look foolish in front of a big audience of people at CEIC. I set up an experiment based on making Flash Cookie modifications that would initiate the same deletion and creation process.  Again, all you need to know for purposes of this blog post is that I was making changes that resulted in near instant file deletion and creation of files with the exact same name in the exact same location.

For example, I used a Windows XP SP2 system that had file system tunneling enabled and obtained these results.


I was making changes that causing these “settings.sol” files to be deleted and then replaced with files of the exact same name. Like before, we are seeing the changes that we would expect from new files being created in that their accessed, written, and modified time stamps reflect the accurate time that they were created. However, all of these new files are keeping the same file created stamp as the original file.  This is file system tunneling at work.

How do we know for sure?  What I did next was to follow the registry modification that I explained before to disable file system tunneling and then to run the experiment again.  Again this was a Windows XP SP2 system, but with file system tunneling disabled.


Now we see that all of the file times are matching the time that the files were created including…finally… the file created times.

And because like all good digital forensics people, I’m always suspicious of what I’m seeing, I repeated the experiment a third time, but this time I used the registry to turn file system tunneling back on which resulted in...


…the exact same file system tunneling behavior that I had observed at the start of all of this.

Sunday, November 25, 2012

AFoD Interview With Carlos Cajigas

I have spent the last several months working on relocating from the New York metropolitan area to Tampa, Florida. Now that I’m starting to get settled into Florida, I will be blogging on a more consistent schedule. Carlos Cajigas is one of the many sharp Florida-based digital forensics people that I have had the privilege to meet in my travels around the state. Carlos is an accomplished digital forensics examiner as well as a bit of an entrepreneur. He’s very passionate about the use of Linux in digital forensics and it didn’t take talking to him for very long to realize that he would be a great interview subject for the blog.

Carlos Cajigas Professional Biography

clip_image002Carlos, a native of San Juan, Puerto Rico, is the Training Director and Senior Forensic Analyst for EPYX Forensics. Additionally, he is employed by the West Palm Beach Police Department (FL) as a Detective/Examiner assigned to the Digital Forensics Unit with over 9 years law enforcement experience. He has conducted examinations on hundreds of digital devices to include computers, cell phones, and GPS devices to go along with hundreds of hours of digital forensics training. His training includes courses offered by Guidance Software (EnCase), National White Collar Crime Center (NW3C), and the International Association of Computer Investigative Specialists (IACIS).

Carlos holds B.S. and M.S. degrees from Palm Beach Atlantic University (FL). In addition, he holds various certifications in the digital forensics field to include EnCase Certified Examiner (EnCE), Certified Forensic Computer Examiner (CFCE) from IACIS, and Certified Digital Forensic Examiner (CDFE) from Mile2. Carlos is a Florida Department of Law Enforcement (FDLE) certified instructor with experience teaching digital forensic classes. He is an active member of both the International Association of Computer Investigative Specialists (IACIS) and Miami Electronic Crimes Task Force (MECTF).

Most recently, Carlos has endeavored in writing a blog for EPYX Forensics that would assist other digital forensic examiners in using free open source Linux-based tools to do their jobs. He hopes to develop and implement course training in this area in the belief that there are alternatives to expensive commercial software and training.

A Fistful of Dongles Blog: What led you into becoming a law enforcement officer?

Carlos Cajigas: Although police work has always appealed to me, the decision to join law enforcement didn’t enter my mind until late in my 20’s.  At the age of 17, I moved from Puerto Rico to Palm Beach County, Florida to pursue a career in baseball. At that time my priorities were anything baseball and my responsibilities were simple: keep making good grades and go to practice. Although I was a fairly talented baseball player, I also knew that I wasn’t the most gifted. From an early age, I learned that hard work could make up for the areas where one lacks talent. That is a lesson that still holds value even to this day. So the answer was simple, I played and practiced as much as I could while working hard every day. Subsequently, I received a baseball scholarship to Palm Beach Atlantic University. 

I continued to work hard and had some success on the field. I broke a few records and became an MVP. It truly was a great experience that I will always remember. Unfortunately, my collegiate baseball career ended after 4 years of eligibility. So there I was - 22 years of age with a decision to make unsure of what I really wanted to do.

On September 11th 2001, halfway through grad school, the events of that day changed many lives forever. The impact that day had on me, led me to join law enforcement. Although law enforcement work always appealed to me, that day I decided that I wanted to make a career out of giving back to the community. I wanted to be part of another team with similar interests and values of mine. Baseball in many ways prepared me for that jump. I finished grad school and applied to the West Palm Beach Police Department. I have been a police officer now for nine years. Throughout my career, I have been part of multiple units and many teams. I have been given opportunities to do some good and have taken advantage of them. Our city is a great city and our Department is top notch. Joining law enforcement is another decision that I am glad I made. 

AFoD: What happened once you joined the West Palm Beach police? What was your initial training like and how did you end up doing digital forensics work?

Cajigas: After completing the Academy and once at the Department, I went through an initial series of stages that I had to pass before being allowed inside of a patrol car. They included training in defensive tactics, driving, and firearms, among others. These were the core primary skills that were taught to trainees. The department was very strict about their minimum qualifications. 

I then progressed into a multi-month field training program that required me to go on patrol while a qualified senior officer sat next to me and evaluated me. My trainers were very good, no-nonsense, seasoned officers.  Learning at this stage was done at a very fast pace. I was taught radio procedures, report writing and everything else.  Every new call brought about a new challenge. On-the-fly problem solving skills were a must have and strict emphasis was given to safety. 

Upon graduation from the field training program, I began responding to calls by myself. I completed a one year probationary period and remained as a road patrol officer for about four years. I then joined a specialized unit created to reduce crime in a specific area of our city. Our team was made up of 6 officers and we were responsible for just about any crime or issue in this area. This unit provided me with an opportunity to conduct investigations from beginning to end. Some investigations required travel to neighboring cities and others undercover work. 

When a position opened up for the Digital Forensics Unit, I interviewed.  After a few months, I was notified that I had been awarded the position.  That was a very exciting day. I have always had a passion for computers and now I was being given the opportunity to combine police work with that passion. A few years later with a few hundred hours of digital forensic training under my belt, I enjoy computers even more.   

AFoD: What is life like working on the Digital Forensics Unit?

Cajigas: Life in the Digital Forensics Unit is full of activity. Our unit is part of the Palm Beach County Internet Crimes against Children (ICAC) Task Force and the Palm Beach County Regional Forensics Task Force. The task forces are made up of investigators and examiners from different participating agencies in the county. Our ICAC task force has a very proactive approach towards pursuing individuals who hurt children. As a result, we conduct an average of one search warrant every other week involving child exploitation. 

The operations that we conduct can take us from one side of the county to the other with very short notice. We travel in our mobile forensics van, and we triage and preview devices on location. In most cases, we can retrieve the necessary evidence that the investigator needs to make an arrest on scene.

As part of our duties to the Regional Forensics Task Force, we provide assistance to neighboring agencies without forensic units. The cases that we assist with can range from simple thefts to homicides and everything in between. On any day, it is quite common that we might get a request to process ten phones and five computers. The amount of devices that we see at the lab daily poses challenges and opportunities to learn something new every day. This is the part of the job that I enjoy the most - the process involving identifying a problem, looking for a solution and then implementing that solution.

Recently during a case, we came across a 3TB hard drive with a corrupt GPT. Once the drive was imaged, our tool of choice was unable to see the directory structure of the volume. We needed specific files, and we wanted to be able to access the volume inside of the drive without restoring the image or editing the image. After a little bit of research, we ended up accessing the volume by mounting the E01 image in Linux and using the program ‘Testdisk’ to point us to the starting sector of the volume.  The mount command then mounted the volume, and the volume became accessible.    

I have learned that Linux can be a little complicated; however, it is powerful and free. In this instance, once we got past the complicated part, we were left with powerful and free. I see the usefulness and versatility that Linux has when used in forensics. On the days that we have time to catch up, I dedicate a few hours to learning Linux in the hopes that it can become another tool in our tool belt in the battle against online criminals.  

AFoD: So how did you discover that Linux could be a powerful tool for digital forensics examinations?

Cajigas: I first got introduced to Linux back in 2007, before learning forensics. I stumbled across Ubuntu version 7.04 out of curiosity and necessity. During those years, I used to spend a lot of time building and fixing computers. I decided to try Ubuntu when a friend requested my help with recovering family photos from his BSOD’d Windows PC.  I burned the ISO to a CD and booted his “dead” computer from the CD drive. Ubuntu loaded and his computer “magically” came back to life.  His drive was still healthy and the directory structure was intact.  All of his family photos were there waiting to be copied. My friend was happy and I was hooked. To this day it still amazes me how the entire OS can run from a CD, just for the cost of the CD.

I began testing and installing as many variants of Linux on as many computers as I could just to see what I could learn. For instance, it took me about a week of testing different distributions before finally getting the right version of xubuntu working on a PS3. As a result, I installed Ubuntu on a flash drive and carried it with me, just in case of a “dead PC” emergency. 

Fast forward a few years and after some forensics training, I decided to try Linux for forensics. I did it out of curiosity and necessity. I saw that great tools like the SIFT workstation were already out there, so I was curious as to how to use them. I needed a second method of doing forensics, so that at the very least, I could use them to validate procedures. I downloaded as many of the forensic distributions as I could and began testing the programs. Just like with any tool, there is always a learning curve.  Unfortunately when it comes to Linux, that learning curve can sometimes be curvier. Just learning the commands for a specific program could take hours of research and trial and error. But once you learn the program, the results can be very rewarding.

There are programs in Linux for just about every action needed in forensics. Some of the smarter minds in forensics build these programs and release them for free for the benefit of the community. Unfortunately, the documentation on how to use programs in Linux can sometimes be difficult to find. I have found myself reading blog after blog gathering bits from one site and pieces from another, while teaching myself how to use these tools.  As a result, I have decided to document procedures on how to use forensic tools on your own standalone Ubuntu 12.04 machine. My intent is to help other digital examiners in using open source tools during the course of their investigations for free.

I started documenting these procedures at the beginning of the year, and I plan on adding many more. So far, I have documented procedures on how to recover Win7 passwords, using Testdisk, acquiring and mounting E01’s, recovering IE history, file carving, extracting files by record number, registry analysis, and parsing the MFT with analyzeMFT. My attempt is to outline the procedures from beginning to the end for users new to Linux, while explaining every step with screenshots. You can find them at

AFoD: What can Linux-based tools do for a digital forensics examiner that the increasingly wide range of Windows-based tools can't do?

Cajigas: They might just save you some money! Windows based tools like EnCase, FTK and X-Ways are simply excellent tools.They combine the  processes of acquiring, indexing, parsing, searching, recovering, and reporting all into one suite. In my opinion, there is no equivalent single program available in Linux that can compare to these great suites. Every lab should have at least one of these tools. What is available in Linux is an accumulation of different tools that when put together can accomplish almost all of the same things that these suites do. Some of the tools can do some tasks better, others, not so well. 

But the tools will always accomplish their tasks for free, and they might help you when you don’t have the commercial tool needed for the job. 

Let’s say for example that you and your case could benefit from an analysis of a timeline and you do not have the tool of choice to build that timeline.  Log2timeline is an excellent open-source framework for automatic creation of super timelines. Log2timeline received the 2012 Forensic 4cast, computer forensic software tool of the year. 

Every examiner that I have talked to can remember that time when their Windows forensic tool of choice crashed and failed to accomplish some sort of task. I can personally recall instances in the lab when Windows based tools failed to image devices, and I reverted to using Guymager in Linux with absolute success. And best of all, these tools can all be run from a ten cent DVD. I recently participated in a Rob Lee webcast were he so accurately described the SIFT workstation as your own mobile computer forensics lab.

Linux-based open-source tools alone can be used to complete forensic examinations. Many of these tools have helped me during my investigations.  They were released free to the community, and I believe that we can all benefit from them.  

AFoD: Let's pick out a couple tools to use as examples. I've sung the praises of log2timeline here on the blog and will continue to do so in the future. Let's focus on a tool that might not be as well known. What is Guymager and why should someone consider using it over a Windows-based tool?

Cajigas: Guymager is an open source forensic imager with an easy to use graphical user interface (GUI). The tool, created by Guy Voncken, was designed to be fast especially on multiple processor machines. It produces DD, E01, or AFF images and conducts verifications upon completion. In addition, it creates an .info file that stores acquisition details to include hashes, bad sectors, and SMART data. Because it runs on Linux, it often succeeds at acquiring those pesky drives that make Windows freeze and/or have trouble showing up in Disk Management.

Since acquisition is the one process that has to be done in 99% of examinations, Guymager is a tool I find myself using a lot of the time. Guymager can be downloaded from the Ubuntu Software Center.

Another open source Windows analysis tool with an easy to use GUI is the Forensic Registry EDitor (FRED). FRED is a registry hive editor created by Daniel Gillen. It can navigate the directory structure of a hive and has a built in hex viewer and data interpreter. Another cool feature built into the tool is that it has automated reporting functions that can give you the “RecentDocs” and the “TypedUrls” out of an NTUser.dat registry hive. FRED can be downloaded from

For their ease of use, these two tools are a good start and a must try for those interested in using Linux based tools.  

AFoD: What do you recommend for someone who wants to learn Linux and get to a point where they can comfortable leverage it for digital forensics examinations?

Cajigas: There will be a lot of reading involved, but these three steps will get you going in the right direction. The first step towards becoming familiar with Linux is to begin using it. As simple as it sounds, installing and using your favorite distribution will teach you a lot about how the OS works and how the directory structure is laid out. Once you know the layout, you are now able to spot the things that look right and the ones that don’t.

The next step is to become familiar with manipulating the shell commands. This is the stage where you learn commands like “cd, cp, rm, mv” and terms like input/output redirection. To redirect the results of one command into a second command and get only one set of results is the linux equivalent of killing two birds with one stone. Redirection is one of the most useful features of the shell. A well written website on learning the linux shell can be found at linuxcommand dot org.

The last step in the familiarization process is to start playing with the forensic tools. This is the fun part. I have written some articles to get you started with the basic forensic tools, and many more can be found on the web. Just like with forensics, in Linux there is something new to be learned every day with tools that are available for just about any task.

The more you play with the tools, the more comfortable you get. Figure out what your need is, and learn how to accomplish that task with Linux. Chances are the next time that you need the same task completed, you will revert back to accomplishing it in Linux.

AFoD: Are their any Linux distros that you recommend over others for the beginning Linux user?

Cajigas: In my opinion, before using any of the forensic live distributions, anyone starting on Linux should start with Ubuntu. Ubuntu was first released in 2004 by a UK based company called Canonical. Canonical provides support, patches, bug and security fixes for a period of eighteen months on each of their new releases, keeping Ubuntu up to date. 

Ubuntu was designed with ease of use in mind and comes with GUI based tools for installation, updating, personalization of the OS, and many more. It has built in support for a lot of different hardware, which translates to a good chance that it will boot and recognize the hardware in your computer. Due to its popularity, a web search will often point you in the right direction towards solving most of the problems you may encounter.

During basic Ubuntu use, mandatory interaction with the terminal is minimal. This gives the user time to get to know the OS before being forced to use the terminal for non-GUI tools. Ubuntu has been selected as the platform for popular live DVD distributions like the SIFT and DEFT. After you become comfortable using your installed version of Ubuntu, graduating to using live distributions no longer feels like unfamiliar territory.  

AFoD: Is there anything else that you'd like people to know?

Cajigas: Open source (Linux) forensic utilities are very useful as a supplement to commercial tools. However, there is the good and bad when it comes to open source. The good - tools are free and they are just as, if not more powerful than commercial tools. The bad – there is a learning curve, and they are harder to use. With that said, as part of EPYX Forensics, my colleagues and I want to bridge that learning curve gap so that forensic examiners are able to take full advantage of open source forensic tools. As I spoke about earlier, I have begun doing this by posting tutorials through the EPYX Blog. We are also currently putting together training courses for law enforcement, government and private sector personnel that we are planning to launch in early 2013.

The world of digital forensics is constantly evolving, and I believe there is a shift towards increased usage of open source forensic utilities, especially with the expenses that come with commercial tools. My hope is for all forensic practitioners to sit down and at least try the open source passage – you just might like what you find.

Saturday, August 25, 2012

Cyber Pearl Harbor

I’m an amateur student of military history who has written several blog posts in the past discussing physical warfare concepts and how they can be applied to the cyber world. For example, I’ve written about how the Battle of the Atlantic and the Battle of Britain provide lessons that can be applied to cyber warfare. Thus, I’m the last person who will say that it’s inappropriate to apply lessons from the physical world to the information technology world. So while it can be an appropriate thing to do, it can also be done in a haphazard manner that doesn’t correctly respect the historical record. The term “cyber Pearl Harbor” is one that is increasingly being used in a manner that just doesn’t make sense from a military history perspective.

120702-N-VD564-016 The Pearl Harbor attack involved the nation of Japan engaging its military to strike a substantial blow against Pacific Fleet of the United States Navy. The damage that was inflicted on the US Navy was such that it provided the Japanese military with a decisive military imbalance in the Pacific that it exploited until the American military was able to rebuild and regroup. In my mind, for an attack to be accurately labeled “cyber Pearl Harbor” it would need to involve a cyber attack that accomplished a similar objective. A reasonable example of “cyber Pearl Harbor” could be a nation-state using a cyber attack to substantially degrade another nation’s ability to respond to future attacks in either the physical or cyber world. An incident where an attacker unexpectedly brings down a power plant without any further attacks isn’t “cyber Pearl Harbor”. That’s certainly a serious incident, but it’s not equivalent to the Pearl Harbor attack.

Another term that I’m having even more trouble with is “cyber terrorism”. The American English version of the Oxford dictionary defines terrorism as “the use of violence and intimidation in the pursuit of political aims.” The Oxford British & World English dictionary defines it as:

the unofficial or unauthorized use of violence and intimidation in the pursuit of political aims:
the fight against terrorism
international terrorism

For a cyber attack to be accurately defined as cyber terrorism, the attack would have to have a violent result or at least some sort of intimidating effect. I just don’t see how a DDoS attack or even something destructive like Stuxnet clears this bar. In my mind, for something in the information technology world to be considered “cyber terrorism”, you’d need a result where you had the loss of life or a substantial and intimidating impact such as taking down a power grid of a major city. An action like a major urban power outage could very well result in indirect loss of life (heat related deaths during summer months) and violence (riots). It’s not that this couldn’t happen, but we just haven’t seen it yet.

Google Takeout

Tom Thomas is the marketing director over at IACIS. He posted to the IACIS email list recently about a Christopher Null authored article he discovered in PC World that explained Google Takeout. Tom was nice enough to give me permission to pass along what he posted to the rest of the team through the blog.

The Google Takeout webpage explains that “Google Takeout allows you to download a copy of your data stored within Google products.” Null explained in his article that:

Google wants you to keep using Search, Docs, and Google+, so it’s trying to play nice, and last June Google introduced a service designed to let you see, in part at least, what Google knows about you with a single click.

Eric Zimmerman’s Tools

Unsurprisingly, the Eric Zimmerman interview generated a tremendous amount of interest in Eric and his tools. One of the popular questions has been how to obtain the various tools that Eric has created and made available to the law enforcement community. You can contact Eric through the FBI’s Salt Lake City Division or you can just send me an email and I’ll pass it along to Eric.

Digital Forensics Email Lists

Another one of the questions that came out of that interview was a request to know which email lists that Eric was participating in since that was mentioned in the interview. I won’t disclose what lists that Eric participates in, but what I will do is contact some of the people who run the various lists that I am on and get permission to post their membership requirements and how to join. Some of these lists have few requirements for membership, but others are more restrictive such excluding people who do criminal defense work.

About The Photo

Photo credit information from the United States Navy:

120702-N-VD564-016 PEARL HARBOR (July 2, 2012) Sailors man the rails aboard the aircraft carrier USS Nimitz (CVN 68) as it passes the USS Arizona Memorial in Pearl Harbor. Nimitz is participating in the biennial Rim of the Pacific (RIMPAC) exercise 2012, the world's largest international maritime exercise. Twenty-two nations, more than 40 ships and submarines, more than 200 aircraft and 25,000 personnel are participating in RIMPAC exercise from June 29 to Aug. 3, in and around the Hawaiian Islands. (U.S. Navy photo by Chief Mass Communication Specialist Keith W. DeVinney/Released)

Thanks to the United States Navy and all of the other services who make these photos available for people like me to use. Thanks to Chief Mass Communication Specialist DeVinney for his service and his excellent photographic work.

Friday, August 17, 2012

AFoD Interview with Eric Zimmerman

Eric Zimmerman is one of the most amazing digital forensics people that I have run across in the last few years. He’s a combination of passion, digital forensics skill, and sharp programming abilities. He has created a whole host of digital forensics tools to help the good guys catch the bad guys. He’s a credit to the digital forensics community as well as his employer who happens to be the Federal Bureau of Investigation (FBI). As Eric points out at the end of this interview, his answers reflect his views and not the views of the FBI unless clearly stated otherwise.

Professional Biography of Eric Zimmerman

Zimmerman Eric 18 Eric Zimmerman is a Special Agent assigned to the Cyber Squad of the Salt Lake City FBI field office where he has been investigating child pornography and computer intrusions since early 2008. He is a member of the Utah ICAC and has provided training and assistance to dozens of local, state, federal and international law enforcement agencies. Eric has a degree in Computer Science and has developed several computer programs to aid in the investigation and prosecution of child exploitation matters.

Eric is an EnCase certified examiner and has several other certifications from CompTIA and SANS.

A Fistful of Dongles Blog: What was your path to the FBI? What made you decide to join?

Eric Zimmerman: My path into the FBI was somewhat out of left field. I moved to Chicago in late 1998 to work at a 3rd party logistics start up company called Con-way Integrated Services. I was the 7th employee and over the years we grew the company to about $70 million/year in revenue. Around 2005, we started the process of merging with a sister company of ours, Menlo Logistics. They were a billion dollar company and as such, their culture won. After this happened, what used to take me two days would take two weeks. Things slowed down dramatically and I was no longer able to move at a pace I was comfortable with. As the years went by, I started asking myself "Who am I benefitting being here?"  I applied to a few other places and did some interviews, but nothing panned out.

In late 2005, my brother, who has been in the Army since he graduated from high school in 1994, recommended I look into such places as the CIA and Secret Service. Neither of those agencies appealed to me for various reasons. I then took a look at the FBI in January 2006 and felt it was a much better fit. I sent in an application that same month and soon after the pieces started falling into place. In September, 2007, I was given a slot in New Agents Class 08-01. I graduated on March 5, 2008 and reported to Salt Lake City soon after that.

The reason I wanted to join the FBI was to be a part of something bigger than myself and to have the opportunity to serve our country. I had lost my passion for what I was doing in the private sector and if you do not love what you are doing, it becomes very hard to get up every morning to go to work. Now I look forward to going to work every day (well, almost every day!).

In short, I left the red tape of corporate America behind to dedicate my career to serving the public, and now, as a FBI special agent, I am able to help the most innocent among us -- our children.

I know there are a lot of other people who go to work day in and day out in the law enforcement field who have enormous responsibilities and more work than they can address effectively due to budget and time restraints. I try to think of ways to make their jobs easier and provide tools and techniques to make their work more efficient at very little to no cost.

I have been given amazing opportunities at the FBI to solve problems and design things to help a lot of people both in the United States and across the world. It hasn't been without its challenges along the way, but it has definitely been worth it and I think the results speak for themselves.

AFoD: Can you tell us what you are currently doing for the FBI and how you came into that position?

Zimmerman: I am currently assigned to the Cyber squad in Salt Lake. We are responsible for both criminal and national security investigations. I have been assigned to this squad since arriving in Salt Lake City as I am a cyber agent. By this I mean my career path is designated ‘cyber’ as opposed to counter intelligence, counter terrorism, and so on. This isn't to say we do not regularly help other squads with things as there is a computer involved at some level in just about any crime these days. My day to day work involves general case work (both criminal and national security as we do not have dedicated squads for each), some programming and the occasional bout of reverse engineering various things.

I have also been actively involved with the Utah Internet Crimes against Children (ICAC) task force since arriving in the division. A lot of the forensic programs I have written were due to my involvement with the ICAC and the FBI Innocent Images program.

Due to the success of these programs, I also spend time supporting these tools to include basic tech support, training, and so on. I provide my cell phone number and email address to everyone who uses my programs so they can contact me any time should they have an issue. I recall sitting in a presentation at the last ICAC conference in Atlanta where a vendor, in all seriousness, essentially said “If you aren't paying for software, its junk.” My guess is he was insinuating free software isn’t supported or kept up to date.

Regardless of what he meant, I feel confident the vast majority of the more than 2360 users (in more than 40 countries) of my software would disagree with that statement. What commercial vendor can you call and speak directly to the developer about a problem? I feel a great sense of responsibility to the users of my software and feel I would be doing a disservice to them and the community if I didn’t make myself available when questions or issues arise, especially when the users are in the field conducting a search and whatnot.

I also teach at several national conferences in the United States (specifically, the national ICAC conference and the Dallas Crimes against Children conference) and at a few international ones as well. I will be teaching at Europol's conference in October of this year for the first time. I am also a member of the Interpol peer to peer technical working group which is comprised of people from various law enforcement agencies all over the world who meet regularly to discuss the most efficient way to combat the exploitation of children on the Internet.

I initially started writing the various software programs out of necessity. Other Special Agents and law enforcement personnel were finding themselves in situations where there were no tools or techniques to aid in their investigations or the tools that were available did not work very well. A lot of the tools we currently have started from a simple phone call and a few hours of work. Over time, more features were added. This process has culminated in a whole suite of cutting edge tools that provide features no one else has at any price. Some of the tools are simple one offs and others have been in active development for several years. I maintain all the tools (currently I maintain 13 programs) in addition to my case load, so it can get quite hectic!

AFoD: So this gets to the heart of why I wanted to do this interview. You're very active in the digital forensics community and anyone who is on the same digital forensics email lists as you can see how knowledgeable and helpful you are. I also want to talk about your development work and the award you were recently given for this work. However, before we dig into that, we have many people who follow the blog who are interested in breaking into digital forensics and cyber investigations. We know your path into the FBI now, but can you explain how you ended up getting assigned to the Cyber squad? Is that something that you can choose to do as a condition of joining the FBI or is it up to fate and the Bureau once you have completed the academy?

Zimmerman: There is currently a list of "critical skills" that the FBI is looking for, at least in regard to the Special Agent position. Some of these include accounting, engineering, and computer science/information technology. If you have a strong background in one of these fields it tends to help move you through the process a bit quicker. My education is in mathematics and computer science, so I came in under the computer science critical skill.

When I attended the FBI Academy, I was assigned to the cyber career path. Other possibilities included counter intelligence, counter terrorism, and so on. While I had some say in my career path in that I ranked my preferences, at the end of the day someone else made the decision for me based on my background and education.

Part of that decision is also based on how many agents the FBI requires in the various career paths as well. While I was sure I would be tracked cyber due to my background and the fact that I had put cyber as my number one choice, there were some people who were tracked cyber who had little knowledge of computers and whatnot. All hope is not lost though. There is opportunity to change career paths once you graduate.

Once you are assigned a career path, FBI Headquarters will assign you to a field office. As with career paths, there is an opportunity to rank the field offices you want to go to, but at the end of the day it is the needs of the Bureau that make the final determination. In my particular case, Salt Lake City was 17th on my list (out of over 50 field offices).

After getting my orders at the academy (about a third of the way through the 21 week course), I received an email message from my soon to be Supervisory Special Agent in Salt Lake welcoming me to the cyber squad. I arrived in the division in March, 2007.

There really isn't a lot that is set in stone before you go to the academy and, without a compelling reason, your orders are your orders. Worst case you could always quit but with how hard it is to get into the FBI academy (last I heard 1 in 5000 Special Agent applicants are given a slot at the FBI academy), very few people choose to do so (at least based on the people in my class). There are always opportunities to go to a different field office as well should people want or need to do so.

For people looking to get into the FBI and work cyber matters, majoring in computer science or some other information technology discipline would certainly be a good start. Being able to program is also a huge help for solving problems we run into every day but I do not know how much that goes into the selection process.

Beyond one’s choice in a particular major, in my mind it is much more important for a person to have a passion for computers. This, more often than not, leads to quite a bit of personal time being spent improving one's skills beyond what is taught in college. Most college classes are not at the cutting edge of what is out there and so, in my experience at least, personal experience and learning often trump what is taught in school.

In short, the FBI decides your career path based upon your background and college major (you must have a bachelor’s degree or better) once you are at the FBI academy.

AFoD: So what does a FBI cyber special agent in the Salt Lake City Field office do on a day-to-day basis?

Zimmerman: Hmm, that’s a difficult one to provide a succinct answer to.

A typical day will almost always involve paperwork of some kind, from
reading email to documenting investigative activities to requesting
authority for something. Other common activities include various types of training, meetings, and operational stuff like search warrants and arrests.

The squad you are assigned to will determine how many arrests and search warrants you are involved with. I have been lucky to have been involved with the Utah ICAC so we do search warrants and/or arrests just about every week. I have been on hundreds of arrests and searches in the time I have been in Salt Lake City. Depending on the nature of the case, you may have to travel for operational needs as well. I recently executed search warrants and an arrest on an Anonymous related case which required traveling to Ohio.

As for day to day activities, I usually have a mountain of email to answer both from internal and external sources. A portion of those emails are related to support questions for the programs I have written, others are asking for direction or how to best handle a particular computer related
matter, and others are for direct support of some initiative in the
office. I also get quite a few phone calls which come at all hours of the day.

Of course sometimes there are fires to put out so those have to be dealt
with as they come up. For example, a few weeks ago, several of us were called out to a search scene to conduct interviews and that ended up consuming the entire day.

Things tend to move in cycles a lot of the time, so there are times when
I will be buried in case work for a few weeks, followed by a period
where you are basically waiting for things to come back (a subpoena
request, search warrant return, etc.) When I am in the slow cycle of
case work, I typically focus on getting some extra programming done or
start looking into a new problem area we are seeing. I like the research
and development (R&D) side of things as it constantly presents new challenges. I tend to get bored doing the same thing over and over, so the time I get to spend doing R&D counteracts the often mundane nature of paperwork.

There are a lot of opportunities to use cutting edge software and
techniques as well. Some of these were developed by the FBI and some are commercial off the shelf software packages. For example, in a recent case I had a need to examine dozens of computers on a network but had to do so in a way that limited our exposure, both physically and on the network. Because of this, walking around to various computers and hooking up hardware to it was out of the question. I ended up using F-Response to facilitate access to all the target machines. This allowed me to view any of the hard drives on the network that were of interest. This access was completely transparent to the users of the workstations. Once the drives were exposed locally, I had a huge amount of flexibility in analyzing and reviewing computers, from taking a forensic image to pulling a few files for review.

Some of the other common activity would be reviewing evidence, imaging computers, reviewing intelligence products, going to firearms to qualify every quarter, taking various training such as legal training, defensive tactics, and specific computer related training (general classes or more specialized classes which result in industry certifications, etc.)

Cyber agents have quite a bit of mandatory training which range from A+
certification to a wide range of SANS courses. The training is divided up into four stages that roughly correspond to the first five years of a cyber agent's career. Outside of the mandatory training, there is opportunity to take various elective courses such as malware analysis and so on. The malware analysis classes are very challenging courses and, like a lot of the other classes, are taught by world class instructors and very accomplished professionals. There are also ways to pursue training which isn’t a part of the official curriculum if it can be articulated that said training is directly applicable to one’s job. For example, I was able to get my EnCase Certification since I do a lot of forensic work.

With all that said, and as I mentioned at the opening of this question, there really isn’t a typical kind of day (and that’s a good thing)!

AFoD: You are making quite a mark in the digital forensics community through your research and development efforts. What can you say in public about the work that you are doing and the investigative benefits that have come from it?

Zimmerman: The response to the tools and techniques has been overwhelming. As mentioned above, several thousand users in over 40 countries have downloaded one or more of my tools. I regularly get email from law enforcement officers and examiners with success stories as a result of using my software. These stories range from successful prosecution stories to the "I never knew that was there" kind of thing. I also regularly hear from people that they couldn't do their job without some of the software I have written.

To date I have released 13 programs ranging from file parsers and hashing tools to network monitoring tools. All of the software I write is provided 100% free of charge and, in most cases, comes with extensive documentation (some of the simpler programs do not require documentation).

Most of these programs sprung from either necessity or the fact that I was not happy with the tools currently out there. A good example of this is my hashing program. In one of the training classes I was in, we were provided a Java based hashing program which handled one kind of hash algorithm and was very unintuitive to use. Over the course of a few evenings in my hotel room, I wrote a replacement for it which included many more algorithms as well as many usability improvements. Over time I kept adding features and polishing the interface. As it stands now, Hasher is the fastest and easiest to use hashing program I am aware of.

I strive to provide intuitive interfaces in my programs which are, as we say in the FBI, "Agent proof." By this I mean the software programs are easy to use and hard to break. This isn't to say people haven't found creative ways to use the programs which I never thought of! Because of this, I have invested a significant amount of resources in providing a way to automatically report errors to me so I can fix issues as quickly as possible. This process includes sending an error report with a single mouse click which includes the complete stack trace, value of all variables, and even the line numbers where the error occurred. With this kind of information, correcting issues is much easier.

Related to this is the automatic updating feature of almost all my software. Gone are the days of having to manually check for updates on a website every few months. Now, assuming you are connected to the Internet, the software will tell you if any updates are available, download the update for you, and, when you tell it to, will apply the update and restart the program for you. This has saved thousands of hours of productivity as people do not need to be trained on the particulars of how to look for and apply updates.

Hundreds of people have downloaded my parsing and hashing utilities, but the most popular piece of software I have released is osTriage. To date, thousands have downloaded osTriage and used it to greatly extend their capabilities in the field. I am not aware of any other software at any price that provides as much information as osTriage does in such an easy to understand and consistently presented format. Most of the newer features are the result of user feedback. In most cases, new features can be added in a few hours. I am currently doing about four releases of osTriage a year, but the amount of new stuff depends on the rest of my case load.

Another major benefit is the improvement from the time a search warrant is executed to the time charges are brought against someone. Previous to osTriage, investigators may have to wait months to get back results of a forensic review. With osTriage, you have most of that data available to you before you leave the scene which can be used to pursue charges. In some states, a full forensic review is not even required anymore due to how well osTriage works.

A good portion of the forensic research I have done to date is directly available to law enforcement and forensic examiners for no cost. In any of my programs which deal with various kinds of forensic artifacts, detailed manuals are included which not only explain how to use the program, but also explain the exact layout of the files the parsers are dealing with. This lets examiners validate the tool using whatever means they choose to.

Most of this material is law enforcement sensitive and as such, I cannot get into details about the specifics of my work in this kind of open interview format. I fear we may lose some readers if we went too far down in the weeds anyways!

What I can say is that most of my work involved reverse engineering proprietary, closed source binary files and network protocols. The tools I used to reverse the files and protocols include packet capture tools, a hex editor, and custom testing programs I wrote to aid in decoding various chunks of data from the raw network captures. Some of the files were more trivial than others to reverse depending on what the purpose of the file was. Reversing the wire protocol was a significant investment of time over several weeks, but the result of the work has significantly improved our investigative efforts on certain networks.

I have also taken open source data (from Twitter for example) and written code to parse that information into a usable format. While Twitter provides a vast amount of data (as the result of a search warrant for example), it is essentially unusable in the format they provide it in. Rather than spend hours trying to correlate userIDs to usernames by hand, I spent a few hours and wrote a program to do it for me. Once that was done all it took was a little time working on a nice front end for it and now any investigator can immediately start using the return data for case work instead of busy work.

This last example is a good illustration of another major area of improvement my software has provided: the ability to automate tasks that used to be very time consuming. This includes such things as deconfliction systems which are used by law enforcement around the world to coordinate efforts related to investigating the exploitation of children online. Previous to my software, deconfliction was handled manually at just about every agency and no one had a means to share this data with each other. This resulted in a lot of duplicated effort and wasted time.

Even with all the gains in efficiency and the reduction of the more tedious aspects of various tasks, by far the best thing to come from my work is the rescue of at least 200 children who were being abused in 2011. We also saw over 300 arrests in 2011 in countries across the world.

AFoD: Can you provide a list of your tools and what they do? 

Zimmerman: Sure, here is a generalized list and a brief description where applicable (in no particular order).

osTriage: Live response tool that, among other things, finds image, video, encryption, virtual machine, archive, and P2P files fast. Live response data is pulled from the registry, via WMI, and various other files. This is by far the most capable live response tool available in my opinion. While it was primarily built for investigations related to child exploitation, it can be used for any investigation involving a computer as it provides such details as every USB device ever plugged into the computer (including make, model and serial #), full browser history for all major browsers, browser search history, dozens of registry keys such as MRU, PIDL, FirstFolder, TypedPaths, and many more, extracts passwords from p2p, email, chat, and other sources, full network details, ARP cache, DNS cache, open ports, running processes, installed software, and on and on. osTriage is very customizable and allows for adding additional items of interest beyond what is included. Anything 'of interest' is highlighted in red and moved to the top for easy visibility. Supports SHA1 base32 or MD5 hashes for image and video matching. There is MUCH more osTriage is capable of, but a full description would be a dozen paragraphs alone.

Hasher: Calculate file hashes using a multithreaded, easy to use interface. I have not found a faster or easier to use tool to date. Supports SHA1 base16, SHA1 base32, MD4, MD5, eMule/eDonkey, TIGER, WHIRLPOOL, SHA-256, SHA-512, RIPEMD-256, and CRC32 hashes and can recursively process files/folders. Also allows exporting of hash results directly to osTriage supported formats, Excel, etc.

Pingaling: Pings one or more IP addresses and/or host names and plays an audible alert when an IP address/host is available. I wrote this tool for use during surveillance of a suspect using a particular IP address at a hotel. Rather than watch command windows and the output of ping, this tool lets you put in the IP, start monitoring, and do whatever else you like. When the IP comes up, you will be alerted and can respond accordingly.

WMISpy: Monitor processes via WMI locally or across the network. Allows for setting up a list of executables to watch. Watched executables have their start and stop times recorded.

Web log parser: Parses Apache access logs, Apache error logs and IIS web server log files and allows for easy analysis. Also supports SSH logs and can perform DNS lookups. Has the ability to set up keywords to search for and will then give you totals of how many times those keywords were seen. I wrote this tool when I was working an Anonymous related case which involved a lot of SQL injection attacks, so I added keywords like sqlmap and Havij and then processed the web logs. I then knew exactly where the attack took place without having to do anything but open the log files. This saved me hours of time.

Twitter Parser: Takes a Twitter search warrant return and cross references Twitter IDs to usernames, allows searching and sorting of messages, and exporting into a variety of formats. Without this tool, Twitter search warrant returns are very hard to analyze.

I have several forensic programs which parse artifacts from various programs. I cannot provide details on those as they are law enforcement sensitive.

I have also written programs to generate certain types of files, monitor networks, and provide for global deconfliction of investigative efforts.

AFoD: Who is eligible to get these tools and how would they go about getting them?

Zimmerman: Anyone working for a law enforcement agency (or a company that directly supports law enforcement) as well as the military is eligible to get the tools. With very few exceptions, my software is available to every country in the world. All of the software is available free of charge.

We also provide hands on training at various conferences such as the national ICAC conference and Dallas Crimes against Children conference as well as more specialized training such as the FBI's Innocent Images basic classes.

There are also many subject matter experts around the country for the software which can provide regional training to agencies who cannot afford to send their personnel to conferences or other remote training venues. We are also looking to do some webinars for some of my programs which will provide another low cost method to provide training to users.

While live training is often ideal, great effort has gone into the manuals in order to allow for self-paced training. In the case of osTriage, reading the manual is the minimum requirement for FBI agents to use the software.

I understand everyone is at different levels of ability and knowledge and as such it was important to me to be able to provide a wide range of training options to people so as not to prohibit the use of a tool until some type of classroom training occurred. I also realize the things learned at training are not used all day, every day after the training is over. This is another reason why detailed, precise manuals are critical as they allow users to refresh their understanding of a program without having to attend some kind of refresher course or follow up training.

As to how to get the software, the URL to access the site will be provided in training or I can provide it directly to people who are interested. Since the site is not a public site, I cannot divulge the URL here, but anyone interested can email or call me and, after vetting the requester, I will provide direction for how to access the installer which allows for downloading all the software. I am easy enough to find on the mailing lists but calling the Sale Lake City field office is another way to contact me if needed.

AFoD: Your work has been so beneficial to the law enforcement community that you were recently recognized with a major award. Can you tell us about that award?

Zimmerman: I don't think I can explain the award any better than the first
paragraph from the press release:

"Every year the National Center for Missing & Exploited Children honors
law-enforcement officers who have demonstrated exceptional efforts in
the recovery of missing children and combat of child sexual exploitation."

My particular award was under the "Law Enforcement Excellence Award
Recipients" category. I was nominated for this award by a peer who
filled out an application which was sent to NCMEC and then reviewed by
NCMEC personnel.

My wife and I were flown to Washington DC where the award ceremony took place. After the ceremony, we were given a tour of NCMEC headquarters in Alexandria, VA. It was a wonderful experience and it was an honor to even be nominated let alone selected as a recipient of the award.

The full press release from NCMEC can be found at:

AFoD: What advice would to give someone who is in high school or college and wants to break into the digital forensics field?

Zimmerman: The best advice I can think of is to have a passion for the subject matter. Without a passion for the work, it will quickly become stale and tedious. Passion is what will take you through the mundane and grinding side of forensics. You can only look at hex for so long without passion before you go crazy.

My particular background education wise is mathematics and computer
science and I (perhaps biased I realize) think those two disciplines
serve as a strong base for a career in forensics. Both require
significant analytical skills which directly translate to working in the
field of forensics.

In my experience I have found much more satisfaction in pursuing things
beyond what is found on a curriculum. Planning and getting a curriculum
approved takes time and by the time those two things happen, the
information is rarely at the forefront of the discipline. With that
said, formal classes certainly serve to provide foundational knowledge
and provide the framework for more original work.

I also see huge value in computers being a hobby outside of formal
studies. Are you interested in protecting a network? Then setting up and
maintaining a mail server, web server, DNS server, etc. on a domain you
control is a great way to get your feet wet and learn the basics.
Protecting systems and data becomes a lot more meaningful when they
belong to you! Similarly, I find it helps immensely in terms of
motivation to have some kind of personal stake in your areas of
research. For me, part of my job is combating child exploitation. It
is very hard to be motivated about something you do not care about.

Another piece of advice I would recommend is this: Do not get hung up in
the ivory tower of pure academia. What works on paper is rarely 100% possible to accomplish in the real world. There are just too many unknowns (especially in the field; on a search warrant, etc.) to not be somewhat flexible in one's approach to forensics. You will quickly find yourself alone on an island (perhaps with a few others who cannot adapt) if you do not accept the fact that it is perfectly fine to deviate from standard operating procedure if the situation calls for it AND you can justify your decisions. As long as you can articulate your position and back it up, I do not see a reason to be fearful of going outside the box.

In other words, be a forward thinker and do not be afraid to question what is accepted as a standard just because it has been done that way forever. I think huge opportunities are missed more often than we realize because of a perceived need to "stay in the lines at all costs."

A good example of this is the movement toward live response from the
"pull the plug and take it to the lab" technique which most of us have
been trained to do for so long. The argument for the latter technique is
of course to avoid making changes to a computer and so on.

In reality, things are changing on a computer regardless if the plug is
pulled or not, so why lose data that can be of value? To me, forensics
should be based on being "minimally intrusive" vs. "do not change
anything." Of course minimally intrusive means being able to show what
changes you made to a system and the easiest/safest way to be consistent is via automation, i.e. live response programs.

Finally, I would like to comment on certifications. When it comes to
certifications I have mixed feelings. Before I joined the FBI, I was burned by people who looked good on paper but in reality could do nowhere near
what their certification made it look like they could do. Competency in
any field is much less a function of being able to pass a test than it
is to have a practical working knowledge of a subject AND the ability to
apply that knowledge to a problem.

I think there is value in certifications as it can be used to demonstrate knowledge and of course some employers like to see candidates with certain letters by their name, but do not get so hung up on getting a certification that you short change yourself in the long run by simply cramming information in your head to pass a test. For those that do, it is readily apparent to others when it is time to deliver.

AFoD: Let's talk a bit about tool development. Let's also go with the same scenario again. You have someone in high school or in college who not only wants to make digital forensics their career, but they also want to follow in your footsteps and develop great tools. How would they prepare themselves to create useful digital forensics tools? Are their certain classes they should take? Are their certain programming languages they should learn?

Zimmerman: As to how to best prepare themselves to create useful tools, the first and most important thing is this: have a problem to solve. I cannot overemphasize that point. It can be anything really, but without a problem to solve, teaching oneself to code is next to impossible as the motivation will just not be there as there will always be other things vying for your time. If spending a few dozen hours learning to program can save you and others around you hundreds of hours, it becomes easy to justify learning to program. Your goal may be as grand as finding the end of the Interwebs to making a simple GUI for your kids that has a few buttons on it that tells jokes when clicked. Either way, having that goal is critical.

Some examples of typical problems in law enforcement include something like taking a flat file and converting it to XML, parsing an Apache log file looking for signs of compromise, or writing a class to parse a binary file to be displayed to an end user.

Knowing your audience is also an important aspect to being a good programmer. Anyone can learn to throw some code together, but creating powerful programs that are easy to use is a skill that takes a lot of trial and error to get right. The only way to achieve this is code, code, code! After a few years, you will look back at your first programs and laugh at what a hack you were!

In my experience, one of the biggest weaknesses in programmers in general is the belief that their users are always nice people. One need look no further than the rampant issues with SQL Injection attacks these days to see how devastating blindly trusting end users can be. What this problem boils down to is programmers not doing adequate input validation (a note to any aspiring programmers: Never, EVER concatenate strings to build SQL statements. ALWAYS use parameterized queries!) of user supplied data.

Closely related to the issue explained above is the belief that end users will use a program exactly how you, the programmer, intended it to be used. It is almost universally true that the programmer will rarely if ever make a mistake when using their own program. By this I mean the programmer knows exactly what they were thinking when writing it and therefore know the exact steps to take when using it.

However, once you give a program to someone else, all bets are off as people are very good about using a program in ways the programmer never thought of. This ranges from everything to not checking the types of data being passed into the program (string vs. integer for example) to ensuring all necessary data is available before continuing.

Finally, finding a mentor or someone to bounce ideas off of is recommended. Anyone who has built anything of significance has failed many times along the way and being able to leverage such experience can do much to improve one's skills than the cruel lessons of trial and error.

While experience is often the best teacher, finding the balance between asking questions and being pointed in the right direction by a mentor can do wonders to keep motivation high and prevent burnout.

When it comes to classes to take, almost every computer science curriculum will include several programming languages as a requirement, so people will more than likely get exposed to the basics in a few different languages. C and/or C++ will almost always be included in courses as well, especially if the class involves Unix or some BSD derivative. While I took a few C classes here and there, I never really developed anything of any significance with it because it takes a lot of work with little to show as far as GUI development goes (at least for the kind of problems I was trying to solve).

A good database class is always helpful too as you will almost always have to persist data in some way in every application. Learning how to properly create a database schema and then manage that schema is a critical skill which will save a lot of pain down the road. When it comes to databases, it is not normal to not normalize.

Taking a class on file systems is highly recommended as you will almost always be dealing with a file system when it comes to forensics. File System Forensic Analysis by Brian Carrier is a fantastic resource for jumping into file systems outside of any class that is offered in a formal curriculum.

There are a lot of excellent resources out there for things other than file systems too. Harlan Carvey's Windows Forensic Analysis is one such example that touches on a lot of key areas not only in regard to Windows, but the incident response process in general.

Personally, I bet I came close to reading one or more books like the ones mentioned above for every college class I took. More often than not, I got more out of the extracurricular reading than what was presented in class.

Finally, which programming language someone chooses to focus on really depends to some degree as to what you want to accomplish. If you plan on writing Windows programs with full blown user interfaces, either C# or would be great choices. I am a developer myself. I started using Microsoft Access and VBScript about 15 years ago and once I outgrew that, moved to VB6. Once .net came out I migrated to that platform.

Similar to the PC vs. Apple debate (the PC is clearly better of course!), there is quite the battle between C# and in some circles. C# has traditionally received new features before VB but once Visual Studio 2012 comes out, the gap between VB and C# will be nearly closed. At the end of the day, any .net language would work as it all ends up as the same intermediate language (MSIL) at some point.

If I was going to recommend a language to someone wanting to learn to program it would almost certainly be Python. Not only is it incredibly powerful, but it can be run just about everywhere and that flexibility will come in handy. I would not want to develop any kind of GUI in Python (though I am sure it’s possible, it just doesn't look polished nor does it have the powerful 3rd party components that are available on the .net platform), but for general scripting and parsing programs, Python is hard to beat. Once you learn the basic constructs of programming (loops, if then statements, functions, etc.) you can then simply learn a slightly new syntax in other languages. By choosing something as commonplace as Python you are ensuring you will be able to execute your code on a wide variety of platforms.

At the end of the day the programming language one decides to use doesn't matter as much as what the program can do for people. You can be an amazing programmer and solve difficult problems, but if your program is hard to use or doesn't provide meaningful information to end users you will find people just won’t use it.

For example, if your program pulls a 64-bit timestamp out of the registry or some binary file and reports it as such to an end user, you just alienated the vast majority of computer users out there as most won’t have a clue what to do with such a number.

Now if you are writing some kind of low level API that carves files into objects or something that is one thing (and it can be argued that you should convert it to a meaningful datetime regardless), but if you write programs with the intention of "normal" users consuming such data you have missed the boat.

As I mentioned earlier, the moral of the story is to know your audience and tailor the output to that audience. Just as with teaching a class or lecturing, if you aren’t bringing things to a level where most can understand it, you aren't being as effective as you should be.

Finally, I am a believer in agile development vs. the classic waterfall approach. Simply put, agile development means letting the actual end users of the product use the tool as it evolves vs. waiting for a program to be 'finished' and then having end users request a myriad of changes as the program doesn’t do what they envisioned it to do.

Remember, the end users are not programmers (if they were they wouldn’t need you), so they will, more often than not, lack the ability to describe exactly what they want in a way that translates to crafting a program. By putting incrementally changing versions of your program in the hands of the people who will ultimately be using the product you will, in my experience, find it much easier to come up with a product people want to use at the end of the development cycle.

My recommendation after identifying a problem is to find a group of people who understand what they want to get from a program in order to accomplish their job. For example, if your problem is developing software to track pedophiles online, find a group of law enforcement officers who are experienced enough to know what they need through all stages of the investigative process AND be able to identify the gaps they currently have to deal with. It is this group of people who should be the ones using your programs as incremental changes are made so they can steer the development along to best solve the problem that caused you to undertake writing a program in the first place.

AFoD: Any final thoughts? 

I would like to close by saying that it has been a privilege to be a part of the forensics community for the past few years. I have met many dedicated professionals who deeply care about their craft and are passionate about computers as well as highly motivated law enforcement officers who are underpaid and underappreciated. Your work impacts people far beyond what get to see every day.

Be safe out there everyone!

P.S. The opinions stated, unless clearly indicated otherwise, are my own and not that of my employer.