Several years ago, I was responsible for interviewing applicants applying for a threat hunting position. I was one of a few managers who would interview the applicants separately and make recommendations. Some of the interviewers focused on knowledge; where is the SHIM cache located, what file stores user passwords in a Windows environment that isn’t domain joined, what are shellbags, etc. Others focused on the individual’s personality; what do they want to do, how do they work with teams, greatest mistake, etc.
I however focused on the applicant’s mindset. Did this individual think like a threat hunter. See it’s easy to teach people knowledge. I can teach an individual what the SAM file is, or which keys in the Registry Hive are typically used to establish malware persistence; but it’s difficult to teach someone how to think like a threat hunter. This is why I would only ask one single question during my interviews. I was confident that this one question was enough to tell me if someone was capable of thinking like a threat hunter (or at least learning to think like one).
Since I’m not a hiring manager anymore, and I’m not disclosing the name of the company, I can now share this question and explain why I thought it was so useful.
When I was interviewing people, I would provide them with this network map and ask them to explain their threat hunting plan to hunt for an unknown adversary in the network. I would provide some additional caveats; specifically, I would inform them that they would be working with Security Onion and only had 2 Sensors to place in the network.
I picked Security Onion because of its opensource vendor agnostic nature, and because Security Onion is a legitimate tool I’ve used in Hunts. However, I would let applicants know that this wasn’t a quiz on their knowledge of Security Onion and if they weren’t familiar with some of the capabilities of the tool that’s fine, I can explain them.
What’s the right answer?
That’s the beauty of this question; there isn’t a right answer. There are a lot of right answers and a lot of wrong answers, but there is no one right answer. So, instead of trying to explain the right answer I’ll address some of the approaches towards answering this question.
The Key Terrain Approach
This is perhaps the best approach to take when preparing to hunt on a new network you know nothing about. I’m going to link this back to my days in the military. In the military we have a doctrine that discusses a topic known as Key Terrain. In the Army doctrine Key Terrain is any terrain that gives a combatant an advantage. Key Terrain Cyberspace (KT-C) is a relatively new and developing concept but it’s an important idea to understand.
When you don’t know anything about the network where should you start looking? Well, you need to know what parts of the network are important to the operations of the business. This is called the Key Terrain, and that’s where you should start hunting.
Take a look at our network here. It’s a warehouse, so its primary functions are to import items, store them, and then ship them out on request. This is the primary mission of the Warehouse (to go with Military terms again). In order to do this mission, the warehouse needs some things. The wireless network which powers the handheld scanners and the SQL server which records all the moving items are both examples of things required for the warehouse to function.
Understanding this key terrain would the suggest that we place one of our sensors at a location where we can collect information from the SQL server.
I don’t provide any of this information upfront because I want to see how an applicant thinks. Do they begin by trying to understand the key terrain and mission requirements of the business they’re hunting? Or do they immediately start deploying sensors?
These interviews were back and forth conversations, so if an applicant spent our entire interview just asking questions about the key terrain, then I would consider that a win and would invite them back from the next phase of interviews.
The Adversary COA Approach
Another approach that I liked to see was the adversary centric, or adversary course of action (COA), approach. In this approach the applicant would create an adversary’s likely course of action and then proceed to deploy sensors in a location where they would be able to detect the COA they came up with.
This approach does not consider KT-C but that’s not always a bad thing. Think again about our warehouse example. How likely is it that the adversary is going to get malware on a handheld scanner? It’s relatively unlikely, so while the handheld scanners may be KT-C they may not necessarily be relevant to the hunt. This is referred to as Mission Relevant Terrain Cyberspace (MRT-C). What equipment in your network is relevant to the mission? Do you spend time trying to figure out how to monitor a handheld scanner or do you focus on things more relevant to what an adversary can actually do?
Look back at our network map. Do you see the break room subnet? These computers are connected to a commercial internet rather than the company’s private intranet; however, the router is configured wrong allowing the break room subnet to talk to the rest of the company’s intranet (something I’d tell the applicant if they asked how the router is configured).
With an Adversary Centric Approach, the applicant might identify that the break room computers (with open access to the internet) are a soft spot an adversary might gain initial access to. That applicant might theorize that a warehouse employee would click on a phishing email while on break and download malware to the break room computer. The applicant might then theorize that an attacker would use exploitation for lateral movement and hit some of the companies more vulnerable systems using high success exploits like EternalBlue.
Using this hypothesis, the applicant would then place their sensors in a location where they could detect the lateral movement between the break room subnet and another system like the SQL server.
I would also consider this approach perfectly valid, and an applicant who took this approach would be asked to come in for follow on interviews with the rest of the team.
The Capabilities Based Approach
With this approach an applicant would first seek to understand what tools they have access to and what capabilities those tools possess. They would also seek to understand what intrinsic capabilities already exist inside of the environment and to what extend those capabilities are being used.
Look at our map; the main router is a pfSense. A pfSense router has a number of different capabilities include Snort and Suricata IDS/IPS, Zeek IDS, and pfBlockerNG-devel. Are these tools installed? If so, are they configured, and how are they configured? What about the switches? They’re Ubiquiti Enterprise switches, which means they’re managed switches. Are the switches configured to SPAN port traffic to some sort of collector?
An applicant using this approach would look to place their sensors in a location where there are gaps in coverage. They’d seek to leverage the capabilities already existing in the environment and then add sensors where the capabilities don’t exist or aren’t sufficient.
This approach would also net the applicant a follow-on interview.
The Data Driven Approach
The last approach I want to talk about is the data driven approach. You may have noticed that I didn’t provide a lot of information about the network. I intentionally gave very little information because that’s an unfortunate reality of some of the situations I’ve found myself in. I’ve been told my team is going to be hunting on a network in a week and we were handed a map with no information with which to plan our hunt.
What were the ACLs on the Router? How are the VLANs set up? What’s the security on the Wireless? Do the end points have a host base security suite on them? What is their patch level? What about EDRs? What OSes and what versions are in the environment? How long is log storage? Do the systems have a DLP policy which blocks USB? What software is running on the systems? What is its patch date of that software? And more.
All of these questions are important and can drastically change how a hunt team prepares and deploys in a network. If I know that the endpoints have a DLP tool which blocks USBs then maybe I don’t need to focus my efforts on hunting for artifacts of a USB malware.
An applicant who starts digging into the network this way would likely be a good hunter as they can start eliminating things the hunt team needs to worry about.
Why these approaches?
I focused on these approaches specifically for two reasons. First because I want a Hunter to do all four of these approaches simultaneously, constantly, and with ever hunt operations they go to. I used to train my analysts to go through these approaches when training.
The second reason is because it’s more important to have a solid framework than a laundry list of facts. Anyone who has done threat hunting knows that you can’t just “find bad”. Sure, maybe if you have a single computer, you’re investigating then pulling all the event logs and scrolling though until you find bad works. It’s not efficient but it can work (lord knows I did that when I started). However, a network of even as few as 5 computers will render this approach completely broken.
I’ll say that again. Simply deploying sensors, pulling back data, and hoping to find bad does not work. This question also gives me the benefits of being able to dig deeper to really understand the applicants through process.
Digging Deeper
With each of these approaches I can dig deeper to see how they’re thinking. Let’s say that the applicant wants to deploy one of their two sensors at the switch between the HR subnet and the main network. I would follow up this question by asking “How do you plan to deploy that sensor?” If the applicant says they’re planning to deploy the sensor in-line, then I would ask them “what considerations do you need to make when deploying in-line?”
The consideration I’m looking for here is the applicant to recognize that deploying in-line means bringing a portion of the network offline for a short time while they deploy sensors. I’ve found that a lot of network owners aren’t thrilled with the idea of you bringing down a portion of their network for even a few minutes to deploy equipment.
If and when an applicant realizes this they’ll have to come up with an alterative way to get the same information. Commonly that would be SPAN porting the switch. If the applicant says that then I’ll ask what considerations do they need to make with a SPAN port, expecting answers like how the total bandwidth of the network might be greater than the speed of a single SPAN port potentially resulting in dropped packets.
These questions can go deeper and deeper until we run out of time or the applicant can’t answer anymore. These deep dive questions aren’t strictly speaking necessary but they give a good understanding of how strategic the applicant thinks. Someone who is able to navigate through these considerations will probably be a good hunter; however, a lot of these considerations are things that come with experience so someone who can’t answer isn’t necessarily a poor choice.
What doesn’t work?
I’m going to quickly list out a few things I’ve seen that don’t work.
- Complaining
- This should be obvious but complaining about how unfair the question is won’t get you a follow-on interview
- Saying there’s not enough information
- As I mentioned above, there have been plenty of times I’ve been given almost no data to work with. A good hunter needs to learn to ask questions and pull in additional information.
- Listing facts of what you’re going to pull and from where
- This is by far the most common response I would get from people who weren’t familiar with hunting but took an Incident Response course. They’ll start telling me how they’re going to check the HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run key to look for a persistence mechanism or they’ll pull the C:\Windows\System32\Config SAM file.
- These factoids are great, but when I dig a bit deeper usually the knowledge fails. Why are you checking the RUN key? What are you looking for? How did an adversary put malware there? What are you looking for in the SAM file?
Issues
I recognize that this method might have some issues. Its possible that an applicant could have extremely technical knowledge but not a good vision for hunting. I consider this situation less important because teaching someone how to think is more difficult then teaching them factoids about systems.
I’ve also had the opposite happen where I approved someone who was good at thinking about hunt, but didn’t understand how to actually do anything they came up with. Generally we can teach people this but occasionally we’ve hired someone who simply didn’t want to learn.
Conclusion
This was my favorite question to ask applicants when hiring for a hunt team. The open ended nature of the question closely resembled real world scenarios I’ve dealt with. Asking a question like this helps me understand if the person I’m interviewing can think like a hunter or not.
What do you think of this question, and how would you answer it?
