My thoughts on a decade of Cyber Security: 10 Lessons I’ve learned

I’ve spent a little more than a decade working in cyber, starting with fixing computers at my local pizza joint to leading a security operations center (SOC) for a multibillion-dollar enterprise with thousands of users and hundreds of thousands of machines. Here are ten lessons I’ve learned along the way.

1. Cyber is risk and nothing else.

It’s important to understand that businesses think in terms of money and risk. If the potential money gain outweighs the risk of money loss, then that idea is likely to go forward; however, the opposite is true. If something is more likely to lose money, then that idea is likely to get stopped. Cyber security falls squarely in the latter category.

Businesses are constantly looking at what is costing them money and what’s bringing in money. Cyber security is a money pit. Money gets poured into cyber security, but it does not make any profit. Some companies have understood how reliant their business is on cyber, especially in the COVID era where online shopping is the safest way to shop, but understanding reliance on cyber does not translate to understanding reliance on security.

A CFO once asked why we spend so much money on antivirus when we haven’t had a virus outbreak in 5 years. This question struck me because the answer seemed obvious; we haven’t had a virus outbreak because we have antivirus. But to the CFO, who’s only looking at the dollar, this seemed pointless. Why spend hundreds of thousands a year to put AV on every desktop when we haven’t had a virus in years. He was thinking reactive, and I was thinking proactive.

This shift in mentality is where I see much room for improvement in traditional SOCs. We need to learn to speak like a finance officer. We need to be able to explain to them how proper cyber security enables their money-making activities.

Going back to my antivirus example, I learned that I needed to speak the language of finance; I needed to talk in terms of money.

2. No one cares about your stats.

This is a direct continuation of the previous point, but it’s important enough to list on its own. No one cares about your stats.

Businesses love stats and metrics. They hire teams of statisticians to make crunch numbers all day and figure out how to squeeze an extra 1% efficiency out of every sector because that extra 1% means more money. The truth is, though, no one cares about your stats because your stats don’t mean anything.

When I worked as a SOC analyst monitoring firewall logs and IDS alerts, I had to report metrics at the end of every shift. We reported things like how many firewall blocks happened, what our top IDS alerts were, how many, if there were any true positives, and several other menial points of data. Out of all that data, there was only one that mattered, and it only mattered to the CISO; were there any true positive IDS alerts.

The metrics we were reporting didn’t matter because they didn’t speak the universal language of business. A statistician can’t take firewall hits and translate that into saved money. We were never reporting AV blocks because it just wasn’t in our SOPs to report it. So, a CFO never saw that he spent $500K a year to blocked thirteen threats a month. Even if he did $500K for thirteen threats sounds like much money for not a high return.

We must translate stats to money. When asked about the AV, I didn’t have a satisfactory answer because I never stopped to think like a CFO. I spent the next month trying to figure out how much an incident cost the company. I spoke with the HR department to figure out what the average pay of an Incident Responder was so I could calculate overtime pay, I spoke to finance to see how much money it cost in server downtime, I talked to DevOps to figure out how long it would take to restore a server, and I spoke to the incident response team to figure out how long it took them to respond. After crunching all the numbers, I figured out that it cost around $50K per incident.

Armed with this knowledge, I looked at how many AV hits we had each month and found that AV saved the company about $650K each month. The CFO was shocked when he heard that number; spending $500K a year saved them almost $7.8M yearly. And that is the point about stats. No one cares how many events your firewall blocks, but they do care about how much money that saves. Learn to translate those stats into cash. Don’t report just failures; report how the security apparatus is working and how much money it’s saving the company on average.

3. Understand that not everyone is as smart as you

It should be obvious to everyone working in this profession, but cyber is broad. Some people are expert malware analysts but can’t make heads from tails of packet capture (PCAP), and consequently, some people can rip a C2 beacon out of a PCAP but can’t find PowerShell usage in Window’s event logs. The point is that cyber is broad, and no one is an expert in all things.

I’m not giving you carte blanche approval to be a jerk but understand that just because you’re a master with Wireshark doesn’t mean everyone else is. What may seem obvious to you may not be evident to someone else. Understand that your peers may not have the same knowledge, experience, or analytical reasoning that you do, and by extension, you don’t have the same that they do.

You may need to take time to explain your thought process to others. It’s not an insult to your intelligence when someone asks you to present your reasoning. It’s far more likely that this is something new to them. At the same time, it’s not incompetence to ask others to explain their reasoning because you don’t understand. I’ve had to explain my thought process on multiple occasions when I open an incident, and on almost every one of those occasions, it was simply because someone else didn’t see the same indicators in the traffic that I did. After a quick explanation, we agreed on opening an incident.

People are human; they miss things. We all miss things. Don’t be an a** when someone asks you to explain your reasoning.

4. Stop with the playbooks.

Playbooks are good, but too many playbooks are a hindrance. It’s essential to have some playbooks for ordinary things; what do you do if ransomware happens? What happens if you get malware on the network? What do you do if there’s a breach of a critical service?

I mention this because I have worked in multiple places that tried to either create too many playbooks or wanted extremely specific playbooks. One office I worked in wanted a playbook for every variant of ransomware. How do we respond to Locky Ransomware? What about for NotPetya? WannaCry?

We also had playbooks for botnets, malware, crypto miners, etc. Every new family of malware needed a new playbook. We spent so much time making playbooks every time we had an infection that we probably missed more than we found. None of these playbooks were indexed correctly, of course, so whenever we had an investigation, we had to hunt through shared drives and folders until we found the appropriate playbook so we could follow the “approved” process for dealing with it. Was Dridex in the botnet folder or the banking folder under malware? Wait, it’s in both? Which one is more updated?

On the opposite side of the house, I’ve had some bosses who wanted playbooks so specific that we hardly had anything usable. You can only use this playbook when the Dridex malware infects exactly three computers on the same VLAN and communicates with a C2 server in China. Playbooks like this were worthless because they never got used since incidents seldom met the criteria.

Playbooks are good, but they need to be at the appropriate level.

5. Read the news for your boss.

Don’t read the news to your boss; read it for your boss. I can’t even begin to count the number of times I’ve had my boss ask me about some ridiculous thing they read about on CNN or Wired.

When the Pulse Secure VPN exploit made significant news with a CVSS score of 10, I got the distinct pleasure of answer a barrage of questions about what we’re doing about this. My answer was simple; nothing. We’re doing nothing because we don’t have Pulse Secure VPNs on our network.

We should never be caught off guard by things that show up in popular news outlets. You can be forgiven for not knowing about something that appeared on an obscure Twitter post, but if something gains enough steam to be reported on by a major mainstream news outlet, then you should know about it before your boss.

Telling your boss that you don’t know about something that hit mainstream news hurts your credibility as a professional, but more than that, it can lead to extra work for you and your team. I’ve been involved in incident response operations that kicked off because someone’s boss read an article on CNN and wanted answers. When his SOC team lead couldn’t speak on the matter, the CISO tasked the incident response team to investigate.

We could have, and did, tell the CISO and the SOC that we weren’t vulnerable to what they read about, but the gears were already in motion, and everyone was wound up, so we went out to respond to a threat that didn’t exist in the first place.

6. Blackhat is mostly pointless.

I’m going to make many people mad here, but I passionately believe that conferences like Blackhat cause more issues than they solve for those of us in the cyber trenches. Blackhat, Defcon, RSA Con, Threat Intelligence Summit, etc., are suitable for introducing the world at large to up-and-coming threats to unique components, but for the vast majority of SOCs, it doesn’t matter.

I’ve said for a long time that the things you were concerned about before Blackhat are probably the things you should still be concerned about after Blackhat. Most of what gets talked about likely won’t apply to you if you’re a traditional IT network. Even if it does apply to you, it’s essential to understand the level at which it applies to you.

Look at the concept of Google’s Row hammer exploit. It made significant news because of its potential on Cloud Virtual Private Server (VPS) providers when introduced. A threat actor could use a vulnerable virtual machine (VM) to manipulate the memory of the physical server to extract data from a different VM on the same physical VPS. This concept made significant headlines, but Microsoft determined it was complicated and generated sub-optimal row activation sequences when tested.

Even though this was technically feasible and potentially devastating, it was not worth worrying about unless you’re dealing with data of National Security. A Blackhat conference may disclose a way to exfil data out of a VM at 1Kbps but is that something you should worry about? At that speeds, an attacker might get a single word document in about a week.

What you were concerned about before Blackhat is what you should still be concerned about after Blackhat.

7. Location, Location, Location

One of the most challenging concepts new SOC analysts seem to deal with isn’t investigating new malware strains; it is understanding the location of our sensors and what they can provide.

There are many logs in your enterprise right now; if your System Information and Event Management (SIEM) is correctly set up, you probably won’t see all the diverse ways logs come to you, but you probably have several. IDS logs, Firewall logs, DNS logs, proxy logs, IPS alerts (if separate from IDS), Window’s event logs, Domain Controller logs, Linux server logs, antivirus logs, and more. Not every source of records shows the same type or amount of data.

In my anecdotal experience, one of the most challenging things for SOC analysts to learn is where to look for distinct types of information. SIEMs help correlates all these logs to give analysts an easier way to move between them, but a good analyst still understands what sensor collects and how to move between sensors in an investigation.

Let’s say that you see a connection to a known bad IP address. Depending on the location of the sensor, you might have more or less information. You might need to query other sensors for what you want.

When I started working in a SOC, we didn’t have complete log ingesting in our SIEM. The SIEM got firewall logs, but we also had an IDS outside the firewall, and different subnets had their edge routers (yes, it was a double NAT) with their IDS behind the router. Depending on where you looked, you might not see the actual host IP address as everything would be behind a NAT.

I’ve also consulted on the location of new sensors to help fill in some gaps we had before I could answer that I needed to know where we already had sensors and what they could see.

8. You’re probably doing threat intelligence wrong.

This one will be a hot take, but you’re probably doing Cyber Threat Intelligence Wrong. Looking up IP addresses in VirusTotal is not threat intelligence. Same with domains and file hashes. Your SIEM should be automatically enriching your data with this type of information. Reading reports and checking Twitter for IOCs to plug into your SIEM because your CFO doesn’t want to pay for Crowstrike’s premium addon isn’t Threat intelligence. True threat intelligence is a nebulous concept, but I’ll try to provide my take on it.

My first job working in a SOC was in the threat intelligence section. My job was to read reporting grab IOCs (mainly IPs, domains, and Hashes) then search them in the SIEM to see connections. The job was frankly a waste of time. Using a feed to enrich our Splunk data quickly made my job obsolete as everything I was doing became automated.

While doing threat intelligence, I asked our incident response team how often they used threat intelligence in their operations. The response was a resounding never. One of the responders even went as far as to say that intelligence actively hurt one of their ops.

I spent some time thinking about what type of intelligence they would need. Your intelligence team needs to understand the critical business assets, the vulnerabilities around those vital assets, what capabilities exist against those vulnerabilities, and what sensors are in place to detect them. Threat intelligence should read about how attackers conduct campaigns, map that to the MITRE ATT&CK matrix, and then overlay the content they can detect. Using all of this, we can create predictive intelligence to help incident response find previously unknown compromises.

9. Don’t write to be understood; write so that you can’t possibly be misunderstood.

Report writing is something we’re all going to have to do at some point. Far too frequently, we write assuming the people reading our reports have the same knowledge level that we do, except that’s not always true.

My team wrote reports that went all the way up to the CEO. And we rewrote the report when the CEO had no idea what we were talking about. Years ago, when I was just a threat analyst, I wrote a report about indications of a worm on the network, only I never mentioned a worm. I forget what the report said, but I remember getting into a heated discussion with my boss about the wording.

She read the report and told me flatly that she didn’t see any threat. I was aghast because I thought it was apparent where the threat was, but she didn’t get it. I explained that this was an indication of a worm, and she told me now she understood, but if this reached the C-Suite folks, they wouldn’t get it. I was an arrogant young analyst and replied that it wasn’t meant for them, so it didn’t matter if they understood it.

Much like my third point in this post, I didn’t consider that others who want to know might not be able to understand what I’m saying. I wrote my report to be understood by a minimal subset of people.

Writing not to be misunderstood also means writing clearly using established terms, frameworks, and a common lexicon. How many times have you heard people refer to the Eternal Blue vulnerability? Eternal Blue was an exploit; the vulnerability was CVE MS17-010. It’s essential when we’re writing to be specific and accurate because you never know who will be reading it.

10. Make friends with your Marketing team.

When I started my career in Cyber Security, I held the, unfortunately, typical belief that art degrees didn’t belong in Cyber Security. Don’t pollute my science with your art; boy, was I wrong.

Art has a place in the realm of cyber security, and marketing can be a significant ally for you when you’re trying to convince the leadership of something.

One day I was trying to explain to my boss how a malware campaign was propagating through our network and why we needed to take specific actions to stop it. She didn’t understand what I was trying to convey. We gave her our 23-page report on the malware, which included many granular details and screenshots of hex code, PCAP, C code, and log files. To this day, I’m convinced she never read it because it was far too long. She wouldn’t advocate for our suggestions because she didn’t understand what we were suggesting.

I was trying to figure out how to get my message across, I consider my public speak skills to be above average, so if I couldn’t convince her with a presentation, I needed something else. My team was brainstorming, and eventually, someone said, “what about an image?”. We all agreed that an image was worth a shot, but none of us were graphic designers, so we were ready to write that idea off or try to make something with PowerPoint when I had the idea of asking an actual graphic designer.

I went over to marketing and explained the situation to their team lead. He listened for a while and then took me over to see their graphics design team. I met with the graphics design folks and explained the situation again when an absolute gem of a human said she would. What followed was three days of back-and-forth communication as she desperately tried to understand what we were saying and translate that into an image. At the end of the three days, she had created a single image masterpiece that clearly illustrated the points we were trying to get across to leadership.

Art and marketing have a place in cyber security. When you think about it from a business perspective, I was trying to market my positions to my boss, and who is better at selling a position than the team explicitly made to market products?

If you look at significant threat intelligence companies like Talos or the Microsoft Security Threat Intelligence Center (MSTIC), you might notice that most of their reports also have some clear graphics to go with them. A well-designed image can convey the same information as a 23-page report.

Conclusion

My time in cyber security has been a wild journey, and I’ve learned a lot. I hope these ten lessons will also be helpful to you on your journey through this career.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

WordPress Appliance - Powered by TurnKey Linux