You probably haven’t heard but there’s a minor new vulnerability making the rounds. This silly vulnerability (CVE-2021-44228) affects a couple of devices here and there with a relatively difficult single line of code input anywhere that writes to the log file. I don’t really see the big issue here, it’s just….an exploit that allows complete remote code execution with a single line of text that can be entered anywhere.
So the above text is a bit sarcastic if you couldn’t tell, but sometimes you need a bit of humor with something so ridiculous.
Log4J was a Zero Day released a few days ago which has been taking the world by storm. Log4J is a Java plugin built into, well almost everything. This plugin has a vulnerability in the Java Name and Directory Interface (JDNI) which interprets certain strings as a command to query an attackers LDAP server to download and execute a malicious binary. The concern with this vulnerability is that the string can be entered into any input field. This means that any comment, username field, fillable form, or other text field on a server with a vulnerable version of Log4J where you can put this string can be a potential attack vector. That sounds bad on it’s own, but it’s not limited to just websites; anything that uses Log4J can be a potential victim. Defenders across the internet have been reporting attackers Wardriving with a Wi-Fi name set and exploiting devices which log Wi-Fi names.
This weekend has been a mess of checking servers, patching, and planning mitigations for devices that can’t be patched. I can’t discuss the mitigations of my work due to an NDA but I can discuss the ways I’ve mitigated at my house.
Finding Log4J
Software auditing and management is a major problem in the cyber security world. Applications and programs aren’t individual bits of code anymore. A single program may call hundreds of different library of code and functions. A basic WordPress website, like this one, is more than just the code of WordPress. I have an Apache server, MySQL, Php, themes, and plugins. All of 5 of those things also relies on different libraries and different code bases.
I have a lot of different servers in my house, and not very good documentation on what underlying subsystems they were running. Before I could mitigate this vulnerability I first had to find it.
Thankfully my servers are Linux, so if Log4J existed somewhere on the system it would be a simple binary, meaning I should be able to find it fairly easily.
sudo apt update && sudo apt install locate && sudo upgradedb && locate log4jThis command on my servers would pull back the location of log4j binaries loaded in the server:
/opt/tplink/EAPController/lib/log4j-api-2.8.2.jar
/opt/tplink/EAPController/lib/log4j-core-2.8.2.jar
/opt/tplink/EAPController/lib/log4j-slf4j-impl-2.8.2.jar
/opt/tplink/EAPController/properties/log4j2.properties(This was a sample of what the output would look like).
Apply Patches
With my problem files located I could manually patch all of my servers or I could try script it. Since I had a lot of servers I decided to go with the scripting option. I haven’t finished this script as I’m having difficulty getting replacing the files in an automated way.
#!/bin/sh
#This script is incomplete
sudo su
apt update
apt install locate
updatedb
locate log4j | tee /tmp/log4jlocations | grep -oE "(log4j-[a-z0-9]{1,5]-?([a-z0-9]{1,6}-)[1-2]\.[0-9]{1,2}\.[0-9]\.jar)" | tee /tmp/log4jversions | grep -i "api" | tee /tmp/log4japis
cat /tmp/log4jversions | grep -i "core" | tee /tmp/log4jcores
cat /tmp/log4jversions | grep -i "slf4j" | tee /tmp/log4jslf4j
wget https://dlcdn.apache.org/logging/log4j/2.15.0/apache-log4j-2.15.0-bin.tar.gz
tar -xf apache-log4j-2.15.0-bin.tar.gz
mkdir -p /tmp/updatefiles/apis
mkdir /tmp/updatefiles/cores
mkdir /tmp/updatefiles/slf4j
for i in $(cat /tmp/log4japis); do cp ./apache-log4j-2.15.0-bin/log4j-api-2.15.0.jar $i
mv *.jar /tmp/updatefiles/apis; done
for i in $(cat /tmp/log4jcores); do cp ./apache-log4j-2.15.0-bin/log4j-core-2.15.0.jar $i
mv *.jar /tmp/updatefiles/cores; done
for i in $(cat /tmp/log4jslf4j); do cp ./apache-log4j-2.15.0-bin/log4j-slf4j-impl-2.15.0.jar $i
mv *.jar /tmp/updatefiles/slf4j; done
rm -rf ./apache-log4j-*
The script currently downloads a patched version of the Log4J and renames it the the specific versions used in all of my programs. From there I manually placed moved the renamed patched Log4J over the unpatched version previously there. Reboot the process and now the patch is in place.
I’m still working on improving the script so I can automatically have the right file moved to the right location but right now, but that’s a project for a later day.
I also want to specify that this may not be the ideal way to patch these system. On every system I’ve tested so far this has worked, but I can’t guarantee that simply replacing the jar file will work on every system. If you have a system that requires high availability then I wouldn’t recommend this method. You’ll want to wait for the vendor to issue a patch.
Blocking at the Edge
For the systems I wasn’t able to patch, or wasn’t sure if the patch would work (like Docker environments) I went about deploying mitigations at the edge using Snort and CloudFlare.
CloudFlare has already begun deploying mitigations in their WAF to block malicious packets, but I have some servers not behind CloudFlare (because they would violate CloudFlare’s ToS). I needed to look at how to block these packets. I took a series of steps to block this, starting with Snort
Snort Rule
In most of the documents I’ve read the attack has a few commons. It’s a User-Agent, with JDNI, LDAP, and usually an IP address.
drop any any any -> $home_net 443 (msg:"Log4shell attack"; pcre:"/User-Agent: \${jdni: (ldap|dns):\/\/(([0-9]{1,3}\.){3}[0-9]{1,3})\/.+}; sid:5555555; rev:1; )Firewall Rules
From what I’ve researched the attack relies on the vulnerable server making an LDAP connection out to an attackers domain. Based off this I went through and blocked LDAP and LDAPs on my network by blocking ports 636 and 389.
Drop IPv4 TCP/UDP * * * 636 *
Drop IPv4 TCP/UDP * * * 389 *IoT Devices
As I started to think through this attack more I can to a disturbing realization. I had no way of checking if my IoT devices were vulnerable to this or not. I started thinking through ways that an attacker could compromise an IoT device and I came up with the following.
An IoT device could be scanning the RF spectrum logging Wi-Fi names when a hacker Wardriving rolls by. The IoT device sees the Wi-Fi name and logs it using Log4J. The IoT device makes an LDAP query out to the attacker server and downloads the malicious payload for execution. Now the IoT device is infected and controlled by an adversary.
IoT Devices are a special challenge when securing any environment. They are often left unpatched by the vendor, they sometimes have hard coded credentials built in, they are notoriously difficult to monitor because they’re almost entirely encrypted, and they’re difficult to control because they often use hard coded DNS and handle CnC with their normal check-in methods (which also doesn’t use standard ports normally). This put me in a difficult situation as I wasn’t able to use my normal method of locating the Log4J file on these, or any way of telling if they were scanning for different Wi-Fi networks in the background.
Nessus has a module that can potentially scan these devices for the vulnerability, but I don’t have the money for Nessus on my own, and GreenBone scanner (formerly OpenVAS) is lacking in a lot of ways. So ultimately the answer here was to assume compromise and attempt to mitigate preemptively.
I had already placed the IoT devices on a VLAN and configured my VLANS to prevent intervlan routing so thankfully any compromise was going to be limited to that specific subnet; however, I wanted to try and prevent compromises. I can’t control whether the IoT device scan for Wi-Fi networks and I’m not going to turn my house into a Faraday cage to stop outside wireless network but I can stop the IoT devices from connecting to an adversary server by blocking LDAP.
I’d already blocked LDAP at the edge router for all of the network so that was taken care of, but I also noticed that some attackers tried using DNS in place of LDAP.
I can mitigate the DNS concern by setting up a DNS redirection on the IoT Network to force all port 53 traffic to my DNS server, thus preventing any outbound DNS by other means.
With those steps in place I also set up more detailed logging from that network. On my switch I configured a SPAN port to copy over all of the IoT subnet traffic into my logging server for trend analysis to spot any connections to something that isn’t normal.
End
I spent all weekend patching and protecting against Log4Shel and the fight isn’t over. I still have to continuously monitor my IoT network to look for any attempts at exploiting Log4Shell, and unfortunately I’ll probably have to do this monitoring for the rest of the life of the device (or until the manufacturer patches, which I seriously doubt). This is not the first time I’ve had to do network wide remediation actions, but it’s the first time I’ve had to do it on my own home network.
It’s made me think about how bad the overall cyber security posture of the world really is. I knew how to do these things because I’ve working in a SoC before, but the average user isn’t going to know any of this. The average user with a flat network, lots of IoT devices, and no knowledge is going to be vulnerable to this attack for years.
