The Bad Guys Only Have to Be Wrong Once

There is a saying in security that good guys have to be right every time, but the bad guys only have to be right once.

This adage is known as the defender's dilemma. It speaks to the unpleasant truth that preventing an attack requires a 100% win rate. But what if you could flip the script so that the bad guys had to be right 100% of the time? That's what we call the intruder's dilemma, an advantage that can be gained by adding the visibility needed to detect an intruder's actions as they move toward their target.

During a recent training session, we took a look at the Suspicious User Agent detection that had popped up in ExtraHop Reveal(x). The "User Agent" is a part of an HTTP request and contains bits of information such as the browser and operating system. For example: Firefox 72 identifies itself as "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0." A Nintendo Switch identifies itself as "Mozilla/5.0 (Nintendo Switch; ShareApplet) AppleWebKit/601.6 (KHTML, like Gecko) NF/4.0.0.5.9 NintendoBrowser/5.1.0.13341".

A smart attacker will manipulate or remove the User Agent (an evasion tactic known as User Agent spoofing) to hide or disguise their actions, but our intruders tipped their hats with a careless Nmap scan that provided all the details we needed to know, proving why, with the help of Reveal(x), attackers can be caught and ousted with just a single slip up.

It's not just browsers and connected devices that emit User Agents. Security tools also emit them. The popular scanning tool Nmap, for example, identifies itself as "Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)."

In this screenshot from a simple Nmap conducted in the lab, you can see the "User-Agent" field in red:

Note in this next screenshot that the User Agent has been manipulated, as any smart attacker would do. It's now "Apple ][."

OMG!!!! The Steves (Jobs and Wozniak) are hacking me!

The User Agent attribute is useful, but only if the bad guy doesn't change the User Agent or, as can sometimes be the case, doesn't know how to change it.

Back to the ExtraHop…

Detecting the Threat

We scrolled down the detection and saw that we had Related Records—this is a killer feature that allows you to go from the high-level detection immediately into the supporting data so you can go after and respond to active threats faster.

Click.

The Client IP Address appeared to be the load balancer, but something else was going on.

Revisiting the offender and victim we see that it looks like the F5 is running Nmap, but that's not the case:

Side note: Modern load balancers (F5, Netscaler, A10, etc) will terminate the_front side_ connection, perform magic, then rewrite the transaction and carry the transaction further inside the network. In the process of rewriting the transaction, the load balancer will often inject an X-Forwarded-For (or similar) header. X-Forwarded-For is the load balancer's way of saying, "This is who I think the client is." It's an industry-standard thing, and ExtraHop automatically detects and extracts the X-Forwarded-For header and saves this field as the Origin. The HTTP Origin is tremendously useful for following bad guys.

(super sophisticated artwork to depict technical concept)

We scrolled to the right a bit and looked closer at the URI, User Agent, and Origin fields:

The User Agent is indeed Nmap. The Origin address is 152.32.145.20.

As of this writing, that address is out of a cloud hosting company in Hong Kong:

Investigating Unwanted Activity

Now that we know the User Agent and the Origin address, we can begin to have fun.

We pivot through Records, and in a few minutes, we are staring at multiple indicators of unwanted activity.

1. The User Agent: This is easy, and we can weaponize this. ExtraHop can message this customer's Palo Alto firewall and say, "We saw Nmap, drop the hammer on this source IP and make it go away for an hour." Buh bye!
2. The Origin address: We adjust the query filters to look at this entire /24 as the Origin address. Bad guys tend to live in bad neighborhoods. When one IP goes rogue, often adjacent IPs in the same network block go rogue. We go from 4 records to several hundred.
3. Targets: By looking at a wider swath of Origin addresses (moving from a single IP to an entire /24) we found external entities targeting a couple of external-facing web servers still in service that had been marked as decommissioned. Whoops.
4. The URI: What really stands out is the direct IP URI being accessed. Normal web transactions look like www.company.com/foo.html. The URIs we are seeing are direct IP (10.20.30.40/foo.html). Further discussion indicates that this web property is a business-to-business application and the entities using these web services are explicitly told to access via a fully qualified domain name ((www.company.com)). So any direct IP access is automatically suspicious.
5. The server port: Note that the URI is specifying port 443. This ExtraHop customer is using Reveal(x)'s secure decryption features, so even though the bad guys are using SSL, ExtraHop is able to decrypt the transaction at line rate and inspect the decrypted HTTP transaction.
6. The URI being accessed: We saw a lot of requests for wp-login.php and admin.php. This customer doesn't use PHP in this web tier. An example of BFMI (brute force, massive ignorance) by the bad guys. If you're robotically asking for PHP resources from a web farm that doesn't use PHP, you're probably a bad guy. Again, we used ExtraHop's automation features to say, "If an external IP address requests a PHP resource from a web farm that does not use PHP, block that source IP for an hour."

When an Intruder Confronts the Dilemma

As we followed the actions of this external IP address, we noticed that the user agent changed from "Nmap" to "Firefox" to "" (empty). It seemed the bad guys realized they screwed up. By that time, it was too late for them. We had their IP and multiple indicators to detect their activity and automatically block it.