BlogQuick! Before you start! There's a really interesting discussion going on over at Hacker News around this whole deal, and as of this update the fake Postman extension has been removed from the Chrome extension store.
We recently uncovered a Chrome extension that was using the name of a popular Chrome App that developers use to test their REST APIs. The extension was attempting to aggregate and exfiltrate data from our environment, but was detected by ExtraHop Reveal(x) in time to prevent any data loss. Since we like to eat our own dogfood, we used ExtraHop gear for the full investigation response to this potential threat, which only took a couple of hours to confidently resolve.
Here's what happened...
How Did It Start?
On Nov 6, 2018 one of our engineers deployed an experimental new detector to our Reveal(x) security product. The detector immediately showed a long lived persistent HTTP websocket connection to an external IP address on a strange port (6332). We investigated with our own ExtraHop tools and found plaintext HTTP traffic that was clearly obfuscated being emitted from an employee's computer.
Our IT staff jumped immediately into action and disabled the machine that had this persistent connection.
How Did We Investigate?
We had found one machine that was definitely exfiltrating data of some kind. We also had an Indicator of Compromise (IOC); the extension always seemed to send the data over port 6332. ExtraHop keeps flow records for all IP transactions. Using these records, we discovered two other machines with the exact same traffic, presumably a second copy of the extension. One was a second machine from the first user and a machine from a second user. The second user was not in the office either, so the same operation happened again; the machine was unplugged and the user was forcibly logged out of all accounts.
Because ExtraHop gathers flow records and makes them easy to search through, this part of the investigation was simple.
Flow records for the suspicious connection, detected by ExtraHop.
We also used our own ExtraHop packet capture appliances to archive all the packets from the time the extensions were installed. We archived these packets for investigation and retention purposes.
By examining the packets using Wireshark, we found that the HTTP traffic included a header from a Chrome extension called Postman. This definitely isn't the Postman that is popular for testing REST APIs and doing web development. Immediately, we became suspicious about the extension because its name hijacked a popular tool.
Another of our engineers downloaded the extension, did some reverse engineering and de-obfuscation and discovered that the extension was indeed exfiltrating URL information that was XOR'ed with 6332 (0x18BC). URLs with internal hostnames would be considered sensitive data and should not be exfiltrated from our network.
How Did the Extension Get Into Our Environment?
The extension is published under the same name as a very popular development tool, Postman. However, the development tool is an application, rather than a Chrome extension like the malicious, false version. This distinction would be easy to miss, even for sophisticated users. It is made even more confusing since the legitimate Postman application has an associated Chrome extension called Postman Interceptor. Two separate users at ExtraHop went to download the Postman app and found the malicious extension of the same name instead. Other extensions by the same author were investigated and the same code was found in one other extension.
The Chrome extension store listing for the fake "Postman" extension that attempts to exfiltrate data from host machines.
Here's the listing for the real Postman application, which has far more users and reviews than the malicious version.
How We Followed Up
On November 7, 2018 once all investigation had been completed, I reported these extensions to the Google Chrome Extension store, asking the store to take it down. I followed up with another report before this was published.
Also on November 7, the offending IP address was reported to Digital Ocean, the hosting provider.
As of December 6, 2018, we haven't heard back from from either vendor.
How Did We Use Our ExtraHop Gear?
Shortly after our Reveal(x) system detected the unusual long-lived connection on a strange port, we were able to use ExtraHop to answer the following questions:
- When did the network activity start (i.e. when was the extension installed on the different machines)?
- What ports and IPs were used?
- What protocol was used?
- What is causing this strange, long-lived connection?
- What data is being attempted to be exfiltrated across the unusual connection?
The fact that this investigation took less than 90 minutes of focused investigation to answer all of our questions is remarkable. Without flow records, this would have been very difficult. Without large PCAP disks for lookback, we might not have been able to verify our results.
How Are We Sure No Sensitive Data Was Compromised?
We quickly reviewed all flows and PCAPs associated with the suspicious connections during our investigation, and we can confidently say that no sensitive data was successfully exfiltrated. During the initial investigation, when we believed we might have lost credentials or session cookies, we audited all google authentication logs and could find no evidence that either account was accessed from any unexplained IP addresses.
Our investigation allowed us to fully understand the extent of the data loss, and much less information was exfiltrated than initially assumed.
Could We Have Used Another Tool?
It's also not clear how any other tool would have detected the long-lived, persistent outbound connection with relatively low bandwidth. This is the fulfillment of the promise of machine learning, especially our approach at ExtraHop. Given the right data inputs and algorithms, our system can sift through vast amounts of data and find small but highly meaningful anomalies.
For the investigation, if we had a SIEM with flow records, we could have potentially done all the tracing of the connections, but getting the end-to-end sequence of events and full packet capture for forensic evidence would likely have taken much longer.
Conclusion
Fake apps masquerading as real, more popular apps have plagued app and extension stores for years. Attackers only get more sophisticated at leveraging these systems, and users grow numb to the telltale signs of fake malicious extensions. Enterprises must assume that these types of risks will be present in their environments, and must be prepared to detect, investigate, and resolve them as rapidly as possible.
Less than 90 minutes elapsed between our initial investigation of this threat and a completed investigation and remediation process, with forensic-level details including full packets. Had we been using piecemeal tools, or had any of the components (PCAP, flow, ML detectors) been absent or siloed off in a different part of the organization, we may not have been able to validate our understanding of the threat quickly or at all.
A day after this post went live, the malicious "Postman" extension was removed from the Google Chrome extension store after being downloaded over 27,000 times.
One of the best successes of this investigation is that I didn't even have to look at the packets until much later in the night, once all investigation was completed. The visibility provided within the ExtraHop interface was enough to confidently resolve the investigation, and viewing the packets was a secondary confirmation step just to verify the truth of what we could see in the code and in our own ExtraHop devices.
As the Deputy CISO of ExtraHop, I'm very proud of the team that investigated and resolved this threat, and the company as a whole for building a product that enables this unprecedented level of sophistication in incident investigation and response.
