eSentire White Logo

Blog | Feb 01, 2019

Using false negatives to illuminate unknowns, presented at BlackHat Europe 2018

One of the most frustrating problems every security practitioner faces is a lack of data about false negative rates. What are those things that just get by our defensive technologies? What are the breach risks that we don’t know about? Where are the likely gaps in my defenses? And everyone usually says, “we just don’t know what we don’t know, so there’s not much we can do. My environment is unique and no one else is likely to have data on it.”

To a large extent, that’s true. But for a strategic managed detection and response company like eSentire, it’s different. We have a mountains of data on breaches from hundreds of unique environments ... and anonymized metadata at that scale can shed real light on the unknowns facing defenders because we can talk about trends and probabilities. This puts us in a unique position to provide really interesting insights on the problem.

So, we made it the focus of a study and recently presented our results to the security community at BlackHat Europe 2018. As background, we provide MDR services using a pool of sensors monitoring 2,000+ different locations across 600+ distinct client environments and 28 countries. Each day, 365 days a year, we forensically investigate tens of thousands of potential threats and respond to contain breaches still in the earliest stages just establishing a beachhead inside the environment.

While we focus on monitoring and actively hunting for things in supposedly “clean” environments, we still gather rare data to make sense of within the evolving cybersecurity space. It is a collection of false negative data points that all the deployed defensive tech in real-world configurations have missed, with forensic data and human analysis to back up each incident data point (zero false positives).

Statistically we used eSentire’s data set of detected breaches-in-progress on the “clean” side to reduce uncertainty of “we just don’t know” areas that face many defenders today to measurable bands of risk at particular confidence intervals. That’s especially helpful for blue teamers that do not yet run their own threat hunting or monitoring efforts and have to trust perimeter defenses and want to talk meaningfully about the chances of something getting through.

In our study, we treated eSentire’s deployed sensors as a convenience sample from the general population - about what you’d find if you did a threat hunting/monitoring effort on a random site. With over 2,000 sites from which to draw data, we have a healthy sample size and a narrow margin of error at high confidence intervals for a similar population. Our demographic does have biases (see presentation for details), but we invite defenders and other researchers to apply their own weights to our sample set to help them get a feel for their own chances and relative risk across a group of similar peers.

So, in the case of a typical mid-sized company, what probability is it that an attacker breached existing automated defenses over this past year to the point that manual intervention by a security specialist would have been needed?

With our data set, for a client with four sites the rate of live malware showing up somewhere on the clean side generally floats between 8 and 15 percent each month, with a CI of +/- ~3 percent at 95 percent confidence. Looking cumulatively over all of 2018, the chance of malware showing up somewhere on the clean side of all your normal defensive tech and doing damage was about 52.4 percent likely (48.1-56.6 percent @ 95 percent CI).

In our view, that answer is a hell of a lot better than “we just don’t know what we don’t know.”

Alexander Feick

Alexander Feick

Technical Director, Security Services Architecture