Finding Deep Patterns at Enterprise Scale
Hypergraphs provide a formal representation of multi-dimensional data. ENSIGN uses a patent-pending high-performance implementation of decomposition algorithms from the mathematics of multilinear algebra to enable the unsupervised discovery of subtle undercurrents and deep, cross-dimensional correlations within these structures. Unsupervised multidimensional decompositions accept records of human and machine activity and produce components – weighted fragments of data that each capture a specific pattern. These components are the product of computationally intensive model-fitting routines that, with ENSIGN, have been aggressively optimized for the cyber domain. From unsupervised discovery, domain knowledge attaches meaning to a handful of components each isolating a key contributing pattern to the overall network flow. In most cases, the story underpinning the existence of a component is self-evident, easily recognizable patterns of expected, benign activity. However, in other cases, patterns emerge among one or more dimensions – regular time intervals, a common destination, a common request type – that indicate a deeper, more directed, intent.
Advanced Threat Detection
ENSIGN extends R-Scope®, a scalable and hardened network security monitor based on Bro that provides the network visibility and rich metadata – the added dimensions – crucial to the success of unsupervised discovery:
- Protocol queries and responses
- Protocol identifiers
- File transfers with type and size
- Versioning information
R-Scope analyzes network traffic at scale to provide contextual network and security metadata and real-time security event alerts. ENSIGN analyzes the historical record of this same metadata to provide daily, weekly, or monthly reports on patterns of activity. These patterns capture emergent behavior before the need for new rules is even recognized. The ability to see aggregate traffic as overlapping patterns complements R-Scope’s real-time capability and provides unparalleled network visibility to enterprise security professionals.
ENSIGN is poised to turn the field of cyber analytics on its head with an approach to advanced threat detection enabled by R-Scope that is rooted in pattern discovery rather than incident detection. Funded by the Department of Defense to deliver mathematically sound unsupervised discovery in large-scale multi-dimensional data and now adapted to the cyber domain, ENSIGN reduces vast logs of information into a set of true, unbiased, visually concise stories about what is actually happening on a network. Demonstrated at Supercomputing Conference 2017, these stories reveal activity and threat intent that would otherwise go unnoticed by methods limited to signature-based discovery alone.