Knowledge-guided Tensor Decomposition for Baselining and Anomaly Detection

We introduce a flexible knowledge-guided penalty for incorporating known or expected patterns of activity into tensor decomposition. Our modified tensor decomposition both enables efficient identification of semantically-related patterns across data sets and provides a means for identifying anomalous patterns. Specifically, we modify the loss function for a CP tensor decomposition such that a subset of the columns of the factor matrices are guided to align with the provided knowledge. We validate the effectiveness of the method for separating baseline patterns from anomalous patterns in the context of cyber network traffic logs. After performing daily decompositions of tensors formed from network logs, we derive a set of expected components describing baseline behavior. We then decompose a new tensor, created from network logs and containing a known anomaly, providing the baseline as knowledge guidance. Notably, the anomalous behavior appears among the unguided components, resulting in drastic reduction in the search space for anomalies. Additionally, we show that our knowledge-guided decomposition is robust to incorrect knowledge in that any enforced components that are not found in the original data have low weight. Our method, implemented in the tensor software package ENSIGN, is an efficient routine that reduces the post- processing needed for an anomaly detection workflow.

For information on Reservoir’s technology related to this paper, visit ENSIGN.