Lossy network compression for distributed network intrusion detection applications

Abstract

In distributed network intrusion detection applications, it is necessary to transmit data from the remote sensors to the central analysis systems. Transmitting all the data captured by the sensor would place an unacceptable demand on the bandwidth available to the site. Most applications address this problem by sending only alerts or summaries; however, these alone do not always provide the analyst with enough information to truly understand what is happening on the network. Lossless compression techniques alone are not suffcient to address the bandwidth demand. This dissertation presents research into lossy compression techniques. It explores several ways in which the maliciousness of network traffc may be rated including entropy, magnitude, fow position, and a combination of N-grams and Bloom flters. These rating methods are combined into a tainted fow rating system. This tainted fow method was used to compress synthetic and competition data sets from 1998 until 2017 to a small percentage of their original size without signifcation loss of Snort alerts.