A hybrid intelligence/multi-agent system for mining information assurance data

Abstract

Organizations across all domains and of all sizes wrestle with the problem of "coping with information overload," or CwIO. They ingest more and more data, in new and varied formats every day, and struggle increasingly vigorously to find the nuggets of knowledge hidden within the vast amounts of information. Furthermore, due to the various and pervasive types of noise in the haystack of data, it is becoming increasingly difficult to discern between shiny false shards and the true needles of knowledge. Although the costs of data storage, memory and processing have dropped, this decline has not maintained parity with the unprecedented increase in the amount and complexity of data to be examined. This problem is especially challenging in the world of network intrusion detection. In this realm, one must not only deal with sifting through vast amounts of data, but it must also be done in a timely manner even when at times one is not sure what exactly it is being sought. In efforts to help solve this problem, this research demonstrates that in the realm of offline network forensic datamining, several different datamining algorithms (hybrid intelligence) working within a multi-agent system, will yield more accurate results than any one datamining algorithm acting on its own. Toward that end, this paper outlines the steps taken to generate and prepare suitably minable test data, compare and contrast the capabilities/output of various types of datamining algorithms (hybrid intelligence), and finally discuss the architecture and construction of a SPADE based multi-agent system to semi-autonomously perform multi-path datamining tasks.