Semantic Interpretation of Structured Log Files

Abstract

Data from computer log files record traces of events involving user activity, applications, system software and network traffic. Logs are usually intended for diagnostic and debugging purposes, but their data can be extremely useful in system audits and forensic investigations. Logs created by intrusion detection systems, web servers, anti-virus and anti-malware systems, firewalls and network devices have information that can reconstruct the activities of malware or a malicious agent, help plan for remediation and prevent attacks by revealing probes or intrusions before damage has been done. While existing tools like Splunk can help analyze logs with known schemas, understanding log whose format is unfamiliar or associated with new device or custom application can be challenging. We describe a framework for analyzing logs and automatically generating a semantic description of their schema and content in RDF. The framework begins by normalizing the log into columns and rows using regular expression-based and dictionary-based classifiers. Leveraging our existing work on inferring the semantics of tables, we associate semantic types with columns and, when possible, map them to concepts in general knowledge-bases (e.g. DBpedia) and domain specific ones (e.g., Unified Cybersecurity Ontology). We link cell values to known type instances (e.g., an IP address) and suggest relationships between columns. Converting large and verbose log files into such semantic representations reveals their meaning and supports search, integration and reasoning over the data.