CyBERT: Contextualized Embeddings for the Cybersecurity Domain

Thumbnail Image

Files

1117.pdf (335.35 KB)

Links to Files

https://ieeexplore.ieee.org/document/9671824

Permanent Link

https://doi.org/10.1109/BigData52589.2021.9671824
 http://hdl.handle.net/11603/25498

Collections

UMBC Computer Science and Electrical Engineering Department
UMBC Faculty Collection
UMBC Student Collection

Author/Creator

Author/Creator ORCID

https://orcid.org/0000-0002-6437-1324
 https://orcid.org/0000-0002-8641-3193
 https://orcid.org/0000-0002-6593-1792

Date

2022-01-13

Type of Work

conference papers and proceedings
preprints
Text

Department

Program

Citation of Original Publication

P. Ranade, A. Piplai, A. Joshi and T. Finin, "CyBERT: Contextualized Embeddings for the Cybersecurity Domain," 2021 IEEE International Conference on Big Data (Big Data), 2021, pp. 3334-3342, doi: 10.1109/BigData52589.2021.9671824

Rights

© 2022 IEEE.  Personal use of this material is permitted.  Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Subjects

UMBC Ebiquity Research Group

Abstract

We present CyBERT, a domain-specific Bidirectional Encoder Representations from Transformers (BERT) model, fine-tuned with a large corpus of textual cybersecurity data. State-of-the-art natural language models that can process dense, fine-grained textual threat, attack, and vulnerability information can provide numerous benefits to the cybersecurity community. The primary contribution of this paper is providing the security community with an initial fine-tuned BERT model that can perform a variety of cybersecurity-specific downstream tasks with high accuracy and efficient use of resources. We create a cybersecurity corpus from open-source unstructured and semi-unstructured Cyber Threat Intelligence (CTI) data and use it to fine-tune a base BERT model with Masked Language Modeling (MLM) to recognize specialized cybersecurity entities. We evaluate the model using various downstream tasks that can benefit modern Security Operations Centers (SOCs). The fine-tuned CyBERT model outperforms the base BERT model in the domain-specific MLM evaluation. We also provide use-cases of CyBERT applications in cybersecurity-based downstream tasks.