KSAT: Knowledge-infused Self Attention Transformer - Integrating Multiple Domain-Specific Contexts
Loading...
Permanent Link
Author/Creator
Author/Creator ORCID
Date
2022
Type of Work
Department
Program
Citation of Original Publication
Rights
This item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author.
Subjects
Abstract
Domain-specific language understanding requires integrating multiple pieces of
2 relevant contextual information. For example, we see both suicide and depression3 related behavior (multiple contexts) in the text “I have a gun and feel pretty bad
4 about my life, and it wouldn’t be the worst thing if I didn’t wake up tomorrow”.
5 Domain specificity in self-attention architectures is handled by fine-tuning on
6 excerpts from relevant domain specific resources (datasets and external knowl7 edge - medical textbook chapters on mental health diagnosis related to suicide
8 and depression). We propose a modified self-attention architecture Knowledge9 infused Self Attention Transformer (KSAT) that achieves the integration of multiple
10 domain-specific contexts through the use of external knowledge sources. KSAT
11 introduces knowledge-guided biases in dedicated self-attention layers for each
12 knowledge source to accomplish this. In addition, KSAT provides mechanics for
13 controlling the trade-off between learning from data and learning from knowledge.
14 Our quantitative and qualitative evaluations show that (1) the KSAT architecture
15 provides novel human-understandable ways to precisely measure and visualize the
16 contributions of the infused domain contexts, and (2) KSAT performs competitively
17 with other knowledge-infused baselines and significantly outperforms baselines
18 that use fine-tuning for domain-specific tasks.