KSAT: Knowledge-infused Self Attention Transformer - Integrating Multiple Domain-Specific Contexts

Abstract

Domain-specific language understanding requires integrating multiple pieces of 2 relevant contextual information. For example, we see both suicide and depression3 related behavior (multiple contexts) in the text “I have a gun and feel pretty bad 4 about my life, and it wouldn’t be the worst thing if I didn’t wake up tomorrow”. 5 Domain specificity in self-attention architectures is handled by fine-tuning on 6 excerpts from relevant domain specific resources (datasets and external knowl7 edge - medical textbook chapters on mental health diagnosis related to suicide 8 and depression). We propose a modified self-attention architecture Knowledge9 infused Self Attention Transformer (KSAT) that achieves the integration of multiple 10 domain-specific contexts through the use of external knowledge sources. KSAT 11 introduces knowledge-guided biases in dedicated self-attention layers for each 12 knowledge source to accomplish this. In addition, KSAT provides mechanics for 13 controlling the trade-off between learning from data and learning from knowledge. 14 Our quantitative and qualitative evaluations show that (1) the KSAT architecture 15 provides novel human-understandable ways to precisely measure and visualize the 16 contributions of the infused domain contexts, and (2) KSAT performs competitively 17 with other knowledge-infused baselines and significantly outperforms baselines 18 that use fine-tuning for domain-specific tasks.