KSAT: Knowledge-infused Self Attention Transformer - Integrating Multiple Domain-Specific Contexts

Simple item page
dc.contributor.authorRoy, Kaushik
dc.contributor.authorZi, Yuxin
dc.contributor.authorNarayanan, Vignesh
dc.contributor.authorGaur, Manas
dc.contributor.authorSheth, Amit
dc.date.accessioned2022-11-03T15:54:12Z
dc.date.available2022-11-03T15:54:12Z
dc.date.issued2022
dc.description.abstractDomain-specific language understanding requires integrating multiple pieces of 2 relevant contextual information. For example, we see both suicide and depression3 related behavior (multiple contexts) in the text “I have a gun and feel pretty bad 4 about my life, and it wouldn’t be the worst thing if I didn’t wake up tomorrow”. 5 Domain specificity in self-attention architectures is handled by fine-tuning on 6 excerpts from relevant domain specific resources (datasets and external knowl7 edge - medical textbook chapters on mental health diagnosis related to suicide 8 and depression). We propose a modified self-attention architecture Knowledge9 infused Self Attention Transformer (KSAT) that achieves the integration of multiple 10 domain-specific contexts through the use of external knowledge sources. KSAT 11 introduces knowledge-guided biases in dedicated self-attention layers for each 12 knowledge source to accomplish this. In addition, KSAT provides mechanics for 13 controlling the trade-off between learning from data and learning from knowledge. 14 Our quantitative and qualitative evaluations show that (1) the KSAT architecture 15 provides novel human-understandable ways to precisely measure and visualize the 16 contributions of the infused domain contexts, and (2) KSAT performs competitively 17 with other knowledge-infused baselines and significantly outperforms baselines 18 that use fine-tuning for domain-specific tasks.en_US
dc.description.urihttps://scholarcommons.sc.edu/cgi/viewcontent.cgi?article=1571&context=aii_fac_puben_US
dc.format.extent9 pagesen_US
dc.genrejournal articlesen_US
dc.genrepreprintsen_US
dc.identifierdoi:10.13016/m24nqo-eefi
dc.identifier.urihttp://hdl.handle.net/11603/26256
dc.language.isoen_USen_US
dc.relation.isAvailableAtThe University of Maryland, Baltimore County (UMBC)
dc.relation.ispartofUMBC Computer Science and Electrical Engineering Department Collection
dc.relation.ispartofUMBC Faculty Collection
dc.rightsThis item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author.en_US
dc.subjectUMBC Ebiquity Research Group
dc.titleKSAT: Knowledge-infused Self Attention Transformer - Integrating Multiple Domain-Specific Contextsen_US
dc.typeTexten_US

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
KSAT_ Knowledge-infused Self Attention Transformer - Integrating.pdf
Size:
567.17 KB
Format:
Adobe Portable Document Format
Description:
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
2.56 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

UMBC Computer Science and Electrical Engineering Department
UMBC Faculty Collection