Transferring Semantic Knowledge Into Language Encoders
| dc.contributor.advisor | Ferraro, Francis | |
| dc.contributor.author | UMAIR, MOHAMMAD | |
| dc.contributor.department | Computer Science and Electrical Engineering | |
| dc.contributor.program | Computer Science | |
| dc.date.accessioned | 2022-09-29T15:37:54Z | |
| dc.date.available | 2022-09-29T15:37:54Z | |
| dc.date.issued | 2021-01-01 | |
| dc.description.abstract | We introduce semantic form mid-tuning, an approach for transferring semantic knowledge from semantic meaning representations into transformer-based language encoders. In mid-tuning, we learn to align the text of general sentences---not tied to any particular inference task---and semantic representations of those sentences that were automatically generated by FrameNet and PropBank Semantic Role parsers. We show that this alignment can be learned implicitly via classification or directly via triplet loss. Our method yields language encoders that demonstrate improved predictive performance across inference, reading comprehension, textual similarity, and other semantic tasks drawn from the GLUE, SuperGLUE, and SentEval benchmarks. We evaluate our approach on three popular baseline models, where our experimental results and analysis concludes that current pre-trained language models can further benefit from structured semantic frames with the proposed mid-tuning method, as they inject additional task-agnostic knowledge to the encoder, improving the generated embeddings as well as the linguistic properties of the given model, as evident from improvements on a popular sentence embedding toolkit and a variety of probing tasks. | |
| dc.format | application:pdf | |
| dc.genre | theses | |
| dc.identifier | doi:10.13016/m2ztl9-dlsv | |
| dc.identifier.other | 12405 | |
| dc.identifier.uri | http://hdl.handle.net/11603/25979 | |
| dc.language | en | |
| dc.relation.isAvailableAt | The University of Maryland, Baltimore County (UMBC) | |
| dc.relation.ispartof | UMBC Computer Science and Electrical Engineering Department Collection | |
| dc.relation.ispartof | UMBC Theses and Dissertations Collection | |
| dc.relation.ispartof | UMBC Graduate School Collection | |
| dc.relation.ispartof | UMBC Student Collection | |
| dc.rights | This item may be protected under Title 17 of the U.S. Copyright Law. It is made available by UMBC for non-commercial research and education. For permission to publish or reproduce, please see http://aok.lib.umbc.edu/specoll/repro.php or contact Special Collections at speccoll(at)umbc.edu | |
| dc.source | Original File Name: UMAIR_umbc_0434M_12405.pdf | |
| dc.subject | Language Modeling | |
| dc.subject | Linguistics | |
| dc.subject | Machine Learning | |
| dc.subject | Natural Language Processing | |
| dc.subject | Semantic Knowledge | |
| dc.subject | Semantic Representations | |
| dc.title | Transferring Semantic Knowledge Into Language Encoders | |
| dc.type | Text | |
| dcterms.accessRights | Distribution Rights granted to UMBC by the author. | |
| dcterms.accessRights | Access limited to the UMBC community. Item may possibly be obtained via Interlibrary Loan thorugh a local library, pending author/copyright holder's permission. |