POQue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events

Files

2022.emnlp-main.594.pdf (4.77 MB)

Links to Files

https://aclanthology.org/2022.emnlp-main.594/

Permanent Link

http://hdl.handle.net/11603/26543
 http://dx.doi.org/10.18653/v1/2022.emnlp-main.594

Collections

UMBC Computer Science and Electrical Engineering Department
UMBC Faculty Collection
UMBC Student Collection

Author/Creator

Author/Creator ORCID

Date

2022-12-05

Type of Work

journal articles
preprints
Text

Department

Program

Citation of Original Publication

Sai Vallurupalli, Sayontan Ghosh, Katrin Erk, Niranjan Balasubramanian, and Francis Ferraro. 2022. POQue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8674–8697, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. 10.18653/v1/2022.emnlp-main.594

Rights

This item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author.
Attribution 4.0 International (CC BY 4.0)

Subjects

UMBC Ebiquity Research Group

Abstract

Knowledge about outcomes is critical for complex event understanding but is hard to acquire. We show that by pre-identifying a participant in a complex event, crowdworkers are able to (1) infer the collective impact of salient events that make up the situation, (2) annotate the volitional engagement of participants in causing the situation, and (3) ground the outcome of the situation in state changes of the participants. By creating a multi-step interface and a careful quality control strategy, we collect a high quality annotated dataset of 8K short newswire narratives and ROCStories with high inter-annotator agreement (0.74- 0.96 weighted Fleiss Kappa). Our dataset, POQue (Participant Outcome Questions), enables the exploration and development of models that address multiple aspects of semantic understanding. Experimentally, we show that current language models lag behind human performance in subtle ways through our task formulations that target abstract and specific comprehension of a complex event, its outcome, and a participant’s influence over the event culmination.