CIS²: A Simplified Commonsense Inference Evaluation for Story Prose

Abstract

Contextual Commonsense Inference (CCI) is the problem of inferring causal relations between the events of a text, such as a story. Like other commonsense reasoning tasks, CCI is a problem of language understanding, rather than language generation. We show that prior work, in using language generation to perform CCI, trains models that struggle on the CCI task in isolation. This conflation of tasks is further exacerbated by evaluating with wordmatching based metrics such as BLEU. In order to isolate CCI from language generation, we reframe CCI as a classification problem. Our system, which we call CIS2 , forces the model to focus on CCI directly by providing it the original text of the story to use for understanding while having it generate only the bare minimum: indices to sentences. We look at the GLUCOSE (Mostafazadeh et al., 2020) dataset and compare against their task for predicting CCI between story sentences. We find that models trained on CIS2 index labels achieve a 4.3% higher CCI accuracy than those trained for generating full phrases, such as in the original GLUCOSE task.