SKGHOI: Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection
Loading...
Links to Files
Author/Creator ORCID
Date
2023-03-15
Type of Work
Department
Program
Citation of Original Publication
Rights
This item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author.
Subjects
Abstract
—Detecting human-object interactions (HOIs) is a
challenging problem in computer vision. Existing techniques
for HOI detection heavily rely on appearance-based features,
which may not capture other essential characteristics for accurate
detection. Furthermore, the use of transformer-based models
for sentiment representation of human-object pairs can be
computationally expensive. To address these challenges, we propose a novel graph-based approach, SKGHOI (Spatial-Semantic
Knowledge Graph for Human-Object Interaction Detection), that
effectively captures the sentiment representation of HOIs by
integrating both spatial and semantic knowledge. In a graph,
SKGHOI takes the components of interaction as nodes, and
the spatial relationships between them as edges. Our approach
employs a spatial encoder and a semantic encoder to extract
spatial and semantic information, respectively, and then combines
these encodings to create a knowledge graph that captures the
sentiment representation of HOIs. Compared to existing techniques, SKGHOI is computationally efficient and allows for the
incorporation of prior knowledge, making it practical for use in
real-world applications. We demonstrate the effectiveness of our
proposed method on the widely-used HICO-DET datasets, where
it outperforms existing state-of-the-art graph-based methods by
a significant margin. Our results indicate that the SKGHOI
approach has the potential to significantly improve the accuracy
and efficiency of HOI detection, and we anticipate that it will
be of great interest to researchers and practitioners working on
this challenging task.