Leveraging Deep Learning and Multimodal Large Language Models for Near-Miss Detection Using Crowdsourced Videos

Abstract

Near-miss traffic incidents, positioned just above "unsafe acts" on the safety triangle theory, offer crucial predictive insights for preventing crashes. However, these incidents are often underrepresented in traffic safety research, which tends to focus primarily on actual crashes. This study introduces a novel AI-based framework designed to detect and analyze near-miss and crash events in crowdsourced dashcam footage. The framework consists of two key components: a deep learning model to segment video streams and identify potential near-miss or crash incidents and a multimodal large language model (MLLM) to further analyze and extract narrative information from the identified events. We evaluated three deep learning models—CNN, Vision Transformers (ViTs), and CNN+LSTM—on a dataset specifically curated for three-class classification (crashes, near-misses, and normal driving events). CNN achieved the highest accuracy (90%) and F1-score (89%) at the frame level. At the event level, ViTs delivered a strong performance with a test accuracy of 77.27% and an F1-score of 67.37%, while CNN+LSTM, although lower in overall performance, demonstrated significant potential with a test accuracy of 78.1% and an F1-score of 68.69%. For a deeper analysis, we applied GPT-4o to process critical safety events (near-misses and crashes), utilizing both zero-shot and few-shot learning for narrative generation and feature extraction. The zero-shot learning method performed better, achieving an accuracy of 81.2% and an F1-score of 81.9%. This study underscores the potential of combining deep learning with MLLMs to enhance traffic safety analysis by integrating near-miss data as a key predictive layer. Our approach highlights the importance of leveraging near-miss incidents to proactively enhance road safety, thereby reducing the likelihood of crashes through early intervention and better event understanding.