Neurophysiological Variations and Pattern Recognition in Food Decision-Making Process

Abstract

Studies showed that understanding the underlying mechanism of the decision-making or factors involved could potentially benefit people in various aspects, such as education, mental health. Though the results of the naturalistic decision making(NDM) framework from psychological studies provide novel insights into the process of decision making, lacking concrete and practical data in this theory limited the development of derivative applications and researches in the long-term. In the collaboration between engineers and psychologists, this theses proposes a state-of-the-art sensor-driven approach to recognize and understand significant patterns that correlate with factors triggering the decision, and validate the results technically and theoretically. In the experiment, participants were equipped with different sensors, such as prefrontal cortex functional near-infrared spectroscopy (fNIRS) for capturing neurological data, electrocardiography (ECG) for heart rate and variability, galvanic skin response (GSR) for skin resistance, while they were selecting foods under both virtual reality and real-world settings. But we only used fNIRS for the data analysis because in current phase we specifically interested in the brain activation patterns in the decision-making process. Later on, we extracted raw fNIRS data and transformed it to spectrogram and performed motion artifact removal. The processed spectrogram were then input in the developed deep learning model. After obtained 74% accuracy on the deep learning model, we adopted Grad-CAM to visualize significant parts of the input spectrogram that are contributing to the prediction. The result demonstrated that the left inferior frontal gyrus is significantly active while selecting foods.