Action unit based micro-expression recognition framework for driver emotional state detection

Understanding a driver’s emotional state is critical for ensuring road safety and public well-being. Emotions such as anger, fear, disgust, sadness, or happiness can significantly influence driving behavior and decision-making. Facial micro-expressions reveal genuine feelings that people attempt to mask or conceal, offering valuable cues for detecting these emotional states, as they tend to be universally expressed across cultures. This study presents a micro-expression recognition framework designed to identify emotional variations in drivers by analyzing facial Action Units (AUs) based on the Facial Action Coding System (FACS). FACS decomposes expressions into AU combinations, enabling more accurate and flexible interpretation of emotions. The proposed method combines a Residual Network (ResNet18) for spatial feature extraction with a Bidirectional Long Short-Term Memory (Bi-LSTM) network for temporal pattern learning. In addition, agglomerative clustering of AU combinations was applied to enhance emotion classification. The model was trained and evaluated on two benchmark datasets: SAMM and KMU-FED, achieving recognition accuracies of 96.38% and 95.96%, respectively. Furthermore, case analysis was carried out to detect driver emotional state using the proposed framework, obtaining an accuracy of 91.00%. The experiment indicated that anger, disgust, sadness, and fear are the predominant emotions expressed by drivers while driving. The goal of this study lies in harnessing action units for micro-expression recognition to enhance precision in recognizing the driver’s emotional state.