MENA: A Multimodal Framework for Analyzing Caregiver Emotions and Competencies in AR Geriatric Simulations

Improving the quality of geriatric care is a challenge that requires insights from stakeholders. While simulated trainings can boost competencies, extracting meaningful insights from these practices to enhance simulation effectiveness remains a challenge. In this study, we introduce Multimodal Epistemic Network Analysis (MENA), a novel framework for analyzing caregiver attitudes and emotional responses in an Augmented Reality simulation. By integrating a multimodal Emotional State Classifier, MENA extends traditional epistemic network analysis to reveal complex relationships between caregiving competencies and positive emotions. Applied in a pilot study (N = 20) comparing caregiver interactions with an unaware versus an aware virtual geriatric patient (VGP), MENA visualizations demonstrated how awareness in the VGP fostered more supportive and person-centered caregiving behaviors. These findings suggest that MENA not only enhances the analysis of multimodal interactions but also provides a powerful tool for designing emotionally intelligent training systems that prepare caregivers for the nuanced demands of real-world practice. The code and setup to reproduce the experiments are publicly available here, and data is available upon request.