“I don't know”: An uncertainty-aware machine learning model for predicting patient disposition at emergency department triage

Background: Machine learning (ML) models are widely used for predicting patient disposition at emergency department (ED) triage. However, these models generate predictions regardless of the level of uncertainty, potentially leading to overconfident outputs that can compromise clinical decision-making. Objective: To develop a conformal prediction model for ED triage that provides uncertainty-aware patient disposition predictions. Methods: This retrospective study analyzed 560,486 adult ED visits (March 2014 – July 2017) from one academic and two community hospitals. An extreme gradient boosting (XGBoost) model was trained, validated, and conformalized to introduce a “Don't know” prediction for high-uncertainty cases. The model was tested on a random sample of 56,000 ED cases. Results: The standard XGBoost model achieved an AUC of 0.9307 (95% CI: 0.9285 – 0.9329), with sensitivity of 0.72 and specificity of 0.94. With conformal prediction at a lower confidence threshold of 60%, the model indicated “Don't know” in 4.9% of cases while returning sensitivity and specificity values of 0.74 and 0.95, respectively. As confidence thresholds increased, the model returned more “Don't know” predictions and fewer misclassifications. At 90% confidence, the model returned “Don't know” in 34.5% of cases while returning sensitivity and specificity values of 0.88 and 0.99, respectively. This trade-off highlights a balance between model confidence and prediction accuracy. Conclusion: Incorporating uncertainty-awareness in ML models improves reliability in ED triage. By acknowledging uncertainty, clinicians receive more interpretable insights, reducing the risk of overconfident predictions and enhancing patient safety.