Objective: Accurate triage in the emergency department (ED) is critical for medical safety and operational efficiency. We aimed to predict the number of future required ED resources, as defined by the Emergency Severity Index (ESI) triage protocol, using natural language processing of nursing triage notes.

Methods: We constructed a retrospective cohort of all 265,572 consecutive ED encounters from 2015 to 2016 from 3 separate clinically heterogeneous academically affiliated EDs. We excluded encounters missing relevant information, leaving 226,317 encounters. We calculated the number of resources used by patients in the ED retrospectively and based outcome categories on criteria defined in the ESI algorithm: 0 (30,604 encounters), 1 (49,315 encounters), and 2 or more (146,398 encounters). A neural network model was trained on a training subset to predict the number of resources using triage notes and clinical variables at triage. Model performance was evaluated using the test subset and was compared with human ratings.

Results: Overall model accuracy and macro F1 score for number of resources were 66.5% and 0.601, respectively. The model had similar macro F1 (0.589 vs 0.592) and overall accuracy (65.9% vs 69.0%) compared to human raters. Model predictions had slightly higher F1 scores and accuracy for 0 resources and were less accurate for 2 or more resources.

Conclusions: Machine learning of nursing triage notes, combined with clinical data available at ED presentation, can be used to predict the number of required future ED resources. These findings suggest that machine learning may be a valuable adjunct tool in the initial triage of ED patients.

J.S. is the cofounder of Vital Software, Inc, a company engaged in developing artificial intelligence clinical decision support products for the emergency department. The other authors declare no conflicts of interest.

👁 FIGURE 1

FIGURE 1

Distributions of “number of resources”…

FIGURE 1

Distributions of “number of resources” category, laboratory orders, imaging orders, and medications administered…

👁 FIGURE 2

FIGURE 2

(A) Test data set confusion…

FIGURE 2

(A) Test data set confusion matrix for number of resources category prediction using…

👁 FIGURE 3

FIGURE 3

(A) Confusion matrix for “number…

FIGURE 3

(A) Confusion matrix for “number of resources” category using predictions of 2 experienced…