UC San Diego Health researchers analyzed a year’s worth of physician edits to AI-drafted replies to patient portal messages, finding that promised time savings from the technology have not consistently materialized in practice.
The study, published in NEJM AI on Sept. 17, examined AI-generated draft responses sent to more than 1,000 physicians across specialties through a system integrated into the Epic EHR. Researchers built a 15-category taxonomy of physician edits using a large language model paired with iterative expert review, then measured response time as a proxy for clinical workload.
Scheduling and rescheduling changes were the most common edit, making up 38.5% of all modifications. Lifestyle and non-pharmacologic guidance followed at 18.2%, and empathy or emotional-support edits accounted for 16.1%. Discontinuing or tapering medications was the least frequent edit, at 2.4%.
Edits requiring clinical judgment carried the heaviest per-message time cost. Interpreting radiology results added 70.1% more time, clarifying or ruling out a diagnosis added 63.9%, and interpreting laboratory results added 60.8%.
“Editing burden is not uniformly distributed: Edits requiring clinical judgment impose the greatest per-message time burden, while high-frequency administrative edits contributed the greatest cumulative workload across the health system,” the study authors wrote.
The findings mark an early step toward helping health systems understand how physicians actually modify AI-generated content, information UC San Diego Health says is critical to realizing the technology’s intended value.
At the Becker’s 32nd Annual Meeting: The Business and Operations of ASCs, taking place October 29-31 in Chicago, ASC leaders, surgeons and healthcare executives will explore strategies to drive growth, enhance operational performance, navigate reimbursement challenges and prepare for the future of ambulatory surgery. Apply for complimentary registration now.
