Image
The Rise of AI Scribes in Healthcare Raises New Ethical Questions
Season two of HBO’s Emmy award-winning show "The Pitt" features a storyline surrounding the ethics and efficiency of using artificial intelligence-based tools on the emergency room floor. The show’s new attending doctor, Baran Al Hashimi, proposes that emergency room staff use AI to speed up the process of charting patient ailments.
Adding AI into the equation may save medical staff time, but it also raises questions around doctor-patient confidentiality. AI scribes can efficiently record spoken patient input and summarize highly sensitive health information, but where does that data go, and who has access to it? What if an AI tool gets a diagnosis wrong and a patient doesn’t receive life-saving treatment? We spoke with Tom Hartvigsen, assistant professor of data science at the University of Virginia, to learn more about the usage of AI scribes in the healthcare system.
Q: Why has the medical industry been so quick to adopt AI scribes rather than hire more healthcare professionals?
AI scribes are pitched as solutions to a problem that is very real: Clinicians are overworked, and their burnout rates are high. Plus, the risks are hard to measure because most AI scribe errors won’t cause catastrophes. However, hospitals operate on very thin margins, so they don't always prioritize quality of care. For example, high costs of care systematically prohibit care for patient populations. When hiring new health care professionals is more expensive than AI subscriptions, administrators can choose AI. Clinicians also reportedly love using AI scribes (again, because the problems are real), even though large-scale effects on quality of care remain relatively unstudied as far as I know.
Q: Would the efficiency of AI scribes result in unrealistic expectations of care or increase the volume of patients seen per hour in a hospital setting?
This is a big risk of AI scribes. Efficiency is more easily measured than quality of care, so therefore easier to optimize. I worry that AI scribes may indeed accelerate care, but towards mediocrity instead of excellence. The AI scribe companies certainly claim error rates are low, but I haven’t seen a measurement of quality that I’ve felt convinced by yet.
Q: Patients are not giving consent to release personal data to an AI application. Could this sensitive information be used as a training tool, or potentially sold to third parties?
Patient data is absolutely at risk of being trained on and sold when ingested by AI scribes. Any time a generative machine learning model is trained on patient data, there is a risk that the data is then exposed in the future in inappropriate contexts. I am also unaware of any possibility that a high-quality AI model can be produced without training on multiple patients’ data.
Q: How can patients advocate for their data to be protected in healthcare settings?
If patients want to protect their data, they should ask their care providers whether their data are being ingested by AI. If so, ask about any benefits they’ve observed and what protections are in place. It’s such new technology that many of these conversations start at the ground up, as we’re still quickly identifying opportunities and challenges.
Tom Hartvigsen is an assistant professor of data science at the University of Virginia. He works to make machine learning trustworthy, robust, and socially responsible enough for deployment in high-stakes, dynamic settings.
Hartvigsen's research has been published at many major peer-reviewed venues in machine learning, natural language processing, and data mining. He is active in the machine learning community, serving as the general chair for the Machine Learning for Health Symposium in 2023, helping organize the 2023 Conference on Health, Informatics, and Learning, and co-chairing workshops on time series and generative AI at NeurIPS'22 and ICML'23.
Prior to joining UVA, Hartvigsen was a postdoctoral associate at MIT's Computer Science and Artificial Intelligence Laboratory. He holds a doctorate and master's degree in data science from Worcester Polytechnic Institute and a bachelor's degree in applied math from SUNY Geneseo.
