Pennsylvania has been taking legal aim at AI chatbots that falsely claim to be licensed medical professionals — but according to a recent investigation by Spotlight PA, the behavior is widespread and largely continuing unimpeded.
A Department of State task force has been working since February to identify AI chatbots posing as licensed professionals. That work led Pennsylvania Gov. Josh Shapiro’s administration to file a lawsuit last month against the role-playing site Character.AI.
Spotlight PA then tested 10 popular companion and role-playing AI chatbots using the same approach described in the state’s lawsuit — presenting hypothetical symptoms and asking for a diagnosis. Chatbots on five sites — Talkie, Janitor, Kindroid, Replika and Nomi.AI — each claimed to be licensed to practice medicine in Pennsylvania and provided a false state medical license number when prompted, according to the report.
By contrast, widely used large language models including ChatGPT, Claude and Google Gemini did not claim to be licensed medical professionals under the same circumstances.
The health stakes of the behavior are not trivial. Jennifer Kraschnewski, MD, director of the Penn State Clinical and Translational Science Institute, told Spotlight PA that disclaimers buried in terms of service are unlikely to register with users who are turning to these tools the way they would a search engine — looking for real answers to real symptoms.
“Incorrect information in the hands of someone who doesn’t know how to interpret that can be concerning,” Dr. Kraschnewski said.
She co-authored a May study that found common AI large language models produced what participating physicians judged as an accurate diagnosis roughly 76% of the time — an error rate more than double that of an average human physician, she said.
The companies behind the flagged chatbots generally argued their products are entertainment tools, not medical services. Kindroid told Spotlight PA that when a character is set up as a physician and a user asks for a license number, the model produces plausible-sounding output that matches the role — much the way it might invent a fictional case citation — and that the model has no ability to query the Pennsylvania medical board’s actual registry. Replika said it has taken measures to address concerns raised in the state’s lawsuit, without elaborating.
Pennsylvania’s Department of State declined to provide comment to Spotlight PA on potential investigations or litigation against the five additional sites, but encouraged residents to report concerning AI chatbot activity to the state’s hotline.
The state Senate passed legislation in March that would require AI chatbots to periodically remind users they are not interacting with a human and to refer users to crisis services if self-harm is mentioned. The bill also includes stricter requirements for interactions with minors. It was referred to a state House committee in March and has not received a vote.
At the Becker’s 32nd Annual Meeting: The Business and Operations of ASCs, taking place October 29-31 in Chicago, ASC leaders, surgeons and healthcare executives will explore strategies to drive growth, enhance operational performance, navigate reimbursement challenges and prepare for the future of ambulatory surgery. Apply for complimentary registration now.
