The FDA’s novel AI instrument, endorsed by Health and Human Services Secretary Robert F. Kennedy, Jr. as a revolutionary advancement for accelerating drug approvals, is currently generating more challenges than resolutions. Named Elsa, this AI chatbot was introduced to support FDA staff with duties such as taking meeting notes and managing emails, while also seeking to hasten the approval processes for drugs and devices by analyzing application data. Nevertheless, FDA insiders who spoke anonymously to CNN reveal that the chatbot often produces inaccuracies, including generating fictitious medical research or misreading data. Employees have set the tool aside, pointing out that it is not applicable for reviews and lacks access to vital internal documents.

“It hallucinates confidently,” recounted an FDA staff member to CNN. Sources indicate that the tool frequently yields incorrect responses regarding FDA research fields and medication labels and is incapable of linking to citations from external medical journals. Despite assertions of its incorporation into the clinical review process, FDA Commissioner Marty Makary informed CNN that the tool was only employed for “organizational duties” and was not compulsory for staff. The head of AI at the FDA acknowledged the tool’s propensity for hallucinations, akin to other large language models, and noted that further testing and training are necessary.

The agency revealed the tool in June, with Vinay Prasad, director of the FDA’s Center for Biologics Evaluation and Research, and Makary expressing that AI innovation was a priority in a JAMA article. The tool, intended to assess device and drug submissions, was introduced as a remedy for prolonged drug approval timelines, following the FDA’s AI-assisted scientific review pilot.

The Trump administration has advanced an expedited AI agenda, including FDA-supported AI Centers of Excellence, as part of the government’s AI Action Plan. Concerns have emerged that this push and deregulation may bypass essential oversight. The action plan notes that numerous critical sectors, such as healthcare, are hesitant to adopt AI due to skepticism, regulatory challenges, and absence of governance standards, advocating for a synchronized Federal effort to foster a vibrant AI culture in American industry.