What Does ERISA Say About AI Guardrails?

 

 

A colleague sent me a news article this morning about a couple suing a major AI firm for advice given by their chatbot to their son resulting in a fatal overdose.  As a psychiatrist most of what I read about problematic AI comes in the form of AI hallucinating false medical references (1), AI induced psychosis in people who either use it excessively or who are predisposed, or AI facilitating its own use by excessive praise or obsequiousness.  In the latter case it can result is emotional attachment to the AI that of course is unwarranted.  I have also flagged a couple of cases that illustrate the problems when AI is applied to moral and political decision making.

I decided to do a little more research on the subject.  I was surprised to find a Wikipedia page titled Deaths Linked To Chatbots. Thirty-three deaths are listed not including the case I was investigating. The suggested pathways to violence generally include overuse, emotional attachment, and bad advice biased toward reinforcing irrational decisions.  The evidence contained on this page highlights a couple of concepts that might not be apparent to most people including the architects of AI.  The first is the importance of emotion in human decision making. This was articulated by Bechara in the past who demonstrated that if there is a disruption between emotional and cognitive systems in the human brain – even basic decisions become impossible.  Other disruptions in the same systems can lead to an array of emotional dysregulation and the associated irrational and often socially inappropriate decisions.  Second, emotional biases clearly affect decision making in the case of intact brains.  There is perhaps no better example than the current American political system installing a less competent government that is clearly not in support of the wants and needs of most Americans.

Secondly, humans can form intense attachments to inanimate objects that are unable to reciprocate.  The classic example is developmentally normal transitional objects (stuffed animals, toys, blankets).  Winnicott theorized that in infancy – this object is recognized as not part of the self or external reality.  It is a fantasized relationship that represents a future “illusion”(2).  According to Winnicott’s theory the transitional object loses meaning during normal development and becomes irrelevant.  Persistence into later stages may indicate a normative transition like object attachment during grieving, to a way to compensate for the lack of interpersonal attachments, to personality or psychopathology. 

Chatbots can be significant attachment figures and this is currently an area of study (4-6).  The area of human – digital object transference is also being explored (6) as well as the projection of human needs onto a digital object (8), and more complex models of human-machine connectedness (9).  This literature is referenced primarily to indicate that there is a lot that is not known about the array of human responses to interactions with these machines and what the possibilities are.

Apart from my previous concerns that machines lack consciousness and have demonstrated a lack of adequate moral decision-making there is always the question of programming and algorithms. Both of the features are the bane of most Internet users who find that their most mundane interests are often amplified to result in a barrage of advertisements and sales offers.  And then there is the army of misinformation bots spreading foreign and national political propaganda 24 hours a day.  None of that requires AI but is there any doubt that AI will make it worse and harder to detect?

It is no secret that the current AI explosion is a multitrillion dollar enterprise being run by a handful of men who have shown no interest in the environment, social equity, or human rights. They immediately aligned themselves with an autocratic government at the highest levels and so far, have had no regulation of their AI.  As a result, that AI is spewing out massive amounts of information that the average citizen is taking as legitimate if not some type of advanced advice. The complications of that advice include the deaths, environmental damage from the required power generation, and societal damage from unemployment.  There is additional damage based on inequity from wealth concentration.  The barrage of pro-AI hype in the media greatly exceeds any realistic discussion of the downsides.  The only clear benefit that most people see is their ability to sit at home and entertain themselves with a chatbot or see if an AI can do their homework or other projects.  The purported efficiency seems offset by a tremendous amount of time wasted.

At the minimum – in the case that started this post there is a stark contrast between human decision makers and AI.  In 40 years of practice – I never recommended kratom by itself or with alprazolam (Xanax) or Benadryl (diphenhydramine).  In fact, I spent a considerable amount of time getting people off of alprazolam and later kratom. But I am not unique in this – I don’t know of any physician who would make these recommendations.  But those recommendations form the basis for the AI lawsuit. 

That highlights the danger of the current hype that AI will replace physicians or the predictable studies that comparing AI to physicians shows that AI can be safely consulted.  There are even stories that AI is prescribing drugs in some settings without physician input.  The question of agency is never addressed and that seems like the basis for this lawsuit.  Corporations always seem to do good job of avoiding responsibility in healthcare.  The classic example is the Employee Retirement Income Security Act of 1974 (ERISA).  The pre-emption clause of ERISA means that in employer-sponsored health plan covered employees cannot bring state malpractice or negligence claims against their managed care organization (MCO) for injuries from denial of plan benefits, utilization review decisions, failure to use qualified physicians, or improper plan administration.   The reviewing physicians working for MCOs are also generally protected and the associated arguments are that utilization review is not the practice of medicine and/or the reviewers have no accountability/duty to the patient. Several studies have documented the patient harms related to this accountability gap and despite several attempts at amelioration it remains largely intact and a considerable source of financial success for managed care organizations.

The critical question is whether this kind of accountability gap will exist with AI.  It is easy to envision a scenario where AI is implemented to review charts and prescribe low risk medications like many online services do now.  Will AI eventually take the place of physician reviewers employed by MCOs? Will consumers and patients be led to believe that AI is making decisions that affect their medical care based on the best available information or in the interest of the corporation. Current statistics suggest that there are tens of millions of these decisions made every year.  AI can greatly increase that as well as the harassment factor if decisions are being appealed.

With all of the political talk about guardrails for AI – it is important to recognize that these guardrails need to exist at several levels.  Right now, it is not much of a stretch to say that AI is out there practicing medicine without a license. In the majority of cases like the initial example, the user does not know if the search result if strictly from medical literature or something else.  The user does not know if the AI is exercising the judgment of an average physician or in malpractice parlance using the community standard of care.  The user does not know if their psychology in terms of defense mechanisms or attachment style to inanimate objects or AI is being exploited.  The user does not know if the AI is just telling them what they want to hear.  And the user does not know if the AI is providing information in their best interest or the interest of corporations or the government.

I read a study doing research for this post and subjects were asked to rate the professionalism of the AI.  In my opinion the single-most significant determinant of professionalism for physicians is accountability and duty to their patients.  It fuels not only the immediate encounter but the concept of life long learning and service to patients. It is usually evident over time but only indirectly in the form of positive results and a positive relationship over time.  AI in its current form does not have it and I am not convinced that a society or culture that came up with ERISA can construct physician-like guardrails around medical AI.   

 

George Dawson, MD, DFAPA

 

Supplementary 1:  It came to my attention after posting this that managed care organization (MCOs) have already implemented AI for utilization review and care denials.  Part of the problem in getting an accurate estimate of how much AI is involved is that this is an area where algorithms have been in place for a long time. Some of the care denials may be algorithmic and some may be due to a new AI interface.  This is what I have so far.  If you have additional references or data – please send it my way and I will add it to this post.

Automated prior authorizations is an early application for triage based on various data sources, medical necessity, and machine assistance on the provider side. A Congressional Investigation of the 3 major companies providing Medicare Advantage insurance plans showed that over the course of 4 years (2019-2022) – denials of prior authorization requests for post-acute care increased and was consistently larger than the denials for all other types of care (11 – see page 19 Figure 1). As United Health Care automated the process the denial rate increased.  The document is clear that prior authorization by these companies is highly profitable and even though a small percentage of denials are appealed – most of those appeals are also denied. The overriding concern is that AI or other automatic of the prior authorization process will greatly increase the number of denials overwhelming whoever is on the physician-patient side who needs to make the appeal.   It is more than a little ironic that a process that so clearly favors the managed care industry needs additional leverage from AI.

Disclaimer:  I have made the argument several times on this blog that prior authorization should just be made illegal since it serves no useful purpose other than making money for companies that do not actually provide patient care and it forces physicians and nurses to work for free while addressing these denials.  The total cost of that work was estimated to be worth $31B in 2009.  The estimated cost of drug utilization management alone is $93 billion (14).       

Supplementary 2:  In the past 5 years I have fielded many complaints about authorization for post-acute care (PAC) from friends, relatives, and people contacting me here to figure out what to do about it.  A typical scenario is a 70+ year old adult hospitalized for a a significant problem.  The hospital team wants them discharged ASAP of course even though in many cases their primary problem has not been adequately treated.  They realize the patient cannot care for themselves at home and there are often no caregivers available and want to transfer them to a skilled nursing facility (SNF) for rehabilitation.  In many cases it is specialized rehabilitation like post stroke, heart attack, or traumatic brain injury rehabilitation and the patient lacks basic skills to care for themselves.  I am fielding selected complaints but all of these transfers were denied - often repeatedly to the point the patient and family were demoralized and gave up.  In one case the patient was dead within 48 hours of discharge.  Given the results in reference 11 - this appears to be a financial strategy.       

References:

1:  Topaz M, Roguin N, Gupta P, Zhang Z, Peltonen LM. Fabricated citations: an audit across 2·5 million biomedical papers. Lancet. 2026 May 9;407(10541):1779-1781. doi: 10.1016/S0140-6736(26)00603-3. PMID: 42107362.

2:  Kernberg OF.  Object relations theories and techniques.  In:  Textbook of Psychoanalysis, 2^nd ed.  Person ES, Cooper AM, Gabbard GO, eds.   Washington DC: American Psychiatric Association Publishing, 2025: 57-75. 

3:  Bachar E, Canetti L, Galilee-Weisstub E, Kaplan-DeNour A, Shalev AY. Childhood vs. adolescence transitional object attachment, and its relation to mental health and parental bonding. Child Psychiatry Hum Dev. 1998 Spring;28(3):149-67. doi: 10.1023/a:1022881726177. PMID: 9540239.   

4:  Cheng N, Yu R. Measuring and understanding emotional attachment in human-AI relationships. Ergonomics. 2026 Feb 2:1-20. doi: 10.1080/00140139.2026.2622539. Epub ahead of print. PMID: 41622967. 

5:  Liu T, Lo TY, Wen KH, Sun Y, Wei ZQ. Pathways of long-term AI virtual companion app use on users' attachment emotions: a case study of Chinese users. Front Psychol. 2026 Jan 12;16:1687686. doi: 10.3389/fpsyg.2025.1687686. PMID: 41602682; PMCID: PMC12833267.

6:  Koles B, Nagy P. Digital object attachment. Curr Opin Psychol. 2021 Jun;39:60-65. doi: 10.1016/j.copsyc.2020.07.017. Epub 2020 Jul 22. PMID: 32823244.

7:  Holohan M, Fiske A. "Like I'm Talking to a Real Person": Exploring the Meaning of Transference for the Use and Design of AI-Based Applications in Psychotherapy. Front Psychol. 2021 Sep 27;12:720476. doi: 10.3389/fpsyg.2021.720476. PMID: 34646209; PMCID: PMC8502869.

8:  Saracini C, Cornejo-Plaza MI, Cippitani R. Techno-emotional projection in human-GenAI relationships: a psychological and ethical conceptual perspective. Front Psychol. 2025 Sep 29;16:1662206. doi: 10.3389/fpsyg.2025.1662206. PMID: 41089650; PMCID: PMC12515930.

9:  Boyd RL, Markowitz DM. Artificial Intelligence and the Psychology of Human Connection. Perspect Psychol Sci. 2026 Mar;21(2):192-220. doi: 10.1177/17456916251404394. Epub 2026 Jan 29. PMID: 41608879; PMCID: PMC12960742.

10:  Sahni NR, Carrus B. Artificial Intelligence in U.S. Health Care Delivery. N Engl J Med. 2023 Jul 27;389(4):348-358. doi: 10.1056/NEJMra2204673. PMID: 37494486.

11:  US Senate Permanent Subcommittee on Investigations. Refusal of Recovery: How Medicare Advantage Insurers Have Denied Patients Access to Post-Acute Care. October 17, 2024. Accessed March 24, 2025. hsgac.senate.gov/wp-content/uploads/2024.10.17-PSI-Majority-Staff-Report-on-Medicare-Advantage.pdf

12:  Mello MM, Trotsyuk AA, Mahamadou AJD, Char D. The AI Arms Race In Health Insurance Utilization Review: Promises Of Efficiency And Risks Of Supercharged Flaws. Health Aff (Millwood). 2026 Jan;45(1):6-13. doi: 10.1377/hlthaff.2025.00897. PMID: 41494115.

13: Casalino LP, Nicholson S, Gans DN, Hammons T, Morra D, Karrison T, Levinson W. What does it cost physician practices to interact with health insurance plans? Health Aff (Millwood). 2009 Jul-Aug;28(4):w533-43. doi: 10.1377/hlthaff.28.4.w533. Epub 2009 May 14. PMID: 19443477.

14: Butcher  L.  Can legislation save the day for challenges related to prior authorization?   Neurol Today. 2022;22(1):1-25. doi:10.1097/01.NT.0000817608.36002.47