Anecdotes Are Not Evidence And Other "Evidence-Based" Fairy Tales

A lot depends on the kind of anecdote that you are talking about.  When we talk about anecdotes in medicine that generally means a case or a series of cases.  In the era of "evidence based medicine" this is considered to be weak or non-existent evidence.  Even 30 years ago when I was rounding with a group of medicine or surgery residents: "That's anecdotal..." became a popular part of roundsmanship as a way to put down someone basing their opinion on a case or series of cases.  At the time, series of cases were still acceptable for publications in many mainstream medical journals.  Since then there has been and inexorable march toward evidence-based medicine and that evidence is invariably clinical trials or meta-analyses of clinical trials.  In some cases the meta-analyses can be interpreted in a number of ways including interpretations that are opposed to the interpretation of the author.  It is easy to do if a biologically heterogeneous condition is being studied.

There has been some literature supporting the anecdotal but it is fairly thin.   Aronson and Hauben considered the issue of drug safety.  They made the point that isolated adverse drug reactions are rarely studied at a larger epidemiological level following the initial observation.  They argued that they could establish criteria for a definitive adverse event based on a single observation.  In the case of the adverse drug event they said the following criteria could be considered "definitive":  extracellular or intracellular deposition of the drug, a specific anatomic location or pattern of injury, physiological dysfunction or direct tissue damage, or infection as a result of an infective agent due to contamination (1).  They provide specific examples using drugs and conclude: "anecdotes can, under the right circumstances, be of high quality and can serve as powerful evidence.  In some cases other evidence may be more useful than a randomized clinical trial.  And combining randomized trials with observational studies, or with observational studies and case series, can sometimes yield information that is not in clinical trials alone."  This is essentially the basis for post marketing surveillance by the Food and Drug Administration (FDA).  In that case, monitoring adverse events on a population wide basis, increases the likelihood of finding rare but serious adverse events compared with the the original trial.

Enkin and Jadad had a paper that also considered anecdotal evidence as opposed to formal research (2).   They briefly review the value of informal observation and the associated heuristics, but also point out that these same observations and heuristics can lead some people to adhere to irrational beliefs.  The values of formal research is a check on this process when patterns emerge at a larger scale.  Several experts have applied Bayesian analysis to single case results, to illustrate how pre-existing data can be applied to single cases analyses with a high degree of success.  Pauker and Kopelman (3) looked at a case of hypertension and when to consider a search for secondary causes - in this case pheochromocytoma.  They made the interesting observation that:

"Because the probability of pheochromocytoma in a patient with poorly controlled hypertension who

does not have weight loss, paroxysms of sweating, or palpitations is not precisely known, the clinician often makes an intuitive estimate.  But the heuristics (rules of thumb) we use to judge the likelihood of diseases such as pheochromocytoma may well produce a substantial overestimate, because of the salient features of the tumor, its therapeutic importance, and the intellectual attraction of making the diagnosis."   

They take the reader through a complex presentation of hypertension and likelihood ratios used to analyze it and conclude:

"Hoofbeats usually signal the presence of horses, but the judicious application of Bayes' rule can help prevent clinicians from being trampled by a stampeding herd that occasionally includes a zebra."

In other words, by using Bayes rule,  you won't subject patients with common conditions to excessive (and risky) testing in order to not miss an uncommon condition and you won't miss the uncommon condition.  Looking at the data that supports or refutes that condition will make it clear, if you have an idea about the larger probabilities.

How does all of this apply to psychiatry?  Consider a few vignettes:

Case 1:  A psychiatric consultant is asked to assess a patient by a medicine team.  The patient is a 42 year old man who just underwent cardiac angiography.  The angiogram was negative for coronary artery disease and no intervention was necessary.  Shortly afterwards the patient becomes acutely agitated and the intern notices that bipolar disorder is listed in the electronic health record.  He calls the consultant and suggests an urgent consult and possible transfer to psychiatry for treatment of an acute manic episode.  The consultant arrives to find a man sitting up in bed, appearing very angry and tearful and shaking his head from side to side.

Case 2:  A 62 year old woman is seen in an outpatient clinic for treatment resistant depression.  She has been depressed for 20 years and had had a significant number of medication changes before being referred to a psychiatric clinic.  All of the medications were prescribed by her primary care physician.  She gives the history that she gets an annual physical exam done each year and they have all been negative.  Except for fatigue and some lightheadedness, her review of systems is all negative.  She is taking lisinopril for hypertension, but has no history of cardiac disease.  She has had electrocardiograms (ECGs) in the past but no other cardiac testing.  The psychiatrist discusses the possibility of a tricyclic antidepressant.

Case 3:  An inpatient psychiatrist has just admitted a 46 year old woman who is concerned about the FBI.  She has been working and functioning at a high level until about a month ago when she started to notice red automobiles coming past her cul de sac at a regular frequency.  She remembered getting into an argument at work about American military interventions in the Middle East.  She made it very clear that she did not support this policy.  She started to make the connection between the red automobiles and the argument at work.  She concluded that an employee must have "turned her in" to the federal authorities.  These vehicles were probably the FBI or Homeland Security.  She noticed clicking noises on her iPhone and knew it had been cloned.  She stopped going into work and sat in the dark with the lights out.  She reasoned it would be easier to spot somebody breaking into her home by the trail of light they would leave.   There is absolutely no evidence of a mood disorder, substance use disorder, or family history of psychiatric problems.  She agrees to testing and all brain imaging and laboratory studies are normal.  She agrees to a trial of low dose antipsychotic medication, but does not tolerate 3 different medications at low doses.

These are just a very few examples of the types of clinical problems that psychiatrists encounter on a daily basis that require some level of analysis and intervention.  A psychiatrist over the course of a career is encountering 20 or 30 of these scenarios a day and ends up making tens of thousands of these decisions.  What is the "evidence basis" of these decisions?  There really is nothing beyond anecdotes with the availability of various strengths of confirmation.  What kinds of evidence would the evidence based crowd like to see?  In Case 1, a study that looked at behavioral disturbances after cardiac catheterization would be necessary, although Bayes would suggest the likelihood of that occurring would be very low.  In Case 2, a large trial of treatment approaches to 62 year old women with depression and fatigue would be useful.  I suppose another trial of what kinds of laboratory testing might be necessary although much of the literature suggests that fatigue is very nonspecific.  In most cases where patients are being seen in primary care - fatigue and depression are indistinguishable.  Extensive testing, even for newer inflammatory markers yields very little.  Further on the negative side,  evidence-based authorities suggest that a routine physical examination and screening tests really adds nothing to disease prevention or long term well being.  Case 3 is more interesting.  Here we have a case of paranoid psychosis that cannot be treated due to the patient experiencing intolerable side effects to the usual medication.  Every practicing psychiatrist knows that a significant number of people can't take entire classes of medications.  Here we clearly need a clinical trial of 40 year old women with paranoid psychoses who could not tolerate low dose antipsychotic medication.

By now my point should be obvious.  None of the suggested trials exist and they never will.  It is very difficult to match specific patients and their problems to clinical trials.  Some of the clinical occurrences are so rare (agitation after angiography for example) that it is very doubtful that enough subjects could be recruited and enrolled in a timely manner.  And there is the expense.  There are very few sources that would fund such a study and these days, very few practice environments or clinical researchers that would be interested in the work.  Practice environments these days are practically all managed care environments where physician employees spend much of their time administrative work, the company views clinical data as proprietary, and research is frequently focused on advertising and marketing rather than answering useful clinical questions.

That brings us to the larger story of what the "evidence" is?  The anecdotes that everyone seems to complain about are really the foundation of clinical methods.  Training in medicine is required to experience these anecdotes as patterns to be recognized and classified for future work.  They are much more than defective heuristics.  How does that work?  Consider case #1.  The psychiatric consultant in this case sees an agitated and tearful man who appears to be in distress.  The medicine team sees a diagnosis of bipolar disorder and concludes the patient is having an acute episode of mood disturbance.  The consultant quickly determines the fact that the changes are acute and rejects the medical team's hypothesis that this is acute mania.  After about 5 questions he realizes that the patient is unable to answer, pulls a pen out of his pocket, and asks the patient to name of the pen.  When he is not able to do this, he performs a neurological exam and determines the patient has right arm weakness and hyperreflexia.  An MRI scan confirms the area of an embolic stroke and the patient is transferred to neurology rather than psychiatry.  The entire diagnostic process is based on the past anecdotal experience of diagnosing and treating neurological patients as a medical student, intern, and throughout his career.  Not to labor the point (too much) - it is not based on a clinical trial or a Cochrane review.                

The idea that the practice of medicine comes down to a collection of clinical trials that that are broken down according to a proprietary boilerplate and generally conclude that the quality of most studies is low and therefore there is not much to draw conclusions on is absurd.  Trusting meta-analyses for answers is equally problematic.  You might hope for some prior probability estimates for Bayesian analysis.  But you will find very little in the endless debate (4) about whether or not antidepressants work or they are dangerous.  You will find nothing when these studies are in the hands of journalists who are not schooled in how to practice medicine and know nothing about treating biologically heterogeneous populations and unique individuals.  No matter how many times the evidence based crowd tells you that you are treating the herd - a physician knows that if they treat the herd - it is one person at a time.  They also know that screening the herd can lead to more problems than solutions.

Treating the herd would not allow you to make a diagnosis of complete heart block and immediate referral to Cardiology for pacemaker placement (Case 2) or psychotherapy with no medications at all and eventual recovery (Case 3).  If you accept the results of many clinical trials or their interpretation by journalists - you might conclude that your chances of recovery from a complex disorder are not better than chance.  There is nothing further from the truth.

That is why most of us practice medicine.



George Dawson, MD, DLFAPA


References:

1:   Aronson JK, Hauben M. Anecdotes that provide definitive evidence. BMJ. 2006 Dec 16;333(7581):1267-9. Review. PubMed PMID: 17170419; PubMed Central PMCID: PMC1702478.

2:  Enkin MW, Jadad AR. Using anecdotal information in evidence-based health care:  heresy or necessity? Ann Oncol. 1998 Sep;9(9):963-6. Review. PubMed PMID: 9818068.

3:  Pauker SG, Kopelman RI. Interpreting hoofbeats: can Bayes help clear the haze? N Engl J Med. 1992 Oct 1;327(14):1009-13. PubMed PMID: 1298225.

4:  Scott Alexander.  SSRIS: MUCH MORE THAN YOU WANTED TO KNOW.  Slate Star Codex posted on July 7, 2014.

Attribution:

Photo at the top is Copyright Gudkov Andrey and downloaded from Shutterstock per their Standard License Agreement on 2/3/2016.

Data On Drug Price Comparisons - And The Myth Of Compromised Physicians

See Attributions For Reference

ND = no data

The response to my last post so far was as predicted by what I said in the post.  It is very difficult for people to get around the idea that they have heard for the past two decades - namely "Damn you Big Pharma!"  Over the past 20 years we have repeatedly heard all of the concerns about physicians essentially being bribed by Big Pharma in the form of speaker's fees, free lunch, various trinkets, ghost written research, and free vacations.  We have seen physicians criticized by a member of Congress for failing to disclose income from sources outside of their academic appointments.   We have seen psychiatrists selected out from other physicians with regard to Big Pharma financing despite the work of a well known non-partisan watchdog showing that they are nowhere near the top of the list in terms of total reimbursement or frequency.  Many people have made a career out of adding various conspiracy theories to the basic Big-Pharma<->physician conspiracy and how it has added unnecessary costs to the health care system, put patients at unnecessary risk,  and compromised professional ethics.   The only major change that I have detected is the elimination of the free lunch at Grand Rounds.  I do so appreciate that.  There was nothing that triggered my misophonia more than the sounds of mastication while I was trying to listen to the lecturer.  Now that all of those evil Big Pharma incentives have been eliminated and the risk of public shaming is in place through at least two databases, it would follow that Big Pharma should be hurting - right?  We should finally be getting reasonable priced pharmaceuticals - right?  Not if the following slide from the Kaiser Family Foundation is to be believed:

See Attribution Section Below For The Full Credit For This Graphic

It seems that the public shaming of physicians and eliminating the various forms of the Big Pharma free lunch have not led to the Utopian state of better pharmaceutical pricing.  The really telling information is in the tables at the top.  This data is widely quoted in a number of sources, but is also readily available from the original source.  The US has the market cornered when it comes to the absolute maximum drug prices.  In some cases other countries are only paying about a quarter as much.  My table also removed the maximum prices in the US that are in some cases much higher than is quoted in this table.   This data illustrates why taking physicians out of the equation has has done nothing.  Of course it will be interesting to look at the data over  time databases and make sure that there is the expected lack of correlation.  This data as well as the data on prescription pricing explodes the myth that physician collusion with Big Pharma had anything to do with pharmaceutical company profits.  For years we have had to tolerate vague rhetoric from Pharmascolds like: "If they (Big Pharma) didn't get a return (on their various trinkets, meals, other incentives) - they wouldn't do it."  There was the associated argument that getting free pens would make you start prescribing the advertised drug like you were a Big Pharma Manchurian candidate.  Neither of those arguments had any traction with me, but then again I had not talked with a pharmaceutical rep in over 20 years.  Compare these arguments with the clinical reality that physicians face every day and that is being harassed by managed care companies if they do not prescribe the least expensive drugs.  Any physician prescribing only the latest antidepressant would spend most of their time on the phone with pharmaceutical benefit managers.  They would not be able to practice.

The third argument was the moral one.  That it was somehow unethical to work for a pharmaceutical company or accept anything from them because it represented a conflict of interest.  Notice I did not use the term appearance of conflict of interest.  That is because the Institute of Medicine has decided for all of us that it is so hard to determine a real conflict of interest from the appearance of conflict of interest - why bother?  Consider it all to be conflict of interest.  To me that always seemed like a variation of the automaton argument - I have accepted pizza or a pen and now I can no longer think for myself - I will just automatically prescribe the suggested drug.  Nobody ever examined the strong reinforcement associated with the idea that:  "I don't eat the free lunch and therefore I am morally superior to you."  That unexamined thought seemed palpable on many blogs and websites where daily outrage about these practices was common.

The fallout from this lack of examination has been significant:

1.  Fewer physicians wanting to work with the industry - medicine is probably the only technical profession that makes this suggestion.  In many professions standards are set by active collaboration with industries.  I don't know how a pharmaceutical company can look for new molecular entities without an eye to problems that clinicians encounter and a solid knowledge of the shortcomings of current therapies.  You can't find that in a lab.

2.  Overgeneralizations about psychiatry based on the predictably negative press - psychiatry takes more of a negative hit on just about anything than the rest of medicine.  The Myth of Compromised Physicians has allowed an absurd level of criticism to be leveled at the field and ignore even basic realities that psychiatry specialty organizations and psychiatrists are hardly the most involved specialists with Big Pharma.  You would not get that impression by reading the popular press or the various antipsychiatry sites in the Internet.

3.  An absurd emphasis on evidence based medicine - as though that could somehow save us from the evils of Big Pharma or ourselves.  There have been endless politically biased analyses to prove that psychiatric treatments do or do not work.  In many cases, the result of the study can be predicted by the author's bias.  In many cases the author's bias is evident even without financial conflict of interest disclosures, all that you have to do is read their previous writing.  Many of these papers are foregone conclusions.  They naturally add nothing to the field because they either lack scholarship or that was not their intent in the first place.  They miss on three standards.  The first involves the intent of regulation of drugs in the United States and the science of pharmaceutical research.  There are no perfect drug trials and the results don't have to be perfect to get approval.  In some cases the results are far from perfect and the drug is approved, even against the vote of the involved scientific committee.  Safety considerations are often clarified in post marketing surveillance.   The second involves the positive experience of clinicians using the drug.  Drugs are often prescribed off label with great success and experienced clinicians have often treated many more patients by themselves than were in the original trial.  They may have better results in the trial largely by their experience using the drug and more comprehensive treatment than is available in drug trials.  There are many reasons why the experience of clinicians using the drug would be expected to be better than the trial, but the trial is considered the gold standard of whether or not a drug "works."  The third involves the safety considerations of the physicians using the drug.  There have been some studies that go back and look at all of the side effects of the drug in clinical trials and try to recalculate risks or side effects and adverse outcomes or to prove the pharmaceutical company or researchers were covering something up - they weren't transparent.  Any clinician who studies the FDA approved package insert for the drug and pays close attention to what their patients tell them, will know much more about the dangers of the drug and its side effects, how to detect and treat them better than any group of people reading research reports.  To think otherwise is folly.  

4.  A serious lack of appreciation of what the real problems are in clinical trials and that is biological heterogeneity.  Any number of polygene determined illnesses will understandably not yield positive and uniform results with great effect sizes in response to a treatment.  I don't care if the illness being studied is depression or asthma or diabetes mellitus.  Why is that shocking or surprising?  Why would it be surprising that some researchers want to break these large heterogeneous groups into small subgroups and see if the treatment response can’t be refined?

5.  A stunning lack of examination of the real problem.  That real problem is quite simply special access to all levels of government on the legislative and regulatory side by industry lobbyists.  Industry in this case includes insurance and managed care companies, and pharmaceutical companies.  Physician professional organizations have no similar access.  Nothing guarantees profits more than lobbyists sitting in a smoke-filled room and writing legislation that regulates your industry.

Take physicians out of the loop and what do you have?

The most expensive prescription drugs (by far) in the world.



George Dawson, MD, DFAPA




Supplementary:

Doctor databases: These databases are there to list payments to physicians from pharmaceutical or medical device manufacturers.  I refer to them as public shaming databases because that is  what they are used for in the press and blogosphere.  There are also obvious comparisons for similar databases that exist for Congress and the obvious fact that transparency doesn't work.  Feel free to look for my name, but I can tell you in advance that you won't find it:

Open Payments - The Official US Government Web site - https://openpaymentsdata.cms.gov/

ProPublica - Dollars For Docs



Attributions:

1:  The Table "2013 Drug Prices In Various Countries" is from a report by the International Federation of Health Plans.  The report is titled: "2013 Comparative Price Report Variation in Medical and Hospital Prices by Country."  It is quoted in many places including the reference below and the report is freely available as a PDF document.  It was accessed on 1/30/2016.

2:  The graphic of "Growth In Prescription Drug Spending......" was downloaded as a Power Point Presentation entitled Attachment-Rx-Spending-and-Use-UPDATED 12.31.2015.  Author is The Henry J. Kaiser Family Foundation. It was accessed on 1/30/2016 and is used by a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License

Asthmatics Held Hostage

I never thought in my wildest dreams that I would end up paying $275 for 0.3 mg of epinephrine.  My allergist told me that the range of the patients he sees is about $150-200.  My allergic friends tell me that in some cases they are paying as low as $80 depending on insurance coverage.  Most Americans are on high-deductible insurance plans like I am.  That means that for the first half of the year or longer - expect to pay the full retail price for medications.  Despite the fact that the price of the autoinjectors has skyrocketed in the past few years, they still contain just a few dollars worth of drug in a fancy injector.  Most people who have anaphylaxis experience occasional reactions when exposed to a certain allergen and do not need to use the pens on a routine basis.  They are warned that the pens expire at a certain date and need to be replaced.  A 2014 paper (1) that looked at an emergency department intervention for anaphylaxis commented on the costs of the auto-injector weighed against both inadequate treatment of anaphylaxis and the potential adverse consequences of administering the wrong dose.  They looked at Average Wholesale Price (AWP) of the autoinjector containing 0.3 mg and 0.15 mg doses of epinephrine ($75.00) and a 1 mg vial of 1:1000 epinephrine ($3.00).  Obviously if you are having an anaphylactic reaction it is useful to have a viable and easily usable form handy to address the problem.  On the other hand what is it about packaging that results higher costs?  Until recently there has been no competition for epinephrine auto-injectors.

The other factor driving the cost is that manufacturers and insurance companies know that patients who need this device are over a barrel.  They need the medication and will pay for it.  If a steep copay or paying retail price due to a deductible is required - many will pay that price rather than risk death.  I said many because the high price of the auto-injector is preventing some patients from refilling their prescriptions.  A letter (2) found that only 40% of adults and teens and 60% of children refilled the auto-injectors at the appropriate time and suggested that cost may have been a factor.  In many ways this could be viewed as an unintended consequence of high deductible insurance and contrary to the pop economic theory that consumers will be more price sensitive when they are spending their own money.  That can only happen when there are price competitive products.

For that reason I have included the Advair Diskus Inhaler.  I have suggested in previous posts that asthma and most severe mental illnesses have a lot in common because of complicated genetics, diverse mechanisms, phenotypic heterogeneity, poor treatment response and chronicity.  Contrary to what most of us were taught in medical school most asthmatics are symptomatic.  Although many of the precipitants of worsening asthma control are known - there seems to be very little interest in doing much about it.  From a marketing standpoint that creates a lot of demand for products to address those issues.  Although there is no clear evidence that one product is superior to others,  Advair Diskus Inhalers have demanded a large market share.  During some recent years they have sold over $13 billion of product in a market where one billion in sales is considered a blockbuster drug.  In 2013, it was the third highest in pharmaceutical sales.  The inhaler pictured at the top of this post cost me $345.  The critical deductible period of my health insurance plan is the reason why I am paying that much cash to take a medication that I take all year long.  Once again it is all in the packaging.  The patent on the fluticasone/salmeterol drug combination ran out in 2010.  The unique disk delivery system remains on patent until mid-2016.  There is still debate about what will constitute a generic substitute.

I think that there are valuable lessons in the marketing and sales of both Advair Diskus and EpiPen that are relevant now that pharmaceutical sales are spiking again.  There is also a valuable lesson about the "market forces" argument for high-deductible insurance.  Packaging alone in these two cases is enough to capture all of that $3000-6000 deductible from anyone who wants to keep taking a prescribed medication.  It also illustrates that rather than being an intermediary for cost effectiveness, insurance companies are much more likely to step aside, and let their customers absorb the  full retail cost of a medication rather than negotiating better prices with manufacturers.  It may take a while for an American public that has been accustomed to Big Pharma and physician conspiracy theories to realize what is really going on.

There is also an associated lack of data on exactly how much people are paying.  I have been describing health care costs as a hidden tax (even before the Obamacare penalty) because it easily exceeds property taxes for most people and is their second highest expenditure after income taxes.  A recent paper by the Urban Institute backs that opinion up with data.  The authors looked at health care costs based on income as a percentage of the Federal Poverty Level (single person = $11,770, 2 person family = $15,930).   The authors looked at a simulation of various income brackets for people enrolled in Affordable Care Act (ACA) compliant non-group plans.  They found that people in both the median and 90th percentile in each bracket pay substantial amounts for premiums and out-of-pocket expenses as a percentage of their income.  As an example, a couple earning 300-400% of the FPL earns about $48,000-$64,000 per year.  At the median income they would pay 14.5% of their income for health care premiums and out-of-pocket expenses.  At the 90th percentile, they would pay 22.2% of their income.   According to this simulation, only those at < 200% of the FPL or > 500% of the FPL would pay less than 10% of their income toward health care expenses.  The range for all two person families was 10.8-13.4% for incomes of $32,000 to $80,000 per year.   That number easily exceeds the average nation wide property tax figure of $2,800 per year.

The title of this post was intentionally dramatic.  I hoped to illustrate just how incredibly expensive health care costs are here in the United States.  I encourage rethinking some of the issues related to this cost.  In this post, it is apparent that business leverage still carries the day.  The pharmaceutical and insurance/managed care businesses can still make as much money as they want to.  It is possible by the pharmaceutical funding and insurance rules that exist out there and of course they are all approved or written by business friendly politicians.  The second shocking fact are the estimates of health care cost for middle income Americans as 10.8-13.4% of their income.  That easily exceeds property tax costs and for most Americans it also exceeds their state income tax bill.

That leads me to conclude that health care costs are really the second largest tax on all Americans, exceeded only by federal taxes.  That fact should stop anyone in their tracks and lead them to think about all of the free market rhetoric and the relationship between this industry and Congress.  It should be clear that there really is no market competition or savings as a result of the current managed care system.  The only management that is being done is managing to put a significant portion of your personal finances in the control of the healthcare industry.  The best solution is to get rid of this system rather than continue to tax Americans to subsidize companies whose products have value only in an artificially inflated marketplace - inflated by this health care tax.       


George Dawson, MD, DFAPA

References:

1: Manivannan, Veena et al. “A Multifaceted Intervention Increases Epinephrine Use in Adult Emergency Department Anaphylaxis Patients.” The Journal of Allergy and Clinical Immunology. In practice 2.3 (2014): 294–299.e1. PMC. Web. 30 Jan. 2016.

2: Westermann-Clark E, Fitzhugh DJ, Lockey RF.  Increasing cost of epinephrine autoinjectors. J Allergy Clin Immunol. 2012 Sep;130(3):822-3. doi: 10.1016/j.jaci.2012.06.018. Epub 2012 Jul 22. PubMed PMID: 22828415.

3:  Linda J. Blumberg, John Holahan, and Matthew Buettgens.  How Much Do Marketplace and Other Nongroup Enrollees Spend on Health Care Relative to Their Incomes?  Robert Wood Johnson Foundation; Urban Institute.  December 2015.  Accessed on January 29, 2016.

The Real Solution To Burnout

One of my favorite things these days is the concept (or is it diagnosis?) of burnout.  It seems to be a popular topic in medical and psychiatric news these days.  In the Psychiatric Times January 2016 edition, Editor in Chief Allan Tasman, MD published a column on burnout entitled My New Years Prescription for You.  He goes on to detail the syndrome and what can be done about it.  He points out the high prevalence of burnout in physicians including house staff, physicians in general and psychiatrists.  These studies generally depend on checklist surveys of symptoms suggestive of "burnout."  Dr. Tasman points out that they are relatively nonspecific and people may not see psychiatrists about burnout until there are more recognizable syndromes of anxiety or depression.

My problem with the concept of burnout is that it doesn't accurately describe the problem.  As I think back on some of my most engaging clinical rotations in training - the teams frequently worked to the point of exhaustion.  The attending came in the next day.  There was an air of collegiality and a lot of learning occurred.  There was a lot of dark humor on the part of house staff.  There was an understanding that all of this exhausting work would end some day when you made the transition to a staff or attending physicians and could work more normal hours.  That was the late 1980s and early 1990s.  As politicians and business people wrested control away from physicians, suddenly most physicians continue to work like they are house staff.  Senior physicians in their 60s are suddenly taking all night call and working 60-70 hours per week.  Hospitalist services were invented requiring physicians to work 7 days on and 7 days off - another exhausting schedule.  I have observed to many of these physicians that they are working like they did when they were house staff - interns and residents.  They numbly shake their heads in the affirmative when I ask them that question.  They also acknowledge the fact that by day 6, their cognitive capacity is markedly diminished.  Suddenly it takes them twice as long to do tasks especially all of the documentation.

The reference to Studer in the Tasman article is interesting.  I don't know if any other physicians have had to suffer through a business consultant-based inservice on how to improve "customer satisfaction scores".  There are discussions on how to introduce yourself to the "customer".  There are the usual business based mnemonics.  Physicians may actually have to demonstrate that they know how to introduce themselves to "customers"!  Think about that for a second, especially if you are a psychiatrist who was trained for years in how to interact with patients rather than customers.  If you are a psychiatrist who passed the oral boards,  you know that failing to make the appropriate introduction led to an immediate failure on that exam.  Now flash forward to the bizarre world where patients are "customers" and now there is a formula designed by business people who know relatively nothing about interacting with patients in a therapeutic manner.   You are expected to demonstrate competency in this shallow business paradigm that is setup to optimize results on customer satisfaction surveys.  This is a great example of how physicians are stressed on a regular basis in health care organizations and their time is wasted.  It is also a great example of how public relations, rather than the latest medical knowledge is the dominant performance metric for healthcare organizations.  If there is a recipe for burnout - this is it.

The dynamics of burnout are the dynamics of many clinical situations that psychiatrists try to address.  The referrals are people with chronic depression or depression that seems to have occurred as a result of a sudden change in their life circumstances.  A common scenario is an unreasonable employer or work supervisor.  I will understand it if the employers jump in here and say that they are entitled to tell people how they want them to work for their salary or that their employees are free to find another job.  Those are political arguments that I don't really care about.  Those arguments are also improbable ways of addressing burnout.

When I am faced with person who is chronically anxious and depressed, chronically sleep deprived due to forced swing shifts or double shifts, is dealing with an obnoxious demanding boss, and is not able to change jobs for economic or insurance reasons - I know the patient and I are up against a wall.  I speculate that there are millions of people in this situation who are diagnosed with one anxiety or depressive disorder or another or chronic insomnia and who are trying to get some kind of treatment to alleviate this stress.  There is no evidence that I am aware of that treatment that targets what is basically a chronic stress response is effective.  There may be some small incremental changes if people feel supported and are getting active feedback in therapy about how to deal with the stress in realistic ways, how the dynamics may have personal meaning, and how to reframe the stressful relationships but many people are likely to stay in treatment for the diagnosis for months or years and have little to show for it.  Many people have the expectation that there is a medication that will restore their ability to function in this situation and not require any significant changes on their part.  That is completely false.

That brings me to the issue of physician burnout.  Burnout is more than the clinical diagnoses that are used to describe people who are experiencing chronic workplace stress.  The current work environment for physicians is designed to produce burnout, anxiety, depression, and all of the associated comorbidity.  One of the central dynamics is administrators with no medical knowledge creating an environment that moves physicians away from patients and creates an onerous clerical and administrative burden.  The large increase in managers has created an environment that is both hostile and full of busy work.  The idea that this is something that can be overcome with medications, meditation, exercise, lifestyle management or psychotherapy leaves a lot to be desired.  It is time that psychiatrists focus on an optimized environment for mental and physical well being rather than than trying to treat the fallout from some of those horrific scenarios.

Addressing burnout in physicians is more than a health and wellness consult.  It is more than a weekend retreat to a local resort.  It is more than "lifestyle changes" when you don't have enough time to have a life.  It is a lot more than going on vacation and realizing that on the day you come back - it is like you never left.  Optimizing the work environment for physicians rather than treating burnout is a good place to start.  Recognizing this when it happens in our patients is also more useful than treating it like depression.



George Dawson, MD, DFAPA





  

Certitude - Moral and Otherwise

Like many Americans - my wife and I recently watched the Netflix series Making A Murderer.  Some of the the critical reviews have called it riveting.  A few have suggested that not all of the evidence was given - making it an overly favorable presentation for the defendants.  In this series two Wisconsin men are accused of murdering a young woman and then burning her body.  The case is sensational because one of them - Steven Avery was in the process of suing the county where the crime occurred for wrongful prosecution in a previous case where he was sentenced for rape and served 18 years in prison.  He was subsequently exculpated on DNA evidence and released from prison.  The other defendant was his nephew Brendan Dassey.  The central theme of the series is whether or not law enforcement including the District Attorney had made up their minds that the defendants were guilty and did everything they could to arrive at that verdict.  We see for example what is essentially a coerced confession from Dassey and one that he later withdraws.  A key witness in that case recants her testimony during cross examination and says that she made up her previous statement based on television broadcasts.  The documentary presents law enforcement, prosecuting attorneys, and judges making a long series of assumptions and decisions that all seem squarely focused on putting the defendants away for life without the chance of release in Avery's case and for at least 42 years in the case of Dassey.  The defense attorneys do an excellent job of illustrating the bias involved in this case and emphasizing how the legal system is designed to function on a theoretical level and how it really functions in Manitowoc and Calumet counties in Wisconsin.  In episode 9, Dean Strang makes the above statement about certitude.

As a psychiatrist I have had a lot of first-hand contact with the legal system through civil commitment hearings, hearings involving forced medication and electroconvulsive therapy, guardianship hearings, competency hearings, and as an expert witness in malpractice cases.  I also testified as an expert in a criminal proceeding involving homicide and a diminished competency defense.  All together I have testified in hundreds and possibly thousands of these hearings.  In the vast majority of those hearings I was not compensated and it was seen as just part of my job. The lack of compensation is a fairly good basis for humility in court cases.  In many of the cases involving a certain legal threshold, I refused to make assumptions and asked that the attorneys ask me the specific question involving a legal standard.  In some of these cases, it seemed like a moving target and there always seemed to be pressure to say things that I did not really want to say.

In some of these cases, I was reprimanded by the judge for stating clear facts.  I remember one case involving a dead animal where the judge interrupted me and told me that under no circumstances was I to mention that dead animal again.  In this case, it was central to the process and there was no jury hearing the case only the judge.  To this day. I don't know why the judge told me to eliminate it from my testimony.  In another case, I was asked a question about medication side effects that were clearly listed on court paperwork and the judge reprimanded me for "reading off the boilerplate".  I did not have the form in my hands and was testifying extemporaneously.  The entire court proceeding in most of these hearings is typed up on court documents that are "boiler plate."  The medication side effects form was no different from all of the rest.

I am used to fielding the questions from attorneys that start with "Doctor - isn't it true......" knowing that those statements are seldom true.  I know that when the attorney starts to address me as "Mister" rather than "Doctor" - he or she is usually trying to annoy me and I am not annoyed.  I have watched the same courts swing from a permissive state where everyone gets committed to a situation where there are hotly contested hearings and people with serious problems are released.  That dynamic is usually changes due to a catastrophic outcome when a person has been released by the court.  The system I have dealt with in my career has been highly idiosyncratic but necessary.  There is no way that vulnerable people with severe mental illness could get treated without it.  The big difference is that there is usually no bad guy.  At its worst, something bad has happened or is about to happen as a result of mental illness or addiction.  

Certitude is a word I have not encountered in a long time.  A standard definition is: The state of being certain or convinced of something; complete assurance; confidence.  There are few things in medicine that can be stated with certitude.  Certitude may also be a discerning point between physicians and many other groups.  If you are clueless enough to start out in the field with that general attitude and continue to maintain that position - you will be beaten down in a hurry.  There is simply no way that you can be certain with a high degree of confidence of diagnoses, treatments or outcomes in any field of medicine.  Even if the diagnosis is 100% correct, individual response to treatment can make outcomes unpredictable.  A physician may find that their patient is unable to tolerate any of the top three recommended treatment modalities and they are forced to not treat the problem or use a tertiary sub optimal solution.  There is far too much biological variability built into the system.

Dean Strang's comment on certitude is an indictment of the legal system that also applies to the managers of the health care system, the politicians that manipulate it, and most certainly the vast majority of the so-called critics of psychiatry and medicine.  The physicians are doing what they can to deal with a high degree of uncertainty and the bureaucrats are pretending that everything can be neatly measured and that outcomes are highly predictable.  They are also pretending that good work can be done in ridiculously brief encounters.  Politicians have come up with a long series of non-solutions to health care.  The end result is a system of large companies that are essentially guaranteed large profits by political mandates and statutes.  They have the power to order physicians to do low quality work and generally waste their time.  Health care costs are probably the second largest tax on Americans and many Americans are paying more than the per capita health care expenditure in the US - by far the largest expenditure of any country in the world.

Psychiatry is a special case when it comes to certitude.  There are a large number of prominent for profit critics that are certain about all of the negatives in psychiatry.  They mock psychiatrists, presume to tell us what is wrong with the field, and don't hesitate to come up with poorly thought out moral arguments against the field.  There is a slightly less aggressive group that act like we are not there when they implement changes in systems design consistent with the general trend of rationing more services to people with mental illnesses.  Their common bond is a lack of scholarship.  Despite that lack of scholarship and clinical experience - they all act with certitude.

I thank Dean Strang for another important dimension to use in analyzing these dynamics.  If you happen to see this series - you will realize how important this idea is.


George Dawson, MD, DFAPA

The CMS Investigation Of Anoka Metro Regional Treatment Center

In a previous post I discussed a recent local news article that pointed out the increase in incidents of aggression at one of the state's major psychiatric facilities and a threatened loss of Medicare funding unless certain deficiencies were corrected.  The deficiencies were determined by an investigation of the facility by the Centers for Medicare & Medicaid Services (CMS).  No specifies from the report were available from the news article or the Minnesota Department of Human Services.  They did provide me with a contact person at CMS and after another forwarded e-mail, I was sent 4 attachments detailing the results of the investigation.   I will report on those reports in this post.  The documents were all typed on a standard government form as noted in the graphic below.  The entire CMS report is written in the column labelled "Summary Statement of deficiencies...".  No comments were written in the column labelled "Provider's Plan of Correction...":

I have coded them AMRTC 1-4 for convenience and will refer to them that way in the summaries below.

AMRTC-1 is a 34 page document that states the visits was done to see if the hospital was in compliance with 42 CFR Part 482 for acute care hospitals.  The survey was conducted from 10/19 to 10/23/2015.  The report indicates that there is a 108 patient capacity at the facility and that 30 records were reviewed as the basis for the report.  Problems were found in 2/30 cases with regard to patient care.  There were additional administrative problems that also resulted in noncompliance with the federal standard.  There were problems noted  It was determined that the hospital was not in compliance with the Conditions of Participation of 42 CFR Part 482.  The main finding of the first report is that The Governing Body of the hospital failed to ensure that services provided by staff or contracted staff were proved in a safe and effective manner.  The highlighted areas include failure to assure that quality processes were in pace to minimize or prevent medical errors, failure to assure that comprehensive nursing plans were developed, and a patient's rights condition that occurred when a patient was given forced medications that were prohibited by a court order.

The Quality Assessment Performance Improvement (QAPI) programs extended across a number of clinical and nonclinical disciplines.  In some cases,  they involved the administration not doing what they stated they would do in their descriptions of quality improvement.  The best example I can think of is the reference to Six Sigma.  I have always found it a questionable practice to apply engineering management processes to any medical field.  I sat through a presentation of this paradigm in a previous job and it just seemed like the standard management buzzwords that we hear in different iterations by people who think they are inventing management every 5-10 years of so.  At that job we suffered through a couple of presentations and printed Powerpoints and it faded as soon as it came up.  We moved on to a different paradigm.  Since it was widely promoted, the Six Sigma approach has been shown to not be uniformly effective in business and manufacturing models.  What the proponents of Six Sigma to medical fields don't seem to understand is that measurement is a limiting factor and it has nowhere near the precision or accuracy of measuring products in electronics or automobiles.  At the philosophical level the administration probably made the common error of espousing a philosophy that they could not live up to.  I am not aware of any major healthcare corporation that uses the Six Sigma management model and they probably have many more resources than a state hospital. 

One of the case examples cited was an agitated patient who was physically aggressive and received olanzapine and then intramuscular haloperidol despite a court order excluding haloperidol and risperidone.  The psychiatrist and nurse involved were questioned and said they were unaware of the order at the time the medication was administered.  The patient got this medication for a period of 3 days before it was discontinued.  CMS investigators comment how the physician in this case could be held in contempt of court for ignoring a District Court judge's order.  There was a question of whether or not there were two different orders and the one barring the medications showed up later.  As a physician who has worked with different court orders in these cases for over 20 years, I can attest to the fact that they are not necessarily clear.  In many cases there is a temporary order until the final document can be typed up.  It would seem that the quality process here would be to appoint a person to make sure the latest order is in the chart and read by the attending physician before any medication orders are written.  There is also a question of how paper documents from the court are placed in an electronic record and how easily they can be read in that record.

At the end of the document problems with the care of 10 different patients with different diagnoses and problems are reviewed.   These clinical examples were given to illustrate that that patient with varied problems were all given treatment plans that were not comprehensive, even in the case of patients with aggressive or self injurious behavior.  The reports describes this as: 

"Interventions on the Patient Treatment Plan were generic and were normal functions of the professional disciplines involved in the patient's care and were not individualized to the patient."  

What does all of this mean?  A recent article in the StarTribune (1) had quotes from several mental health experts and advocates about the state of affairs at AMRTC.  The commentary seemed to vary in the level of outrage expressed as "egregious" and "appalling" and "no excuse."  As an expert - when I read the report it seems to scratch the surface.  Would correcting the deficiencies in the report right the ship out at AMRTC?  Possibly - but the previous news report suggests there is a much bigger problem.  That report was about incidents  of aggression, how they were increasing, and there was an opinion that aggressive inmates transferred based on new legislation was the main reason.  A union representative was quoted as saying that some of the inmates transferred from correctional facilities had "taken over" and that they were more aggressive than non-correctional patients.  None of those problems are specifically addressed in the report.  The report comments on problems in the care of specific individuals, only one of whom seem to be as aggressive as two of the patients mentioned in the original article (2).  The errors in the report may be largely documentation and reading errors, but administrators always emphasize "if it isn't documented it did not happen."  Some of the problems at AMRTC have been decades in the making.

For a long time the message given to most professionals in the state is that the state hospital system including AMRTC (like practically all other hospitals in the state system) was going to be shut down. Only the practical fact that there is always a backlog of committed patients waiting to get in to AMRTC prevents it from being shut down.  But the key question remains - is this really the attitude of managers at the level of the State of Minnesota?

The second problematic attitude that I have heard about constantly is written about in the recent article (1):

"Nearly half of the 101 patients currently there no longer meet the hospital-level criteria for care but are kept at the hospital because they have nowhere to go in the community. In 2013 alone, patients spent a total of 13,800 unnecessary days at Anoka-Metro after they were treated — enough to care for another 140 patients, according to a state legislative report."

This is a good example of circular reasoning.  The reason why patients spend so-called "unnecessary days" at AMRTC is that there are no other facilities that can manage their behavior.  I am aware of programs where very aggressive individuals are managed in very small settings (2 to 4 resident group homes) and the staff is taught to physically restrain them when they become very aggressive.  That is really an unacceptable long term solution to the problem for many reasons.  It is time to stop pretending that long term hospitals are acute care hospitals and that they should be managed like acute care community hospitals.  A transient reduction in symptoms does not mean that a patient at AMRTC is spending "unnecessary days" at the hospital.  If they cannot successfully transition to a community placement - they probably need to be there.

The real and unaddressed issues (beyond the CMS report):

1. The effect of the message that state hospitals should all be closed: As a psychiatrist in the state, this is what I have been hearing for a long time. It is really not possible to develop a quality of care focus or have the necessary stable staffing patterns of experienced staff, when those same staff are hearing that the state is trying to close down the facility and that many people at the facility don't need to be there. Instead - the facility should be managed as one that can provide state-of-the-art care to patients with complex problems including violence and aggression. Another aspect of that is eliminating the positions of experienced staff to save money. You will never have a high quality program using this approach and yet the state has used this approach.

2. The effect of management from higher levels: This seemed to stand out as I read the issue of "generic treatment plans" from the CMS report. At some level all treatment plans become "generic treatment plan". The evidence is that you can purchase treatment planning texts for nursing, psychotherapy and to a lesser degree psychiatry that will show you generic treatment plans for an entire list of problems. Is the problem really a generic treatment plan that covers most interaction or the lack of a treatment plan that addresses a high degree of aggression? I would contend that it is the latter.

Complicating that issue are previous stories about how plans were implemented by state administrators with no psychiatric experience to address patient aggression. I sat in on one of these sessions that suggested that a focus on the aggressive person as a psychologically traumatized individual was the best way to proceed, but not much specifics after that. Is at least part of the problem that state hospital staff have inadequate guidance on what to do about aggression? Are they reluctant to intervene early or clearly document what happened and their response because the response from administrators is inconsistent? Are they being advised to use interventions that are ineffective?

3. The lack of teamwork and possibly a split staff: One of the most dangerous problems in any inpatient psychiatric environment is staff splitting - some of the staff are praised and well liked and other are criticized and disliked. This emotional environment in inpatient care leads to problems in patient care. Splitting needs to be minimized or eliminated largely by recognizing that professionalism and the objective analysis and treatment of problems is the real priority. I have been in treatment environments where staff were disliked or falsely accused and that lead to major problems in patient care and episodes of aggression. It also leads to staff turnover.  The attitude of administrators can be particularly insidious and create an immediate rift among the staff.

4. The influx of inmates into AMRTC that is caused by the current public policy of rationing community psychiatric care and the resulting shift in the cost of care to the correctional system: Instead of addressing the widespread problem of rationing psychiatric care for the severely mentally ill - the solution is currently to dump at least some of them from law enforcement facilities to a rationed long term care facility. How is that a solution to anything?

These are the real problems at AMRTC and within the state system as far as I can tell. This is all based on what I read in the papers, the CMS report, and my extensive inpatient and out patient experience as well as experience treating aggressive people. The CMS report while noting significant problems does not come close to addressing these issues and makes it seem that addressing problems in patient care or documentation will correct the problem with aggression within this system.

I doubt it is that easy.



George Dawson, MD, DFAPA


1:  Chris Serres.  Anoka state mental hospital violated basic rules for patient care, feds say generic treatment plans, other issues put mental hospital's federal funding at risk. StarTribune January 16, 2016.

2: Chris Serres. State psychiatric hospital in Anoka threatened with loss of federal funding. Minneapolis StarTribune January 4, 2016.

Guns In Psychiatric Hospitals - Texas Has The Worst Possible Idea

Almost on cue, USA Today came out with a story at about the same time that I responded to a post about secure environments in psychiatric hospitals.  My response provided a specific reason why these places need to be a firearm free environment and why armed peace officers sitting at bedside or in the hallway are not really more of a deterrent to criminals with goal directed aggressive behavior or patients with mental illness who have aggressive behavior.  My personal experience with firearms in psychiatric settings is fairly extensive.  It varies from visiting a primary care physician in his office early in my career and being shown a closet full of firearms turned into him to working in settings where mental health professionals or law enforcement professionals were killed by the use of a firearm.  It has occurred in both inpatient settings and outpatient clinics.  Even without firearms I have worked on inpatient units with highly aggressive individuals that on several occasions basically rioted and took control of the hospital unit until enough law enforcement staff came on the scene to restore order.  In one situation an entire unit was disrupted by one individual and law enforcement had to be called.  Against this backdrop, I was more than a little puzzled by new legislation in the state of Texas that allows visitors to carry weapons on units in Texas psychiatric hospitals.  

The USA Today article states that although staff and patients are not allowed to have weapons, visitors are now allowed to openly carry firearms.  Signs suggesting that these weapons need to be left in cars or concealed needed to be removed.  A hospital spokesman quoted in the article makes the understatement of the year by saying that it is generally not a good idea to expose hospitalized patients to weapons of any kind.  Even police officers entering these hospitals do not carry in weapons, probably because it is standard police protocol to not carry weapons in an environment where there are large numbers of potentially aggressive people with impaired judgement in close proximity.  In my previous post, I also point out that firearms are not a deterrent to people who are aggressive and have severe impairments in judgment or see them as a means to escape or perpetrate violence. Law enforcement officers involved are also not able to maintain a high enough level of vigilance to prevent an unexpected attack. A hospital environment is not generally a very stimulating environment. There may be a significant amount of background noise, but there are not a lot of events that require focused attention - like very low frequency aggressive events.

The best protection against these events are physical barriers to protect people from the aggressive person and maintaining a therapeutic environment with multiple interventions to reduce violence. The barriers include jail cell units where incarcerated patients who need acute medical treatment can be transferred to and entire 18-20 bed units that specialize in treating aggressive men. In the case of open units, staff must be available and out there with the patients to provide therapeutic interactions and also frequent assessments. In this era of the electronic health record, it is common to see people sitting in unit offices charting on computers all day long. That is not an approach that optimizes the therapeutic environment. The units themselves have to be staffed with people who are comfortable dealing with aggression and who know how to address it. The environment has to be secured against contraband weapons and drugs and all material coming into the unit needs to be searched. Metal detectors are also employed to detect any weapons coming into the unit.  I have also witnessed incidents where visitors have become physically aggressive and threatening to staff.  One of the logical flaws of gun advocates is that anyone who is licensed to carry a firearm always acts in a rational manner.  You don't have to be a psychiatrist to see that as an unrealistic statement.

The real problem in visitors carrying weapons into a psychiatric facility is the potential adverse impact on individual patients who are being treated there.  To cite a few examples:

1.  Patients with a history of trauma and in some cases post traumatic stress disorder.   These patients are hypervigilant and scanning the environment for the slightest hint of danger.  What would appear more dangerous than a person walking in with a gun?

2.  Patients who constantly expect to be harmed or killed - paranoid patients.  During inpatient work it is common to have many people with this problem.

3.  Suicidal patients who may have those thoughts under fairly good control until these see a highly lethal method within arms reach.

4.  Aggressive patients who may have been involved with weapons prior to admission and immediately gravitate towards anyone carrying a weapon.

There are more examples, but in our society guns are powerful symbols.  Any powerful symbols tend to be amplified in many predictable and unpredictable directions by psychopathology.   The other unappreciated fact is that there is a psychological environment in any hospital setting.  That environment is the conscious and unconscious product of every staff person, patient and visitor in that facility.  Unless that environment is actively managed for safety and affiliative rather than confrontive interpersonal communications there is the potential for major problems.   Carrying firearms into a psychiatric facility is more than a bad idea.  It is an inexcusable use of a psychiatric facility for political purposes at the cost of a therapeutic environment.

In order to get more details about this legislation and the positions of Texas psychiatric organizations I sent an e-mail to the Texas Psychiatric Federation - a website that lists Texas Society of Psychiatric Physicians, the Texas Academy of Psychiatry, and the Texas Society of Child and Adolescent Psychiatry as the major professional organizations in the state.  I am interested in getting feedback on the positions that these organizations are taking as well as the position of the American Psychiatric Association.  I delayed posting this for a few days but so far have not received a reply.  I will post information in the comments section as it becomes available.

Every psychiatric professional organization and every psychiatrist should know what is wrong with this picture and demand safe and therapeutic hospital environments for our patients.
.


George Dawson, MD, DFAPA


References:

Rick Jervis.  Texas allows guns into state mental health hospitals.  USA Today August 8, 2016.
http://www.usatoday.com/story/news/2016/01/08/texas-open-carry-psychiatric-hospital/78522138/


Attributions:

The graphic at the top is downloaded from Shutterstock via their Standard License Agreement and is copyrighted by Bob Orsillo.