Saturday, October 4, 2014

Life Is Not A Randomized Double Blind Controlled Clinical Trial

Or  what is wrong with placebo controlled clinical trials and transparency?

I was in an imaginary meeting with a bunch of Internists and Psychiatrists.  We were debating the available and meager literature on the use of trazodone for sleep.  We got into one of my favorite topics - the double blind placebo controlled trial and that catch all "Evidence based medicine."  The dialogue went something like this:

Internist:  "The evidence from double blind placebo controlled trials is weak for trazodone....."

Me:  "Do I need a double blind placebo controlled trial to tell me to prescribe trazodone for sleep?"

Internist: "Are you saying you don't need evidence.  Oh wait, I was at a conference where the head of the society stood up and and gave evidence that clinical trials have their limitations.  Like they are subsequently proven to not be true...."

Me:  "That is not what I am talking about.  I am talking about all of the hype out there that only 40% of people recover from antidepressants or that they are no better than placebo........"

Internist: "But you do have the algorithmic approaches that show...."

Me:  "Yada, yada, yada - for every algorithm, there is somebody with a meta-analysis to show it is wrong.  No I am talking about the ridiculous notion that psychiatrists could stay in business or want to stay in business if only 40% of the people they treat got better.  If that was true I would have become a recluse 20 years ago and just walked away."

General laughter

Internist: "I would kill for a 40% response rate for some of the problems that I treat."

And so on........

This informal conversation among colleagues is illustrative of the recent arguments that focus on physicians and generally psychiatrists more than other physicians that treatments are ineffective.  They are based on an oversimplified view of placebo controlled clinical trials and conflict of interest.  Many times there are no clear points of demarcation between what is a scientific issue (the technical aspects of the clinical trial) and the ethical issue (conflict of interest issues that may compromise the scientific results).   You can read all about current technical problems with clinical trials, like the common observation that there is a lack of generalizability to clinical populations compared with the highly selected and trial sample that in the case of psychiatry generally has much milder forms of the illness being studied.  Here are a few of the concerns that I don't see being discussed anywhere, especially when it comes to clinical trials of psychiatric interventions:

1.  The crude state of clinical trials:  Clinical trials in psychiatry are tremendously unsophisticated. The trial result generally depends on rating scale or clinician global rating scale results that grossly oversimplify the condition and measure parameters that are irrelevant in clinical settings.  The best example I can think of is depression rating scales that list DSM criteria for depression and then apply a Likert dimension to those symptoms.  In clinical practice it is common to see hundreds of patients with the same score on this scale who have a full spectrum of disability from absolutely none to totally disabled.  Which population might be more likely to exhibit an antidepressant effect?  It is also common to see medically ill patients with insomnia who may score as mildly to moderately depressed who are physically ill and not depressed at all.   The same problems exist with clinical trials of schizophrenia, anxiety and Alzheimer's disease.   When this weak technology is combined with a selection process that eliminates clinical populations with the most severe illness, it should not be surprising that any treatment being studied has a weak effect.

2.  A weak clinical trials data base:  One of the clarion calls of so-called evidence based medicine is the Cochrane Collaboration.  I have looked up hundreds of their reviews and the majority of those reviews of both medical and psychiatric studies is: "insufficient results and methodologically unsound".  I suppose it is good to have somebody make that statement, but if you have proclaimed that you are an evidence based physician - you have nothing to go on.  In fact, at some point you stop going to Cochrane because the results are fairly predictable.   Even in the case where you have a result does it apply to the patient you are seeing?  I don't see many Cochrane studies depressed patients that have right bundle branch block, ventricular premature contractions, or complex atrial fibrillation - all common patients seen in clinical practice.  How many more research papers do I have to read that conclude that "more research is needed" while continued inadequate trials are being published and analyzed by Cochrane?  From the current trends and political correctness of evidence based medicine this will go on forever.  The practical aspects are the very large costs of trials.  That is the real reason that pharmaceutical companies (Big Pharma) sponsor these trials.  The political system in the US has decided to farm out clinical trials to private for-profit companies in the exact manner that the US government has farmed out management of the entire health care system.  In these systems Big Pharma absorbs a disproportionate amount of criticism relative to managed care companies.  Managed care has equated "medically necessary" care with care that can be provided in the form of a pill.

3.  The politicalization of clinical trials:  No clinical trials of psychiatric medications can be done these days without an eye to the politically relevant dimensions.  A new antidepressant needs to get as many FDA approved indications as possible in order to beat the political restrictions of utilization review by managed care companies.   That would include indications not only for depression, anxiety, panic, and social anxiety but also chronic pain and possibly attention deficit-hyperactivity disorder (ADHD).  The best way to beat utilization review is to have the only FDA approved indication in the class.  That is also applied to atypical antipsychotics and that fact seems to escape the critics who focus on the number of prescriptions and what that implies.  Physicians are pawns in a game when there are no suitable tolerated medications for bipolar disorder depression and there are atypical antipsychotics for that indication.  These current political factors in clinical trials preclude a focus on cognition, functional capacity, and endophenotypes - all potentially much more valuable than a focus on diagnostic criteria or rating scales based on those criteria.

4.  A characterization that the average physician is ignorant - at best:  Any political movement is associated with ugly rhetoric.   There has clearly been an effort to stampede any physicians with reservations about evidence based medicine into a Neanderthal category.  The irony is that the criticism often comes from sources like managed care companies, medical certification authorities and the press (bloggers) - all entities with their own high levels of conflict of interest.  Common rhetoric used against psychiatrists has been: "You just don't want to be measured".  If the criticism originates from a government, managed care company or administrative authority there is often the attached explicit threat: "Those  days are over!".  The obvious implicit observation is that medicine and psychiatry is now being run by people who don't know anything at all about medicine and there is plenty of evidence on this blog to back that up.

Many critics seem to get a lot of mileage out of the position that they seem to be particularly anointed to criticize the field.   That seems especially true if they happen to be a physician as in: "My colleagues certainly seem to be a bunch of chumps and therfore can be rejected on a wholesale basis or of course listen to me for enlightenment".  I have never witnessed that level of incompetence in any group of physicians where I have practiced across multiple settings.

5.  The use of statistics for rhetorical purposes:  At this stage after reading research for that past 35 years, I am convinced that you can come up with a meta-analysis that will show whatever you want it to show.  Several years ago in the New England Journal of Medicine there was a study that looked at how well meta-analyses predict the results of available large scale clinical trials.  That study showed 2/3 times.  It is common to see a result and then see a meta-analysis that "disproves" the clinical trials result.  Nobody seems very interested in looking in detail at how sound that meta-analysis was.

Today we have a large number of questionably valuable clinical indicators or quality care markers that are often based on the political rhetoric of the government and managed care organizations.   They may be invalid on the face of it, but there is a second equally important aspect.   These same organizations have no valid approach to looking at the statistics of longitudinal data.  They will look at clinic results or even results from the same physician and convert them all to a numerator and a denominator.  Whether that fraction means anything or not is anybody's guess, but there is an administrator somewhere who will be happy to tell you all about it.

6.  False assumptions about the expertise of physicians:  Much of the rhetoric above is focused on physicians.   Psychiatrists as the most politically disenfranchised group are a frequent target.  In the past years we have endured the ideas that medical treatments being prescribed are ineffective and overprescribed.  That brings us full circle to the opening imaginary conversation.  Physicians are trained to focus on several goals including acute treatment, treatment of chronic problems, and providing care for the dying.  The psychiatrists and physicians that I have had the pleasure of working with have been highly effective is achieving those goals.  When I look at the modest results of poorly designed clinical trials all I can do is shake my head.  I would have quit a long time ago if my diagnostic or treatment capabilities were accurately described in clinical trials and most physicians would have.  Informed clinical treatment is a series of often rapid changes in course, rejecting poorly tolerated treatments and looking for things that work better along the way.  I can change the course of treatment depicted in a clinical trial in one day.  In the trial that result is carried to the end as a failure.  How is it that a clinical trial or this evidence would predict my treatment results by mean?  If a treatment is ineffective or not tolerated in my practice, I don't stop treatment and call that person an unsuccessful intent-to-treat subject.  I work with them to establish effective treatment - often in the same time frame as a clinical trial.  Is it quite literally absurd to suggest that clinical trials accurately describe what will happen in clinical practice and yet that is the state of discourse.

7.  The false promise of transparency:  Anyone who followed politics knows that disclosing potential conflicts of interest  is meaningless in the case of politicians.  The general idea is that politicians would refrain from either accepting money from sources where they are involved politically or just not be involved in that area of legislation.  In reality that does not happen.  Sometimes the politicians involved will speak out against any suggestion that there is a connection between the money they receive and their legislative record even when their activities are consistent with a financial conflict of interest.  The sunlight of transparency that many of the critics talk about simply legitimizes ongoing involvement in areas of potential conflict of interest.   Disclosure is a stamp of approval for involvement.  All of that can be researched on Congressional watchdog sites.   The new CMS web site that posts payments to doctors is hyped as a way to appease all of the critics who seem clueless about transparency.  For an eye opener take a look at their decision point on conflict of interest.  CMS seems much more charitable than the typical blogger, politician or journalist with this disclaimer:

"Sharing information about financial relationships alone is not enough to decide whether they’re beneficial or improper.  Just because there are financial ties doesn’t mean that anyone is doing anything wrong.  Transparency will shed light on the nature and extent of these financial relationships and will hopefully discourage the development of inappropriate relationships.  Given the complexity of disclosure and the importance of discouraging inappropriate relationships without harming beneficial ones, CMS has worked closely with stakeholders to better understand the current scope of the interactions between physicians, teaching hospitals, and industry manufacturers."

8.  A lack of appreciation of the regulatory environment:  In the rush to condemn Big Pharma and anyone associated with them, critics often have an idealized version of regulation in their minds.  If only they had access to all of the clinical trials data to analyze for themselves.  They would personally be able to hold Big Pharma's feet to the fire and only allow drugs to be approved that they deemed to be safe and effective.   They are smarter and more ethical than anyone doing the actual trials and certainly smarter than the regulators.  This is at the minimum a failure to recognize that pharmaceutical regulation is after all a political process.  Like all politically directed regulation there are broad goals of safety and efficacy but they are relative and the regulatory mandate takes that into account.  On this blog I have pointed out several cases of medications that not only I,  but the Scientific Committee employed by the FDA recommended against based on safety considerations, but that were approved by the FDA.   Many drugs that I approved as a member of a Pharmacy and Therapeutics Committee (P&T) were expensive but had minimal efficacy.  We approved it based on political considerations (small but vocal advocacy group and untreatable illness) rather than pure efficacy or safety considerations.  The regulatory environment is currently designed to get promising agents into the hands of clinicians for clinical use.  The limited exposure of patients in clinical trials means that rare but serious side effects will not be recognized until post marketing surveillance occurs.  Every person taking an FDA approved medication should realize that.  The regulatory process is not designed to provide perfect medications because there are none.

There is a lot more to say about this subject like a detailed analysis of how the politicalized version about how physicians work and think has nothing to do with the way physicians actually work and think.  But for today I am stopping here.

Life is not a randomized double blind controlled clinical trial and that is generally a good thing.

George Dawson, MD, DFAPA


  1. Seems issues arise not just with medications. Somewhat old article, but pertinent.

    Also, what do you think of the new "blood test" for depression?

    It seems like too much hype before an actual product. Similar to what they did with ketamine. So, I am very skeptical of it.

  2. Hype is all part of the American culture and it extends broadly to science and not just medicine. As I have posted here before, in 35 years of reading Science and Nature regularly - two of the most highly respected journals in science I have read thousands of breakthrough papers that never really panned out. Very few do and hence Ioaniddes is correct again.

    You can find for example, a paper in Science that proclaims to make psychiatric diagnoses by a computerized algorithm. That was prematurely sold as a device that made the same claim. After hundreds (thousands?) of these devices were sold it soon became apparent that they did not come close to living up to that hype. But this all was based on a paper in Science - one of the most prestigious scientific journals.

    Anyone who thinks that there is no hype, not just about medications and devices but also theories - just has not been around the block.

    If someone tells me they have a test for depression - I would want to know what kind of depression they are talking about. If they told me it correlated with a PHQ-9 score I would tell them to take a hike.

  3. I'm much more interested in data transparency than COI transparency. I think the raw data should be published and I think negative trials must be published. I think we can all agree that that is reasonable.

    Drug approval process is too expensive and time consuming. I would love to see phase 3 trials eliminated (the fact that 20 percent of meds are prescribed off label shows how useless they are) in exchange for data transparency reforms.

    I agree with you that RCTs need to be done only on sicker patients and not on outliers who are really just having self limited conditions. As I had opined at 1BOM, archangels could be in charge of current clinical trials as they are currently designed and they would still be flawed. That's why older TCA MAOI studies were more robust...they were done with depressed inpatients with more sx. PHQ-9 as a research measure is crazy, sleep disturbance and anhedonia both count as "one". These flaws probably account for 7% increases per decade in the placebo response rate.

  4. What do you think of the Montgomery-Asberg Depression Rating Scale, and its use during clinical trials?

    1. Same song different verse.

      The MADRS differs in that it is a clinical interview that includes some observational data. I am not aware of any study that looks at that issue, but in clinical practice it is significant. The original version did not specify a time frame for rating the symptoms but I notice online versions specify one week. Additional criticisms include that there is an overemphasis on cognitive symptoms and there was no original manual or instruction set but a scoring guide and interview guide seems to be available after the fact. It is apparently copyrighted by the British Journal of Psychiatry.

      In short, it seems to have the usual deficiencies I have alluded to in the discussion of the technical difficulties of clinical trials. It is quite literally to the point that I could write up my own rating scale - let's call it George's Rating Depression Scale of the GRDS. I could easily show that it correlated with the Ham-D, Beck, MADRS, etc.

      And voila - we have another paper and pencil descriptor of DSM criteria for depression. The only DSM feature that is missing in all of this is Criterion for Clinical Significance (p 21). That is the dimensions of whether or not the symptoms that are acknowledged by the patient actually affect their ability to function or are being endorsed for some other reason.

      As long as we use these scales or ones that anyone else can invent (even myself) we have to acknowledge we are using a severely limited technology.

    2. What about a good old fashioned MMPI-2 2 depression scale? Another advantage is it also screens out fake good and fake bad.

    3. Some of the items are used in the CES-D. I notice that when psychologists use it they sometimes also give the BDI. I am not aware of why that is necessary. Also used in some cases to validate a depression scale. No matter how "good" it is - it is just another permutation of DSM symptoms. In the days when I did some psychopharm research "severe" personality disordered patients were screened out. This is also a problem comparing one clinical trial of depression to another and certainly another reason for drug nonresponse. I am not aware of any studies that looked at the MMPI in comparing responders versus non-responders. It does seem like it is being used less as a screen for general psychopathology. As an example, some professional schools that used to require it no longer do.

  5. BTW, the problems in psychiatry are not limited to the field and are part of a broader trend of fraud and bad methodology in science in general:

    Academia has really been going to hell in the last twenty years. Too many universities, too little talent.

    1. This comment has been removed by the author.

    2. Academia going to hell has very little to do with too many universities or too little talent. The degradation of academic departments seems like the direct result of mismanagement on the business side to me. The idea that various inpatient speciality services that were prime training grounds for academics are displaced by hospitalists working seven days shifts and admitting and discharging patients as fast as they can according to some poorly conceived business plan is one thing. There could be an argument that it frees up the more academic clinicians to do research. In fact, the professors need to "support" their positions with productivity based positions - sometimes out in the hinterlands. A productivity RVU based system is no better for academics that it is for clinicians. It stifles intellectual pursuits and creativity in both and really creates a huge deficit in quality teaching for medical students, residents, and fellow.

      Just to provide an example, I wanted to use deidentified brain images (CT and MRI) to develop a teaching file for my annual Neurology Boards review for residents. I was told I needed permission from an administrator to do that. When I asked that administrator about it the reply I got was:

      "Why would we want to let YOU use our images?"

      How about because its the way it has been done for about the last 50 years? A far cry from my Neurosurgery rotations where we would spend 3 hours every Saturday morning reviewing films with the Chief of Neurosurgery. Managed care (the dominant administrative paradigm) views academic medicine as a nuisance. They are waiting for the day when physicians will just fall into lock step with whatever their case managers tell them to do.

      Contrary to the ideas about Big Pharma and the associated corruption, Big Managed Care is the real problem and they seem to have gotten a pass from all of the anti-Pharma blogs out there.

  6. Well both are a problem, and I don't think the KOL corruption issue can be swept under the rug. Some of the commenters of those blogs are definitely antipsychiatry or simply full of themselves (like the hypocrites hiding behind an alias demanding full disclosure). And I don't like reflexive pharama bashing. On the other hand the correct response to COI influence tainting research is not that it's OK because politicians and investment banks also have COI. The correct response is that politicians and investment banks need to be accountable for COI as well. I mean if the ethical standard for psychiatric research is not higher than modern politicians, it's all just a big waste.

    Not that I disagree with any of your points about the dumbing down of academia. But I'm not sympathetic on the issue of RVUs when they saddled us with MOC. And I was speaking of academia in general with my comment about too many universities and too little talent, not just psychiatry. At some point when colleges become like Starbucks, there's going to be a dilution, just like there is when sports leagues expand too much.

  7. Take-down of the blood test for depression here:

    Debate continues in the comments there too.