Showing posts with label teachers. Show all posts
Showing posts with label teachers. Show all posts

Sunday, October 9, 2016

Big Data - What Is It Good For?




Big Data and Data Science have been buzzwords in science and industry for over a decade.  A Medline search shows over a thousand current references to Big Data in healthcare.  A good starting point is consider what is meant by Big Data and then discuss the implications.  A quick scan of the references shows that they vary greatly in technical complexity.  A standard definition from Google is: "extremely large data sets that may be analyzed computationally to reveal patterns, trends, and associations, especially relating to human behavior and interactions."  These techniques developed because of the widespread availability of digitized data and the ease with which sets of behavioral choices (in the form of mouse clicks) can be collected on web sites.  In many cases specific data collection paradigms can be used to elicit the information, but there is also a wealth of static data out there as well.  In health care, any electronic health record is a massive source of static data. Financial, real estate, educational records and records of all of these transactions are also a significant source.

Most  Americans logging in to set up a Social Security account online (ssa.gov)  in the past couple of years would be surprised at what it takes to complete the job.  After the preliminary information there is a set of 5 security questions.  Four of those questions are about your detailed personal credit history - home mortgage information and credit card history.  When Social Security was initially set up in 1936 there was widespread concern that Social Security Numbers would become national identifiers.  At one point Congress had to assure the electorate that the number would not be used for that purpose.  Since then the SSN has been used for multiple identification purposes including credit reporting.  At this point it seems that we have come full circle.  Congress invented the SSN and told people it would not be an identifier.  They mandated its use as an identifier. Congress authorized and basically invented the credit reporting system in the United States.  The federal government currently uses the credit reporting system to quiz taxpayers wanting to set up a Social Security account online.  In the meantime, large amounts of financial, legal, and health care data are being collected about you under your SSN in data systems everywhere.  At this point the full amount of that data and the reasons why it is being collected for any person in the US is unknown because it is all collected without your knowledge or your consent.  It is impossible to "opt out" from this data collection.  The federal government does have an initiative to remove SSNs from health records, but there are so many other identifiers out there right now, this effort is too little and too late.

Additional sources of data include your online foot print including sites that you may have visited and what you seem to be interested in.  A visit to Amazon for example and a quick look at an expensive digital camera may result in that same camera with a link to Amazon in the margin of very other web page you see for the next two weeks.  Expensive digital cameras of a different brand than the one you originally looked at may start showing up.  You may notice product ads showing up in your Facebook feed that you mentioned casually to your friends during a conversation there.  The conversation could be as generic as bicycle seats and suddenly you are seeing a flurry of ads for bicycle seats.  Any number of web sites encourage to sign in with other accounts and then share your account information with them.  All of this data provides companies with what they need to fuel their predictive algorithms to sell you a product.  It provides the major advertisers in this space like Google and Facebook with a huge revenue source because based on the scale and personalization of these ads - they are effective.  Big Data seems to be very good for business.  But is there a downside?        

That brings me to a current resource on the nefarious uses of Big Data written by an expert in the field.  Cathy O'Neil is a PhD in mathematics.  Her PhD work was in algebraic number theory.  She started work as an academician but subsequently worked for a hedge fund, work as a data scientist for several firms and currently heads the Lede Program in Data Journalism at Columbia University.  I am familiar with her work through her blog MathBabe.  Her newly released book Weapons of Math Destruction takes a look at the dark side of Big Data specifically how data collection and biased algorithms can be good for administrators, politicians and business but bad for anyone who falls under the influence of those agencies and their work.  In the introduction she leads of with the example of teacher assessments.  I was familiar with a scattergram that she had posted on her web site showing that year to year teacher assessment scores were essentially uncorrelated or random.  In the book she describes the human toll in this case a teacher fired because of this defective algorithm.  In another example later in the book, an experienced teacher scored a 6 out of 100 on a "value-added" teacher evaluation.  Only tenure kept him from getting fired.  The scoring algorithm was opaque and nobody could tell him what had happened.  The next year he scored the 96 out of 100.  But the algorithm was so flawed he knew that score was no more legitimate than the last one.  With the politicized environment surrounding teaching the proponents of teacher "accountability" like this variation since it fits their ideas about the system retaining incompetent teachers that need to be weeded out.  In fact, the algorithm is defective and like many is based on erroneous assumptions.

I personally know that physicians are subjected to the same processes as teachers, but so far it is less technologically advanced.  O'Neill points out that there is nothing magical about algorithms.  That they frequently incorporate the biases of the people who design and contract for them.  Opacity and a lack of correction by feedback is another feature.  I worked for the same employer for a number of years when physician "accountability" measures were put in place.  The "algorithm" for salary went something like this RVU Productivity + Outside Billing + Citizenship = Pay.  RVUs were the total number of patients seen according the the biased government and managed care billing schemes.  Outside billing was any consulting work done outside of the clinical work that was billed through the department.  Citizenship included teaching and administrative duties as well as any Grand Rounds or CME lectures that were done.  In other words apart from the subjectively based billing scheme all of the inputs are almost totally subjective and influenced by all kinds of pseudoaccountability measures along the way.  For example, in parallel with the teacher ranked on the algorithm, I was told one year that I had achieved the top rank in terms of documentation in a group of about 25 physicians.  The next year - making no changes at all in terms of that documentation - I was dead last.  My conclusion, like the teacher in the example was that the rating scheme was completely bogus and with that kind of a scheme who cares about the results?

The number of based algorithms applied to physicians has eerie parallels to those mentioned in WMD.  Here are a few that I picked out on the first read:

1.  The algorithm is based on faulty data - the teacher evaluation algorithms were based on a faulty interpretation of data in the Nation at Risk report.  The report concluded that teachers were responsible for declining SAT scores between 1963 and 1980.  When Sandia Labs reanalyzed the data 7 years later they found that an great expansion in the number of people taking the test was responsible for decreased average score but subgroup analysis by income group showed improved scores for each group (p. 136).  The only reason that teachers are still being blamed is political convention.  I posted here several years ago that the top ranked students in the world in Finland are taught by teachers who are assumed to be professionals and who are not critiqued on test results.

The parallel in medicine was the entire reason that medicine is currently managed by the government and the healthcare industry.  It was based on criticism in the 1980s that doctors were lining their pockets by performing unnecessary procedures and that work quality was poor.  That should sound familiar because that criticism has been carried forward despite a major study that showed it was completely wrong.  The massive Peer Review Standards Organizations (PRSO) in each state in the 1990s conducted rigorous reviews of all Medicare hospitalizations and concluded that there was so little overutilization and so few quality problems that it would not pay to continue the program.  The only reason that managed care companies exist today is by political convention.

2.  An effective teacher like an effective doctor is too complex to model - When that happens only indirect measures or "crude proxies" (p. 208) can be used to estimate effectiveness.  In medicine like teaching - the proxy measures are incredibly crude.  They generally depend on diagnosis, poorly account for comorbid illness, and the outcome measures are heavily influenced by business rather than medical decision making.  The best examples are length of stay parameter and readmission parameters.  Every physician knows that there are set payment schedules based on the supposed ideal length of stay for a particular illness.  The business influence in the discharge decision is so malignant these days that non-physician case managers are present to pressure physicians into discharging patients.  If the discharge beats the length of stay parameter - the hospital makes money.  I sat in a meeting at one point and asked the obvious question: "OK - we have completed the discharge checklist - do we know the outcomes?  How do the patients do when they are discharged by this process?  How many of them die?"  Dead silence followed.  Most people would be shocked to hear that what passes for evidence based medicine is often a checklist that has no meaning in the real world.  Making the points on the checklist is good for advertising though.

3.  There is a lack of transparency in the overall process - The teachers in WMD who were blindsided by the algorithm were never told how that conclusion was reached.  I encountered the same problem in a managed care organization when it was clear to me that administrators with no knowledge of psychiatry were telling us what to do.  In some cases, "consultants" were brought in to write reports to confirm the most recent administrative edicts.  When I asked my boss if I could talk with the people sending out the edicts I was informed that there was a "firewall" between clinicians and upper management.  This lack of feedback is another critical dimension of algorithms gone astray.  If you are writing an algorithm biased toward a business goal - why would you want feedback from clinicians?  Why would you want any humanity or clinical judgment added especially in the case of psychiatric care?  Let's just have a dangerousness algorithm and leave it at that.  Those are the only people who get acute treatment, even though it is patently unfair relative to how the rest of medicine works.             

Big Data is good for science.  We can't do elementary particle physics or genomic analysis very well without it.  Big Data is also good for business is much different ways.  There are clearly people out there who cannot resist buying items online if the Amazon algorithm shows it to them enough times across a number of web pages.  Big data in business can also come up with billing algorithms that have less to do with reality than making a profit.  Similar programs can be found for employee scheduling, performance analysis, and downsizing.  The problems happens when the business biases of Big Data are introduced to science and medicine.  Those techniques are responsible for an array of pseudoquality and pseudoaccountability measures for physicians, hospitals, and clinics.

Unfortunately physicians seem to have given up to the political conventions that have been put upon us.  Some administrator somewhere suggests that quality care now depends on a patient portal into an electronic health record and a certain number of emails sent by patient to their physician every month.  Across the country that will result in hundreds of millions of emails to physicians who are already burned out creating highly stylized documentation that is used only for billing purposes.  Terabytes of useless information that nobody will ever read again - the product of a totally subjective billing and coding process that started over two decades ago.  Is there any data that email communication is tied to the effectiveness or technical expertise of the physician?  I doubt it.  I worked with great physicians long before email existed.

It is about time that somebody pointed out these manipulations provide plenty of leverage for the management class in this country at the expense of everyone else.  It is well past the time that doctors should be confronting this charade.  


George Dawson, MD, DFAPA




References:

1.  Cathy O'Neill.  Weapons of Math Destruction - How Big Data Increases Inequality And Threatens Democracy.  Crown Publishing Group.  New York, NY, 2016.  I highly recommend this book for a look at the other side of Big Data.  It is written in non-technical language and is very readable.







Attribution:

The photo at the top is a Server Room in CERN By Florian Hirzinger - www.fh-ap.com (Own work (Florian Hirzinger)) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0) or GFDL (http://www.gnu.org/copyleft/fdl.html)], via Wikimedia Commons"   href="https://commons.wikimedia.org/wiki/File%3ACERN_Server_03.jpg"><img width="512" alt="CERN Server 03" src="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d7/CERN_Server_03.jpg/512px-CERN_Server_03.jpg"/></a>

  CERN Server 03

Wednesday, November 27, 2013

Fantasy Foundation For The Preservation of Psychiatry

Psychiatry is on the ropes.  The content of this blog illustrates the prevalent biases against the field that all eventually trickle down to less resources to work with and managed care companies rationing those meager resources in order to make money.  One of my favorite fantasies lately is to think about what I would do to save psychiatry if I ran a foundation with significant resources.  I have thought about it long enough and hard enough to come up with a number of guideposts:

1.  Save the teachers - probably the most beleaguered people in the field these days are the teachers of psychiatrists.  There are a lot of bloggers out there complaining about the "ivory tower" academics who just don't know how life is on the front lines.  The usual gripe is that they make too much money or are in some kind of shady consulting deal.  How dare they dictate to the rest of us how to practice?  That has not been my experience, and I have probably taught as much to medical students and residents as the next guy.  I see people trying to make a living and teach at the same time.  I see people needing to meet absurd "productivity" expectations and teach at the same time.  Teaching in generally is not counted as "productivity" in a managed care environment.  I see people who give up their ability to type up more patient notes at noon so that they can give a lecture to mostly disinterested medical students or fatigued residents.  They end up typing those notes at night on what is supposed to be their own time.

When I ask myself what would help them the most it comes like a flash - free high quality graphics for PowerPoints.  I have a parallel blog with some ideas, but there is nothing like great graphics that are free to use and save your faculty hours of sleep trying to come up with their own and not violate somebody's copyright.  You would think that professional organizations, like the American Psychiatric Association (APA) would support this idea.  Like everybody else, they produce downloadable PowerPoint slides for their major journals.  If you read the small print, you are supposed to go to the CopyRight Clearance Center and pay a fee.  I paid a fee of $45 for a lecture to a class of 12 and $85 to lecture a class of 42.  That was to project the slide and include it in my PowerPoint for the day.  I currently give about 32 lectures a year.  Considering the reimbursement I get for the lecture, it is not a commercial presentation, and I have been paying lots of money to the APA for about 30 years - you would think I could get a break.  As the head a a great foundation, I would purchase the rights to several good resources like Blumenthal's Neuroanatomy Through Clinical Cases or Atlas' MRI of the Brain and Spine and make them freely available to any instructors of psychiatrists.

2.  Free neuroscience conferences - there need to be much better basic science courses to bring clinical psychiatrists up to speed on the latest neuroscience and how it applies to the field.  Typical conferences are centered around some clinical activity that most of us are doing anyway.  Do we really need to hear more about something that we are doing everyday?  Something that we know everything about including the usual limitations?  Why not expand back into a consciousness based discipline looking at innovative ways to conceptualize problems and solutions.  Neuroscience is critical to that and there are several very articulate voices in the area.  I would plan a conference every years that was free to psychiatrists for 2 - 4 days of neuroscience.  There is a lot of neuroscience out there and I would ask some of the top journals like Nature, Science, Neuron, Biological Psychiatry, and Molecular Psychiatry to submit a program of Neuroscience for psychiatrists.  I would award the grant competitively to the best submitted program.

3.  Free computerized psychotherapy and an affiliated institute of psychotherapy using computers - I previously posted about John Griest's work in computerized psychotherapy and its effectiveness.  The whole point of the post was to emphasize a significant source of non-medication based treatment that is essentially not limited by manpower requirements.  There are several groups who have implemented this already, but to my knowledge none of them are major U.S. health care organizations or managed acre companies.  The commonest managed care approach is to give everyone a non specific depression rating scale, call that a quality marker, and then put as many people on antidepressants as soon as possible.  There is enough IT available that a foundation could take the lead in this area, develop the programs, and accept referrals from psychiatrists across the country for specific types of computerized psychotherapy.   

4.  Free clinical workgroups -  I have posted on the University of Wisconsin Memory Clinics collaborative clinical network across the state that focuses on maintaining a high level of expertise in all of the cooperating clinics for the diagnosis and treatment of Alzheimer's Disease and other dementias.  There is no reason that model cannot be extended to Depression, Bipolar Disorder, Post Traumatic Stress Disorder, or Attention Deficit Hyperactivity Disorder.  When people talk about collaborative care, they are usually talking about a managed care model that marginalizes psychiatrists.  A recent post suggested that some of the promoters of the managed care model have challenged naysayers to come up with an alternative.  I am a naysayer to anything that resembles managed care and the UW model is definitely a competing model that emphasizes psychiatrists at the top of their game in diagnosing and treating mental disorders.  That would be my priority over a managed care model that is so watered down, you don't even need a psychiatrist on the premises.

5.  An independent certification process - The American Board of Medical Specialties (ABMS) has a chokehold on all board certification processes with the exception of the American Board of Addiction Medicine (ABAM).  ABAM has their own certification and recertification process.  The current controversy involves the recertification process and whether it should be a standard blind exam with no learning aspects and a review of patients in a physicians practice or not.  I have posted some details about this to show how highly politicized it has become.  There is really no good evidence that recertification beyond the usual CME requirements is needed.  Although the American Board of Psychiatry and Neurology (ABPN) and the APA has gone along with ABMS ideas, most members find the process onerous and not conducive to learning, especially when they are in a labor intensive work environment that allows little time for study.  Any professional organization should be innovative enough to come up with an ideal process that would keep members up to speed professionally while not intruding on their limited time.  My foundation would develop a recertification system based on the APA's Focus journal an develop a process that would allow members to study on their own time and recertify by taking the Focus examinations.  It should eventually be possible to incorporate modules from the ongoing neuroscience seminars and what is learned in the computerized psychotherapy lab as study modules.

Using these innovations and hopefully more, my foundation would seek to improve the technical expertise of all psychiatrists, highlighting what is possible for the future and bring every clinician out of the current misery of political overegulation and managed care overproduction.  The whole idea that we currently have a professional organization and a specialty board that are not protective of psychiatrists is one thing.  The idea that they are actually doing things that are counterproductive to the ongoing professional education of psychiatrists and increasing burnout by creating a more stressfull practice environment is another.

My fantasy foundation would hope to reverse those trends.

George Dawson, MD, DFAPA

Wednesday, February 22, 2012

Lessons from Finland on Professional Report Cards


The New York Review of Books this week contains a review by Diane Ravitch entitled “Schools We can Envy”.  In it she reviews “Finnish Lessons: What Can the World Learn from Educational Change in Finland?” by Pasi Sahlberg. 

It turns out that the Finland has one of the highest performing school systems in the world.  That occurs in the context of very high professional standards for teachers and a lack of attention to standardized tests.  From the article:

“Because entry into teaching difficult penetrating is rigorous, teaching is a respected and prestigious profession and Finland. So selective and demanding is the process that virtually every teacher is well-prepared. Sahlberg writes that teachers enter the profession with a sense of moral mission and the only reasons they might leave would be "if they were to lose their professional autonomy" or if "a merit-based compensation policy tied to test scores were imposed".   Meanwhile the United States is now doing to his teachers what Finnish teachers would find professionally reprehensible: judging their worth by the test scores of their students.”

As expected, blaming the teachers is currently popular in the United States but it does not fly in Finland.

And what implications does this have for blaming the doctors? I could easily make the argument that the variance in patient outcomes for a particular physician is probably much less under the control of that physician than the variance in student outcomes for any teacher.

It is time to let the public know that the "report cards" on doctors is another poorly thought out idea from the government and the managed-care cartel and they are probably even less valid than report cards on teachers.  I will provide all of the details in subsequent posts.

George Dawson, MD

Diane Ravitch.  Schools We Can Envy.  New York Review of Books