“Watch Out! Unsophisticated AI in Healthcare Can Perpetuate Bias”


As the healthcare industry continues to hype about AI’s potential to be transformative, it is important to remember that even software can harbour a certain amount of unconscious bias.

Carla Smith champions transformational strategy, governance, and policy in the health sector, particularly in the areas of digital health and health IT. In this article, she warns us not to get carried away with the shiny offerings of AI before all bugs are ironed out, and to do that all stakeholders must be involved. 

There are myriad places where AI, while exciting in many ways for health and healthcare, has dangerous potential for perpetuating existing, or introducing new, bias.  


In the Fall of 2018, Amazon scrapped its self-developed AI recruitment tool designed to find highly-qualified software engineers.


Problem was, the tool didn’t like women.  


Amazon discovered the problem in 2015, just a year after launching it.  They had programmed their AI tool to conduct its analysis based upon patterns it observed in resumes submitted to Amazon over a 10-year period.  The tool was supposed to filter through hundreds of resumes and find the handful that were right for a software engineering position at Amazon.


Because the tech industry is male-dominated, the bulk of those resumes came from men. So, the system taught itself that male candidates were preferable.  Amazon’s tool penalized resumes that included the word “women” as in “captain of the women’s chess club”. And, the AI system downgraded resumes from candidates who attended two all-female colleges.


Amazon’s programming team edited the AI programming to neutralize the impact of some terms but were never able to confirm that the tool was, and would remain, free of bias.  Ultimately, leadership lost faith and shut it down.


There’s a similar story of AI-perpetuated bias against blacks in the US criminal justice system, uncovered by ProPublica in 2016. Both are cautionary tales.  There are myriad places where AI, while exciting in many ways for health and healthcare, has a dangerous potential for perpetuating existing or introducing new, bias.  Barbara Grosz, an AI professor at Harvard put it well when she said: “Stop thinking about robots taking over. We have more to fear from dumb systems that people think are smart than from intelligent systems that know their limits.”


According to David Magnus, Executive Director of Stanford’s Center for BioMedical Ethics, bias can impact AI-related health data and decision-making in three ways: (1) human bias; (2) bias introduced by design; and, (3) bias in ways health professionals use the data.  Plus, once an AI system launches, it learns things that affect its future decision-making. As it learns, the tool’s knowledge base becomes extremely difficult, if not impossible, for human programmers to discern and understand.


Let’s use just one healthcare example – AI predicting which potential patients have the best/worst ability to pay.  That could introduce bias skewing access to care away from those who are deemed to be less-able to pay.  As a result, certain demographics may become less healthy, which could, in turn, lead to public health crises and terrible situations for patients and their families.


What if we use AI to determine where to deploy our clinicians in the field? What if we use AI to predict which patients are most/least likely to comply with evidence-based treatment plans?  All of these examples have the potential for bias to be built-in, or resulting from, the use of AI.


Here are just a few of the pilots, plans, and explorations underway in the health sector:

  • Nigam Shah is leading an algorithm pilot at Stanford to predict the need for a palliative care consultation.  The team, which includes an ethicist, is working to ensure that the incorporation of predictions into the care path guarantees that the physician “…has full understanding that the patient’s problems are answered and well-understood”.  
  • Mount Sinai in New York fed data from more than 700,000 patients to its AI algorithm including doctors’ visits and patients’ test results. When the Mount Sinai team tested its efficiency, the AI algorithm was “pretty accurate” at predicting diseases based on a patient’s records. But, Joel Dudley who leads the AI team said that, while the team can build the AI models, they have no idea how they work.
  • FujiFilm, Lunit, and Salud Digna (a diagnostic service provider in Mexico) are partnering to bring radiologists into an evaluation of FujiFilm/Lunit’s AI technologies for diagnostic imaging.  According to Tak Shimomura at FujiFilm Medical USA, Salud Digna was selected as a provider partner because of their “…focus on training, technology, and commitment to accessible, quality care for economically-disadvantaged patients.”
  • Parsa Mirhaji is leading Montefiore’s strategy to build an AI system that “…harvests every piece of data that we can possibly find, from our own EMRs and devices to patient-generated data to socioeconomic data from the community.  It’s extremely important to use anything we can find that can help us categorize our patients more accurately.”
  • Based upon a study of 180 “qualified professionals in provider and non-provider settings”, HIMSS Media is now advocating for providers to “…not only invest, but also take on a proactive role in developing new tools.”
  • Len Usvyat of Fresenius Medical Care North America leads the team in multiple dialysis-related AI pilots.  One that’s in the third phase of development extracts 1,600 data variables from nursing notes daily to predict if a patient may go into the hospital within in the next week.  And, a new Fresenius pilot involves predictive modelling of a home dialysis patient’s probability of getting peritonitis within one month.


Only one of the above examples (Stanford) mention an ethicist as a member of the development team.  None mentioned having a patient involved – just that they use patient data. And, one example is actually advocating for providers to hurry up and get involved in AI now.  


Hold on.  We’re moving very fast all of a sudden.  I would argue that we lack the right voices at the table.  As a result, we risk breaking the fundamental rule of healthcare: “First, do no harm”.


To combat the potential to perpetuate – or introduce new – bias into AI-related health decision-making, clinicians, programmers, patients, and ethicists must be in lock-step from the very first conceptual conversation.  AI teams must ensure they’ve got the all these stakeholders actively involved, a written set of values by which the team will operate, and as transparent a process as it is reasonably possible to create.


Do you know of an AI pilot that includes all the stakeholders?  Tell us – we’d love to hear how it’s going.

Related Content

Latest Report