Bias in algorithm to detect sepsis

In 2019 a group of researchers in Duke University Hospital’s emergency department started developing an algorithm to help predict childhood sepsis. It is a medical condition that is curable with antibiotics but is fatal for nearly 10% of kids in the United States. However, diagnosis is challenging because typical early symptoms (fever, high heart rate and high white blood cell count) can mimic other illnesses including the common cold. The research team has been aware of possible bias problems, and they spent a lot of effort to teach the algorithm to identify sepsis based on vital signs and lab tests. They implemented quality control tests to ensure the algorithm found sepsis equally well regardless of race or ethnicity. However, after almost three years, they discovered possible bias. They found that doctors at Duke University Hospital took longer to order blood tests for Hispanic kids eventually diagnosed with sepsis than for white kids. One possibility for that was that the physicians were perhaps taking illnesses in white children more seriously than those of Hispanic children, but another reason could also be that the need for interpreters slowed down the process or ordering blood tests. This delay inaccurately taught AI that Hispanic kids develop sepsis slower than other kids. And that time difference resulting from the bias could be fatal.

At Duke University Hospital the team immediately started to solve the problem and after eight weeks they fixed the algorithm to predict sepsis at the same speed for all patients. But more generally, this event increased awareness that transparency is crucial to determine if an algorithm is unbiased enough to safely use on patients. More and more researchers are now starting to share best practices on how to tackle bias and there is also a push for new regulatory requirements. Because the result of the bias in AI algorithms is the bias in inequities in health care and this is something that we all want to prevent.