Medical diagnosis AIs can be tricked into telling whose data trained them

AI models used to help diagnose medical conditions have a problem: They’re ready and willing to identify patients whose data was used to train them.

German researchers reported in a Nature paper published Wednesday that discriminative AI models - those used to classify data and make predictions about new inputs based on their training sets - are particularly susceptible to membership inference attacks (MIAs) that query the models in an attempt to figure out whether a particular datapoint is included in their training sets. 

What that means for medical AI models is that any patient whose data is used to educate the bot could be exposed, leading to details about their medical history and diagnoses being leaked. In an analysis of seven medical AI datasets consisting of images, ECG records, and general electronic health records, the team determined that individual patients targeted by such attacks can be identified with “near-perfect attack success,” which they explain flies in the face of how such models are evaluated for safety. 

“The fact that MIAs can achieve near-perfect success rates for individual patients is not adequately captured by the standard evaluation protocol, which measures attack success in aggregate across records,” the researchers said. Based on their findings, they conclude, reporting standards for AI privacy audits need to change. 

It gets worse, too: Patients in the dataset are generally easy to identify and, unsurprisingly, those underrepresented in medical AI training data are even easier to finger than those whose data doesn’t stand out. 

Underrepresented groups can include those in a number of sensitive categories: Race, insurance status, sex, the protocol used to conduct medical imaging, and certain disease statuses can all function as outliers that make it easier to identify individuals. 

“Generally speaking, privacy risks from MIAs become more severe as a model’s training cohort becomes more specific,” Technical University of Munich AI in Healthcare and Medicine chair and paper lead author Moritz Knolle told The Register in an email conversation. “You could imagine … scenarios where membership in a training dataset reveals that someone has a dormant genetic condition such as Huntington's disease, depression, or attended a specific, specialised treatment clinic.” 

In other words, exposing healthcare AI training data could be used to identify those with sensitive health conditions, spill secrets they may not want public, or otherwise fuel discrimination. 

To make things even worse again, the larger the dataset, the easier it is to expose records, and “the magnitude of this change in patient-level risk was previously unknown” in larger models. 

The privacy devil in the data details

This is bad and all, but it’s not necessarily the end of the world, as performing an MIA attack on a medical AI model supposes the attacker already has a few things at their disposal, namely at least some medical data belonging to the people they want to identify. 

“To conduct a MIA an attacker needs access to a target data point,” Knolle confirmed to us while also noting that their paper revealed access to a full patient data point isn’t needed, in contrast to what was previously believed. “In our paper we show that an attacker with partial access can still successafully conduct MIAs.” 

The MIA attack itself, as detailed in the paper, relies on medical AIs being more certain of their predictions if the input data is already part of their training set. A potential attacker, then, simply peppers an AI model with obtained patient data, checks the confidence level, and surmises that said patient is part of the training data. 

“An attacker conducting a MIA does not need to know who the data belongs to that they are trying to conduct the MIA with,” Knolle explained. “In fact, all the dataset we use in our study were anonymized.”

Anonymized in the datasets, but not the target data, that is. As explained in the paper their MIA attacks were largely error-free at the individual patient level, meaning confidence levels are an accurate way to figure out if a particular patient's data is part of a training set. 

“The attacker would simply need access to someone’s blood test results, or part of these results” in order to infer inclusion, Knolle said. 

Of course, they have to get that data first, but given how frequently healthcare data is exposed in breaches, it’s not exactly hard to imagine a bad actor getting ahold of something they can use.  

“Given that medical data is not always securely stored it is not unthinkable that an attacker could get access, for example, by gaining unauthorized access to the database of your general practitioner after they performed a routine blood test,” Knolle said. 

How to protect patient data?

Asked what he hopes this research accomplishes, Knolle told us he just wants the medical world to understand that AI training data needs to be better secured. 

“I hope that the medical AI community will start to take privacy risks seriously and that risk mitigation techniques are used in situations where they are necessary,” Knolle said. 

The researchers make several recommendations for how to do this, like through the use of differential privacy frameworks that are designed to mathematically guarantee training data remains anonymous - a key consideration if medical AI firms want patients to trust them with their data. 

As mentioned above, the team also wants to see privacy audit standards change to consider individual-level data, not just aggregate privacy risks. Alternatively, medical AI training data could just be compiled so that underrepresented groups are better represented, Knolle said. 

“There are many situations where a successful MIA represents a small or negligible privacy violation,” Knolle noted. “These are situations where AI models are trained on large, general populations in which both healthy and diseased individuals are represented in sufficient numbers.”

Representation, in other words, definitely matters when it comes to keeping patient data private. ®