Bias in Government Fraud Detection Systems

In 2019 it was revealed that the Dutch tax authorities had used a self-learning algorithm to create risk profiles in an effort to spot child care benefits fraud. But the algorithm was not working properly and has been wrongly labelling people as fraudsters. After several years of using this biased algorithm, the country’s privacy regulator opened an investigation. They found out that the tax authorities focused on people with “a non-Western appearance” (more specifically, they targeted people with Turkish or Moroccan nationality), and among risk factors were also having dual nationality and a low income. Authorities penalized families over a mere suspicion of fraud based on the system’s risk indicators. The results were disastrous. Dutch tax authorities demanded that people flagged as fraudsters pay back their child care allowances, and tens of thousands of families were pushed into poverty because of exorbitant debts to the tax agency. Some victims committed suicide and more than a thousand children were taken into foster care.

After the investigation Dutch data protection agency fined the Dutch tax administration with 2.75 million EUR in December 2021 for the “unlawful, discriminatory and therefore improper manner” in which the tax authority processed data of child care benefit applicants. In April 2022 Dutch data protection agency imposed another 3.7 million EUR fine on the Tax Administration for illegally processing personal data over a period of years in its “fraud identification facility”. But several experts are warning that something similar could happen again. When the scandal came to light, the Dutch government resigned, but they regrouped 225 days later.