The report Xenophobic Machines exposes how racial profiling was baked into the design of the algorithmic system used to determine whether claims for childcare benefit were flagged as incorrect and potentially fraudulent. Tens of thousands of parents and caregivers from mostly low-income families were falsely accused of fraud by the Dutch tax authorities as a result, with people from ethnic minorities disproportionately impacted. While the scandal brought down the Dutch government in January, sufficient lessons have not been learnt despite multiple investigations.
Curated from amnesty.org · 26 October 2021 →
Between 2005 and 2019 the Dutch tax administration used a risk model to decide which childcare benefit claims to investigate, and dual nationality was one of the things it scored on. Around 26,000 families were wrongly accused of fraud and ordered to repay years of benefit in full, with no hardship provision and, in practice, no route of appeal, since the model's reasoning was not disclosed to the families or to the officials acting on it. Households were driven into debt and lost homes and jobs, and more than three thousand children were taken into care. The Rutte cabinet resigned over it in January 2021. It is the largest documented harm from an automated decision system anywhere, and there is no model in it: this is a scoring rule with a protected characteristic in the feature list.
