According to a report by Reuters, Amazon began developing an automated system in 2014 to rank job seekers with one to five stars. But last year, the company scrapped the project after seeing it had developed a preference for male candidates in technical roles. [...] It would reportedly penalize resumes containing the word "women's" or the names of certain all-women colleges. Although Amazon made changes to make these terms neutral, the company lost confidence that the program was indeed gender neutral in all other areas.
Curated from technologyreview.com · 10 October 2018 →
The oldest case on this page and still the clearest. Amazon trained a hiring model on ten years of its own applications, which were mostly from men, and the model learned that being a man was a signal. It marked down the word women's, as in women's chess club captain, and graduates of two women's colleges. Reuters broke it in October 2018. Amazon says the tool was never the sole basis of a hiring decision and that it was abandoned. The reason it is worth reading now is the last sentence of the quote: Amazon could neutralise the specific terms it had found, and could not establish that there was nothing else, which is the general problem with correcting a model after the fact rather than the data before it.
