What Happened
In 2014, Amazon began developing an AI system to automate the review of job applicants’ resumes. The system was trained on patterns in resumes submitted to the company over the previous 10 years — a period during which the tech industry, and Amazon in particular, was overwhelmingly male. The AI learned that being male was correlated with hiring success and began systematically penalizing resumes that indicated the applicant was female.
The system downgraded resumes containing the word “women’s” (as in “women’s chess club captain”) and penalized graduates of all-women’s colleges. It had effectively learned that the historical pattern of gender discrimination was a feature to replicate rather than a bug to avoid.
Timeline
Amazon’s machine learning team began the project in 2014. By 2015, the company had discovered that the system was not rating candidates in a gender-neutral way. Engineers attempted to modify the system to be neutral on gender-specific terms, but found that the bias was embedded throughout the model’s learned patterns and could not be easily isolated. By 2017, Amazon disbanded the team. Reuters broke the story publicly in October 2018.
Impact
The Amazon hiring tool became perhaps the most cited real-world example of AI perpetuating historical discrimination. It demonstrated clearly that AI trained on biased historical data will reproduce and potentially amplify those biases. The case appeared in virtually every subsequent discussion of AI fairness, from academic papers to congressional hearings.
The incident also dampened enthusiasm for AI-driven hiring tools across the industry. Companies that had been eagerly adopting AI screening faced scrutiny from regulators, civil rights organizations, and their own legal teams.
Response
Amazon never used the tool for actual hiring decisions, deploying it only experimentally to compare its recommendations against human recruiters’ choices. When engineers could not eliminate the gender bias, the company scrapped the project entirely. Amazon emphasized that the tool was never used to make actual hiring decisions, though this distinction was lost in much of the public discourse.
Lessons Learned
The Amazon hiring case illustrated the fundamental problem of training AI systems on historical data that reflects past discrimination. The AI was working exactly as designed — finding patterns in historical data — but the patterns it found were discriminatory. This created a philosophical challenge: how do you build predictive systems from historical data while preventing them from perpetuating historical injustices?
The case also demonstrated that bias in AI can be systemic rather than isolated. Amazon’s engineers could not simply remove gender-specific terms from the model because the bias was woven throughout the learned patterns in complex, non-obvious ways. This finding influenced subsequent research on AI fairness and debiasing techniques.