United States

Biased AI Resume Screening at Amazon

Amazon has been using computer programs to review job applicants’ resumes since 2014. At one point they developed an experimental hiring tool that used artificial intelligence to give job candidates scores ranging from one to five stars. This tool has been very useful for the human resources department, however in 2015, they realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way. The problem was that Amazon’s AI models were trained on applications submitted to the company over a 10-year period. And while most of the applications came from men, the algorithm concluded that male applicants were preferred and penalized resumes that indicated that the applicant was female.

OUTCOME

When Amazon internally (by 2015) discovered its system for evaluating applicants for software development jobs and other technical positions was biased towards women, they made the necessary changes to remove the bias. Engineers first tried to neutralize identified terms that introduced bias, however they realized the model could develop new, harder-to-detect biases. But the management lost faith in the initiative, because they became aware that other biases can occur. So, in 2017 they stopped using AI for reviewing job applicants’ resumes. Amazon retained some parts of the developed solution, for instance duplicate-resume detection, but the biased tool was never broadly reactivated.

REFERENCES

Insight – Amazon Scraps Secret AI Recruiting Tool That Showed Bias against Women (Reuters)
Case Study: How Amazon’s AI Recruiting Tool “Learnt” Gender Bias (Cut The SAAS)

Type of Bias – Technical Point

Training Data / Representation Bias

The model was trained on 10 years of predominantly male resumes, embedding gender bias. It penalized resumes containing terms like “women’s” or those from women’s colleges (American Civil Liberties Union, Reuters).

Algorithmic Bias

The model learned proxy signals (e.g., verb choice) that correlated with male applicants, amplifying skewed preferences (Cut The SAAS).

Type of Bias – Social Sciences Point

Gender Bias

The tool systematically downgraded female applicants, effectively marginalizing women in technical roles.

Implicit Bias

It reflected and reproduced existing gender imbalances in tech culture, reinforcing stereotypes (Forbes).

How many people were affected

Magnitude

Amazon never disclosed an exact count of affected people, however there is estimation that thousands of female applicants were affected. Company-wide trial from 2014–2017; biased behavior discovered in 2015. Amazon halted the tool in 2017, though legacy issues remain industry-wide.

REFERENCES

Insight – Amazon Scraps Secret AI Recruiting Tool That Showed Bias against Women (Reuters)
Case Study: How Amazon’s AI Recruiting Tool “Learnt” Gender Bias (Cut The SAAS)