Auditing AI Systems for Discrimination

1. Introduction

Auditing AI systems for discrimination means systematically examining an Artificial Intelligence (AI), machine-learning, automated decision-making, or algorithmic system to determine whether it produces unfair or discriminatory outcomes for individuals or groups protected by law.

In employment, AI may be used for:

• recruitment and candidate screening;
• CV/resume ranking;
• interview assessment;
• facial-expression or voice analysis;
• employee performance scoring;
• promotion and appraisal;
• salary recommendations;
• attendance and productivity monitoring;
• disciplinary decisions;
• termination decisions;
• workforce scheduling; and
• targeted job advertising.

The important legal principle is that automation does not automatically remove employer responsibility. An employer generally cannot defend discriminatory treatment simply by saying, “the computer/AI made the decision.” The EEOC has specifically recognized that existing employment-discrimination laws continue to apply when AI or automated systems are used.

2. Meaning of AI Discrimination

AI discrimination occurs when an automated system produces an unfair disadvantage because of a protected characteristic such as:

• race;
• colour;
• sex/gender;
• pregnancy;
• religion;
• national origin;
• age;
• disability; or
• another characteristic protected by applicable law.

Discrimination can occur even when the AI system does not explicitly use the protected characteristic.

Example

Suppose an AI recruitment system does not contain a field called “gender.”

However, it uses:

• employment history;
• career breaks;
• particular job titles;
• college attended;
• postcode;
• language patterns; and
• previous hiring outcomes.

If these variables operate as proxies for gender, the system may still produce discriminatory results.

This is particularly important because algorithmic systems can learn patterns from historical data and reproduce existing inequalities. The EEOC has highlighted concerns that automated systems can amplify discrimination rather than eliminate it.

3. Why AI Systems Become Discriminatory

There are several major sources of algorithmic discrimination.

A. Biased training data

AI learns from historical information.

If historical hiring decisions were discriminatory, the AI may learn those patterns.

Example

A company historically hired:

• 80% men for technical positions;
• 20% women.

An AI trained on those records may incorrectly conclude that characteristics associated with previous male employees are indicators of a “successful candidate.”

The AI therefore reproduces historical discrimination.

B. Proxy discrimination

The AI may not use race, gender, religion or age directly but may use variables strongly associated with them.

Examples:

Therefore, an audit must examine indirect variables, not merely protected-characteristic fields.

4. Disparate Treatment and Disparate Impact

An AI discrimination audit should distinguish between two major forms of discrimination.

A. Disparate treatment

This occurs where a protected characteristic directly influences the decision.

Example

An AI hiring program contains a rule:

Reject female applicants for a particular technical position.

That is direct discrimination.

B. Disparate impact

This is more complicated.

A rule may appear neutral but disproportionately disadvantage a protected group.

Example

An AI system requires:

“At least 10 years of uninterrupted employment.”

The rule appears neutral.

However, it disproportionately eliminates applicants who have:

• pregnancy-related career breaks;
• disability-related interruptions;
• caregiving gaps.

The auditor must therefore ask:

1. Who is being rejected?
2. Which groups are disproportionately affected?
3. Is the criterion genuinely job-related?
4. Is it necessary?
5. Is there a less discriminatory alternative?

The principle of disparate impact is particularly important in AI because apparently neutral algorithms can produce statistically unequal outcomes.

5. Objectives of an AI Discrimination Audit

An effective audit should determine:

1. Whether discrimination exists

Does the AI produce different outcomes for different protected groups?

2. Why discrimination occurs

Is the problem caused by:

• training data;
• model design;
• proxy variables;
• thresholds;
• human intervention;
• data quality;
• feedback loops?

3. Whether discrimination is statistically significant

The auditor should compare selection, rejection, promotion, termination or other outcome rates.

4. Whether the system is legally defensible

The organisation should determine whether the AI complies with applicable:

• equality laws;
• employment laws;
• disability-discrimination laws;
• data-protection/privacy requirements;
• contractual obligations;
• internal policies.

5. Whether corrective action is required

An audit should not merely identify bias. It should establish a remediation plan.

6. Major Stages of an AI Discrimination Audit

Stage 1: Identify the AI system

The auditor should document:

• name of the system;
• developer/vendor;
• purpose;
• version;
• date implemented;
• departments using it;
• decisions influenced by it;
• degree of human involvement.

For example:

AI system → recruitment → CV screening → ranking → interview selection.

7. Stage 2: Identify Protected Groups

The audit should identify which protected characteristics may be relevant.

Depending on jurisdiction, these may include:

• sex;
• gender;
• race;
• caste;
• ethnicity;
• religion;
• disability;
• age;
• pregnancy;
• national origin.

The auditor should be particularly careful with intersectional discrimination.

Example

An algorithm may not significantly disadvantage:

• men generally; or
• women generally.

But it might disproportionately disadvantage:

women above a particular age.

Therefore, analysis should sometimes examine intersections such as:

gender + age

or

gender + disability.

8. Stage 3: Audit the Training Data

The auditor should examine:

• who is represented;
• who is missing;
• historical hiring patterns;
• labels used for training;
• quality of the data;
• data collection methods;
• historical discrimination.

Important question

“Was the historical decision itself fair?”

If the answer is no, using that decision as a training label can transmit the discrimination into the AI.

9. Stage 4: Test for Proxy Variables

The auditor should identify variables correlated with protected characteristics.

For example:

postcode → socio-economic characteristics → race

or

employment gap → caregiving/pregnancy → gender

or

graduation year → age

The fact that a protected characteristic is excluded from the model does not necessarily prove that the system is non-discriminatory.

10. Stage 5: Conduct Statistical Testing

An AI discrimination audit should compare outcomes among groups.

For example:

The auditor would investigate why Group B's selection rate is substantially lower.

Useful measurements can include:

• selection-rate comparison;
• adverse-impact ratio;
• false-positive rate;
• false-negative rate;
• approval/rejection rates;
• promotion rates;
• termination rates;
• salary differences;
• error rates by demographic group.

Statistics do not automatically establish unlawful discrimination, but they can provide powerful evidence that further investigation is required.

11. Stage 6: Test the AI Under Controlled Conditions

Auditors can create comparable test cases.

For example:

Candidate A

• same qualifications;
• same experience;
• same education;
• male-associated name.

Candidate B

• exactly the same qualifications;
• exactly the same experience;
• exactly the same education;
• female-associated name.

If the AI produces substantially different scores, the auditor should investigate why.

Similar testing can be performed for:

• age;
• disability;
• race;
• religion;
• nationality.

12. Stage 7: Examine Human Oversight

AI discrimination can occur even when a human technically makes the final decision.

For example:

AI rejects 90% of applicants → HR reviews only the remaining 10%.

In that situation, saying that “a human made the final decision” may not adequately address the problem.

The auditor should ask:

• Does HR review rejected applicants?
• Can HR override AI?
• Is the override documented?
• Are employees trained to identify AI bias?
• Does management monitor override patterns?

13. Stage 8: Audit Explainability

The organisation should be able to explain:

Why did the AI rank Candidate A above Candidate B?

A completely unexplained automated decision creates serious governance problems.

The auditor should document:

• relevant input variables;
• decision rules;
• model version;
• scoring methodology;
• thresholds;
• human intervention;
• reason for final decision.

14. Stage 9: Disability and Accessibility Audit

AI hiring systems may discriminate against disabled applicants.

Examples include:

• speech-recognition systems failing to understand speech impairments;
• facial-analysis systems misinterpreting facial differences;
• timed assessments disadvantaging certain disabilities;
• chatbots failing to provide accommodation information;
• automated tests requiring abilities unrelated to the job.

The U.S. Department of Justice has specifically warned that algorithmic hiring technologies can create disability discrimination and that employers should consider reasonable accommodations.

15. Stage 10: Continuous Monitoring

An AI audit should not happen only once.

Models change.

Data changes.

Workforce composition changes.

Business requirements change.

Therefore, organisations should establish:

Continuous AI discrimination monitoring.

For example:

• monthly monitoring;
• quarterly statistical analysis;
• annual independent audit;
• audit after model modification;
• audit after a discrimination complaint.

16. Important Case Laws

The following cases are particularly useful for understanding AI discrimination auditing.

Case 1: EEOC v. iTutorGroup, Inc. (2023)

This is one of the most important direct examples of AI-related employment discrimination.

The U.S. Equal Employment Opportunity Commission alleged that iTutorGroup's automated recruitment system automatically rejected applicants based on age, including female applicants aged 55 or older and male applicants aged 60 or older.

More than 200 applicants were allegedly screened out.

The matter resulted in a settlement/consent decree, including monetary relief and compliance measures.

Principle

An automated recruitment system can produce unlawful age discrimination.

Audit lesson

Auditors must test:

• rejection rates by age;
• rejection thresholds;
• interaction between age and gender;
• automated knockout questions;
• historical hiring data.

AI automation does not provide immunity from anti-discrimination law.

17. Case 2: Mobley v. Workday, Inc.

Mobley v. Workday, Inc. is a major litigation concerning alleged discrimination arising from algorithmic employment screening.

The plaintiff alleged that Workday's applicant-screening technology could discriminate on grounds including:

• race;
• age; and
• disability.

The litigation is significant because it raises the question of whether an AI/software provider can face legal exposure where its algorithmic tools allegedly contribute to discriminatory employment decisions.

By 2026, the litigation had continued through significant pleading and discovery developments.

Principle

AI vendors cannot necessarily assume that responsibility ends with the employer.

Audit lesson

AI audits should therefore examine both sides of the technology chain:

Employer → AI vendor → training data → algorithm → output → employment decision.

Contracts with AI vendors should also address:

• bias testing;
• audit rights;
• data transparency;
• model changes;
• incident reporting;
• regulatory cooperation.

18. Case 3: Liu v. Uber Technologies

In Liu v. Uber Technologies, the allegations concerned Uber's use of passenger ratings in decisions affecting drivers.

The EEOC argued that a disparate-impact claim could arise where a rating system disproportionately affected non-white drivers.

The important issue was whether the plaintiff could plausibly allege that a seemingly neutral rating practice caused discriminatory effects.

Principle

A neutral-looking algorithmic/rating system may create disparate impact.

Audit lesson

AI auditors must investigate the source of data used by the algorithm.

For example:

Passenger rating → algorithmic score → driver termination.

If the underlying passenger ratings contain racial bias, an apparently neutral algorithm may reproduce that bias.

19. Case 4: Griggs v. Duke Power Co., 401 U.S. 424 (1971)

Although this is not an AI case, it is one of the foundational cases for auditing algorithmic discrimination.

The U.S. Supreme Court considered employment requirements that appeared neutral but disproportionately excluded Black workers.

The Court developed the modern disparate-impact principle.

Principle

A requirement does not escape discrimination scrutiny merely because it is facially neutral.

Relevance to AI

This principle maps directly onto algorithmic systems.

For example:

“The AI does not know the candidate's race.”

That fact alone is insufficient.

The real question is:

Does the AI's neutral-looking rule disproportionately disadvantage a protected group without adequate justification?

Therefore, Griggs is an essential conceptual foundation for AI discrimination audits.

20. Case 5: Watson v. Fort Worth Bank & Trust, 487 U.S. 977 (1988)

In Watson v. Fort Worth Bank & Trust, the U.S. Supreme Court dealt with subjective employment decisions and disparate impact.

The Court recognized that employment decisions can be scrutinized even when they are not based upon completely objective mechanical criteria.

Relevance to AI

AI systems are sometimes presented as:

“objective and mathematical.”

But an algorithm can simply convert subjective historical judgments into mathematical scores.

For example:

Human managers historically judged:

“good cultural fit.”

The company then trains AI on those historical judgments.

The AI produces:

“Cultural-fit score = 82%.”

The number looks objective, but the underlying concept may remain subjective and biased.

Audit lesson

Auditors should investigate how the target variable was originally created.

21. Case 6: Ricci v. DeStefano, 557 U.S. 557 (2009)

In Ricci v. DeStefano, the Supreme Court examined an employment testing system and the tension between test results and concerns about racial disparity.

The case is important for understanding the legal consequences of changing or discarding employment decisions because of demographic outcomes.

Relevance to AI

Suppose an AI recruitment model produces:

• 70% selection rate for Group A;
• 30% selection rate for Group B.

The employer suddenly changes the model solely because the statistical results are unequal.

That action itself can create legal questions.

Audit lesson

AI remediation should therefore be:

• evidence-based;
• legally reviewed;
• statistically justified;
• properly documented.

An organisation should not simply manipulate an algorithm's outputs to achieve a desired demographic result without understanding the legal consequences.

22. Case 7: EEOC v. Amazon — AI Recruitment Example

Amazon's earlier experimental recruitment tool became a widely cited example of algorithmic gender bias.

The system was reportedly trained using historical resumes in a male-dominated technical workforce. It subsequently learned patterns associated with male applicants and disadvantaged women.

Amazon ultimately abandoned the tool.

The EEOC has cited Amazon's experience as a prominent example of the risk that historical data can reproduce gender discrimination.

Principle

Historical hiring data can encode historical discrimination.

Audit lesson

Before deploying an AI recruitment model, auditors should ask:

“Does our historical dataset represent merit, or does it represent our historical hiring preferences?”

This is one of the most important questions in AI auditing.

23. Case 8: Saas v. Major, Lindsey & Africa, LLC

In Saas v. Major, Lindsey & Africa, LLC, the plaintiff alleged that algorithmic recruitment/screening practices contributed to discrimination based on sex and age, including concerns relating to a career or “motherhood gap.”

The litigation illustrates the difficulty of identifying discrimination where recruitment technology and human recruitment processes interact.

Audit lesson

Auditors should not examine AI in isolation.

They should examine the complete recruitment chain:

Advertisement → sourcing → screening → ranking → recruiter review → interview → selection.

Discrimination can occur at any stage.

24. What These Cases Establish

Taken together, these cases demonstrate several important principles:

25. AI Discrimination Audit Checklist

An organisation can use the following checklist.

A. Governance

•  Is there an identified owner of the AI system?
•  Is the purpose documented?
•  Is there an AI-use policy?
•  Has legal review been conducted?
•  Is an audit schedule established?

B. Data

•  Has training data been examined?
•  Is the data representative?
•  Does historical discrimination exist?
•  Are protected characteristics properly understood?
•  Have proxy variables been identified?

C. Model

•  Has the model been independently tested?
•  Are selection rates compared?
•  Are false-positive and false-negative rates measured?
•  Are threshold decisions justified?
•  Are model changes documented?

D. Employment Decisions

•  Does AI influence hiring?
•  Promotion?
•  Pay?
•  Performance?
•  Discipline?
•  Termination?

E. Disability

•  Is the system accessible?
•  Can applicants request accommodation?
•  Are speech/facial-analysis systems validated?
•  Are alternative assessment methods available?

F. Human Oversight

•  Can humans override AI?
•  Are overrides recorded?
•  Are rejected applicants ever manually reviewed?
•  Are managers trained about algorithmic bias?

G. Vendor Management

•  Does the vendor provide sufficient information?
•  Is independent auditing permitted?
•  Are discrimination incidents reported?
•  Are model updates disclosed?
•  Are contractual compliance obligations included?

26. Remediation When Discrimination Is Found

If an audit identifies discriminatory outcomes, the organisation should not simply delete the audit report.

A proper remediation process should include:

Step 1 — Stop or restrict the problematic function

If the AI creates serious discriminatory outcomes, its use may need to be temporarily suspended.

Step 2 — Identify the source

Determine whether the problem comes from:

• training data;
• model architecture;
• proxy variable;
• threshold;
• human interaction;
• vendor system.

Step 3 — Correct the dataset

Remove or correct unreliable and discriminatory training examples.

Step 4 — Re-test

The modified system should undergo new discrimination testing.

Step 5 — Independent validation

Where the decision is high-risk, an independent auditor should review the system.

Step 6 — Document everything

The organisation should maintain:

• audit reports;
• testing methodology;
• datasets;
• model versions;
• corrective measures;
• management approvals;
• complaints;
• remediation results.

27. Importance of Audit Trails

An AI discrimination audit should maintain an audit trail showing:

Input → AI processing → score → recommendation → human review → final decision.

For example:

Candidate applies

↓

AI screens CV

↓

AI gives score 42

↓

Candidate rejected

↓

HR reviews decision

↓

HR confirms rejection

This record allows an organisation to determine where discrimination entered the process.

Without an audit trail, proving why an AI decision occurred can be extremely difficult.

28. Key Principle: “Black Box” Is Not a Complete Defence

An organisation should not simply say:

“We don't know why the AI rejected the candidate because it is a black-box model.”

That is a major governance problem.

If an organisation uses AI for consequential employment decisions, it should establish sufficient controls to understand:

• what data is being used;
• what factors influence outcomes;
• whether discriminatory patterns exist;
• who is responsible;
• how decisions can be challenged.

The modern AI-discrimination debate therefore shifts the focus from merely asking:

“Is the algorithm accurate?”

to also asking:

“Is the algorithm fair, lawful, explainable and appropriately governed?”

29. Indian Legal Context

In India, AI discrimination in employment must be considered against broader constitutional and employment-law principles.

Important constitutional concepts include:

Article 14

Guarantees equality before law and equal protection of laws.

Article 15

Prohibits discrimination on specified grounds in the constitutional context.

Article 16

Provides equality of opportunity in public employment.

Article 21

Protects life and personal liberty and has been interpreted broadly to include dignity, privacy and procedural fairness.

Therefore, where AI is used by government/public authorities or public-sector employers, constitutional equality principles can become particularly important.

For private employers, the legal analysis additionally depends upon applicable labour, employment, contract, disability, privacy/data-protection and other statutory obligations.

30. Conclusion

Auditing AI systems for discrimination is not simply a technical exercise. It is a combined legal, statistical, technological and governance process.

A proper audit should examine the entire AI lifecycle:

Data → Model → Proxy Variables → Algorithmic Output → Human Decision → Employment Outcome → Monitoring

The most important lesson from cases such as iTutorGroup, Mobley, Liu, Griggs, Watson, Ricci, and the Amazon recruitment example is that a decision does not become non-discriminatory merely because it is made or assisted by a computer.

AI can reproduce discrimination that already exists in historical data, create new forms of proxy discrimination, or amplify apparently neutral employment criteria. The EEOC has expressly identified these risks across recruitment, screening, advertising, interviewing and workplace management.

Therefore, organisations should implement pre-deployment testing, periodic independent audits, statistical disparity analysis, accessibility testing, explainability controls, human oversight, vendor due diligence and continuous monitoring.

The central principle can be stated simply:

“Automating a discriminatory decision does not make the decision lawful; an effective AI audit must examine both the technology and the employment consequences it produces.”