1. State v. Loomis (Wisconsin Supreme Court, 2016) — Algorithmic risk scoring in sentencing

Facts

• The defendant was sentenced partly using the COMPAS algorithm, a proprietary risk assessment tool.
• COMPAS predicted likelihood of reoffending using statistical modeling.
• The defendant argued that the algorithm was:
  • Secret (“black box”)
  • Biased against racial groups
  • Not explainable or challengeable in court

Legal Issue

Whether reliance on a proprietary algorithm violates due process rights.

Judgment

The court allowed use of COMPAS but imposed safeguards:

• Judges must be warned that algorithmic tools are not determinative
• Risk scores cannot be the sole basis for sentencing
• The tool’s limitations must be disclosed

Importance for Medical AI

Medical systems often use similar “black box” tools for:

• ICU risk prediction
• Sepsis detection
• Insurance risk scoring

Key legal principle:

If an AI system affects rights or liberty (or healthcare access), lack of transparency can raise due process concerns.

2. Bridges v. South Wales Police (England & Wales Court of Appeal, 2020) — Facial recognition and bias

Facts

• Police used automated facial recognition (AFR) in public spaces.
• The system flagged individuals as potential suspects.
• Concerns arose about:
  • False positives
  • Racial and gender bias
  • Lack of clear legal safeguards

Legal Issues

• Whether AFR deployment violated:
  • Privacy rights
  • Equality principles
  • Data protection standards

Judgment

The court ruled the system was unlawful in its deployment at that time, mainly because:

• No proper equality impact assessment was conducted
• The system lacked adequate safeguards against bias
• Police discretion was too broad

Medical AI relevance

Similar risks arise in:

• AI diagnostic imaging tools
• Patient triage systems
• Automated mental health screening

Key principle:

High-impact AI systems must undergo structured bias and equality impact assessments before deployment.

3. EEOC v. iTutorGroup (United States, 2023 settlement) — AI discrimination in hiring

Facts

• An online education company used AI to screen job applicants.
• The system automatically rejected:
  • Women above a certain age
  • Men above a different age threshold
• Applicants were not informed about algorithmic filtering.

Legal Issue

Violation of the Age Discrimination in Employment Act (ADEA).

Outcome

• The company settled with monetary compensation.
• It agreed to stop discriminatory AI screening practices.

Medical AI relevance

Many healthcare systems use AI for:

• Nurse recruitment
• Doctor credential screening
• Patient eligibility for treatments

Key principle:

If AI replicates or amplifies protected-class discrimination (age, gender, race), the employer/organization is still legally responsible.

4. Mobley v. Workday (United States District Court, ongoing since 2023–2025) — AI hiring bias litigation

Facts

• Workday provides AI-powered hiring tools used by many employers.
• Plaintiff alleges that the AI system:
  • Disproportionately rejected applicants based on race, age, and disability
  • Used historical biased training data
• The case claims the AI acted as a “gatekeeper” decision-maker

Legal Issue

Whether AI vendors (not just employers) can be liable for discrimination.

Key Arguments

• Plaintiff: AI is an “agent” making employment decisions
• Defendant: AI is only a tool used by employers

Importance (Still developing law)

Courts are actively examining:

• Whether AI vendors can share liability
• How “intent” applies when bias is statistical, not human

Medical AI relevance

This directly impacts:

• AI diagnostic vendors
• Hospital decision-support software companies
• Insurance AI systems

Key principle emerging:

Liability may extend beyond users to developers of biased AI systems.

5. Griggs v. Duke Power Co. (U.S. Supreme Court, 1971) — Disparate impact doctrine (foundational)

Facts

• Employer required a high school diploma for certain jobs.
• Requirement disproportionately excluded Black applicants.
• No proof the requirement was necessary for job performance.

Legal Issue

Whether neutral policies that produce unequal outcomes are discriminatory.

Judgment

The Supreme Court ruled:

• Even neutral rules are illegal if they have unjustified discriminatory effects
• Introduced the “disparate impact” doctrine

Why this matters for medical AI

This is the core legal framework used today for AI bias cases.

AI systems may:

• Use neutral variables (ZIP code, health spending, prior diagnoses)
• But still discriminate against racial or low-income groups

Key principle:

Discrimination can exist without intent—impact alone can be unlawful.

6. Healthcare-specific AI bias principle (derived from Obermeyer v. UC Berkeley research, 2019 — influential, not a court case)

Although not a court ruling, it heavily influences litigation and policy.

Finding

• A widely used U.S. healthcare algorithm underestimated the health needs of Black patients.
• It used “health cost” as a proxy for “health need.”
• Because Black patients historically spent less on healthcare due to access inequality, the algorithm systematically underestimated their illness severity.

Legal implication

This kind of reasoning is now used in lawsuits arguing:

• AI in healthcare may violate anti-discrimination laws if proxies encode structural inequality

Key principle:

Using biased proxies (like cost instead of illness severity) can create unlawful discriminatory outcomes in healthcare systems.

Overall Legal Principles from These Cases

Across jurisdictions, courts and regulators are converging on a few key rules:

1. Accountability principle

Even if AI makes the decision, humans and organizations remain legally responsible.

2. Disparate impact applies to AI

If AI disproportionately harms protected groups (race, gender, age, disability), it can be illegal even without intent.

3. Transparency requirement

Black-box systems used in high-stakes decisions face increasing legal pressure.

4. Audit and fairness obligation

Organizations must test AI systems for bias before deployment in sensitive fields like healthcare.

5. Vendor liability expansion

AI developers may also face liability, not just users.