Research On Criminal Evidence Standards When Using Ai-Derived Inferences In Prosecutions
Key Legal / Evidentiary Standards When Using AI‑Derived Inferences in Criminal Cases
Before diving into cases, it's helpful to outline what legal / evidentiary standards are invoked (or ought to be) when AI inferences are used in prosecutions or sentencing:
Due Process / Procedural Fairness
The defendant must have a meaningful opportunity to challenge evidence. If an algorithm’s “black box” nature prevents an effective challenge, that raises due process concerns.
Related to this is the right to be sentenced on accurate information.
Individualized Sentencing
Sentencing should reflect the individual, not just statistical generalizations. If a tool relies heavily on group-level data, the defendant may argue that it undermines their right to be treated individually.
Transparency & Explainability
Courts may require “warnings” or disclaimers about limitations, especially if the algorithm is proprietary.
There is tension between trade secret (or private) algorithms and the defendant’s right to understand how “risk” was computed.
Reliability / Validation
The AI tool should be validated for the population to which it’s applied; if not, its predictions may not be reliable.
There may be need for empirical validation, error rates, and periodic re‑validation.
Weight & Use
Even assuming reliability, how heavily can a court rely on the AI inference? Should it be a determinative factor or merely a tool among many?
Risk assessment tools are often one factor; courts must resist over-reliance (“automation bias”).
Right to Rebut
The defense should have access to underlying data or at least output so that it can rebut or contextualize the AI inference.
Bias / Fairness
AI may embed or exacerbate bias (e.g., racial, gender). Courts may need to ensure that such risk assessment tools are fair and do not produce discriminatory outcomes.
Adversarial Scrutiny / Audit
Courts may allow or require adversarial testing of the algorithm (e.g., independent experts probing its behavior). Some scholars propose “robust adversarial testing.”
Important Cases (and Legal Precedents)
Here are five (plus) leading / illustrative cases and how they have addressed (or failed to fully address) these issues. I focus especially on risk-assessment algorithms because that is where most litigation has occurred so far.
State v. Loomis (Wisconsin, 2016)
Facts: Eric Loomis pleaded guilty to lesser crimes (after a drive-by shooting). At sentencing, the court used COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), a proprietary risk-assessment algorithm, which classified him as “high risk” for recidivism.
Claims: Loomis argued that using COMPAS violated his due process rights because he could not challenge the scientific validity of the algorithm (it was trade-secret), and because the tool used group-level data (statistical generalizations) rather than purely individual data; he also argued that gender was improperly factored in.
Court’s Decision (Wisconsin Supreme Court): The court held that consideration of COMPAS did not, per se, violate due process — if used with awareness of its limitations.
It required certain caveats / warnings in the pre-sentence report: for example, noting that the risk tool may not have been validated in the defendant's specific population, and cautioning about racial bias.
Crucially, the court emphasized that COMPAS should not be determinative: it was one tool among multiple sentencing factors, not the only factor.
Criticism / Analysis:
Scholars argue (and the court itself acknowledges) that the risk of automation bias is real — judges may give undue weight to the algorithmic score.
Some (in concurrence) warned: “the court may consider risk assessments, but it may not rely on them” as the sole basis for sentencing.
Significance: This is perhaps the leading U.S. case on the constitutional limits of algorithmic risk assessment in sentencing. It shows a balancing approach: permitting the tool, but not uncritically accepting it, and requiring transparency and caveats.
Malenchik v. State (Indiana, 2010)
Facts: Anthony Malenchik was convicted of receiving stolen property. Before sentencing, a probation department presentence report included a risk-and-needs assessment (using LSI‑R, the Level of Service Inventory–Revised), a third‑generation risk-assessment instrument.
Issues: Whether a trial court may consider algorithmic risk assessment in deciding a sentence, and how much weight to give it.
Court’s Decision (Indiana Supreme Court):
The court allowed the use of the LSI‑R assessment tool.
Importantly, the court held that these scores supplement — not replace — judicial discretion. The judge must still craft an individualized sentence, and risk assessment does not automatically dictate the severity or length.
On the defense’s argument about reliability (e.g., under an expert-evidence standard), the court noted that in Indiana, the rules of evidence (like Rule 702) do not apply at sentencing.
Significance:
This case illustrates that risk‑assessment tools can be constitutionally and legally permissible at sentencing — but their role is advisory.
It also shows a limitation: defense may have limited ability to challenge the underlying tool when standard evidentiary rules (like expert testimony) are not required in sentencing.
State v. Walls (Kansas, 2017)
Facts: A defendant (Walls) was assessed by a risk-needs tool, which classified him as a “high-risk, high-needs probation candidate.” The sentencing court relied in part on that risk-assessment report but denied the defense access to the underlying questionnaire, scoring, and report.
Issues:
Whether the defense has a due process right to inspect the risk-assessment report (including the questions, the answers, and how the score was computed) when the court relies on that report to make sentencing or probation decisions.
Court’s Decision (Kansas Court of Appeals):
The court held in favor of Walls: denying access to the report “necessarily denied him the opportunity to challenge the accuracy of the information upon which the court relied.”
The court said that to deprive him of such access violated his right to procedural due process.
Significance:
This is a strong precedent that when algorithmic assessments are used in sentencing / probation decisions, defendants may demand transparency and the right to rebut.
The case underscores that black-box risk tools cannot always remain opaque if they influence legal decisions.
State v. Rogers (West Virginia, 2015)
Facts: In this case, a defendant moved for reconsideration of his sentence because the court had not used a risk-assessment instrument in his sentencing.
Issues:
Whether there is a legal (or constitutional) obligation on a sentencing court to use a risk-assessment tool (i.e., whether absence of such an assessment justifies revisiting the sentence).
Court’s Decision (West Virginia):
The court denied the motion (on procedural grounds), but a concurring opinion clarified important limits: risk assessment tools are optional, not mandatory. Trial courts are not required to use them; they are merely one tool among many
Significance:
This helps define the proper role of AI-derived risk-assessment in sentencing: it shouldn’t be seen as necessary, but as an instrument that may assist judges.
It also provides a check against overdependence: judges are not forced to outsource decision-making to algorithms.
Other Legal / Scholarly Developments / Proposals
While not “cases” in the sense of appeals court decisions, some scholarly and policy developments shed important light:
Adversarial Scrutiny / Audit Framework: Scholars like Rediet Abebe, Moritz Hardt, Rebecca Wexler, and others have argued that we need robust adversarial testing of statistical software used in criminal settings (probabilistic genotyping, risk assessment, etc.) so defense counsel can meaningfully challenge output.
Judicial Resistance: Research by Dasha Pruss shows that even when risk-assessment instruments are introduced, judges may resist using them — calling them “useless,” “not helpful,” etc.
Interpretability & Fair Machine Learning: Work by Caroline Wang, Cynthia Rudin, and others shows that interpretable ML models (not black‑box) can predict recidivism with comparable accuracy and greater fairness / transparency.
Normative Critiques: Philosophers and technologists (e.g., in Philosophy & Technology) argue that predictive algorithms at sentencing risk flattening a person’s life into statistical probabilities, undermining the individual nature of justice and embedding risk management as a dominant mode of governance.
Analysis: Challenges & Tensions
Based on the cases and scholarship, here are some of the major tensions and challenges when using AI‑derived inferences in criminal prosecutions / sentencing:
Transparency vs Trade Secrets
Proprietary algorithms (like COMPAS) may resist full disclosure, but defendants need enough information to meaningfully challenge their use.
Courts, as in Loomis, offer a partial solution via required disclaimers, but critics say that is not sufficient.
Reliability & Validation
Even if a tool is validated in one jurisdiction, it may not generalize to another — leading to “zombie predictions.”
Periodic re‑validation and ongoing accuracy checks are often missing.
Weight & Over-Reliance
There is a risk that judges will lean too heavily on algorithmic output, treating it as objective “truth” rather than just one input.
Cognitive biases like anchoring (relying too much on first piece of info) can exacerbate this.
Due Process / Rebuttal Rights
If defense cannot inspect or cross-examine the algorithm’s logic, error rates, or data, its role in sentencing is suspect.
Cases like Walls show that courts may enforce a right to inspect the report, but the exact scope (e.g., access to underlying code) remains debated.
Fairness / Bias
Since many risk tools are trained on historical criminal justice data, they may replicate systemic biases (racial, socioeconomic).
Without transparency and fairness audits, such tools may perpetuate injustice under the guise of “data-driven” decisions.
Institutional Resistance
As research shows, some judges are skeptical or unwilling to use these tools, which may limit their influence or lead to inconsistent application. arXiv
On the other hand, some jurisdictions may feel pressure to adopt risk-assessment tools for efficiency, but without adequate safeguards.
Regulatory / Normative Gap
There is a lack of standardized rules (legislative or judicial) about how AI-derived inferences should be treated.
Although some judicial bodies are starting to consider rules, as of now, much depends on individual cases and local practices.
Conclusion & Recommendations
Legal Standard: Courts should treat AI-derived inferences (such as risk scores) as probabilistic tools, not deterministic machines. They must ensure due process by requiring transparency, opportunities to rebut, and clear limits on the role of such tools.
Best Practices for Courts / Legislatures:
Disclosure Regimes: Mandate enough disclosure of how the tool works (at least input variables, validation, error rates) to enable challenge.
Validation & Revalidation: Require independent validation studies on relevant populations and periodic reassessment to ensure continued accuracy.
Weight Guidelines: Issue judicial guidelines on how much weight to give algorithmic inferences; explicitly caution against overreliance.
Defense Rights: Ensure defendants (through counsel) can access the risk tool’s output and relevant data to cross-examine, challenge, or contextualize.
Auditing & Oversight: Set up mechanisms (possibly adversarial testing) by which the tools are regularly audited by external experts.
Bias Mitigation: Require fairness audits; integrate interpretability in AI tool design; consider using transparent/interpretable models.
Training for Judges: Judges should be trained in the limitations of AI, including cognitive biases like anchoring, so they use risk tools judiciously.
Future Directions:
As AI evolves (e.g., generative AI, deepfakes), courts will need to develop new admissibility standards for synthetic evidence (not just risk-assessment).
There is a growing scholarly consensus (and some policy movement) toward constitutionalizing algorithmic due process: embedding principles like accountability, transparency, and proportionality into law.
RELATED Blog
comments