Discrimination, artificial intelligence, and algorithmic decision-making

This report by Frederik Zuiderveen Borgesius (Council of Europe, Anti-discrimination department) addresses a central question for decision sciences and human-rights governance: how and where algorithmic decision-making and other AI systems can produce discriminatory effects, what legal safeguards currently exist to mitigate those risks, and what recommendations should be made to organizations deploying AI, equality bodies, and human-rights monitoring institutions.

The question matters because AI is increasingly used to make or support decisions with far-reaching consequences for individuals in both the public and private sectors. The report emphasizes that AI can appear neutral or rational, but can still generate unfair or illegal discrimination. It also highlights a compounding mechanism: AI systems can learn from biased human decisions, and many small discriminatory or exclusionary decisions—such as targeted advertising—can aggregate into large-scale social inequality.

The report’s significance lies in its cross-disciplinary framing. Rather than treating discrimination as only a legal concept, it connects discrimination pathways in machine learning to existing European non-discrimination and data protection regimes. It also argues that current law may not fully cover “new types of unfair differentiation” created by AI, especially when differentiation is based on invented categories that do not map neatly onto legally protected characteristics (e.g., race or gender). Consequently, the report positions AI discrimination as both a compliance problem (enforcing existing norms) and a regulatory design problem (potentially requiring sector-specific rules and new enforcement mechanisms).

Methodologically, the report is not an empirical study with a dataset or statistical analysis. It is a literature review and “quick scan” intended to map relevant issues and safeguards. The author synthesizes prior academic work (notably Barocas and Selbst’s framework for disparate impact in machine learning) and uses illustrative real-world cases (e.g., COMPAS in criminal justice; biased AI hiring systems; discriminatory ad targeting; online price discrimination; facial recognition errors; and gender bias in translation tools). The report’s “analysis technique” is therefore conceptual and legal-analytical: it organizes discrimination risks into mechanisms, then evaluates how legal regimes respond to those mechanisms.

Key findings are presented primarily as structured claims and concrete case-based evidence rather than as new quantitative results. The report identifies at least six pathways through which AI can lead to discrimination:

1) Defining the target variable and class labels in ways that embed social disadvantage (e.g., defining “good employee” using outcomes correlated with protected groups). 2) Biased training data, including when historical human decisions were discriminatory. 3) Biased data collection processes, including feedback loops (e.g., policing attention that increases recorded crime in certain neighborhoods). 4) Biased feature selection, where proxy variables for protected characteristics are chosen because they are predictive. 5) Proxy effects and “redundant encodings,” where protected-class membership is encoded in other variables; the report notes that one approach to avoid systematic disadvantage can reduce overall accuracy. 6) Intentional discrimination, where proxies are selected with discriminatory intent.

The report then provides field examples to show how these mechanisms manifest:

• Criminal justice and risk assessment: The COMPAS example is used to illustrate disparate error patterns. The report cites ProPublica’s findings that COMPAS “correctly predicts recidivism 61 percent of the time,” while Black defendants were “almost twice as likely as whites to be labelled a higher risk but not actually reoffend,” and that Black defendants were “twice as likely” to be misclassified as higher risk of violent recidivism. It also discusses the calibration and fairness trade-offs, noting that equalizing error rates can be mathematically impossible when base rates differ.

• Employment and education: The report describes a UK medical school admissions example where an AI system trained on historical admissions decisions reproduced bias against women and people with an immigrant background. It also references Amazon’s decision to stop using an AI recruiting tool after it was found to be biased against women.

• Advertising: The report cites evidence that online ad systems can encode discrimination through targeting and exclusion. Examples include:

• Google ad delivery differences by race-associated names (Sweeney, 2013).
• Simulated-user experiments suggesting gendered ad delivery for high-paying jobs (Datta, Tschantz, Datta, 2015).

• Facebook ad targeting/exclusion by race and other protected or sensitive “ethnic affinities” (ProPublica reporting), and the Dutch Data Protection Authority’s findings that Facebook enabled advertisers to target based on sensitive characteristics.

• Online price discrimination: The report describes Princeton Review’s differentiated pricing across US areas, citing that customers in areas with high density of Asian residents were “1.8 times as likely to be offered higher prices, regardless of income” (Angwin et al., 2015).

• Image search and recognition: The report cites examples where image search results reflect societal stereotypes (e.g., “three black teenagers” vs “three white kids”) and where face recognition systems show higher error rates for darker-skinned groups. It specifically cites Buolamwini and Gebru’s finding that “darker-skinned females are the most misclassified group (with error rates of up to 34.7%),” while “the maximum error rate for lighter-skinned males is 0.8%.”

• Translation tools: The report cites research showing gender bias in machine translation, including that gender-neutral source sentences can produce gendered defaults when translated back into English, and that male defaults are “strong” and “exaggerated” for stereotype-associated fields such as STEM.

Limitations are acknowledged in the report’s framing: it is a “quick scan” constrained by length, based on literature review rather than original empirical measurement. It also explicitly limits scope: it focuses on discrimination risks and does not deeply cover privacy-related questions, automated weapons, unemployment, filter bubbles, or data monopolies. Additionally, the report notes that it is “too early” to assess the practical effect of newer legal provisions (e.g., GDPR automated decision rules) and that enforcement and compliance deficits remain.

Practical implications follow from the report’s legal and governance analysis. The author argues that the most relevant legal tools are non-discrimination law and data protection law, and that effective enforcement could mitigate many AI discrimination risks. However, the report stresses that enforcement is difficult due to (i) AI opacity (“black box” effects), (ii) evidentiary burdens for proving indirect discrimination, (iii) limited resources and enforcement powers of authorities, and (iv) gaps where AI differentiates using invented categories not covered by protected-characteristic statutes.

Accordingly, the report recommends concrete organizational steps: education to reduce automation bias and awareness of discrimination risks; risk assessment and mitigation involving multidisciplinary teams; documenting and monitoring AI projects; and considering impact assessments inspired by GDPR DPIAs. It also recommends transparency and auditability in the public sector, potentially controlled access for researchers, and “sunset clauses” for public AI systems.

For equality bodies and human-rights monitoring bodies, the report emphasizes building AI technical capacity (including hiring or consulting computer scientists), conducting public awareness campaigns, engaging in early consultation for public procurement, cooperating with data protection authorities, and possibly commissioning or supporting research and strategic litigation.

Finally, the report’s regulatory recommendations argue against a single one-size-fits-all AI law. Instead, it suggests sector-specific regulation and a multi-level approach combining broad principles with faster-updating guidelines. It also calls for improved enforcement of existing norms via transparency requirements, auditing capabilities, and adequate investigative powers.

Who should care? The report is aimed at policymakers and regulators (especially equality bodies and data protection authorities), organizations deploying AI in high-stakes domains (public agencies, banks, employers, insurers, and platforms), and researchers and civil society groups that investigate discrimination. For decision sciences practitioners, the report provides a structured map from machine-learning design choices to legal risk, reinforcing that fairness is not only a technical property but also a governance and accountability requirement.

Overall, the core contribution is a synthesized, mechanism-based account of AI discrimination risks tied to the practical reach and limits of non-discrimination and data protection law, culminating in recommendations for enforcement, organizational governance, and sector-specific regulatory evolution.