When AI decides about people

⚡ TL;DR

You are about to ship AI that decides about people — who gets a loan, a diagnosis, a job, a school place. When those systems fail quietly — biased, opaque, wrong in a way no functional test catches — the harm is real, and so is the bill: under the EU AI Act, fines reach €35 M or 7% of worldwide turnover, and a high-risk system with no signed technical file simply cannot be placed on the EU market — no CE marking, no sale, and out of every public tender that requires it. This page is the why before the how: four European systems that hurt real people because a risk was never treated, what non-compliance costs, who is on the hook, and the clock that is already ticking. The punchline is the whole product: compliance you can only get by designing it in — like tests, like security — never by documenting it after the fact.

1 · Why the law exists — four European systems that did exactly what they were told

The EU AI Act did not appear from nowhere. It exists because AI systems have already caused concrete, documented harm in Europe — not through malice, but because a risk was never identified and treated before the system was deployed. The model did exactly what its data taught it, and that was the problem.

• Education — the UK A-level algorithm (2020). With exams cancelled by COVID, an algorithm set the grades and downgraded around 39% of A-level results, hitting students from disadvantaged state schools hardest while lifting private schools — it simply reproduced the inequalities already baked into the data. The public outcry forced one of the biggest U-turns in UK education, back to teacher-assessed grades. The model encoded the past, and the past was unequal. → you meet this risk in Level 2 · student selection.
• Employment — Austria’s AMS algorithm. The public employment service scored each job-seeker’s chance of finding work — and being female lowered your score (so did disability and care duties), routing exactly those people away from the best support. Austria’s data-protection authority banned it in 2020. (A federal court overturned the ban in 2025; the legal dispute is still ongoing.) Existing labour-market inequality, encoded and amplified. → you meet this risk in Level 5 · talent screening.
• Public benefits — the Dutch childcare-benefits scandal (Toeslagenaffaire). The tax authority used risk profiling to flag families for benefits fraud. It was wrong at scale: tens of thousands of families (around 26,000) were falsely accused, ordered to repay benefits, and financially ruined; the fallout brought down the Dutch government in January 2021. (The related fraud-detection system SyRI was ruled unlawful by a Dutch court in 2020 for breaching human rights.) An untreated risk, deployed at national scale.
• Benefits & fairness — France’s CNAF scoring. The family-benefits fund scores ~32 million people for fraud risk from personal traits — a low income, unemployment, a disability allowance or single-parent status all raise your suspicion score. In 2024, 15 organisations (La Quadrature du Net, Amnesty France) took it to the Conseil d’État, arguing it discriminates and breaches GDPR Art. 22 (automated decisions). The Netherlands’ Rotterdam ran a near-identical model whose source code listed even “well-groomed appearance” as a feature. An automated decision about your money, biased by a proxy nobody measured — the same question you meet in Level 4 · consumer credit.

One pattern under all four

None of these was a bug in the usual sense — no failing unit test, no crash. Each model passed its functional checks and still caused harm, because the harm lived in a risk nobody measured before shipping: a subgroup hurt, a proxy learned, a population the data under-represented. That is exactly why a green test suite is not enough for high-risk AI — and why risk, not the feature backlog, has to drive the work.

2 · Who carries the obligation

“High-risk AI” is not someone else’s problem. The AI Act assigns duties along the chain, and you are somewhere on it:

• The provider — whoever develops a high-risk system (or puts their name on it) — carries the heavy obligations: the Articles 9–15 risk, data, logging, transparency, oversight and robustness duties, the technical documentation (Annex IV), and the signed Declaration of Conformity.
• The deployer — whoever uses the system under their authority — has their own duties (human oversight, monitoring, sometimes a fundamental-rights impact assessment).
• Importers and distributors must check the file exists before placing it on the market.

If you build the model, you are most likely the provider — and the documentation, the risk management and the sign-off are yours, not your customer’s.

3 · What it costs to not comply

Two costs, and the second is the one that actually moves budgets.

• Fines (Art. 99). Up to €35 M or 7% of total worldwide annual turnover (whichever is higher) for prohibited practices; up to €15 M or 3% for breaching the high-risk obligations; up to €7.5 M or 1% for giving authorities incorrect information. This is stricter than the GDPR.
• Market access — the real lever. A high-risk system cannot be placed on the market or put into service in the EU without a conformity assessment, the Annex IV technical documentation, the EU Declaration of Conformity (Art. 47) and CE marking (Art. 48). Market-surveillance authorities can order withdrawal or recall (Arts. 79–83). And public buyers increasingly require AI Act conformity — so a non-conformant system is excluded from the tender before price is ever discussed. (That last one is exactly the procurement hand-off you build in Level 6.)

The business case in one line

The strongest argument is not the fine — it is the toll gate. Without the signed Annex IV and Declaration of Conformity, your high-risk system does not ship in the EU and does not win the bid. Compliance is not overhead on the product; for high-risk AI it is market access.

4 · Why now — the clock is already running

This is not a 2030 problem; it is your next system.

Date	What applies
1 Aug 2024	The AI Act enters into force.
2 Feb 2025	Prohibited practices (Art. 5) apply.
2 Aug 2025	Obligations for general-purpose AI models.
2 Aug 2026	Transparency duties (Art. 50).
2 Dec 2027	High-risk systems under Annex III (with the delay proposed in the Digital Omnibus; was 2 Aug 2026).

A high-risk system takes months to govern properly. Starting when the deadline lands means starting too late — which is the whole reason to fold the work into development now, on the system you are building this sprint.

The bridge: design it in, don’t document it after

Here is the trap the three stories share with every late compliance effort. You spend weeks assembling a risk dossier; you hand it in; the next day you retrain the model — and your dossier already describes a system that no longer exists. Post-hoc compliance is born obsolete.

The way out is the same move software already made for correctness (tests) and for safety (security reviews): you make it part of writing the code. Every risk treatment becomes a versioned, signed change, so the evidence always describes the system that actually exists — and catching a fairness regression is just catching a red test in CI, not rewriting a document by hand the night before an audit.

That is compliance by design, and it is what the rest of this site is about.

Compliance by designThe thesis applied: how each risk treatment becomes a signed commit, so the Annex IV is assembled by construction from evidence that is always current — not written by hand after the fact.

What is a risk?The five-minute vocabulary the whole course runs on: likelihood × impact, appetite, residual, and the RED/AMBER/GREEN gate — so 'a risk nobody treated' becomes something you can actually measure.

Quickstart — see it in 5 minutesHands on the keyboard: take a model that fails a gate (RED), fix it with one signed commit, and walk away with evidence a regulator would accept — the why of this page, made concrete.