Where risks come from

⚡ TL;DR

What is a risk? gave you the anatomy — likelihood × impact, appetite, residual, the gate. This page is the step everyone skips: where do the risks actually come from? A risk is not a bug — no failing test, no crash. It is a harm the system can cause while doing exactly what it was built to do, because of its data, its design, or how people use it. The EU AI Act says exactly where to look. Article 9(2) requires you to identify the known and reasonably foreseeable risks the system poses to health, safety and fundamental rights — (a) under its intended purpose, (b) under reasonably foreseeable misuse, and (c) from post-market reality. And the sources sit inside the system: the data (Art. 10), the model (Art. 15) and the human-AI interaction (Art. 14). The four cases on the home page were all born in the data. Identifying that origin is the first step of the loop — before you ever measure a likelihood.

🎯 By the end you'll be able to…

• Name the three origins of risk in Article 9(2): the intended purpose (a), reasonably foreseeable misuse (b, defined in Art. 3(13)), and post-market reality (c, Art. 72).
• Point to the sources inside the system the Act singles out — data (Art. 10), model (Art. 15), human oversight (Art. 14) — and recognise that most real-world harm is born in the data.
• Say what the harm is to — health, safety, fundamental rights — and map that honestly onto the engine’s individual / society / organization impact axes (and where they do not line up).
• Read where this lives in froga.yaml: intended_purpose, potential_misuses, each risk’s domain, and the impact axes — so “identifying a risk” stops being a vibe and becomes a declaration.
• Explain why post-market risk (Art. 72) is the honest GAP: this engine governs ex-ante; runtime monitoring is a future line, not a current capability.

The three origins — Article 9(2)

A risk-management system, says Article 9(2), must identify “the known and the reasonably foreseeable risks that the high-risk AI system can pose to health, safety or fundamental rights” — and it names three places those risks are born:

• (a) The intended purpose. The risk is born in the use itself. A system that scores creditworthiness, screens job applicants, or grades exams carries risk because of what it decides about people — not because the code is faulty. The intended purpose (and the Annex III use-case it falls under) is the first thing you declare, because it sets what is at stake.
• (b) Reasonably foreseeable misuse. Art. 9(2)(b) extends the duty to risks under “reasonably foreseeable misuse” — defined in Art. 3(13) as use not in line with the intended purpose but which can be expected from normal human behaviour. The recruiter who treats a screening score as the final decision; the operator who runs the model on a population it was never validated for. Foreseeable, so it is your risk to anticipate.
• (c) Post-market reality. Art. 9(2)(c) adds risks that emerge once deployed, from the post-market monitoring of Art. 72 — drift, new conditions, harms that only show at scale.

A risk is not a bug

This is the mental shift. A bug is the system doing something it was not built to do. A risk is the system doing exactly what it was built to do — and causing harm anyway, because of its data, its design, or a foreseeable misuse nobody anticipated. That is why a green test suite is not enough for high-risk AI: the tests check does it run as specified?; risk asks what could it get wrong, for whom, and how would we know? The origin is the answer to that question.

The sources inside the system

Origins tell you when risk enters; these tell you where in the system to look. The Act points at three:

• The data — Art. 10 (data governance). The biggest source by far. Biased, incomplete or unrepresentative training data produces discrimination that no functional test detects. The Act requires examining datasets for biases (Art. 10(2)(f)–(g)). Every one of the four cases on the why-it-matters page was born here: the UK A-level grades, Austria’s AMS scores and France’s CNAF profiles all encoded inequality already present in the data. You meet this origin head-on in the data gap.
• The model — Art. 15 (accuracy, robustness, cybersecurity). A model that is inaccurate, brittle on edge cases, or attackable is a source of risk on its own — independent of the data.
• The human-AI interaction — Art. 14 (human oversight). Risk also lives in how the system is used: no meaningful human review, automation bias, an operator who cannot override. The control may be perfect and the risk still real if a person rubber-stamps every output.

What the harm is to

Article 9(2)(a) is explicit about whom the risk threatens: health, safety, or fundamental rights. That is the test for whether something is a risk worth managing — not “is it annoying for the business”, but “could it hurt a person, their safety, or their rights?”

An honest note on the impact axes

The engine scores impact on three axes — individual, society, organization. Map them carefully: the AI Act’s fundamental-rights and health/safety harms live in individual (and, for collective effects, society). organization is the business dimension from ISO 23894 (reputational, financial risk to the provider) — useful for prioritisation, but not an AI Act harm category. So when you read a high organization impact, that is risk to you, not the regulated harm to people. Keep the two straight: the law cares about persons and rights; the org axis is your own skin in the game.

How FROGA makes the origin explicit

The point of naming the origin is that it becomes a declaration in froga.yaml, not a hand-wave:

• intended_purpose (plus classification.basis) — the Art. 9(2)(a) origin: what the system is for, and the Annex III basis that makes it high-risk.
• potential_misuses (each with an addressed_by pointing at a risk) — the Art. 9(2)(b) / Art. 3(13) origin: the foreseeable misuses you anticipated, and which risk treats each. A misuse with addressed_by: [] is an honest open item, not a hidden one.
• Each risk’s domain — data, safety, governance, transparency — names the source of that specific risk (data / model / oversight / disclosure), so the dossier shows where each risk was born.
• impact: {individual, society, organization} — the harm dimensions above.

So “identifying a risk” — the first step of the loop in What is a risk? — is concretely: declare the purpose, list the foreseeable misuses, and for each risk name its source domain and who it can harm. That declaration is what the rest of the engine then measures, treats and signs.

The honest gap: post-market (Art. 72)

Of the three origins, this engine fully serves two. The intended-purpose and foreseeable-misuse origins are ex-ante — declared and governed before the system ships, which is exactly what froga does. The post-market origin (Art. 9(2)(c), Art. 72) is ex-post: it needs runtime telemetry and incident monitoring from the deployed system. That is a future line, not a current capability — the course teaches ex-ante risk identification, and is honest that continuous post-market monitoring lives beyond today’s engine.

Self-check

❓ According to Article 9(2), what are the three origins of an AI system's risks?

(a) The intended purpose — risks the system poses when used as intended (and within its Annex III use-case); (b) reasonably foreseeable misuse — risks from use that is not intended but expectable from normal human behaviour (defined in Art. 3(13)); and (c) post-market reality — risks that emerge once deployed, surfaced by the post-market monitoring of Art. 72. In all three, the risks that matter are those to health, safety or fundamental rights.

❓ Where inside the system does the EU AI Act tell you to look for the source of a risk — and which source produced the four cases on the home page?

Three places: the data (Art. 10 — bias, gaps, non-representativeness), the model (Art. 15 — accuracy, robustness, cybersecurity), and the human-AI interaction (Art. 14 — insufficient oversight). The four European cases (UK A-levels, Austria AMS, France CNAF, the Dutch Toeslagenaffaire) were all born in the data: each one encoded inequality already present in its training data, and did exactly what that data taught it — no bug, a risk nobody measured.

❓ The engine scores impact on individual / society / organization. Which of these is NOT an EU AI Act harm category, and why does that matter?

organization is not an AI Act harm category. The Act protects health, safety and fundamental rights — harms to people — which the engine carries on the individual and society axes. organization is the business risk to the provider (reputational, financial), from ISO 23894; it is useful for prioritising work but is not what the law regulates. Confusing “this is bad for us” with “this harms a person’s rights” is exactly the mix-up the impact axes are designed to keep apart.

Where to go next

What is a risk?The anatomy this page builds on: likelihood × impact, the 5×5 matrix, appetite, inherent vs residual, and the RED/AMBER/GREEN gate. Origin first, then measurement.

The data gap (Art. 10)The most common origin, made concrete: when the risk is born in the data and no code change can close it — the sufficiency gate that stays RED until you acquire more representative data.

When AI decides about peopleThe four European systems whose risks were all born in the data — A-levels, AMS, CNAF, the Toeslagenaffaire — and what it cost when nobody identified the origin before shipping.

EU AI Act — the spineArticle 9 (risk management) in context, alongside Art. 10 (data), Art. 14 (oversight) and Art. 15 (robustness) — the articles that name where risk comes from and what to do about it.