What is a risk?

⚡ TL;DR

A risk is the chance that something goes wrong, weighed by how bad it would be: likelihood × impact. You put it on a 5×5 grid to get a risk level, decide up front how much risk you are willing to live with (your appetite), and then treat it — change something — until the leftover (residual) risk drops to or below that line. The whole course turns on this one loop, and our first example involves nothing more dangerous than confusing two species of flower.

🎯 By the end you'll be able to…

• Define a risk as likelihood × impact, and say why impact is read on three axes (individual / society / organization).
• Read a 5×5 matrix and name a risk’s level (low / medium / high / critical).
• Explain appetite as the acceptance line you draw before you measure anything.
• Tell inherent risk apart from residual risk, and explain why we confirm the residual with a control, not just a promise (the hybrid residual).
• Walk the risk-management loop (identify → analyze → evaluate → treat → residual → monitor) and read the gate colours RED / AMBER / GREEN.
• Apply all of it to one worked example — the Iris species-confusion risk — that you will run hands-on in Level 1 · Iris.

A risk is likelihood times impact

Start with the idea, in one sentence: a risk is the chance that something goes wrong, multiplied by how much it would hurt if it did. Two ingredients, always:

• Likelihood — how probable is the bad outcome? (rare … almost certain)
• Impact — if it happens, how bad is it? (negligible … catastrophic)

A lottery-ticket loss is high-likelihood but tiny-impact; a meteor strike is the opposite. Neither is “a big risk” on its own — you only know once you put the two together. That is why we multiply, conceptually, rather than look at either number alone.

This is the oldest idea in the field, and it is exactly the one the international standard adopts. ISO 31000 — the parent vocabulary for managing risk of any kind — defines risk as the “effect of uncertainty on objectives,” and ISO/IEC 23894 (its 2023 AI-specific sibling, the standard this whole course speaks) carries that vocabulary into machine-learning systems. When we later write iso/23894@2023#6.4 we mean clause 6.4 of that 2023 standard — the part about analyzing risk. We’ll gloss every such citation the first time it appears; the idea always comes first, the reference second.

Why a standard at all, for flowers?

You do not need a law, or even a customer, to benefit from this. Managing risk is good engineering: it is how you find out, on purpose, where your system is weak — before a user does. Our first example (classifying iris flowers) is not covered by any regulation imaginable. We still apply ISO 23894 to it, because the method pays off regardless of whether anyone forces you. That is the whole point of starting with flowers.

Impact is multidimensional

“How bad is it?” begs the question: bad for whom? A single number hides the answer. So we read impact on three axes:

• Individual — harm to one person (a wrongly rejected loan applicant, a missed diagnosis).
• Society — harm to a group or to the public (a model that quietly discriminates against a whole subgroup).
• Organization — harm to the people who build or run the system (a recall, a fine, a reputational hit).

The same failure can score very differently on each axis. A flower classifier that mixes up two species harms no individual and poses no danger to society — at most it is an organizational annoyance (a botanist’s catalogue is wrong). Naming the axes keeps you honest about which kind of harm you are actually managing. ISO 23894 (clause 6.4, “risk analysis”) asks for exactly this multi-dimensional read rather than a single lumped score.

The 5×5 matrix and a risk’s level

Here is the intuition before the formalism: take a 5-step scale for likelihood (rare → unlikely → possible → likely → almost certain) and a 5-step scale for impact (negligible → minor → moderate → major → severe), and lay them on a grid. Where a risk’s likelihood row meets its impact column, the cell’s colour is its level: low, medium, high, or critical.

flowchart LR
  I["Identify<br/>species confusion"] --> A["Analyze + Evaluate<br/>Likely × Moderate = HIGH<br/>(above appetite)"]
  A --> T["Treat<br/>add petal features"]
  T --> R["Residual<br/>Unlikely × Moderate = MEDIUM<br/>(≤ appetite)"]
  R --> P["Approve<br/>James signs"]

The diagram above is the loop you will walk in a moment, applied to our flower example. Read it left to right: we identify a risk (the model confuses two species), place it on the matrix and get HIGH (likely to happen, moderate impact), treat it (give the model better features), re-measure to get a MEDIUM residual that is now acceptable, and only then approve. The matrix is just the bookkeeping that turns two vague words (“likely,” “moderate”) into one comparable number — a level you can hold against a line.

That line is the next idea.

Appetite: the line you draw before you measure

Appetite is how much risk you are willing to accept — decided in advance, before you look at any results. It is the acceptance criterion. You write it down per impact axis, e.g. “individual: LOW, society: LOW, organization: MEDIUM,” meaning: we will tolerate up to a MEDIUM organizational risk, but nothing above LOW that touches an individual or society.

Drawing the line first is what keeps risk management honest. If you only set the bar after seeing the number, you will always be tempted to move it to wherever the model happens to land. Appetite is the commitment that makes a later “this passes” or “this fails” mean something. In ISO 23894 terms (clause 6.5, “risk evaluation”) it is the threshold you evaluate the analyzed risk against.

Inherent vs residual — and why we confirm the residual

Two snapshots of the same risk, before and after you do anything about it:

• Inherent risk — the risk as it stands today, untreated. (Our flower model, fed only weak features, confuses two species — HIGH.)
• Residual risk — what is left over after you treat it. (Give the model better features and the confusion becomes rare — MEDIUM.)

The goal is never zero risk (that does not exist); it is a residual at or below your appetite. Treatment is whatever you change to get there — and crucially, in this course treatment is a versioned change in git: a commit you can point to.

Now the subtle part. How do you know the residual is really MEDIUM and not wishful thinking? Two sources of truth, and we insist on both:

• Declared — a human writes down, in the manifest, “after treating this, the residual likelihood drops to unlikely.” That is a claim.
• Control-confirmed — a measurable control (here, a quality metric) actually measures the treated system and confirms the claim holds.

The hybrid residual is the rule that the residual only counts as real when the declared claim is backed by a passing control. A promise that no measurement confirms is just a promise.

If you are thinking “this is cute, but I don’t classify flowers” — that is exactly the point. The same likelihood × impact you are learning on Iris is what separates a credit model that quietly denies a mortgage through a proxy nobody measured from one you can actually defend. Same loop, higher stakes — see Why it matters.

Why 'hybrid' and not just 'declared'

A declared residual you cannot measure is exactly where compliance theatre lives: a tidy spreadsheet that says “risk: mitigated” with nothing behind it. The hybrid residual refuses that. The control re-runs every time the code changes, so the residual is earned on each commit, not asserted once and forgotten. When the control passes, the gate goes GREEN; when it cannot pass with confidence, the system says so.

The loop: identify → analyze → evaluate → treat → residual → monitor

Putting the pieces in order gives you the whole method as a single loop:

1. Identify — name the things that could go wrong. (Where do they come from? Where risks come from builds it out, anchored in EU AI Act Art. 9(2).)
2. Analyze — for each, estimate likelihood and impact (the 5×5 read).
3. Evaluate — compare the level against your appetite. Above the line? It needs treatment.
4. Treat — change something (a feature, a parameter, a process) to lower likelihood or impact.
5. Residual — re-measure; confirm the leftover risk is now at or below appetite (the hybrid residual).
6. Monitor — keep watching, because data and the world drift; a residual that was fine can degrade.

That sequence is ISO 23894’s risk-management process (clauses 6.4–6.5), and it is the spine of every level of this course. The same six steps apply whether the stake is a misclassified flower or a denied loan — only the numbers change.

Gate colours: RED, AMBER, GREEN

When the engine evaluates a treated system, each blocking control comes back in one of three colours. Idea first, then the nuance:

• GREEN — the control passed, with enough confidence. The residual is confirmed at or below appetite. Good to approve.
• RED — the control failed. The residual is above appetite. Do not approve; treat it (or, if you consciously accept the excess, that is a recorded override with a mandatory reason — covered later in the course).
• AMBER — the most honest of the three. The point estimate may look fine, but the measurement is underpowered: there is not enough data to prove the residual is acceptable. The confidence interval still straddles the threshold. AMBER says “we cannot yet tell,” and the right answer is to gather more evidence, not to wave it through.

AMBER is a feature, not a bug

A system that only ever shows RED or GREEN is hiding the hardest cases. AMBER is what honest uncertainty looks like: the model’s point result passes, but the sample is too small to trust. Calling that GREEN would be the dishonest move. (Our flower example is deliberately simple and closes cleanly GREEN — you will meet your first real AMBER in Level 2, where fairness is measured on a small held-out cohort.)

One worked example: the Iris species-confusion risk

Let’s make all of this concrete on the example you will run in Level 1 · Iris.

The Iris dataset (Fisher, 1936 — 150 real flowers, three species, four measurements each) is the textbook classification problem. Three species: setosa is trivially easy to tell apart, but versicolor and virginica overlap — they are genuinely similar in some measurements. That overlap is a real property of the flowers, not a strawman we invented.

Walk the loop:

• Identify. The model could confuse versicolor with virginica. We call this risk.species-confusion.
• Analyze. A model trained on sepal measurements alone (the sepal is the green outer leaf — the weakly discriminating feature) finds versicolor and virginica nearly indistinguishable. Confusion is likely; impact is moderate (organizational only — a wrong catalogue, no one is harmed). On the matrix: HIGH, above our appetite.
• Evaluate. HIGH organizational risk exceeds the appetite line (which we set at MEDIUM for the organization). It needs treatment.
• Treat. Add the petal measurements (the petal is the inner coloured leaf — and it is the discriminative feature: petals separate versicolor from virginica cleanly). This is a real change, versioned in git: a single parameter flip from sepal-only to all-features.
• Residual. Re-measure. With petals in the model, confusion becomes unlikely; the residual drops to MEDIUM, now at appetite. The control — macro recall, a quality metric, glossed in Level 1 — confirms it: it crosses from below the gate to above it. Hybrid residual satisfied.
• Monitor. The control re-runs on every future commit, so if someone later removes the petal features the gate turns RED again automatically.

The gate goes from RED (sepal-only) to GREEN (with petals), and only then does the system get approved. No law was involved at any step — we did this because it is good engineering, and because it gives us signed evidence that the risk really was treated.

❓ Why isn't a risk just its likelihood, or just its impact?

Because either number alone is misleading. A near-certain but trivial event (losing a lottery ticket) and a near-impossible but catastrophic event (a meteor strike) can look “extreme” on one axis while being negligible overall. A risk is likelihood × impact: you only know how much to worry once you weigh how probable against how bad. The 5×5 matrix is just a way to combine the two into one comparable level.

❓ What is the difference between inherent and residual risk, and why isn't 'residual' enough on its own?

Inherent risk is the risk untreated, as it stands today; residual risk is what is left after you treat it. The aim is a residual at or below your appetite, not zero. But a declared residual (“we fixed it”) is only a claim. The hybrid residual rule says it counts only when a measurable control actually confirms it on the treated system — declared and control-confirmed. Otherwise you have a promise no measurement backs, which is where compliance theatre hides.

❓ A control's point estimate passes the threshold, but the engine shows AMBER. What does that mean, and what should you do?

AMBER means the result is underpowered: the point estimate looks fine, but there is not enough data to prove the residual is acceptable — the confidence interval still crosses the threshold. It is the honest “we cannot yet tell.” The right response is to gather more evidence (widen the cohort, collect more data) and re-measure, not to treat AMBER as GREEN. Calling underpowered results green would be the dishonest move; AMBER exists precisely so you don’t.

Where to go next

You now have the vocabulary the whole course runs on. Put it to work hands-on, with three real people and one real repository, on the simplest possible system — flowers, no law:

Level 1 · Iris — your first risk loop, no law involved →