Level 3 · The arc, by handoffs

The arc, by handoffs

Four beats. As in previous levels, all three roles are present from beat one: 🧭 Nerea at the top (portal · product), 🛡️ James in the middle (portal · quality), 🧑‍💻 Martha on the right (CLI + git). The lesson of this arc is the GREEN that isn’t enough: the sensitivity gate goes RED → clean GREEN (not an honest amber, a real green) — and even so the cycle pauses at “requested”, because a Class IIa medical device cannot close its conformity on its own attestation. The engine signs evidence; it is not a notified body.

The hard data of each beat —the commands, the commit message, the diff and the gate colour— is not hand-written: every GuionStep block reads it from the published repository’s steps/ script. If a step did not exist, the build would fail; what you read here is exactly what the repository does.

1Declare the SaMD (Art. 6(1) + MDR) and the under-diagnosis risk

🧭 Nerea

Nerea is the Product Owner at the clinic deploying the DR screener. She opens the portal and runs the authoring missions that generate the governance manifest froga.yaml. Three things matter, and the second is new:

1. The tier is high, declared by a human, via the product-safety route. The classify-system mission fills in the classification block with the justification: “Art. 6(1) (safety component of a medical device — SaMD MDR Class IIa)”. The high-risk trigger here is Art. 6(1) + Annex I, not Annex III. The engine does not infer it; Nerea declares it and owns it.
2. Two regimes are declared, and one is the overlay. The declare-standards mission writes applicable_standards with the spine (eu/ai-act@2024, eu/pren-18228@2026, iso/23894@2023) plus eu/mdr@2017 — the MDR overlay. No DORA (not a financial entity). Declaring eu/mdr@2017 is what makes the engine emit the MDR crosswalk later.
3. The risk is patient safety. The identify-risks mission adds risk.dr-underdiagnosis — “a false negative deprives the patient of timely treatment, risking irreversible vision loss.” Inherent: LIKELY × CRITICAL (individual). It exceeds appetite and demands treatment. The control that will confirm the residual is DR sensitivity (recall) — a blocking clinical gate, model-dr-sensitivity > 0.80.

The three missions open a PR in the system’s repository. The turn passes to James, who reviews and merges it. In the script, the identity, the framing and the risk programme land as three commits of the froga.yaml manifest —the diff of the last one shows the patient-safety risk with its numeric gate set before any model exists:

Identidad del sistema (froga init + patch)🧭 Nerea

Contexto y estándares aplicables (SaMD MDR Clase IIa, Art.6(1))🧭 Nerea

Programa de riesgos con gates numéricos (antes del modelo)🧭 Nerea

🛡️ James

James receives the PR opened by Nerea’s missions, reviews that the manifest is correct — tier high via Art. 6(1) + Annex I (SaMD MDR Class IIa), standards eu/ai-act@2024 + eu/pren-18228@2026 + iso/23894@2023 + eu/mdr@2017, patient-safety risk in full vocabulary — and merges it. The declared froga.yaml is now in the repository.

↓ relevo calidad → dev · high-risk SaMD via Art. 6(1) + MDR Class IIa; under-diagnosis risk declared

🧑‍💻 Martha

There is nothing to measure yet, but Martha leaves the reproducer ready in git: the uv + DVC project and the real ODIR-5K dataset (labels and metadata versioned; the ~2 GB of images are read from the cache at build time) with its Croissant manifest are already commits of the script:

Repo, identidad, proyecto uv (uv init + deps resnet18/LogReg), DVC, pubkey y README del reproductor🧑‍💻 Martha

Commands⧉ copy

15

• git init -q -b main
• git config user.email marta@demo.seigarrena.dev
• git config user.name "Marta Demo"
• uv init --bare --name vldemo-retina-screening --python 3.12
• patch: pyproject-index.patch
• uv add "dvc>=3" pyyaml froga "venturalitica==0.6.11" "torch==2.12.0" "torchvision==0.27.0" "scikit-learn>=1.4" "pandas>=2.1" pyarrow "pillow>=10.0" "numpy>=1.26" "mlcroissant>=1.0" joblib
• patch: gitignore.patch
• patch: gitattributes.patch
• patch: readme.patch
• uv sync
• uv run dvc init --no-scm -q
• uv run dvc config core.analytics false
• froga pubkey --out .froga/PUBKEY.txt
• git add -A
• git commit -m "init: repo, proyecto uv (deps reales) y README del reproductor"

Dataset ODIR-5K + manifiesto Croissant + versionado DVC🧑‍💻 Martha

With the manifest declared and the scaffolding in git, the turn passes to measuring (beat 2).

Venturalítica·demo.venturalitica.ai

DeclaredNext: run and measure

Venturalítica·demo.venturalitica.ai

Which Annex III category applies?

Education and vocational trainingHR and hiringEssential public servicesCredit scoringHealthcare servicesWelfare services

Which MDR class applies to the device?

Class I (low risk; self-certified)Class I sterile (NB for sterilization)High-risk AIClass I with measurement (NB for metrology)High-risk AIClass IIa (medium risk; NB required)High-risk AIClass IIb (medium-high risk; NB required)High-risk AIClass III (high risk; NB + design dossier)High-risk AI

Any additional regime?

DORA (financial entity)GDPR (personal data)MDR (medical device)ET Art. 64.4.d (employment algorithms)Law 15/2022 (equality)

Venturalítica·demo.venturalitica.ai

Significant risks

IDTitleLikelihoodIndividual impactSocietal impactOrganizational impactAssign treatment to (optional)

1

ID

Title

LikelihoodRareUnlikelyPossibleLikelyAlmost certain

Individual impactLowMediumHighCritical

Societal impactLowMediumHighCritical

Organizational impactLowMediumHighCritical

Assign treatment to (optional)

2

ID

Title

LikelihoodRareUnlikelyPossibleLikelyAlmost certain

Individual impactLowMediumHighCritical

Societal impactLowMediumHighCritical

Organizational impactLowMediumHighCritical

Assign treatment to (optional)

3

ID

Title

LikelihoodRareUnlikelyPossibleLikelyAlmost certain

Individual impactLowMediumHighCritical

Societal impactLowMediumHighCritical

Organizational impactLowMediumHighCritical

Assign treatment to (optional)

4

ID

Title

LikelihoodRareUnlikelyPossibleLikelyAlmost certain

Individual impactLowMediumHighCritical

Societal impactLowMediumHighCritical

Organizational impactLowMediumHighCritical

Assign treatment to (optional)

5

ID

Title

LikelihoodRareUnlikelyPossibleLikelyAlmost certain

Individual impactLowMediumHighCritical

Societal impactLowMediumHighCritical

Organizational impactLowMediumHighCritical

Assign treatment to (optional)

6

ID

Title

LikelihoodRareUnlikelyPossibleLikelyAlmost certain

Individual impactLowMediumHighCritical

Societal impactLowMediumHighCritical

Organizational impactLowMediumHighCritical

Assign treatment to (optional)

2Measure V1: DR sensitivity (recall) comes out red

🛡️ James

Waiting. Nerea has declared the system and James has merged it; now James waits for Martha’s first measurement. He does not touch the CLI: when Martha reports the gate colour, he will see whether the patient-safety risk is real and by how much it exceeds appetite.

↓ relevo calidad → dev · tag retina-v1.0.0-red — DR sensitivity (recall) 0.4165 < 0.80, the patient-safety gate RED

🧑‍💻 Martha

Martha measures the system as it stands — the inherent risk, untreated. First she compiles the gate contract from froga.yaml —which controls will be measured and against what threshold—:

Compilar el plan de evaluación OSCAL (antes del modelo)🧑‍💻 Martha

Commands⧉ copy

• froga compile
• git add -A
• git commit -m "compile: assessment plan OSCAL (contrato del gate, antes del modelo)"

Now she adds the base model —a resnet18 + LogReg screener at the default threshold and with class imbalance— and runs the evidence. froga run reproduces the pipeline, measures the controls and applies the gate:

Modelo resnet18+LogReg base V1 — escribe Y corre (gate RED esperado)🧑‍💻 MarthaExpected verdictRED

📍 reproducir retina-v1.0.0-red

git clone https://github.com/Venturalitica/vldemo-retina-screening
cd vldemo-retina-screening
git switch -c try retina-v1.0.0-red

⧉ copiar

Commands⧉ copy

• patch: params.patch
• patch: dvc.patch
• patch: train.patch
• patch: evaluate.patch
• patch: compliance-eval.patch
• froga runmay fail
• git add -A
• git commit -m "modelo: resnet18+LogReg V1 (train/evaluate + dvc.yaml) — run: V1 evidencia base (gate RED esperado; seed=42)"

The model-dr-sensitivity control comes back RED, and why is the point:

• recall_score = 0.4165, gate > 0.80, severity critical, enforcement block → passed: false.

A recall of 0.4165 means the screener misses well over half of the true DR cases. The command exits non-zero —hence the “may fail” note on froga run— but the bundle is still signed and anchored: a RED result is evidence, recorded honestly.

This RED is real, measured on real images

The miss rate is a property of an imbalanced model on a real cohort (ODIR-5K, public, anonymised), not a strawman engineered to fail. A classifier that doesn’t account for the minority DR class will happily achieve high overall accuracy while missing the cases that matter. The honest move is to surface that as RED, not to lower the 0.80 clinical bar until it passes.

The risk is real and above appetite; it needs treatment. This V1 evidence commit is frozen at the checkpoint above, retina-v1.0.0-red.

Venturalítica·demo.venturalitica.ai

●Evaluated · RED

3Treat the risk: class-balanced retraining + 0.30 threshold → clean green

🛡️ James

Waiting. James has seen the RED — the screener misses too many cases. He does not touch git; he waits for the re-measured evidence, where he will read the residual and the gate colour. He will see not just a colour but why — the engine reports the confidence interval, not only the point estimate.

↑ relevo dev → calidad · tag retina-v2.0.0-green — recall 0.8847, CI low 0.8516 > 0.80: clean green, NOT amber

🧑‍💻 Martha

This is the treatment (ISO 23894 §6.5, “risk treatment”). As at every level, the versioned change is the replacement of train.py; the diff is the treatment, and in a single step Martha treats and re-measures.

The new train.py brings this level’s own mechanism: Martha re-trains the screener with class-balanced weighting so the minority DR class is no longer drowned out, and lowers the decision threshold to 0.30 so a borderline eye is referred rather than missed. Both moves trade a little precision for a lot of sensitivity — the right trade for a screening tool, where a miss is far worse than an over-referral. (In N1 the treatment adds the petal features; in N2 it swaps the learner for fairlearn. The git diff has the same shape —a versioned code change in train.py—; what changes is its nature.) The treated model makes the evaluation stale, so the same step re-trains and re-measures — the GPU step you watch rather than run:

Tratamiento: reentrena balanceado + umbral 0.30 (§6.5) — escribe Y corre (gate VERDE)🧑‍💻 MarthaExpected verdictGREEN

📍 reproducir retina-v2.0.0-green

git clone https://github.com/Venturalitica/vldemo-retina-screening
cd vldemo-retina-screening
git switch -c try retina-v2.0.0-green

⧉ copiar

Commands⧉ copy

• patch: train-treatment.patch
• froga run
• git add -A
• git commit -m "treatment: reentrena el cribador DR con clase balanceada + umbral 0.30 (recupera la sensibilidad recall > 0.80) — ISO 23894 §6.5"

The gate turns GREEN — and this time, cleanly:

• recall_score = 0.8847, gate > 0.80, enforcement block → passed: true.
• Power block: bootstrap 95% CI [0.8516, 0.9127], cluster-bootstrap by patient (n = 1279 eyes, 1160 patient clusters, seed 42).

Read the CI honestly, the way Level 2 taught you: the lower edge is 0.8516, comfortably above the 0.80 line. The whole interval clears the gate, so the engine does not fire inconclusive(). This is a clean GREEN, not an amber.

Why this is clean GREEN here, and was honest AMBER in Level 2

Same machinery, opposite verdict — and the difference is power. In Level 2 the fairness metric was measured on 99 held-out students, so the bootstrap CI straddled the threshold and the engine reported AMBER / INCONCLUSO — it could not prove the residual. Here the sensitivity is measured on a much larger held-out cohort (n = 1279), so the CI is tight and its lower bound (0.8516) sits above the line. The engine reports GREEN because the evidence genuinely proves the residual is acceptable — not because anyone wanted a green. AMBER and clean-GREEN are the same honest rule applied to different amounts of evidence.

Clean green, signed, in git — frozen at the checkpoint above, retina-v2.0.0-green. And yet — as the next beat shows — that is still not enough to close the cycle.

Venturalítica·demo.venturalitica.ai

●Treated · GREEN

4Governance on pause: conformance, request and no approval (notified body missing)

🛡️ James

Martha’s signed, clean-GREEN evidence is in the repository, with conformance already issued (Martha’s column). In Level 1 a clean green meant a clean approve; in Level 2 an honest amber was approved with the gap on the record. Here something different happens — and it is the point of the whole level.

James reviews the bundle, verifies Martha’s signature, sees the patient-safety gate green and the residual within appetite — and does not approve. There is no froga approve. The lifecycle state stays draft; approval is null. The cycle stops at “requested”.

The cycle pauses at 'requested' — the notified-body GAP made concrete

This is not a bug, and it is not James forgetting to click. A Class IIa medical device cannot close its conformity cycle on its own attestation. The MDR conformity-assessment procedure (Article 52) requires an accredited, independent notified body to assess the evidence and issue a certificate before the device may be placed on the market. The engine is not a notified body — it signs evidence with the developer’s key (a fictional Dev Demo signer in this demo), which is self-declaration, the opposite of the independent third-party assessment the MDR demands. So the most honest thing the lifecycle can do is pause at “requested”: everything a manufacturer should do before the notified body is done, signed, and reproducible — and the cycle waits at the boundary where an external body must take over. That pause is the GAP, written into the state machine.

↓ relevo calidad → dev · tag retina-v2.0.0-request — the cycle pauses; NO approval, pending the notified body

🧑‍💻 Martha

Martha closes her part of the cycle from the CLI. With V2 at clean GREEN, she emits the signed conformance and reconstruct over the applicable standards (including the eu/mdr@2017 crosswalk), and requests approval:

Conformance + reconstruct firmados (entitlement + 6 normas)🧑‍💻 Martha

Commands⧉ copy

• patch: entitlement.patch
• froga conformance --standard eu/pren-18228@2026 --outmay fail
• froga conformance --standard iso/23894@2023 --outmay fail
• froga conformance --standard eu/ai-act@2024 --outmay fail
• froga conformance --standard eu/mdr@2017 --outmay fail
• froga conformance --standard eu/pren-18283@2026 --outmay fail
• froga conformance --standard eu/pren-18282@2026 --outmay fail
• froga reconstruct --outmay fail
• git add -A
• git commit -m ".froga: conformance + reconstruct firmados"

Solicitar aprobación (froga request) — verde firmado, ciclo en pausa: sin organismo notificado🧑‍💻 Martha

📍 reproducir retina-v2.0.0-request

git clone https://github.com/Venturalitica/vldemo-retina-screening
cd vldemo-retina-screening
git switch -c try retina-v2.0.0-request

⧉ copiar

git commitsolicitud de aprobación (el acto lo commitea froga request)

And there the script ends. There is no approval step, and there is none on purpose: a Class IIa device’s cycle cannot be closed by the manufacturer at all, so there is no conformance to re-issue over an Approved state that never arrives. Martha produced and signed the clean-GREEN evidence; the rest is not hers to grant, nor James’s to give. The system stays declared, treated, signed and requested — anchored at the retina-v2.0.0-request tag of the checkpoint above —, honestly pending the notified body. That is the end of the arc: a draft, requested, never approved.

Venturalítica·demo.venturalitica.ai

The cycle stays at "requested"

Pending: notified-body assessment (MDR Art. 52)A Class IIa medical device cannot be closed by self-declaration.

---