Tutorial: third-party LLM fine-tuned with LoRA (SGCD credit)

✅ Stable — The reproducer is REAL: the DVC pipeline fine-tunes LittleLamb 0.3B with LoRA on GPU and measures the evidence. The whole arc —red (base V1) → amber (LoRA V2) → approved— is published in the repository and its four tags are watched by the nightly. The accuracy figures cited (AUC) are golden-build observations, not numbers derived in the docs: the course does not claim them as a verdict (future line), the same way Levels 2 and 4 do it.

This tutorial walks through the pilot-banco-atlantico-credito-llm system: an LLM that scores the creditworthiness of consumer-credit applications. It is the reproducer of the public demo repository vldemo-santander-credit, and its story is that of the third-party model as a governed risk:

• The scoring engine is LittleLamb 0.3B (Multiverse Computing, Apache-2.0), compressed from Qwen3-0.6B with CompactifAI technology. The bank does not train it: it integrates it as a third-party GPAI — it is the EU AI Act integrator (Art. 25), bound by the accuracy, robustness and security duties of Art. 15.
• The data is the SGCD (Stressed German Credit Dataset, Santander AI Lab, CC-BY-4.0, SGCD-v0.1): 1,000 real credit applications derived from Statlog German Credit, with documented perturbations («shocks») on two orthogonal axes.
• The operating bank is fictitious — «Banco Demo Atlántico SA», with a made-up LEI. Only the dataset (Santander AI Lab) and the model (Multiverse Computing) are real public sources; the rest is a demonstration governance template, not a real scoring system.

It is a high-risk system: automated credit scoring (EU AI Act Annex III §5(b)) operated by a financial entity → an ICT asset subject to DORA (Reg. EU 2022/2554). The source of truth for this tutorial is the repository’s steps/ script: the commands, commit messages, diffs and every gate colour are not hand-written — each block reads them from the published script. Were a step to not exist, the build would fail; what you read is exactly what the repository does. The commands and commit messages are the real Spanish artefacts of the repo you clone, shown verbatim.

What you’ll learn:

• How a third-party LLM (GPAI, Art. 25) is governed without retraining it: the LoRA fine-tune is a parameter change versioned in git and therefore a risk treatment (ISO 23894 §6.5) that is attributable and reproducible.
• Why base V1 comes out RED (near-random accuracy) and the LoRA lifts it to an honest AMBER (accuracy passes the threshold, but robustness on the contradictory inputs stays inconclusive because of small samples) — without fabricating a green.
• How the SGCD’s four shock risks (F1 missingness · F2 ambiguity · F4 contradiction + an orthogonal cosmetic axis) are declared before the model, each with a gated coverage attesting that the class is represented in the measurement cohort.
• How conformance is gated by entitlement (paid standards require a signed licence) and how approval consciously accepts the amber residual, with the auditor’s reason on record.
• The contrast with Level 4 · Consumer credit: there the model is a logistic regression trained in the pipeline itself; here it is a third-party LLM and the treatment is a fine-tune, not a from-scratch retrain.

The system is live in the portal

You can walk through this same system in the demo portal: demo.venturalitica.ai/demo/systems/santander-credit. The portal renders what the repository has committed: the declaration, the signed evidence, the approval with its reason, and conformance per standard. The public repository is vldemo-santander-credit.

The three roles of the cycle appear from the first beat: 🧭 Nerea (product · declares identity, framing and risks), 🧑‍💻 Martha (development · brings data, measures, treats and signs from the CLI) and 🛡️ James (quality · accepts the residual and approves). The role badge on each block below comes from the script’s who.

---

The arc, beat by beat

Start the repo and declare the identity and the §5(b) + DORA framing

Venturalítica·demo.venturalitica.ai

DeclaredNext: data and risk programme

The repository starts as a uv + DVC project (no SCM yet), with the reproducer’s README. Then the PO declares who the system is and which regulatory role the bank plays: froga.yaml grows through manifest commits — identity (v0), and then the framing with the applicable standards and the human-declared §5(b) + DORA classification.

Repo, identidad, proyecto uv (uv init + deps LLM/LoRA), DVC, pubkey y README del reproductor🧑‍💻 Martha

Commands⧉ copy

15

• git init -q -b main
• git config user.email marta@demo.seigarrena.dev
• git config user.name "Marta Demo"
• uv init --bare --name vldemo-santander-credit --python 3.12
• patch: pyproject-index.patch
• uv add "dvc>=3" pyyaml froga "venturalitica==0.6.11" transformers torch peft accelerate pyarrow pandas scikit-learn
• patch: gitignore.patch
• patch: gitattributes.patch
• patch: readme.patch
• uv sync
• uv run dvc init --no-scm -q
• uv run dvc config core.analytics false
• froga pubkey --out .froga/PUBKEY.txt
• git add -A
• git commit -m "init: repo, proyecto uv (deps reales) y README del reproductor"

Identidad del sistema (froga init + patch)🧭 Nerea

Contexto, estándares aplicables y encuadre🧭 Nerea

The framing diff shows the essentials: context.applicable_standards lists prEN 18228, ISO 23894, EU AI Act and DORA; the (fictitious) DORA entity is declared; and classification.tier: high is justified in words — «credit-scoring §5(b)» + «financial-entity ICT risk (DORA Art. 6)». The engine does not infer the tier: a person declares and owns it.

Bring the SGCD data and declare the risk programme (before the model)

Before any model exists, the real data is brought in and the risk programme with its numeric gates is set. This is RDD in its purest form: first you declare what you’ll measure against; the model comes later.

Dataset SGCD + manifiesto Croissant + versionado DVC🧑‍💻 Martha

Each SGCD application ships in a dual view: a structured json_view (the record as it stands) and a natural-language text_view. The LLM scores over the text_view — and that is where the dataset’s shocks matter, because the model only sees the text:

SGCDataset.jsonl (one record, trimmed)

{
  "id": "SGCD-0000",
  "label": 0,
  "metadata": { "shock_semantic": "F0", "shock_cosmetic": false },
  "value": {
    "json_view": { "duration": 9, "credit_amount": 1366, "age": 22, "personal_status_sex": "A92", "...": "..." },
    "text_view": "Checking account status: < 0 DM. Credit duration: 9 months. ... Personal status: female : divorced/separated/married. ... Age: 22 years old. ... Foreign worker: yes."
  }
}

The featurize step (featurize.py) loads the SGCD, derives sex from personal_status_sex and produces the train and eval parquet files. Still no model or dvc.yaml: only the first link of the pipeline.

Pipeline de rasgos sobre el SGCD — escribe Y corre (dvc repro featurize)🧑‍💻 Martha

And now the heart of this beat: the PO declares the full risk programme (risk.risks[]) with its numeric gates, before a single number is measured. The diff of this commit is the whole programme:

Programa de riesgos con gates numéricos (antes del modelo)🧭 Nerea

The four shock risks are the protagonists. The SGCD documents that its inputs come degraded on two orthogonal axes, and the manifest identifies one sibling risk per class — each with a coverage measure (shock-coverage-*) that attests, with a blocking gate > 0.99, that that shock class is represented in the measurement cohort (ISO 23894 §6.4.2):

Shock class	Risk	What happens to the text_view	Measure (gate > 0.99)
F1 · missingness (12.8%)	risk.data-quality-shock-f1-missingness	a field vanishes from the text; the model decides as if it did not exist	shock-coverage-f1
F2 · ambiguity (11.5%)	risk.data-quality-shock-f2-ambiguity	a precise value turns into vague wording	shock-coverage-f2
F4 · contradiction (6.0%)	risk.data-quality-shock-f4-contradiction	the text asserts a valid value different from the structured one — nothing «looks» broken	shock-coverage-f4
Cosmetic (13.5%, orthogonal)	risk.data-quality-shock-f3-cosmetic	format noise: field order, delimiters, vague numbers («around 13 months»)	shock-coverage-cosmetic

The semantic axis (F1/F2/F4) and the cosmetic one are independent and co-occur: a record may carry both. The manifest also declares the accuracy risk (which base V1 will fail, and which LoRA treats), the robustness risk on F4 (the worst condition, analysed with a bootstrap CI) and the fairness (parity by sex), abstention (the LLM never abstains) and human oversight (Art. 14) risks.

Why the shocks are a risk, not a benchmark trick

In the benchmark each shock is labelled. In production, not: the bank will receive applications with empty fields, vague wording and contradictions without that mark. The gated coverage does not measure «how good the model is under shock» — it attests that the risk source identified at design time is represented in the evidence. If a class were not in the cohort, its coverage would drop to 0 and the gate would block: an honest identification gap, not a green by omission.

Measure base V1: the accuracy gate comes out red

Venturalítica·demo.venturalitica.ai

●Tested · RED

Martha compiles the OSCAL evaluation plan from froga.yaml first —the gate contract: which controls are measured and at what threshold—. froga compile needs no model: it compiles the OSCAL from the declaration.

Compilar el plan de evaluación OSCAL (antes del modelo)🧑‍💻 Martha

Commands⧉ copy

• froga compile
• git add -A
• git commit -m "compile: assessment plan OSCAL (contrato del gate, antes del modelo)"

Now she adds the base V1 model —LittleLamb 0.3B without fine-tuning (mitigate: false in params.yaml)— and runs the evidence. froga run reproduces the pipeline, measures the controls and applies the gate:

Modelo LittleLamb base V1 — escribe Y corre (gate RED esperado)🧑‍💻 MarthaExpected verdictRED

📍 reproducir santander-v1.0.0-llm-base

git clone https://github.com/Venturalitica/vldemo-santander-credit
cd vldemo-santander-credit
git switch -c try santander-v1.0.0-llm-base

⧉ copiar

Commands⧉ copy

• patch: params.patch
• patch: dvc-model.patch
• patch: train-llm.patch
• patch: infer.patch
• patch: evaluate.patch
• patch: characterize.patch
• patch: compliance-eval.patch
• froga runmay fail
• git add -A
• git commit -m "modelo: LittleLamb base V1 (train/infer/evaluate + dvc.yaml) — run: V1 evidencia base (gate RED esperado; seed=42)"

The result is RED. LittleLamb 0.3B is a compressed general-purpose model with no credit-domain tuning: over the text_view it produces near-random scores —in the golden build, an observed clean AUC around 0.42–0.54— below the accuracy minimum an automated credit decision requires (Art. 15, gate > 0.65). froga run exits with a non-zero code —hence the «may fail» note— but the package is left signed and anchored: a RED is evidence too, recorded honestly. The «reproduce» chip clones the santander-v1.0.0-llm-base tag and reproduces this same colour.

This RED is real, not a straw man

The gate is not rigged. A generic 0.3B LLM, without a fine-tune, simply does not separate good from bad payers over the application text. The honest thing is to show it RED, not to lower the threshold until it passes. That RED is the baseline the treatment will be measured against.

Treat with LoRA and re-measure: honest, underpowered amber

Venturalítica·demo.venturalitica.ai

●Treated · AMBER

The treatment does not retrain the model from scratch nor edit the threshold: it turns on the LoRA fine-tune over the bank’s labelled applications. In the repository that change is a single boolean in params.yaml —the diff is the treatment— and, with LoRA active, the same step re-measures: it treats and re-measures in a single step.

Abre la rama de tratamiento + activa el fine-tuning LoRA (§6.5) — escribe Y corre (gate ÁMBAR honesto)🧑‍💻 MarthaExpected verdictAMBER

📍 reproducir santander-v2.0.0-llm-finetuned

git clone https://github.com/Venturalitica/vldemo-santander-credit
cd vldemo-santander-credit
git switch -c try santander-v2.0.0-llm-finetuned

⧉ copiar

Commands⧉ copy

• git checkout -b tratamiento/afina-lora
• patch: params-mitigate.patch
• froga runmay fail
• git add -A
• git commit -m "treatment: afina el modelo LittleLamb con LoRA sobre las solicitudes etiquetadas del SGCD — ISO 23894 §6.5 (tratamiento: REDUCE → re-medición)"

Flipping mitigate to true is the ISO 23894 §6.5 treatment: a parameter change versioned in git —with authorship, date and message— that the engine attributes to the reduction of the accuracy risk. It is the §6.5 of the loop, projectable to whichever standard applies.

The result is an honest AMBER. The fine-tune lifts the observed clean AUC to ≈ 0.71 (golden build) → it passes the accuracy gate. But robustness on the contradictory inputs (F4, the worst condition) stays inconclusive: the AUC drop under F4 is small, but the sample is tiny (n = 60) and the bootstrap CI is wide → it crosses the threshold. The engine marks the control inconclusive and the verdict is AMBER, not a fabricated green.

Honest amber: the treatment moved the number, not the power

LoRA fixed accuracy (red → fit). It does not fix robustness on F4, because that residual is not a model problem but an evidence one: with n = 60 there is no statistical power to prove the degradation is within appetite. Rounding the point estimate up to green would violate the engine’s honesty rule: it never claims more certainty than its evidence carries. It is the same rule as in Level 4, where fairness is measured over n = 200 and is likewise underpowered.

Governance: gated conformance, informed acceptance and re-issue

Venturalítica·demo.venturalitica.ai

Undeclared

Declared

Assured

Tested

Treated

Approved

Marketable

With V2 treated, Martha issues the per-standard conformance + the cycle reconstruct, signed. Here the entitlement appears: froga conformance projects the same package onto each applicable standard, but the paid standards require a signed licence.

Conformance + reconstruct firmados (entitlement + 4 normas)🧑‍💻 Martha

Commands⧉ copy

• patch: entitlement.patch
• froga conformance --standard eu/pren-18228@2026 --outmay fail
• froga conformance --standard iso/23894@2023 --outmay fail
• froga conformance --standard eu/ai-act@2024 --outmay fail
• froga conformance --standard eu/dora@2022 --outmay fail
• froga reconstruct --outmay fail
• git add -A
• git commit -m ".froga: conformance + reconstruct firmados"

The .froga/entitlement.json file is a signed DSSE (in-toto predicate entitlement/v1) declaring which standards the system may emit. Its payload, for this system, enables two paid standards:

.froga/entitlement.json (payload, relevant fields)

{
  "predicateType": "https://venturalitica.ai/entitlement/v1",
  "predicate": {
    "systemId": "pilot-banco-atlantico-credito-llm",
    "org": "demo",
    "entitledStandards": ["eu/pren-18228@2026", "eu/dora@2022"],
    "issuedAt": "2026-07-14T00:00:00Z", "expiresAt": "2099-01-01T00:00:00Z",
    "...": "..."
  }
}

The engine splits standards into two tiers: ISO 23894 is free (it never requires an entitlement — it is the open-core baseline gate, the same boundary the Quickstart teaches); prEN 18228 and DORA are paid and here this signed entitlement enables them. A paid standard without valid entitlement would come out Unlicensed; a standard with no projectable catalogue —like the EU AI Act itself, the root regulation, which is not in the standards registry— produces no verdict («no catalogue»). That is why every froga conformance in the step carries allowFailure: the un-enabled standard comes out Unlicensed and the unregistered one emits no verdict, without aborting the step. In this dossier ISO 23894, prEN 18228 and DORA are emitted; the EU AI Act projection produces no verdict.

The entitlement is the moat, not a key the customer edits

The entitlement is signed by the issuer (Venturalítica), not by the system. The open-core engine verifies it against the issuer key-set and, if it does not check out —invalid signature, standard not listed, expired—, the paid standard comes out Unlicensed. The system cannot «declare itself» enabled by editing a file: the licence travels signed and is checked cryptographically.

Martha requests approval (a lifecycle act; the signed act .froga/acts/NNNN-request.json is emitted by froga):

Solicitar aprobación (froga request)🧑‍💻 Martha

📍 reproducir santander-v2.0.0-request

git clone https://github.com/Venturalitica/vldemo-santander-credit
cd vldemo-santander-credit
git switch -c try santander-v2.0.0-request

⧉ copiar

git commitsolicitud de aprobación del plan de tratamiento (el acto lo commitea froga request)

And now the governance piece: James, quality, approves — but the residual is AMBER, so it is not an automatic sign-off. It is a conscious acceptance, and the portal requires the auditor’s reason on record:

Aprobar🛡️ James

📍 reproducir santander-v2.0.0-approved

git clone https://github.com/Venturalitica/vldemo-santander-credit
cd vldemo-santander-credit
git switch -c try santander-v2.0.0-approved

⧉ copiar

🖊️ James approves

Reason: «Aceptación consciente del residual ámbar: el control bloqueante de precisión (risk.credit-accuracy) certifica en VERDE tras el tratamiento LoRA (V2, AUC limpio ≈ 0.71 por encima del umbral > 0.65), pero la caída de robustez sobre F4 (risk.robustness-shock, n=60) queda INCONCLUSA por IC bootstrap ancho y el residual global agregado excede el apetito MEDIUM; se aprueba con el gap documentado y revisión semestral (P6M). Remedio real: ampliar la cohorte F4 para dar potencia estadística a la medición de robustez (Art.15, línea futura).»

The approval is a git-native, signed act: a DSSE act .froga/acts/NNNN-approve.json —signed ECDSA-P256 and verified against James’s key declared in signers:— that records the approver’s identity (James) and travels with history. The commit that adds it is plain text, not GPG-signed (don’t expect the «Verified» badge when you click it on GitHub), but the act itself is cryptographically signed; and alongside it the evidence (bundle.json.sig, reconstruct.json.sig) and the conformances are signed too —ECDSA-P256 + DSSE. The reason —the robustness drop on F4 (n = 60) inconclusive because of a wide CI, the aggregate residual above appetite and the semiannual review (P6M)— is recorded in the portal record: approval from the UI requires that textarea; the CLI, which does not accept --reason, drops it with a warning. The signed act says who approved by its signature, as an attributable git act; the record keeps why; the signature proves the act and the package are intact.

With the real approval in git history, Martha re-issues the conformance: froga conformance derives the cycle from history and now sees the approval. The risk-management-review clause (prEN 18228 cl. 11, satisfied by the approval evidence) turns from Gap to Covered on reading the approval act .froga/acts/NNNN-approve.json. The overall-residual clause (cl. 10) was already Covered from the first emission: the aggregate residual (HIGH) sits within the declared criterion (overall_residual_criterion: HIGH). What exceeds is the appetite (MEDIUM) — and that signal is exactly what forced the conscious acceptance with the auditor’s reason on record: the dossier does not gloss over the residual, it governs it. The aggregate is re-signed.

Re-emitir conformance tras aprobar (cl.11 = Approved)🧑‍💻 Martha

Commands⧉ copy

• froga conformance --standard eu/pren-18228@2026 --outmay fail
• froga conformance --standard iso/23894@2023 --outmay fail
• froga conformance --standard eu/ai-act@2024 --outmay fail
• froga conformance --standard eu/dora@2022 --outmay fail
• froga reconstruct --outmay fail
• git add -A
• git commit -m ".froga: re-emisión de conformance + reconstruct tras aprobar (cl.11 = Approved)"

---

What this tutorial demonstrates

1. A third-party LLM (LittleLamb 0.3B, GPAI, Art. 25) governed as a high-risk §5(b) + DORA system, without retraining it: the bank is the integrator.
2. LoRA fine-tuning as a treatment (ISO 23894 §6.5): a parameter change versioned in git —the diff is the evidence—, not an opaque retrain.
3. An arc honest at both ends: base V1 is RED (near-random accuracy, real), LoRA V2 is AMBER (accuracy fit, robustness on F4 inconclusive at n = 60) — no fabricated greens.
4. Four shock risks (F1/F2/F4 semantic + orthogonal cosmetic axis) declared before the model, each with gated coverage attesting its representation in the cohort.
5. Conformance gated by entitlement: ISO 23894 free; prEN 18228 and DORA paid, enabled by a signed DSSE licence; a standard with no licence (Unlicensed) or no projectable catalogue (EU AI Act) fails without aborting (allowFailure).
6. Conscious acceptance of the amber residual (within the declared criterion, above appetite) with the auditor’s reason on record (approval as a git-native signed act .froga/acts/NNNN-approve.json, evidence signed) + re-issue that turns cl. 11 to Covered.

The hard figures are golden-build observations

The AUC values cited (≈ 0.42–0.54 on base V1, ≈ 0.71 after LoRA) are golden-build observations on GPU, not numbers the course derives from the signed package. They are given to anchor the red → amber narrative; the verdict that is derived is the colour (RED / AMBER), read from the script’s expect.verdict. Deriving the measured figures and rendering them is a future line, consistent with how Levels 2 and 4 treat it. See Status & gaps.

Level 4 · Consumer credit (DORA)The same domain with a tabular model trained in the pipeline: the contrast with the third-party LLM fine-tuned with LoRA.