Add a scenario (.json)

✅ Stable

Venturalítica’s e2e harness is data-driven: each use case lives in a tests/scenarios/<name>.json file. The harness code is the whole tests/scenario/ module (mod.rs + runner.rs + assertions.rs + env.rs + fixtures.rs, orchestrated by mod.rs); it is shared by all scenarios and is never modified when adding a new one.

Adding a use case = adding a .json (and its resources) + a one-line #[test] function that names it. No pilot_*.rs is written.

---

Structure of a scenario

A scenario file has three top-level sections:

tests/scenarios/my-system.json (skeleton)

{
  "name": "my-system",
  "description": "Description of the use case and what is being tested.",
  "resource_dir": "my-system",
  "steps": [
    { "do": "git_init" },
    { "do": "uv_init", "deps": ["venturalitica==0.6.11", "mlcroissant>=1.0", "dvc>=3"] },
    { "do": "copy", "from": "sei.yaml", "to": "sei.yaml" },
    { "do": "compile", "expect": { "exit": "ok" } },
    { "do": "run", "expect": { "exit": "fail", "gate": "red" } },
    { "do": "verify", "expect": { "exit": "ok" } }
  ]
}

Field	Type	Description
name	string	Scenario identifier (matches the .json filename).
description	string	Free text describing the purpose and invariants. Printed in the trace.
resource_dir	string	Subdirectory under tests/resources/ containing the scenario’s fixture files.
steps	array	Ordered list of steps. Everything that happens on disk is an explicit step.

---

Step verbs (do)

The runner understands the following verbs. Each step may include an expect block with assertions.

Environment verbs

Verb	Effect
git_init	git init + fixture repo identity.
uv_init	Writes pyproject.toml with the given deps and runs uv sync.
dvc_init	dvc init.
copy	Copies from (relative to resource_dir) to to (in the temp repo).
dvc_add	dvc add <path> — versions the file with DVC.
commit	git add -A && git commit -m <message>.
edit_sei	Substitutes literals in sei.yaml (simulates a versioned change).

sei CLI verbs

Verb	Equivalent command
compile	sei compile — risk program (risk: section of sei.yaml) → assessment_plan.oscal.yaml.
run	sei run — reproduces the pipeline, anchors and signs evidence. Exit reflects the risk gate.
status	sei status — freshness + risk gates, without recomputing.
verify	sei verify — verifies the bundle signature.
reconstruct	sei reconstruct [--out] — reconstructs the ISO 23894 cycle by git replay.
conformance	sei conformance --standard <id> [--out] — projects the bundle onto the standard’s clause catalogue.
approve	sei approve --by <by> — management approval with Sei-Approved-by trailer.
impact	sei impact — impact assessment and foreseeable misuse (ISO 42001 §6.1.4).
assess	sei assess — the KAG proposes undeclared risks (live register backlog).

DVC verbs (RDD exploration)

Verb	Effect
dvc_exp	dvc exp run [--set-param ...] — runs the pipeline as a git-stashed experiment (not committed). Accepts expect_metrics to assert that the candidate would close the gate.

---

The expect block

Each step may declare assertions in its expect field. The runner checks each one and fails the test if any is not met.

expect examples

{
  "do": "run",
  "expect": {
    "exit": "fail",
    "gate": "red",
    "classification_tier": "high",
    "risks_nonempty": true,
    "pipeline_lock_present": true,
    "control_count": 10,
    "controls": {
      "unfair-credit-exclusion": {
        "passed": false,
        "enforcement_mode": "block",
        "severity": "high",
        "operator": "lt",
        "actual_gt": 0.03,
        "frameworks": ["eu/dora@2022#art-6"]
      }
    },
    "gate_failures": ["unfair-credit-exclusion"],
    "risk_analysis": {
      "risk.unfair-credit-exclusion": {
        "inherent_overall": "HIGH",
        "requires_treatment": true,
        "cycle": "open"
      }
    }
  }
}

expect field	What it checks
exit	"ok" or "fail" — the command exit code.
gate	"green" or "red" — the risk gate state in the bundle.
classification_tier	System classification level (e.g. "high").
risks_nonempty	The bundle contains at least one risk.
pipeline_lock_present	pipeline_lock_digest in the bundle starts with sha256:.
control_count	Exact number of controls in control_results.
controls	Per control: passed, enforcement_mode, severity, operator, actual_gt/lt, frameworks.
gate_failures	Exact list of blocking controls in red (excluding accepted).
drift	Per section: the expected drift text in stdout (e.g. "recomputado (clase B)").
stdout_contains	List of strings that must appear in stdout.
risk_analysis	Per risk: inherent_overall, residual_overall, requires_treatment, cycle.
overall_residual	Global bundle residual: level, evaluation, contributing.
treatment_events	Count and contents of treatment events in the bundle.
sei_artifacts	Paths relative to .sei/ that must exist, be valid JSON, and have a verifiable signature.

---

One scenario per backend, same loan

To add a new MLOps backend over the loan scenario, simply create a .json file that copies sei_<backend>.yaml as sei.yaml and replaces the pipeline definition file. The resources (compliance_eval.py, train.py, german_credit.*) are the same because they live in resource_dir: loan; the risk program lives in the risk: section of sei.yaml itself.

The three backend variants of the loan scenario (loan.json = DVC, loan_mlflow.json = MLflow, loan_dagster.json = Dagster) are the reference implementation of this pattern (there are 8 JSON scenarios in total today). See Integrate your pipeline for details on each backend.

Register the test

After creating the .json and resources, add a one-line function to tests/scenarios.rs:

tests/scenarios.rs (add)

#[test]
fn my_system() {
    run_scenario("my-system");
}

No other Rust file is touched.

---

Scenario resources

Each scenario declares its resource_dir. The runner looks for copied files (copy) in tests/resources/<resource_dir>/. You can reuse the loan directory for new backends of the same scenario, or create your own directory for different systems.

tests/
  scenarios/
    my-system.json         ← the scenario
  resources/
    my-system/
      sei.yaml             ← system manifest + risk program (risk: section)
      compliance_eval.py
      train.py
      data/my-dataset.csv
      data/my-dataset.croissant.json

---

References

• Integrate your pipeline — how to connect DVC, MLflow, or Dagster to the Reproducer seam
• Reference sei.yaml — manifest fields
• Reference sei CLI — subcommand details