Evaluation results

An evaluation result should make its decision explainable without access to the agent's private context.

That does not mean the result should record private prompts or hidden reasoning. It should record the public inputs needed to reproduce the decision: package identity and digest, evaluator version, workspace revision, selected paths, active rules, findings, and final state.

The result should include the profile digest, scope, findings, evaluator version, and timestamps. A pass is scoped evidence, not a universal quality certificate.

Malformed requests, missing packages, unknown tools, and transport failures are reported by the CLI or MCP boundary as errors. They are not evaluator states and must not be counted as policy findings. evaluation_error is the evaluator's fail-closed state when it cannot produce a trustworthy artifact after the evaluation path begins.

What the benchmark proves

The retained technical benchmark is a reproducible engineering fixture, not a general claim about every model or repository. The current clean artifact records 64 paired scenarios, 128 Docker gates, 128 compiler-control checks, and 15 verified source references. It also records its repository revision and whether the checkout was dirty.

It does not prove that every agent output improves, that ordinary project tests and lint passed, that a source author endorses a Mind, or that the system has long-running production telemetry. Independent review and human adoption remain release gates.

Reading a finding

When a result contains a finding, read it in this order:

1. Confirm the file and line are inside the selected scope.
2. Confirm the rule identifier and severity match the active package.
3. Read the message and suggested next action.
4. Decide whether to change the code, change the package through review, or fix the evaluation environment.

Do not solve a required finding by deleting the file from the scope. If the scope is wrong, fix the scope and rerun the evaluation. The new result should make that change visible.

Example result shape

{
  "state": "needs_revision",
  "profile": { "id": "example.profile", "digest": "sha256:..." },
  "scope": { "base": "abc123", "files": ["src/boundary.rs"] },
  "findings": [
    {
      "rule": "errors.keep-cause",
      "severity": "required",
      "path": "src/boundary.rs",
      "message": "The boundary drops the original error."
    }
  ]
}

The exact serialization can evolve with the protocol version. The important property is that a person can connect the state to a rule and a file without asking the agent what it meant to do.

Proof matrix and real-world demos

This is the catalog of LMP proof work. It distinguishes runtime qualification, real-OSS paired scenarios, and internal repository demonstrations. A scenario listed here is implemented in an executable runner; it does not mean the latest run passed. A release claim must link to the generated JSON artifact for the exact commit, profile, and machine that produced it.

Docker coverage is mandatory. If the daemon or lmp-sandbox:local image is unavailable, the result is blocked, not complete.

Runtime qualification

The qualification suite covers clean and prohibited source, malformed syntax, mixed Rust and TypeScript workspaces, invalid signatures, missing policy layers, deterministic compilation, changed and untracked Git files, non-Git fallback, oversized or symlinked inputs, advisory and audit modes, MCP initialization and JSON-RPC errors, daemon watching, 5,000-file brownfield scans, 32 profile definitions, and selected-path Docker mounts.

cargo build --workspace
docker build -t lmp-sandbox:local orchestrator
python3 orchestrator/qualification_suite.py \
  --output ./lmp-test-results/qualification-result.json
python3 -m unittest discover -s orchestrator -p 'test_*.py'

Real-OSS benchmark

The executable benchmark creates paired baseline and guided candidates across eight public repositories and eight task classes: 64 scenarios in total. Each scenario records its public URL, resolved revision, task, cited source, baseline result, guided result, compiler-control result, rule IDs, and privacy metadata.

The task classes are API behavior, dependency choice, input validation, security boundary, regression testing, performance judgment, CLI workflow, and architecture/refactoring. Every class is paired with a cited public source describing the relevant method, philosophy, or trade-off. That citation proves provenance, not endorsement or access to private reasoning.

cargo build --workspace
python3 orchestrator/real_world_benchmark.py \
  --lmp target/debug/lmp \
  --output ./lmp-test-results

A complete technical benchmark requires status: "complete", exactly 64 scenarios, all 128 paired Docker gates passing, all 128 compiler-control checks passing, allTransitionsPassed: true, and every source verification entry marked verified: true. A partial, failed, or blocked run is not a completed benchmark. The raw JSON artifact remains the source of truth; dashboards are views over that artifact.

Internal repository demos

The internal demo applies the onboarding and Mind workflow to three pinned public repositories:

cargo build --workspace --target-dir /tmp/lmp-internal-demo-target
python3 orchestrator/internal_demo.py \
  --lmp /tmp/lmp-internal-demo-target/debug/lmp \
  --output ./lmp-internal-oss-matrix

A successful run requires all three baseline-to-guided transitions to pass. The report preserves the Mind digest, checked-file count, selected and rejected options, source revision, and privacy fields.

Evidence limits

These demos prove that the implemented LMP loop can load a selected profile, inspect a bounded workspace, produce deterministic findings, and record an explainable transition for tested candidates. They do not prove that every model output improves, that a Mind is universally correct, or that static evaluation replaces human review. The independent human pilot remains a separate release requirement.