Gateway and Contract¶

Local gateway¶

The local gateway is a lightweight HTTP server that reads run_record snapshots from disk and serves the embedded workspace UI. It is the bridge between your Writer output and the workspace.

Start it with:

cemi gateway --save-dir .cemi

The gateway binds to 127.0.0.1 only. It is a local developer tool, not a shared network service.

Save directory layout¶

save_dir/
  runs/
    <run_id>.jsonl
  artifacts/
    <run_id>/
      <filename>

The Writer and gateway must point at the same save_dir. The gateway reads runs/*.jsonl on each request — it does not cache or watch the directory.

API endpoints¶

Endpoint	Purpose
`GET /workspace`	Serve the embedded workspace SPA
`GET /api/health`	Gateway health, mode, port, and active save directory
`GET /api/projects`	Discover projects from run files on disk
`GET /api/projects/<project>`	Project details
`GET /api/projects/<project>/runs`	List runs in a project
`GET /api/projects/<project>/metrics/registry`	Metric definitions across all runs in a project
`GET /api/projects/<project>/contract`	Load contract JSON (optional `?path=` query param)
`GET /api/projects/<project>/recommendation`	Evaluate the project contract against all runs; returns per-run verdicts
`GET /api/runs/<run_id>`	Full normalized run snapshot
`GET /api/runs/<run_id>/metrics`	Metric events; supports `?name=`, `?fromStep=`, `?toStep=` filters
`GET /api/runs/<run_id>/params`	Run parameters
`GET /api/runs/<run_id>/artifacts/<name>`	Download a stored artifact
`POST /api/events`	Append a `run_record` event to a local JSONL file

Health response¶

{
  "status": "ok",
  "mode": "local",
  "save_dir": "/path/to/.cemi",
  "port": 3141
}

The workspace reads this response on startup. When mode is "local", it skips the hosted login flow and goes directly to the local workspace experience.

Compatibility normalization¶

The gateway normalizes run payloads for UI consumption:

payload.metrics.events is flattened into run.metrics for the workspace
summary_metrics is reconstructed from metrics.summary entries when needed
Tags are normalized to a list of key/value pairs
Artifact URIs are adjusted so relative paths resolve through the active gateway port

The `run_record` event schema¶

Each line in runs/<run_id>.jsonl is a run_record event. The Writer appends one complete snapshot per emit_run_record() call. The gateway uses the last valid line as the current view of that run.

Schema version¶

Current local contract: "2.0". Contract result extension (planned v0.2.0+): "2.1".

Event shape¶

{
  "type": "run_record",
  "schema_version": "2.0",
  "payload": { ... }
}

Payload shape¶

{
  "run_id": "local-abc123",
  "project": "ptq-sweep",
  "name": "mobilenetv2-ptq-int8",
  "stage": "benchmark",
  "status": "succeeded",
  "created_at_ms": 1730000000000,
  "created_at": "2026-04-13T10:00:00Z",
  "started_at": "2026-04-13T10:00:01Z",
  "ended_at": "2026-04-13T10:05:00Z",

  "tags": { "model": "mobilenetv2", "quant": "int8" },

  "context": {
    "case":   { "suite": "ptq-benchmark", "task": "classification", "dataset": "imagenet-val" },
    "policy": { "name": "accuracy-first", "objective_metric": "final_accuracy", "objective_direction": "higher_is_better" },
    "device": { "board": "stm32h7", "runtime": "onnxruntime", "memory_budget": "512MB" }
  },

  "parameters": [
    { "key": "quantization_scheme", "value": "ptq-int8", "value_type": "string" },
    { "key": "calibration_samples", "value": 1000, "value_type": "number" }
  ],

  "metrics": {
    "events": [
      {
        "name": "loss",
        "value": 0.42,
        "step": 1,
        "timestamp_ms": 1730000001000,
        "unit": "",
        "role": "custom",
        "aggregation": "raw",
        "direction": "lower_is_better",
        "tags": {}
      }
    ],
    "summary": [
      {
        "name": "latency_p99_ms",
        "value": 18.4,
        "unit": "ms",
        "role": "performance",
        "aggregation": "p99",
        "direction": "lower_is_better"
      }
    ]
  },

  "summary_metrics": {
    "final_accuracy": 0.934,
    "model_size_mb": 4.2,
    "peak_ram_mb": 480
  },

  "artifacts": [
    {
      "kind": "model",
      "name": "model.onnx",
      "uri": "http://127.0.0.1:3141/api/runs/local-abc123/artifacts/model.onnx",
      "media_type": "application/octet-stream",
      "size_bytes": 4300000
    }
  ],

  "contract_result": {
    "contract_id": "accuracy-sla",
    "verdict": "pass",
    "evaluated_at_ms": 1730000300000,
    "gates": [
      { "id": "accuracy-gate", "verdict": "pass", "metric": "final_accuracy", "threshold": 0.9, "actual": 0.934 }
    ]
  }
}

Contract rules¶

The local JSONL contract is strict:

One file per run ID: runs/<run_id>.jsonl
One JSON object per line; no trailing commas
Every emit_run_record() call appends a complete snapshot — no incremental diffs
The gateway uses the last valid run_record line as the current view
Project discovery is derived from run files, not from a separate registry

Contracts¶

A contract is a JSON file that specifies machine-checkable gates a run must pass to be considered deployment-ready. It is what separates benchmarking from verification.

Run cemi verify --contract contract.json --run <run_id> to evaluate. Exit code 0 = all gates pass. Exit code 1 = at least one gate failed.

Contract schema (v0)¶

{
  "contract_id": "accuracy-sla",
  "project": "ptq-sweep",
  "name": "Accuracy and latency SLA for MobileNetV2",
  "baseline": {
    "run_id": "local-abc000"
  },
  "gates": [
    {
      "id": "accuracy-gate",
      "role": "quality",
      "metric": {
        "name": "final_accuracy",
        "source": "summary_metrics"
      },
      "direction": "higher_is_better",
      "absolute": { "min": 0.90 }
    },
    {
      "id": "latency-budget",
      "role": "performance",
      "metric": {
        "name": "latency_p99_ms",
        "source": "summary",
        "aggregation": "p99",
        "tags": { "scenario": "offline" }
      },
      "direction": "lower_is_better",
      "absolute": { "max": 25.0 }
    },
    {
      "id": "accuracy-degradation",
      "role": "quality",
      "metric": {
        "name": "final_accuracy",
        "source": "summary_metrics"
      },
      "direction": "higher_is_better",
      "relative_degradation": { "max_pct": 1.0 }
    },
    {
      "id": "memory-limit",
      "role": "resource",
      "metric": {
        "name": "peak_ram_mb",
        "source": "summary_metrics"
      },
      "direction": "lower_is_better",
      "absolute": { "max": 512 }
    }
  ]
}

Gate types¶

Absolute gates¶

Assert that a metric value satisfies a hard threshold:

{
  "id": "accuracy-floor",
  "role": "quality",
  "metric": { "name": "final_accuracy", "source": "summary_metrics" },
  "direction": "higher_is_better",
  "absolute": { "min": 0.90 }
}

For lower_is_better metrics, use "max" instead of "min".

Relative degradation gates¶

Assert that a metric has not regressed more than a percentage from a baseline run:

{
  "id": "accuracy-degradation",
  "role": "quality",
  "metric": { "name": "final_accuracy", "source": "summary_metrics" },
  "direction": "higher_is_better",
  "relative_degradation": { "max_pct": 1.0 }
}

The contract must include a "baseline" field with the baseline run_id for degradation gates to evaluate. If the baseline run is not found, the gate is skipped.

Absolute degradation instead of percentage:

"relative_degradation": { "max_abs": 0.01 }

Metric source: `summary_metrics` vs `summary`¶

Source	Description
`"summary_metrics"`	Reads from `payload.summary_metrics` (flat key/value dict). Use this for values logged with `log_summary_metric()` or `log_summary_metrics()`.
`"summary"`	Reads from `payload.metrics.summary[]` (typed entries with role, aggregation, direction). Use this for values logged with `log_summary()`. Supports `aggregation` and `tags` filtering.
`"metrics"`	Reads from `payload.metrics.events[]` and aggregates across matching events. Supports `aggregation` and `tags` filtering.

Aggregation for `summary` and `metrics` sources¶

When source is "summary" or "metrics", the gate can specify an aggregation to use when multiple matching entries exist:

"metric": {
  "name": "latency_ms",
  "source": "metrics",
  "aggregation": "p99",
  "tags": { "scenario": "interactive" }
}

Supported aggregations: last, min, max, mean, p50, p90, p95, p99, sum, count.

Cost gates¶

Derive cost per inference from throughput and an hourly pricing model:

{
  "cost": {
    "currency": "USD",
    "output": {
      "metric": { "name": "outputs", "source": "summary_metrics" },
      "fallback": 1
    },
    "rates": {
      "default_per_hour": 1.0,
      "by_tag": [
        { "tags": { "device": "cpu" }, "per_hour": 0.25 },
        { "tags": { "device": "A100" }, "per_hour": 6.0 }
      ]
    }
  }
}

Cost per output is derived as:

cost_per_output = (rate_per_hour / 3600) / throughput_outputs_per_second

The gateway exposes cost evaluation results at GET /api/projects/<project>/recommendation.

Contract schema (v1 — plan style)¶

An alternative schema for ASR and NLP benchmarks with dedicated quality, performance, and resource sections:

{
  "version": "1",
  "contract_id": "asr-server-sla",
  "project": "asr-benchmark",
  "name": "ASR Server SLA",
  "baseline": { "run_id": "local-baseline-001" },
  "quality": {
    "metric": "delta_wer",
    "direction": "lower_is_better",
    "operator": "<=",
    "threshold": 0.05,
    "notes": "WER degradation must stay under 5% of baseline"
  },
  "performance": [
    {
      "scenario": "server",
      "metric": "latency_p99_ms",
      "operator": "<=",
      "threshold": 300,
      "unit": "ms"
    },
    {
      "scenario": "offline",
      "metric": "throughput_words_per_s",
      "operator": ">=",
      "threshold": 1000
    }
  ],
  "resources": [
    { "metric": "peak_gpu_mem_gb", "operator": "<=", "threshold": 24 }
  ]
}

Evaluation output¶

cemi verify --output json produces:

{
  "contract_id": "accuracy-sla",
  "run_id": "local-abc123",
  "verdict": "fail",
  "generated_at": "2026-04-13T10:05:00Z",
  "gates": [
    {
      "id": "accuracy-gate",
      "role": "quality",
      "metric": "final_accuracy",
      "verdict": "pass",
      "run_value": 0.934,
      "explain": "0.934 >= 0.90"
    },
    {
      "id": "latency-budget",
      "role": "performance",
      "metric": "latency_p99_ms",
      "verdict": "pass",
      "run_value": 18.4,
      "explain": "18.4 <= 25.0"
    },
    {
      "id": "accuracy-degradation",
      "role": "quality",
      "metric": "final_accuracy",
      "verdict": "fail",
      "run_value": 0.934,
      "baseline_value": 0.961,
      "explain": "degradation_pct 2.81% > 1.0%"
    }
  ]
}

Each gate entry includes:

verdict: "pass" or "fail"
run_value: the actual metric value from the run
baseline_value: the baseline value, when a relative degradation gate is evaluated
explain: a human-readable string showing exactly why the gate passed or failed

Using contracts in CI¶

steps:
  - name: Run benchmark
    run: cemi start --project $PROJECT --save-dir .cemi -- python benchmark.py

  - name: Verify deployment contract
    run: |
      cemi verify \
        --contract contract.json \
        --run $RUN_ID \
        --save-dir .cemi \
        --output json \
        --output-file result.json
    # Exit code 0 = pass, 1 = fail, 2 = parse error

  - name: Upload verification result
    uses: actions/upload-artifact@v4
    with:
      name: contract-result
      path: result.json

The CEMI_RUN_ID environment variable is set by cemi start. Your benchmark script can read it to pass the ID downstream.