Suite format

A suite is a JSON file. Add "$schema" for editor autocomplete and validation (behavtest schema prints the schema).

{
  "$schema": "https://unpkg.com/behavtest/schema/suite.schema.json",
  "name": "support-bot",
  "description": "Regression suite for the support assistant",
  "defaults": { "judge": "anthropic:claude-sonnet-5", "repeat": 1, "timeoutMs": 30000, "concurrency": 4 },
  "pipeline": {
    "adapter": "http",
    "config": {
      "url": "${PIPELINE_URL:-http://localhost:4000/pipeline}",
      "headers": { "Authorization": "Bearer ${PIPELINE_TOKEN}" }
    }
  },
  "cases": [
    {
      "id": "refund-policy",
      "input": "Can I return an item after 40 days?",
      "expected": "No: returns are accepted within 30 days.",
      "tags": ["policy"],
      "scorers": ["llmJudge", "latencyCost"],
      "scorerConfig": {
        "llmJudge": { "rubric": "Does the answer state the 30-day limit and avoid promising an exception?" },
        "latencyCost": { "maxLatencyMs": 3000 }
      }
    },
    {
      "id": "order-status",
      "input": { "messages": [{ "role": "user", "content": "Where is order 123?" }] },
      "expected": "Your order shipped on Monday.",
      "scorers": ["exactMatch"],
      "repeat": 5
    }
  ]
}

Field	Meaning
`cases[].id`	Unique and stable across runs: it is how future comparisons match cases. Letters, digits, `.`, `_`, `-`.
`cases[].input`	A string, `{ "messages": [...] }` (chat history), or any object (sent as-is to HTTP pipelines; LLM adapters need `inputTemplate`).
`cases[].expected`	Reference answer. Required by `exactMatch`; optional context for `llmJudge`.
`cases[].expectedDocs`	Ids of the documents a correct retrieval returns, for the `retrieval` scorer.
`cases[].scorers`	Names of scorers to run. `scorerConfig.<name>` holds that scorer's options.
`cases[].tags`	Labels for filtering with `--tag`.
`cases[].repeat` / `timeoutMs`	Per-case overrides. CLI flags beat case values, which beat `defaults`.
`${VAR}` / `${VAR:-default}`	Environment placeholders, allowed in any string of `pipeline.config`. Secrets belong here, never in the file. A missing variable stops the run before anything is sent.
`pricing`	Optional extra/override model prices (see cost).
`variants`	Optional: run the suite once per variant (see matrix runs). Each is `{ name, description?, pipeline?: { adapter?, config? } }`.

Unknown keys are rejected, so typos like scorer (instead of scorers) fail loudly, with the JSON path.