Configuration
Everything about a run comes from four places: the suite file, per-case fields in it, command-line flags and environment variables. When two set the same thing, the more specific one wins:
| Setting | Resolved in this order (first one set wins) | Built-in default |
|---|---|---|
| Attempts per case | --repeat → the case's repeat → suite defaults.repeat | 1 |
| Timeout per attempt | --timeout → the case's timeoutMs → suite defaults.timeoutMs | 30,000 ms |
| Attempts in flight | --concurrency → suite defaults.concurrency | 4 |
| Judge model | the case's scorerConfig.<scorer>.judge → --judge → suite defaults.judge → BEHAVTEST_JUDGE | none (a judged case without one is a configuration error) |
| Model prices | --prices <file> → suite pricing → the bundled price table | bundled table |
| Results database | --db <path> | .behavtest/results.db |
- Secrets and URLs go in the environment, referenced from
pipeline.configas${VAR}or${VAR:-default}; a missing variable stops the run before anything is sent. Provider keys are read fromOPENAI_API_KEYandANTHROPIC_API_KEY(or the variable named byapiKeyEnv), andOPENAI_BASE_URL/ANTHROPIC_BASE_URLredirect the built-in adapters. - Which cases run is filtered with
--tagand--case(both repeatable), and in a matrix suite--variant. - What gets saved is controlled with
--no-trace(don't store pipeline steps),--label(a name for the run) and--export/--compact(also write a run file). - Editor support: add
"$schema": "https://unpkg.com/behavtest/schema/suite.schema.json"to a JSON suite for autocomplete and validation;behavtest schemaprints the same schema.
The fields of a suite are in suite format, every flag is in the CLI reference, and each adapter's options are in adapters.