BehavTest › Reference

Baselines and CI: fail the pull request that made things worse

compare needs a run to compare against, and a CI job starts with an empty .behavtest/ directory. Run files fill that gap: a portable JSON copy of a run, with every attempt's case hash, so compare still tells a changed case from a regression.

behavtest run suite.json --repeat 3 --export run.json    # write a run file as part of a run
behavtest export <run> --out run.json                    # or export a saved run (no --out: print it)
behavtest compare base.json head.json                    # compare two files: no database needed
behavtest compare behavtest.baseline.json                  # a file alone is the base, vs the latest run of its suite
behavtest report <run> --against behavtest.baseline.json --out report.html
behavtest import run.json                                # load a full run file into the database

Compact run files (--compact) keep only what a comparison needs: case ids and hashes, attempt statuses, latency, cost and each scorer's pass/fail. They leave out inputs, expected answers, outputs, error messages, judge reasoning and traces, so they are small and safe to commit. They can be compared against, but not imported or turned into a report of their own.

(A --json report is not a run file: it has no case hashes, so it can't be used as a baseline.)

Recipe 1: a committed baseline (recommended)

On GitHub, the GitHub Action does this recipe for you. The steps below do the same with the CLI, for other CI systems or more control.

Keep behavtest.baseline.json in the repository. Every pull request compares against it, and moving the baseline is an ordinary, reviewed commit, so the git history doubles as the history of your pipeline's quality.

Create or update the baseline when the pipeline is in a state you accept:

behavtest run behavtest/suite.json --repeat 3 --export behavtest.baseline.json --compact
git add behavtest.baseline.json && git commit -m "Update BehavTest baseline"

Then gate pull requests (.github/workflows/behavtest.yml):

name: BehavTest
on: pull_request

jobs:
  behavtest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
      # Start your pipeline here if the suite calls it over HTTP.
      - name: Run the suite
        # Exit 1 (some cases failed) is fine here: the comparison decides. Exit 2 (bad config) still fails.
        run: npx behavtest@0.8 run behavtest/suite.json --repeat 3 || test $? -eq 1
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - name: Compare with the baseline
        run: npx behavtest@0.8 compare behavtest.baseline.json --fail-on-regression --md behavtest.md
      - name: Job summary
        if: always()
        run: cat behavtest.md >> "$GITHUB_STEP_SUMMARY"

Use the same --repeat for the baseline and the pull request runs: more attempts per case give the comparison more power (see how it decides). If the pull request deliberately changes cases, they show as modified and don't fail the gate; update the baseline in the same pull request.

Recipe 2: the latest run on main as the baseline

No committed file: every push to main uploads its run, and pull requests compare against the newest one. Less ceremony, but the baseline moves without review.

name: BehavTest
on:
  push:
    branches: [main]
  pull_request:

jobs:
  behavtest:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      actions: read   # to download main's run
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
      - name: Run the suite
        run: npx behavtest@0.8 run behavtest/suite.json --repeat 3 --export run.json --compact || test $? -eq 1
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - name: Keep main's run as the baseline
        if: github.event_name == 'push'
        uses: actions/upload-artifact@v7
        with:
          name: behavtest-baseline
          path: run.json
      - name: Compare with main
        if: github.event_name == 'pull_request'
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          id=$(gh run list --workflow behavtest.yml --branch main --event push --status success --limit 1 --json databaseId --jq '.[0].databaseId')
          gh run download "$id" --name behavtest-baseline --dir baseline
          npx behavtest@0.8 compare baseline/run.json run.json --fail-on-regression --md behavtest.md
          cat behavtest.md >> "$GITHUB_STEP_SUMMARY"