User guide

Roundtable runs several LLMs as a team of specialists on one objective. A Python orchestrator owns the workflow; the models never talk to each other. Each one receives a purpose-built context, replies with schema-validated JSON, and the orchestrator decides what happens next. Every call, verdict, file change, and test result is logged to SQLite.

Install

git clone https://github.com/bronette/roundtable
cd roundtable
uv sync                       # Python 3.12+, creates .venv
uv run roundtable --help

Log in to the CLIs you want to use, once, in your own terminal:

claude login       # claude.ai subscription
codex login        # ChatGPT
grok login         # xAI
agy                # Google Antigravity (Google AI Pro); sign in once interactively
ollama serve       # local models, then: ollama pull qwen3:30b-a3b

No API keys are needed for those. For Google models, use agy (Antigravity) with a Google AI Pro login; the older Gemini CLI no longer accepts personal logins, so it and the Gemini API provider need GEMINI_API_KEY from https://aistudio.google.com/apikey (free tier available). Other API providers need the variable named in the config (ANTHROPIC_API_KEY, OPENAI_API_KEY, XAI_API_KEY).

First run

uv run roundtable providers -c examples/project.yaml

prints each configured provider and whether it can be used right now. Then:

uv run roundtable run examples/project.yaml

The example objective is a small pure-Python function with tests. A full run at autonomy 2 makes six to nine model calls and takes ten to twenty minutes, most of it Grok thinking. The console shows each stage as it happens; the report lands at runs/dd-demo/<run_id>/report.md.

What a run does

INIT        prepare the workspace (worktree, copy, or empty dir), all on a new git branch
PROPOSE     proposer: scope check, plan, assumptions, risks, acceptance criteria
CRITIQUE    critic: attacks the proposal; ACCEPT / REVISE / REJECT / NEEDS_EVIDENCE / NEEDS_EXPERIMENT
REVISE      proposer answers every problem (fixed / rejected / deferred) and reissues the plan
            ... CRITIQUE ↔ REVISE up to budget.max_rounds ...
            on ACCEPT (or REVISE with no blocker at the round limit) the criteria are LOCKED;
            a blocker still open at the round limit stops the run with status needs_input
IMPLEMENT   engineer: writes code and tests in the workspace; the orchestrator commits the diff
TEST        orchestrator runs project.test_command; output is recorded verbatim
FIX         engineer sees the failing output and tries again (budget.max_fix_rounds times)
REVIEW      validator: checks every locked criterion against the files and the real test output
SYNTHESIZE  synthesizer: what was built, what the tests say, disagreements, risks, next step
DONE        report written

Anything can halt the run: a budget cap, a provider outage, a usage limit, invalid output twice in a row. A halted run still gets a report; the synthesizer is allowed one call past the budget for it, and if it cannot run either, the report is built from the database alone.

What "solved" means

A run is only a success when four observable things are all true:

1. every locked acceptance criterion is marked satisfied by the validator with a citation (a file and line, a test name that appears in the output, or a quoted output line);
2. the test command was actually executed by the orchestrator and exited 0;
3. no blocker-severity problem is open from the critic or the validator;
4. the synthesizer's overall verdict is ACCEPT, with any disagreements it could not resolve listed rather than hidden.

A model saying the work is done is never evidence. The validator deliberately does not see the proposer's reasoning or the critic's opinions; it judges the artifact and the test output.

Who is in charge

The orchestrator, which is code. It picks the next stage from a fixed transition table. Within a stage one role owns the output: the proposer owns the plan, the engineer owns the diff, the validator owns pass or fail, the synthesizer owns the report. The critic owns nothing; it can only block. No agent can declare victory or skip a stage.

Reading the report

report.md has, in order:

• Status and verdict. implemented means tests passed and the validator ran. Other statuses: implemented_with_objections, tests_failing, plan_accepted (autonomy 1), rejected, needs_input, needs_decomposition, advised (autonomy 0), halted_budget, halted_usage_limit, halted_error.
• What was built, test results, disagreements, remaining risks, recommended next experiment. Written by the synthesizer from the run's records. Disagreements list both positions; a criticism the proposer rejected or deferred always appears here.
• Proposal history. Every proposal and the critique it received, with each problem and its severity, and every criticism the proposer rejected or deferred with the reason.
• Acceptance criteria. The locked list with its timestamp.
• Implementation. Branch, commits, changed files, known gaps, diff artifacts.
• Test runs. Command, exit code, counts, duration, per attempt.
• Validator review. One row per criterion: satisfied or not, with the cited evidence.
• Decisions. Every state transition with its reason.
• Usage. Calls, tokens, and cost per provider. Cost shows only for models in the pricing table.

For the full audit trail:

uv run roundtable calls <run_id> -c examples/project.yaml            # one line per call
uv run roundtable calls <run_id> --failed                             # only invalid or errored attempts
uv run roundtable calls <run_id> --role critic --json                 # filter, machine-readable
uv run roundtable call-show <call_id>                                 # one call in full: messages, reply, command line

Every call shows role, provider, model, attempt, tokens, latency, and cost with its source: table when the model is in pricing.yaml, reported when the CLI supplied its own estimate.

Resuming a halted run

A run that halted (budget, provider outage, usage limit, invalid output twice) can be restarted at the stage that failed. Nothing completed is repeated or paid for again: the proposals, critiques, locked criteria, workspace, commits, and test results are all reloaded from the database, and the run continues from there.

uv run roundtable resume <run_id> -c examples/project.yaml
uv run roundtable resume <run_id> --max-calls 24 --max-cost 10      # raise caps if the halt was a budget

Budget counters continue from the run's recorded usage, so a budget halt needs a higher cap to make progress. A resumed run records "resumed by operator" in its decisions and rewrites the report at the end. Finished runs cannot be resumed.

Retries

Transient provider errors (timeouts, 5xx, dropped streams) are retried with backoff, three seconds then nine then twenty-seven, up to retries times per seat (default 2). Every failed attempt is in the audit log with its error. Usage-limit errors, missing keys or binaries, and invalid output are not retried: those go to the repair round or the fallback provider instead.

Working on an existing project

Set project.repo to the directory. If it is a git repository, each run gets its own worktree on a new branch roundtable/<run_id>; the engineer works there and the orchestrator commits there. Your checked-out branch and working tree are never touched.

When the run finishes:

cd ~/your/repo
git log main..roundtable/<run_id>          # what the run committed
git diff main...roundtable/<run_id>        # the full change
git merge roundtable/<run_id>              # if you want it
git worktree remove runs/<project>/<run_id>/workspace   # clean up (path is in the report)
git branch -D roundtable/<run_id>          # if you do not

A directory that is not a git repo is copied into the workspace (skipping .git, .venv, node_modules, caches) and given a local git repo so diffs and commits still work. Omitting repo gives an empty workspace, which is what the example uses.

Set project.test_command to whatever proves the work (python -m pytest tests/ -q, npm test, make check). Set project.python to the repo's own interpreter when the tests need its dependencies.

Engineer modes

Agent mode (default for CLI providers): the CLI is launched inside the worktree with its own tools, exactly as you would run codex or claude there by hand. It reads, edits, and runs tests itself, then returns a short JSON summary. The orchestrator captures the resulting diff. Containment: Codex runs under its workspace-write sandbox; Claude Code auto-accepts edits inside the directory and shell is limited to python, pytest, and uv; Grok has no sandbox, only the worktree boundary and the prompt. Nothing outside the worktree is writable by design, and the worktree is disposable.

Answer mode (default for API providers, or mode: answer): the model returns complete files as JSON; the orchestrator validates paths (no .., no absolute paths, nothing under .git) and writes them. Slower to converge on real repos because the model cannot explore, but it works with any backend including local models.

Budgets and quota

Caps in budget are checked before every call. Two things to know about CLI providers:

• Each call carries the tool's own system prompt, roughly 10k to 15k tokens for Claude and Grok and about 14k for Codex, before your content. Budget by calls as much as by tokens.
• Subscriptions have rolling five-hour and weekly limits shared with your interactive use. A usage-limit error halts the run with status halted_usage_limit; the report is still written.

The reported dollar figures for CLI calls are the vendor's list-price estimates, marked billing: subscription in the database. They tell you what a run would cost on the API; they are not a bill.

Choosing roles

Observed so far on the example project:

• Grok as critic is the strongest reviewer of the three but slow (two to four minutes and 10k+ output tokens per critique at medium effort) and occasionally returns nothing on long prompts. Give it fallback: claude_cli so a stalled critique does not end the run.
• Claude as proposer, reviser, and synthesizer is fast (30 to 50 seconds) and writes precise criteria and honest reports.
• Codex as engineer in agent mode edits and tests inside the worktree.
• A local model as validator is free and independent, but weak: qwen3 30B got a simple drawdown question wrong twice. It is fine for checking test output against criteria; do not expect it to catch subtle logic errors. Any backend can take any role; swap in the config.

Point critic and proposer at different vendors when you can. Independence is the point.

Troubleshooting

Layout on disk

runs/<project>/
  roundtable.db                    every run for this project
  <run_id>/
    workspace/                     the worktree or copy the engineer worked in
    artifacts/                     diffs and test output, one file per artifact id
    report.md

Audits and blinded data

read_in_place: true gives read-mode seats the whole live checkout, including ignored data, logs and reports. That is what makes an audit accurate, and it is also a hazard: a proposer that opens a dataset a pre-registration declared unread has peeked on the operator's behalf, and that registration can no longer be run clean. Before an in-place run on a repo that holds blinded data, name the off-limits paths in the objective ("OFF LIMITS: do not open data/x.db"), or run in worktree mode, where ignored files are absent. This was learned the hard way on 2026-09-26, when a sports hunt sized a maker margin from ProphetX depth ladders that SB-004 had declared unread.

Experiments (pre-registered)

For research questions, especially trading, the unit of work is an experiment whose criteria are fixed before any result exists. This is the mechanism that stops "tune until the backtest looks good". Three commands:

uv run roundtable experiment preregister prereg.yaml -c project.yaml   # lock it: E1, hash, pinned commit
uv run roundtable experiment run E1 -c project.yaml                     # execute the command, record the real output
uv run roundtable experiment interpret E1 -c project.yaml               # read the numbers against the locked criteria

prereg.yaml mirrors the hypothesis-gate one-pager:

hypothesis: The slow-category NO slice has positive net edge after fees.
expected_result: mean_edge >= 0.01 per contract at n >= 300
method: run the study's own analyzer on the frozen settled cohort
data: data/study.db, settled 2026-06-01..2026-09-15, never used for tuning
success_criteria: ["mean_edge >= 0.01 with n >= 300"]
failure_criteria: ["ci_lower <= -0.01 with n >= 300"]
n_trials: 1                                  # configs searched; feeds multiple-testing corrections
execution_model: taker at ask, Kalshi fee schedule 2026-07, no partial fills
command: python scripts/analyze.py --json    # writes JSON to $ROUNDTABLE_METRICS or prints a JSON line
repo: ~/code-projects/SomeTrader             # optional; pinned to its current commit at lock time

What each step guarantees:

• preregister hashes the whole pre-registration and records the repo commit. Any later edit to the stored criteria makes run and interpret refuse with "altered after locking".
• run executes the command once, in a worktree at the pinned commit, with API keys stripped and a timeout. Metrics come from the JSON file named by ROUNDTABLE_METRICS or the last JSON line on stdout. An experiment with a recorded result cannot be run again; pre-register a new one.
• interpret has one model read every success and failure criterion against the recorded metrics, citing the value that decides each. The decision rule is then applied by code, not by the model: any failure criterion met is KILL; else any success criterion met is PASS; else INCONCLUSIVE. A model that flinches from a KILL is overridden and the override is recorded. A second model reviews the reading; if it disagrees on the decision, the experiment is recorded INCONCLUSIVE with both positions and an open question for you.

experiment list and experiment show E1 print the ledger. Roles: interpreter and experiment_critic if configured, else the synthesizer and critic seats.

Trading projects

Set project.domain: trading. Two things change:

• The proposer must state the execution model (fills, fees, spread, slippage, latency, capacity), the search budget (n_trials), the benchmark, and the in-sample/out-of-sample split.
• The critic must fill in a fifteen-item checklist on every proposal (look-ahead bias, survivorship bias, data leakage, overfitting, multiple-hypothesis testing, unrealistic fills, commissions and fees, spread, slippage, liquidity and capacity, queue position, execution latency, sample size, regime dependence, parameter sensitivity), each marked ok, concern, or not applicable with a reason. A critique missing any item fails validation and is repaired.

A model saying a strategy works is never evidence; only an executed experiment is. A proposal that claims an edge without one gets NEEDS_EXPERIMENT, which points you at the commands above.