@devries-cameron20 / t3code-1 / commits / 0fa2b92f8d
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

fix(evals): T3CODE_EVAL_MODEL_PROVIDER picks the GPT evals' proxy provider (#39505)

Julius Marminge committed Nov 20, 2026 at 15:00 UTC 0fa2b92f8d2fc729556fea15c2919b91d50c2f10
3 files changed +7 -5
evals/mcp-tool-use/README.md
+1 -1
@@ -13,7 +13,7 @@ pnpm mcp-tool-use --model claude-opus-5-5 --out runs/opus
13 ```
14
15 - `ANTHROPIC_API_KEY` in the repo root `.env` pays for Claude models and the Haiku shell simulator.
16 -- Codex models (`gpt-*` in `MODELS` in `run.ts`) go through a CLIProxyAPI server, set up as the Codex model provider: `model_provider` in `$CODEX_HOME/config.toml` (default `~/.codex/config.toml`) when it names a `[model_providers.<name>]` table, otherwise `[model_providers.vibeproxy]`. That table needs `base_url` (the proxy's OpenAI-compatible URL, such as `http://localhost:8317/v1`) and a key in `experimental_bearer_token` or the variable named by `env_key`. Keep the proxy under about 12 samples in flight across all runs.
16 +- Codex models (`gpt-*` in `MODELS` in `run.ts`) go through a CLIProxyAPI server, set up as the Codex model provider: `model_provider` in `$CODEX_HOME/config.toml` (default `~/.codex/config.toml`) when it names a `[model_providers.<name>]` table, otherwise `[model_providers.vibeproxy]`. Set `T3CODE_EVAL_MODEL_PROVIDER=<name>` to use another table, such as `vibeproxy` when your everyday `model_provider` is a different custom provider. That table needs `base_url` (the proxy's OpenAI-compatible URL, such as `http://localhost:8317/v1`) and a key in `experimental_bearer_token` or the variable named by `env_key`. Keep the proxy under about 12 samples in flight across all runs.
17 - `--shell-model` picks the model that plays the shell: `claude-haiku-5-5` by default, or `gpt-6.1-terra` / `gpt-6-luna` through the same proxy, so Codex runs need no Anthropic key. Prefer Terra: Luna often contradicts the case's world (it calls named checkouts missing or files empty). Compare runs only when they used the same shell model.
18 - Flags: `--case <ids>`, `--cases <file.ndjson>`, `--samples <n>` (default 2, and a 3rd when the first 2 disagree), `--concurrency <n>` (samples in flight, default 8), `--max-steps <n>`, `--effort <level>`, `--dry-run` (prints request bodies, no model calls), and `--context` (prints the size of T3's text and schemas, no key needed).
19 - `--baseline <results.jsonl> --stop-on-regression` stops as soon as a case that passed every baseline sample fails. Use it to test one copy change against a baseline run.
evals/shared/proxy.ts
+5 -3
@@ -11,8 +11,8 @@ const decodeProxyConfig = Schema.decodeUnknownSync(
11 ]),
12 );
13 /**
14 - * Reads the CLIProxyAPI provider from `$CODEX_HOME/config.toml` (default `~/.codex`): the
15 - * configured `model_provider` when it is a custom provider, otherwise `vibeproxy`.
14 + * Reads the CLIProxyAPI provider from `$CODEX_HOME/config.toml` (default `~/.codex`): the one
15 + * `T3CODE_EVAL_MODEL_PROVIDER` names, else `model_provider` when it is custom, else `vibeproxy`.
16 */
17 export const proxyConfig = () => {
18 const config = decodeProxyConfig(
@@ -31,7 +31,9 @@ path = home / 'config.toml'
31 if not path.is_file(): fail(f'No Codex config at {path}.')
32 config = tomllib.loads(path.read_text())
33 providers = config.get('model_providers', {})
34 -name = config.get('model_provider') if config.get('model_provider') in providers else 'vibeproxy'
34 +override = os.environ.get('T3CODE_EVAL_MODEL_PROVIDER')
35 +name = override or (config.get('model_provider') if config.get('model_provider') in providers else 'vibeproxy')
36 +if override and name not in providers: fail(f'T3CODE_EVAL_MODEL_PROVIDER names {name!r}, but {path} has no [model_providers.{name}].')
37 if name not in providers: fail(f'{path} has no custom model provider to use (model_provider is {config.get("model_provider", "unset")!r}).')
38 p = providers[name]
39 if not p.get('base_url'): fail(f'[model_providers.{name}] in {path} has no base_url.')
evals/test-signal/README.md
+1 -1
@@ -12,7 +12,7 @@ pnpm test-signal --out runs/baseline
12 pnpm test-signal --out runs/cut --prompt /path/to/variant.md --case new-table-row
13 ```
14
15 -- Models go through a CLIProxyAPI server, set up as the Codex model provider: `model_provider` in `$CODEX_HOME/config.toml` (default `~/.codex/config.toml`) when it names a `[model_providers.<name>]` table, otherwise `[model_providers.vibeproxy]`. That table needs `base_url` (the proxy's OpenAI-compatible URL, such as `http://localhost:8317/v1`) and a key in `experimental_bearer_token` or the variable named by `env_key`. Default models: `gpt-6.1-terra` and `gpt-6-luna`. Pass `--model` once per model to change them.
15 +- Models go through a CLIProxyAPI server, set up as the Codex model provider: `model_provider` in `$CODEX_HOME/config.toml` (default `~/.codex/config.toml`) when it names a `[model_providers.<name>]` table, otherwise `[model_providers.vibeproxy]`. Set `T3CODE_EVAL_MODEL_PROVIDER=<name>` to use another table, such as `vibeproxy` when your everyday `model_provider` is a different custom provider. That table needs `base_url` (the proxy's OpenAI-compatible URL, such as `http://localhost:8317/v1`) and a key in `experimental_bearer_token` or the variable named by `env_key`. Default models: `gpt-6.1-terra` and `gpt-6-luna`. Pass `--model` once per model to change them.
16 - Flags: `--prompt <file>` (a variant of the check, default the real one), `--case <id>` (repeatable), `--samples <n>` (default 2, and a 3rd when the first 2 disagree), `--concurrency <n>` (default 6), `--reasoning <level>` (default `low`, as the check runs).
17 - A bad case passes when a comment lands inside a target test. A guard case passes when the check posts nothing and replies exactly `All clear`.
18 - `summary.md` gives the prompt hash and word count, then per model: bad cases caught, guards clean, errors, and tokens.