The best way to control Claude Code, Codex and more from any computer or phone
Mirrored from https://github.com/Camo67/t3code.git
Synced Nov 21, 03:37 UTC
| 1 | # Test signal |
| 2 | |
| 3 | Runs the Macroscope `test-signal` check ([`.macroscope/check-run-agents/test-signal.md`](../../.macroscope/check-run-agents/test-signal.md)) on real and synthetic test diffs, and checks that a cheap model flags low-signal tests and leaves good ones alone. Use it to cut the check to the fewest words that still work. |
| 4 | |
| 5 | The model gets the check body, the PR's test diff, `read_file` and `grep` at the PR head, and `post_comment`. The wrapper prompt around the body is a stand-in for Macroscope's, which is not public. |
| 6 | |
| 7 | ## Run |
| 8 | |
| 9 | ```sh |
| 10 | cd evals |
| 11 | pnpm test-signal --out runs/baseline |
| 12 | pnpm test-signal --out runs/cut --prompt /path/to/variant.md --case new-table-row |
| 13 | ``` |
| 14 | |
| 15 | - Models go through a CLIProxyAPI server, set up as the Codex model provider: `model_provider` in `$CODEX_HOME/config.toml` (default `~/.codex/config.toml`) when it names a `[model_providers.<name>]` table, otherwise `[model_providers.vibeproxy]`. Set `T3CODE_EVAL_MODEL_PROVIDER=<name>` to use another table, such as `vibeproxy` when your everyday `model_provider` is a different custom provider. That table needs `base_url` (the proxy's OpenAI-compatible URL, such as `http://localhost:8317/v1`) and a key in `experimental_bearer_token` or the variable named by `env_key`. Default models: `gpt-6.1-terra` and `gpt-6-luna`. Pass `--model` once per model to change them. |
| 16 | - Flags: `--prompt <file>` (a variant of the check, default the real one), `--case <id>` (repeatable), `--samples <n>` (default 2, and a 3rd when the first 2 disagree), `--concurrency <n>` (default 6), `--reasoning <level>` (default `low`, as the check runs). |
| 17 | - A bad case passes when a comment lands inside a target test. A guard case passes when the check posts nothing and replies exactly `All clear`. |
| 18 | - `summary.md` gives the prompt hash and word count, then per model: bad cases caught, guards clean, errors, and tokens. |
| 19 | - Cost: proxy quota only, about 0.5M tokens for a full run on both models. |
| 20 | |
| 21 | ## Add a case |
| 22 | |
| 23 | Add an entry to `cases.ts`. A `real` case names a merged commit and its test paths. A `synthetic` case inserts one test after an anchor string in a real file. Bad cases list the titles of the tests to flag. |