Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

next
md 20 lines 1.44 KB
Rendered Raw
1 # Skill decisions
2
3 One directory per decision about the `composio-cli` skill or the CLI text agents read: why the change was made, what was measured, and the batches it was measured with. `results/` is git-ignored; this directory is where the evidence behind a kept or dropped change stays.
4
5 A baseline has a directory of its own, named `baseline-*`, when the harness changed and the older batches can no longer be compared. It holds no decision: it is the line later decisions are measured from. See [`baseline-grader-guards`](./baseline-grader-guards/README.md).
6
7 Each decision has:
8
9 - `README.md`: the problem seen in the runs, the change, the commands that measured it, the comparison, and the decision.
10 - `<batch id>/`: `batch.json`, `runs.jsonl` and `summary.md` of every batch the decision rests on, copied with `bun run eval.ts keep <batch> <decision>`. The full logs are not kept; rerun the batch to get them.
11
12 To check a decision again, compare the kept batches, then run the same commands on the current checkout and compare the new batch with the kept baseline:
13
14 ```sh
15 bun run eval.ts compare evals/<decision>/<baseline id> evals/<decision>/<candidate id>
16 bun run eval.ts run --model claude-haiku-5-5 --runs 5 --label recheck
17 bun run eval.ts compare evals/<decision>/<candidate id> <new batch id>
18 ```
19
20 `compare` refuses batches graded by a different harness. When the journeys or the checks changed since, measure a new baseline at the decision's revision instead.