@devries-cameron20 / composio Mirror · read-only
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 04:45 UTC

next
md 1.44 KB

Skill decisions

One directory per decision about the composio-cli skill or the CLI text agents read: why the change was made, what was measured, and the batches it was measured with. results/ is git-ignored; this directory is where the evidence behind a kept or dropped change stays.

A baseline has a directory of its own, named baseline-*, when the harness changed and the older batches can no longer be compared. It holds no decision: it is the line later decisions are measured from. See baseline-grader-guards.

Each decision has:

  • README.md: the problem seen in the runs, the change, the commands that measured it, the comparison, and the decision.
  • <batch id>/: batch.json, runs.jsonl and summary.md of every batch the decision rests on, copied with bun run eval.ts keep <batch> <decision>. The full logs are not kept; rerun the batch to get them.

To check a decision again, compare the kept batches, then run the same commands on the current checkout and compare the new batch with the kept baseline:

bun run eval.ts compare evals/<decision>/<baseline id> evals/<decision>/<candidate id>
bun run eval.ts run --model claude-haiku-5-5 --runs 5 --label recheck
bun run eval.ts compare evals/<decision>/<candidate id> <new batch id>

compare refuses batches graded by a different harness. When the journeys or the checks changed since, measure a new baseline at the decision's revision instead.