@devries-cameron20 / composio Mirror · read-only
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 07:15 UTC

next
md 80 lines 5.62 KB
Rendered Raw
1 # Verification Map
2
3 The maintained argv and working directories are in `scripts/agent-profiles.json`.
4 Run from the repository root with its pinned mise toolchain, Bun and the normal
5 or Python development environment prepared. Profiles run existing checks and
6 their normal environment setup (including nox environment creation); they do not
7 choose a model.
8
9 The CLI and TypeScript SDK profiles use package-scoped Turbo tasks so prerequisite
10 workspace builds run before tests and type checks, including in a fresh checkout.
11
12 | Profile | Existing checks | What remains outside its proof |
13 | ------------------ | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
14 | `workflow` | Bun task integration tests, their TypeScript check and both agent-guidance validators | Agent judgment, product behavior, hosted CI |
15 | `cli` | CLI package tests, boundary validators and TypeScript checks | Built-binary behavior, real backend behavior |
16 | `sdk-typescript` | Core SDK unit tests and type checks | Changed providers, runtime packaging, live API behavior |
17 | `sdk-python` | Existing `make chk` and `make tst` nox sessions in `python/` | Built artifacts, live API behavior, providers outside those sessions |
18 | `release-metadata` | Existing release-workflow checks | Hosted checks, merges, tags, registries and clean consumer installation |
19
20 Use the package skills for narrower tests or additional affected providers. For
21 packaging or runtime changes, use existing Node/Cloudflare or CLI Docker E2E
22 checks after reading `typescript-testing` or `cli-e2e`. For publication, follow
23 `cli-release` or `python-release`; a local receipt never establishes publication.
24
25 ## Carry a task between sessions
26
27 The TypeScript CLI runs with Bun and uses `effect/cli` for argument validation and
28 generated help. Start with `bun run agent:task --help` or
29 `bun run agent:task init --help` to inspect commands and required flags.
30 Task commands print JSON; verification streams check output to stderr.
31
32 ```sh
33 bun run agent:task init cli-output --goal "Reproduce the reported CLI output bug and verify the fix" --profile cli --source https://github.com/ComposioHQ/composio/issues/123 --authority "Implement locally; publication requires the current request"
34 bun run agent:task status cli-output
35 bun run agent:task verify cli-output
36 bun run agent:task list
37 ```
38
39 The id names `.artifacts/agent-tasks/<id>/task.json`. Update its `nextStep` and
40 `blocker` fields as work progresses. Keep the goal observable; record the real
41 source issue or PR URL when one exists. The authority field preserves context;
42 it is never permission to exceed the current user request or repository rules.
43
44 Receipts retain the task, profile, source-content fingerprint and exact HEAD at
45 verification time. A changed task acceptance contract, tracked or untracked source file, or profile
46 makes previous passing evidence stale. Ignored artifacts and Claude Code's
47 `.claude/scheduled_tasks.lock` are excluded, so a
48 receipt does not cover ignored binaries or environment state changing later.
49 Fresh external observations are still required for production and registry claims.
50 Changing only `nextStep` or `blocker` preserves verification evidence.
51
52 Each attempt has its own receipt and a private output log capped at 64 KiB. Longer
53 output keeps its first 16 KiB and last 48 KiB around an omission marker, so final
54 errors survive. The log is saved while commands run, so it keeps recent output when the
55 runner is killed. Attempts are numbered in order, not by clock time. Failed attempts survive subsequent success. Missing tools, failed commands, interruption
56 and source edits during verification cannot produce a passing attempt. Remaining
57 commands after the first failure were not run. An abruptly killed runner reads as
58 interrupted, including when its latest attempt has no finished receipt. A retry
59 recovers a dead runner's lock once its old command group is gone. If that group
60 remains alive, or the runner died before saving the command's process group, the
61 error identifies the command and receipt to inspect before retrying. Live-runner
62 locks are never recovered automatically. An empty or corrupt `verify.lock` is
63 replaced only when no task has a running attempt whose runner or command group is
64 alive. A command that exits while workers it started are still running does not end
65 its step: the runner stops those workers before the next command or lock release.
66
67 Verification is serialized per worktree because profiles share build and nox outputs.
68 Use separate worktrees for parallel tasks. An interrupted lock acquisition can leave
69 `.artifacts/agent-locks/acquire.lock`; inspect its PID and remove it only after
70 confirming that process has ended.
71
72 ## Learn from interventions
73
74 `test/agent-task.test.ts` turns concrete workflow failure classes into runnable
75 checks: context survives resumption; edits invalidate proof; failures preserve
76 evidence; missing tools fail rather than skip. These exercise the tool's behavior,
77 not an LLM's planning quality. For a new recurring mistake, add the smallest
78 behavioral check at the real failure boundary and demonstrate its failure before
79 the fix. Keep unrelated passing cases as retention checks. Assess judgment-heavy
80 changes on new tasks before claiming they reduce supervision.