@devries-cameron20 / composio-1 / commits / 74ec3ad46
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

feat(agents): add resumable SDK and CLI verification (#5103)

## Summary Substantial CLI and SDK tasks can resume across coding agents with their goal, authority context and verification evidence intact. The tooling is TypeScript executed with Bun, and reuses the repository's existing checks. ## Changes - preserves local task state through `bun run agent:task` and exposes the checks each verification profile actually proves - binds receipts to the checked revision, source contents and acceptance criteria; retains failed attempts and rejects stale proof - serializes verification within a worktree and retains command-group ownership through cancellation and crash recovery - provides typed commands, generated help and runnable examples through `effect/cli` - adds the `verify-change` skill, Bun integration tests and strict tooling typecheck; ports the two touched guidance validators to Bun and TypeScript ## Type of change - [x] New feature - [x] Refactor/Chore - [x] Documentation ## How Has This Been Tested? The 15 Bun workflow tests cover generated help, argument validation, repeated flags and literal task text, alongside task retention, stale source/profile/acceptance/revision evidence, preserved failures, missing tools, source mutation, force-killed runners, competing tasks and resistant descendants. The strict agent-tooling typecheck, both guidance validators, routing probes and formatting checks pass. Both maintained package profiles also pass: CLI tests and type checks, and core SDK tests and type checks, with prerequisite builds handled by package-scoped Turbo tasks. Run the checks with the mise-pinned toolchain: ```sh bun run test:agent-workflow bun run typecheck:agent-workflow bun run validate:agent-skills bun run validate:skill-routing ``` ## Screenshots (if applicable) Not applicable to repository tooling. ## Checklist - [ ] I have read the Code of Conduct and this PR adheres to it - [ ] I have read and understand every line of this diff, including any LLM-generated code - [x] Any third-party docs links or promotional content follow the third-party links policy - [x] I ran linters/tests locally and they passed - [x] I updated documentation as needed - [x] I added tests or explain why not applicable - [ ] I added a changeset if this change affects published packages ## Additional context Task records and logs stay private under ignored `.artifacts/`. A passing local profile establishes only its named checks; hosted CI, live API behavior and publication require their own evidence. Published SDK and CLI code is unchanged, so no changeset is needed. Hosted CI: the agent-guidance job and the dependency audit pass. This PR adds root development declarations for existing Effect catalog pins and preserves production dependency resolutions and audit policy.

Alberto Schiabel committed Oct 29, 2026 at 18:11 UTC 74ec3ad465ec421a2b0b1b74c78806571860b111
16 files changed +1585 -60
.agents/skills/effect-v4/SKILL.md
+1 -1
@@ -63,7 +63,7 @@ pnpm typecheck
63 pnpm --filter @composio/cli test
64 ```
65
66 -Run `pnpm validate:agent-skills` and `pnpm validate:skill-routing` after editing this
66 +Run `bun run validate:agent-skills` and `bun run validate:skill-routing` after editing this
67 skill or its descriptions, and compile the TypeScript blocks in this skill and in
68 `typescript-testing/references/effect-v4-cli.md` against the pinned packages with:
69
.agents/skills/repo-guidance/references/repository-workflow.md
+1 -1
@@ -30,7 +30,7 @@ pnpm typecheck
30 pnpm test
31 pnpm test:e2e
32 pnpm test:e2e:cli
33 -pnpm validate:agent-skills
33 +bun run validate:agent-skills
34 ```
35
36 Python commands run from `python/`:
.agents/skills/skill-maintenance/references/skill-format.md
+3 -3
@@ -37,12 +37,12 @@ Rules:
37 Run:
38
39 ```bash
40 -pnpm validate:agent-skills
41 -pnpm validate:skill-routing
40 +bun run validate:agent-skills
41 +bun run validate:skill-routing
42 ```
43
44 Both validators are deterministic, and `.github/workflows/agent-substrate.yml` runs them in CI on every push and pull request — the stale-guidance scan covers the whole repo, so any change can affect the result.
45
46 `validate:agent-skills` checks frontmatter, names, reference links, compatibility symlink state, stale guidance references, and known command names.
47
48 -`validate:skill-routing` is a deterministic routing smoke test (`ts/scripts/test-skill-routing.mjs`): for representative tasks it asserts the expected skill is the unique top match by distinctive trigger phrases, and fails if any skill has no probe. When you add or rename a skill, or rewrite a `description`, add or update its probe so routing stays covered.
48 +`validate:skill-routing` is a deterministic routing smoke test (`ts/scripts/test-skill-routing.ts`): for representative tasks it asserts the expected skill is the unique top match by distinctive trigger phrases, and fails if any skill has no probe. When you add or rename a skill, or rewrite a `description`, add or update its probe so routing stays covered.
.agents/skills/verify-change/SKILL.md new
+26
@@ -0,0 +1,26 @@
1 +---
2 +name: verify-change
3 +description: Maintain resumable task records, select repository verification profiles, inspect revision-bound receipts, and carry work between coding agents. Use for substantial SDK or CLI work spanning sessions, evidence review, or carrying work between coding agents; use package-specific skills for implementation and ordinary small edits.
4 +---
5 +
6 +# Verify Change
7 +
8 +Write agent-tooling scripts and tests in TypeScript and execute them with Bun; do not add `.mjs` or Node entrypoints. Existing SDK languages and package-management commands keep their repository ownership.
9 +
10 +Use [the verification map](references/verification-map.md) to select the existing
11 +checks and preserve a task's goal, authority, next step and blockers across sessions.
12 +
13 +1. For substantial work, initialize one task with the requested outcome and a
14 + matching profile. Keep the actual user request authoritative.
15 +2. Inspect `status` when resuming, update `nextStep` and `blocker` in its local
16 + `task.json`, and refresh external issue, PR and CI state before acting.
17 +3. Run the profile after implementation. Read the receipt and its boundary before
18 + reporting what passed. A passing profile establishes only its named checks.
19 +4. Treat a recurring intervention as a candidate for an existing behavioral test,
20 + type or lint check. Prove the check rejects the original mistake. Preserve
21 + one-off judgment locally instead of expanding global instructions.
22 +
23 +Do not create task records for trivial edits. Private task data and command output
24 +stay under ignored `.artifacts/`; never attach them wholesale to a PR. This tool
25 +records context and evidence, and does not grant commit, push, merge or release
26 +authority.
.agents/skills/verify-change/references/verification-map.md new
+80
@@ -0,0 +1,80 @@
1 +# Verification Map
2 +
3 +The maintained argv and working directories are in `scripts/agent-profiles.json`.
4 +Run from the repository root with its pinned mise toolchain, Bun and the normal
5 +or Python development environment prepared. Profiles run existing checks and
6 +their normal environment setup (including nox environment creation); they do not
7 +choose a model.
8 +
9 +The CLI and TypeScript SDK profiles use package-scoped Turbo tasks so prerequisite
10 +workspace builds run before tests and type checks, including in a fresh checkout.
11 +
12 +| Profile | Existing checks | What remains outside its proof |
13 +| ------------------ | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
14 +| `workflow` | Bun task integration tests, their TypeScript check and both agent-guidance validators | Agent judgment, product behavior, hosted CI |
15 +| `cli` | CLI package tests, boundary validators and TypeScript checks | Built-binary behavior, real backend behavior |
16 +| `sdk-typescript` | Core SDK unit tests and type checks | Changed providers, runtime packaging, live API behavior |
17 +| `sdk-python` | Existing `make chk` and `make tst` nox sessions in `python/` | Built artifacts, live API behavior, providers outside those sessions |
18 +| `release-metadata` | Existing release-workflow checks | Hosted checks, merges, tags, registries and clean consumer installation |
19 +
20 +Use the package skills for narrower tests or additional affected providers. For
21 +packaging or runtime changes, use existing Node/Cloudflare or CLI Docker E2E
22 +checks after reading `typescript-testing` or `cli-e2e`. For publication, follow
23 +`cli-release` or `python-release`; a local receipt never establishes publication.
24 +
25 +## Carry a task between sessions
26 +
27 +The TypeScript CLI runs with Bun and uses `effect/cli` for argument validation and
28 +generated help. Start with `bun run agent:task --help` or
29 +`bun run agent:task init --help` to inspect commands and required flags.
30 +Task commands print JSON; verification streams check output to stderr.
31 +
32 +```sh
33 +bun run agent:task init cli-output --goal "Reproduce the reported CLI output bug and verify the fix" --profile cli --source https://github.com/ComposioHQ/composio/issues/123 --authority "Implement locally; publication requires the current request"
34 +bun run agent:task status cli-output
35 +bun run agent:task verify cli-output
36 +bun run agent:task list
37 +```
38 +
39 +The id names `.artifacts/agent-tasks/<id>/task.json`. Update its `nextStep` and
40 +`blocker` fields as work progresses. Keep the goal observable; record the real
41 +source issue or PR URL when one exists. The authority field preserves context;
42 +it is never permission to exceed the current user request or repository rules.
43 +
44 +Receipts retain the task, profile, source-content fingerprint and exact HEAD at
45 +verification time. A changed task acceptance contract, tracked or untracked source file, or profile
46 +makes previous passing evidence stale. Ignored artifacts and Claude Code's
47 +`.claude/scheduled_tasks.lock` are excluded, so a
48 +receipt does not cover ignored binaries or environment state changing later.
49 +Fresh external observations are still required for production and registry claims.
50 +Changing only `nextStep` or `blocker` preserves verification evidence.
51 +
52 +Each attempt has its own receipt and a private output log capped at 64 KiB. Longer
53 +output keeps its first 16 KiB and last 48 KiB around an omission marker, so final
54 +errors survive. The log is saved while commands run, so it keeps recent output when the
55 +runner is killed. Attempts are numbered in order, not by clock time. Failed attempts survive subsequent success. Missing tools, failed commands, interruption
56 +and source edits during verification cannot produce a passing attempt. Remaining
57 +commands after the first failure were not run. An abruptly killed runner reads as
58 +interrupted, including when its latest attempt has no finished receipt. A retry
59 +recovers a dead runner's lock once its old command group is gone. If that group
60 +remains alive, or the runner died before saving the command's process group, the
61 +error identifies the command and receipt to inspect before retrying. Live-runner
62 +locks are never recovered automatically. An empty or corrupt `verify.lock` is
63 +replaced only when no task has a running attempt whose runner or command group is
64 +alive. A command that exits while workers it started are still running does not end
65 +its step: the runner stops those workers before the next command or lock release.
66 +
67 +Verification is serialized per worktree because profiles share build and nox outputs.
68 +Use separate worktrees for parallel tasks. An interrupted lock acquisition can leave
69 +`.artifacts/agent-locks/acquire.lock`; inspect its PID and remove it only after
70 +confirming that process has ended.
71 +
72 +## Learn from interventions
73 +
74 +`test/agent-task.test.ts` turns concrete workflow failure classes into runnable
75 +checks: context survives resumption; edits invalidate proof; failures preserve
76 +evidence; missing tools fail rather than skip. These exercise the tool's behavior,
77 +not an LLM's planning quality. For a new recurring mistake, add the smallest
78 +behavioral check at the real failure boundary and demonstrate its failure before
79 +the fix. Keep unrelated passing cases as retention checks. Assess judgment-heavy
80 +changes on new tasks before claiming they reduce supervision.
.github/workflows/agent-substrate.yml
+11 -6
@@ -3,11 +3,11 @@ name: Agent Substrate
3 # Guards the repo-level AI-agent guidance — the files coding agents read —
4 # so it cannot silently drift from reality:
5 #
6 -# * `pnpm validate:agent-skills` — SKILL.md frontmatter and reference links,
6 +# * `bun run validate:agent-skills` — SKILL.md frontmatter and reference links,
7 # the .agents/skills <-> .claude/skills compatibility symlink, required
8 # nested AGENTS.md files, stale guidance references, and every pnpm/make/
9 # nox command mentioned in guidance checked against its real definition.
10 -# * `pnpm validate:skill-routing` — deterministic routing smoke test: each
10 +# * `bun run validate:skill-routing` — deterministic routing smoke test: each
11 # skill keeps a probe asserting it is the unique top match for a
12 # representative task, so description edits cannot silently break routing.
13 #
@@ -15,8 +15,7 @@ name: Agent Substrate
15 # point. It runs on every push and PR with no path filters: the
16 # validate:agent-skills stale-guidance walk scans every text file in the
17 # repo, so any change can affect the result. Both scripts are
18 -# dependency-free Node programs, but the standard toolchain setup is kept
19 -# so the pnpm script wiring is exercised too.
18 +# Bun TypeScript programs using the repository's existing dependencies.
19
20 on:
21 push:
@@ -52,10 +51,16 @@ jobs:
51 run: pnpm install --frozen-lockfile
52
53 - name: Validate agent skills and guidance invariants
55 - run: pnpm validate:agent-skills
54 + run: bun run validate:agent-skills
55
56 - name: Validate skill routing
58 - run: pnpm validate:skill-routing
57 + run: bun run validate:skill-routing
58 +
59 + - name: Typecheck agent tooling
60 + run: bun run typecheck:agent-workflow
61 +
62 + - name: Verify resumable task and evidence contracts
63 + run: bun run test:agent-workflow
64
65 - name: Verify issue triage security boundary
66 run: pnpm test:issue-triage
AGENTS.md
+6 -3
@@ -30,6 +30,7 @@ Use the smallest relevant skill:
30
31 - `repo-guidance`: repo layout, branch/PR workflow, changesets, generated files, or cross-repo maintenance.
32 - `skill-maintenance`: adding or changing local skills, skill references, compatibility mirrors, or validation.
33 +- `verify-change`: resumable task records, verification profiles, revision-bound receipts, or carrying work between coding agents.
34 - `bug-fixing`: fixing a defect and choosing regression tests.
35 - `cross-sdk-parity`: aligning TypeScript and Python behavior, generated client bumps, or API contract drift.
36 - `docs-decisions`: docs site work, changelogs, docs decisions, or docs automation.
@@ -78,11 +79,13 @@ pnpm typecheck
79 pnpm test
80 pnpm test:e2e
81 pnpm test:e2e:cli
81 -pnpm validate:agent-skills
82 -pnpm validate:skill-routing
82 +bun run validate:agent-skills
83 +bun run validate:skill-routing
84 +bun run typecheck:agent-workflow
85 +bun run test:agent-workflow
86 ```
87
85 -Both validators are dependency-free and deterministic. `.github/workflows/agent-substrate.yml` runs them in CI on every push and pull request — the stale-guidance scan covers the whole repo, so any change can affect the result.
88 +Both validators are deterministic Bun TypeScript programs; the guidance validator uses the existing Zod dependency to read package manifests. `.github/workflows/agent-substrate.yml` runs them in CI on every push and pull request — the stale-guidance scan covers the whole repo, so any change can affect the result.
89
90 Python commands run from `python/` unless otherwise noted:
91
CONTRIBUTING.md
+3 -3
@@ -168,13 +168,13 @@ This repository ships its own agent guidance, and CI keeps it accurate. An agent
168 Two deterministic checks guard this guidance:
169
170 ```bash
171 -pnpm validate:agent-skills # frontmatter, reference links, stale guidance refs, command names
172 -pnpm validate:skill-routing # routing smoke test over skill descriptions
171 +bun run validate:agent-skills # frontmatter, reference links, stale guidance refs, command names
172 +bun run validate:skill-routing # routing smoke test over skill descriptions
173 ```
174
175 `validate:agent-skills` parses `package.json`, `python/Makefile`, and `python/noxfile.py`, then verifies that every command mentioned in guidance exists. Both checks run in CI through `.github/workflows/agent-substrate.yml`.
176
177 -If you add or rename a skill, or rewrite a skill description, run both checks and add a routing probe in `ts/scripts/test-skill-routing.mjs`. Read [`.agents/skills/skill-maintenance/SKILL.md`](.agents/skills/skill-maintenance/SKILL.md) before you author or edit a skill.
177 +If you add or rename a skill, or rewrite a skill description, run both checks and add a routing probe in `ts/scripts/test-skill-routing.ts`. Read [`.agents/skills/skill-maintenance/SKILL.md`](.agents/skills/skill-maintenance/SKILL.md) before you author or edit a skill.
178
179 Using an agent doesn't change your responsibility: [read the code you submit](#read-the-code-you-submit).
180
package.json
+8 -3
@@ -32,6 +32,8 @@
32 "test:provider-compatibility": "bun test test/provider-compatibility.test.ts",
33 "test:toolchain": "bun run test/mise-bun-pin.test.ts",
34 "test:context7": "node --test test/context7-check.test.mjs",
35 + "agent:task": "bun scripts/agent-task.ts",
36 + "test:agent-workflow": "bun test --timeout 30000 test/agent-task.test.ts",
37 "check:context7": "node scripts/context7-check.mjs",
38 "test:examples": "tsx ts/scripts/validate-examples.ts",
39 "test:e2e": "turbo test:e2e --filter='@e2e-tests/*'",
@@ -54,9 +56,9 @@
56 "check:peer-deps": "tsx ts/scripts/check-peer-deps.ts",
57 "check:provider-compatibility": "tsx ts/scripts/check-provider-compatibility.ts",
58 "update:peer-deps": "tsx ts/scripts/update-peer-deps.ts",
57 - "validate:agent-skills": "node ts/scripts/validate-agent-skills.mjs",
59 + "validate:agent-skills": "bun ts/scripts/validate-agent-skills.ts",
60 "validate:changesets": "node ts/scripts/validate-changesets.mjs",
59 - "validate:skill-routing": "node ts/scripts/test-skill-routing.mjs",
61 + "validate:skill-routing": "bun ts/scripts/test-skill-routing.ts",
62 "prepublish": "pnpm run build:packages",
63 "changeset": "changeset",
64 "changeset:version": "changeset version",
@@ -64,7 +66,8 @@
66 "changeset:pre-enter": "changeset pre enter",
67 "changeset:pre-exit": "changeset pre exit",
68 "clean": "git clean -xdf .turbo node_modules ts/packages/**/.turbo ts/packages/**/node_modules ts/packages/**/dist",
67 - "clean:workspace": "pnpm turbo clean"
69 + "clean:workspace": "pnpm turbo clean",
70 + "typecheck:agent-workflow": "bun node_modules/typescript/bin/tsc -p tsconfig.agent-workflow.json"
71 },
72 "devEngines": {
73 "runtime": {
@@ -80,10 +83,12 @@
83 "@composio/anthropic": "workspace:*",
84 "@composio/core": "workspace:*",
85 "@composio/openai": "workspace:*",
86 + "@effect/platform-bun": "catalog:",
87 "@types/bun": "catalog:",
88 "@types/node": "^26.6.4",
89 "@types/semver": "^7.8.0",
90 "@vitest/ui": "catalog:",
91 + "effect": "catalog:",
92 "husky": "^9.1.7",
93 "lint-staged": "^17.6.0",
94 "oxlint": "1.89.0",
pnpm-lock.yaml
+6
@@ -153,6 +153,9 @@ importers:
153 '@composio/openai':
154 specifier: workspace:*
155 version: link:ts/packages/providers/openai
156 + '@effect/platform-bun':
157 + specifier: 'catalog:'
158 + version: 4.0.5(effect@4.0.5)
159 '@types/bun':
160 specifier: 'catalog:'
161 version: 1.4.3
@@ -165,6 +168,9 @@ importers:
168 '@vitest/ui':
169 specifier: 'catalog:'
170 version: 5.0.5(vitest@5.0.5)
171 + effect:
172 + specifier: 'catalog:'
173 + version: 4.0.5
174 husky:
175 specifier: ^9.1.7
176 version: 9.1.7
scripts/agent-profiles.json new
+71
@@ -0,0 +1,71 @@
1 +{
2 + "workflow": {
3 + "boundary": "offline workflow contracts and agent guidance",
4 + "commands": [
5 + {
6 + "cwd": ".",
7 + "argv": ["bun", "run", "test:agent-workflow"]
8 + },
9 + {
10 + "cwd": ".",
11 + "argv": ["bun", "run", "typecheck:agent-workflow"]
12 + },
13 + {
14 + "cwd": ".",
15 + "argv": ["bun", "ts/scripts/validate-agent-skills.ts"]
16 + },
17 + {
18 + "cwd": ".",
19 + "argv": ["bun", "ts/scripts/test-skill-routing.ts"]
20 + }
21 + ]
22 + },
23 + "cli": {
24 + "boundary": "offline CLI unit tests and TypeScript checks; no binary or live API proof",
25 + "commands": [
26 + {
27 + "cwd": ".",
28 + "argv": ["bun", "run", "turbo", "run", "test", "--filter=@composio/cli"]
29 + },
30 + {
31 + "cwd": ".",
32 + "argv": ["bun", "run", "turbo", "run", "typecheck", "--filter=@composio/cli"]
33 + }
34 + ]
35 + },
36 + "sdk-typescript": {
37 + "boundary": "offline TypeScript core SDK unit tests and type checks; no provider or runtime packaging proof",
38 + "commands": [
39 + {
40 + "cwd": ".",
41 + "argv": ["bun", "run", "turbo", "run", "test", "--filter=@composio/core"]
42 + },
43 + {
44 + "cwd": ".",
45 + "argv": ["bun", "run", "turbo", "run", "typecheck", "--filter=@composio/core"]
46 + }
47 + ]
48 + },
49 + "sdk-python": {
50 + "boundary": "offline Python SDK and selected provider checks through existing nox sessions",
51 + "commands": [
52 + {
53 + "cwd": "python",
54 + "argv": ["make", "chk"]
55 + },
56 + {
57 + "cwd": "python",
58 + "argv": ["make", "tst"]
59 + }
60 + ]
61 + },
62 + "release-metadata": {
63 + "boundary": "offline release workflow and metadata checks; no hosted CI, publication, registry or consumer-install proof",
64 + "commands": [
65 + {
66 + "cwd": ".",
67 + "argv": ["bun", "run", "test:release-workflow"]
68 + }
69 + ]
70 + }
71 +}
scripts/agent-task.ts new
+653
@@ -0,0 +1,653 @@
1 +#!/usr/bin/env bun
2 +import { createHash, randomUUID } from 'node:crypto';
3 +import { execFileSync, spawn } from 'node:child_process';
4 +import fs from 'node:fs';
5 +import path from 'node:path';
6 +import type { ChildProcessByStdio } from 'node:child_process';
7 +import type { Readable } from 'node:stream';
8 +import { Effect, Option, Runtime } from 'effect';
9 +import { Argument, Command as CliCommand, Flag } from 'effect/cli';
10 +import * as BunServices from '@effect/platform-bun/BunServices';
11 +
12 +type State = 'pending' | 'running' | 'passed' | 'failed' | 'interrupted' | 'stale';
13 +interface Command {
14 + cwd: string;
15 + argv: string[];
16 +}
17 +interface Profile {
18 + boundary: string;
19 + commands: Command[];
20 +}
21 +interface Task {
22 + version: number;
23 + id: string;
24 + goal: string;
25 + profile: string;
26 + source: string | null;
27 + authority: string | null;
28 + nextStep: string | null;
29 + blocker: string | null;
30 + createdAt: string;
31 +}
32 +interface Subject {
33 + head: string;
34 + fingerprint: string;
35 +}
36 +interface CommandResult extends Command {
37 + startedAt: string;
38 + finishedAt: string | null;
39 + exitCode: number | null;
40 + signal: NodeJS.Signals | null;
41 + pid?: number | null;
42 + error?: string;
43 +}
44 +interface Receipt {
45 + version: number;
46 + id: string;
47 + runnerPid: number;
48 + startedAt: string;
49 + finishedAt: string | null;
50 + status: State;
51 + taskDigest: string;
52 + profileDigest: string;
53 + subject: Subject | null;
54 + boundary: string;
55 + commands: CommandResult[];
56 + bunVersion: string;
57 + error?: string;
58 + logTruncated?: boolean;
59 +}
60 +type StoredReceipt = Partial<Receipt> & Pick<Receipt, 'id' | 'status'> & { file: string };
61 +interface LockOwner {
62 + pid: number;
63 + task: string;
64 +}
65 +const errorCode = (error: unknown) =>
66 + error instanceof Error && 'code' in error ? error.code : undefined;
67 +const errorMessage = (error: unknown) => (error instanceof Error ? error.message : String(error));
68 +
69 +const root = path.resolve(import.meta.dir, '..');
70 +const tasksRoot = path.join(root, '.artifacts/agent-tasks');
71 +// Locks live outside the task namespace so no task id can collide with them.
72 +const locksRoot = path.join(root, '.artifacts/agent-locks');
73 +const logHeadLimit = 16 * 1024;
74 +const logTailLimit = 48 * 1024;
75 +const readJSON = <T>(file: string): T => JSON.parse(fs.readFileSync(file, 'utf8'));
76 +const digest = (value: unknown) =>
77 + createHash('sha256')
78 + .update(JSON.stringify(value) ?? 'null')
79 + .digest('hex');
80 +const taskDigest = ({ version, id, goal, profile, source, authority }: Task) =>
81 + digest({ version, id, goal, profile, source, authority });
82 +const now = () => new Date().toISOString();
83 +const processGroup = (pid: number) => (process.platform === 'win32' ? pid : -pid);
84 +const print = (value: unknown) => console.log(JSON.stringify(value, null, 2));
85 +const git = (args: string[], cwd = root) =>
86 + execFileSync('git', args, { cwd, maxBuffer: 64 * 1024 * 1024 });
87 +const writeJSON = (file: string, value: unknown) => {
88 + fs.writeFileSync(`${file}.tmp`, `${JSON.stringify(value, null, 2)}\n`, { mode: 0o600 });
89 + fs.renameSync(`${file}.tmp`, file);
90 +};
91 +
92 +const taskIdPattern = /^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/;
93 +function taskDirectory(id: string) {
94 + if (!taskIdPattern.test(id ?? ''))
95 + throw new Error(
96 + 'Use a task id of 1-64 letters, digits, dots, underscores or hyphens, starting with a letter or digit.'
97 + );
98 + return path.join(tasksRoot, id);
99 +}
100 +
101 +function profileFor(task: Task) {
102 + const profile = readJSON<Record<string, Profile>>(path.join(root, 'scripts/agent-profiles.json'))[
103 + task.profile
104 + ];
105 + if (
106 + !profile ||
107 + typeof profile.boundary !== 'string' ||
108 + !Array.isArray(profile.commands) ||
109 + profile.commands.length === 0
110 + )
111 + throw new Error(`Unknown or empty profile: ${task.profile}`);
112 + for (const command of profile.commands) {
113 + if (
114 + !Array.isArray(command.argv) ||
115 + command.argv.length === 0 ||
116 + command.argv.some(arg => typeof arg !== 'string') ||
117 + typeof command.cwd !== 'string'
118 + )
119 + throw new Error('Profiles require cwd and a nonempty argv array.');
120 + const relative = path.relative(root, path.resolve(root, command.cwd));
121 + if (relative.startsWith('..') || path.isAbsolute(relative))
122 + throw new Error('Profile cwd must stay inside the repository.');
123 + }
124 + return profile;
125 +}
126 +
127 +function alive(pid: number | null | undefined) {
128 + if (typeof pid !== 'number' || !Number.isSafeInteger(pid) || pid === 0) return false;
129 + try {
130 + process.kill(pid, 0);
131 + return true;
132 + } catch (error) {
133 + if (errorCode(error) === 'ESRCH') return false;
134 + if (errorCode(error) === 'EPERM') return true;
135 + throw error;
136 + }
137 +}
138 +
139 +// Attempts are numbered under the worktree lock, so their order does not depend on the clock.
140 +const attemptNumber = (name: string) => Number.parseInt(name, 10);
141 +function latestAttempt(directory: string) {
142 + const attempts = path.join(directory, 'attempts');
143 + return fs.existsSync(attempts)
144 + ? fs
145 + .readdirSync(attempts)
146 + .filter(name => Number.isSafeInteger(attemptNumber(name)))
147 + .sort((a, b) => attemptNumber(a) - attemptNumber(b))
148 + .at(-1)
149 + : undefined;
150 +}
151 +
152 +function latestReceipt(directory: string): StoredReceipt | null {
153 + const attempts = path.join(directory, 'attempts');
154 + const latest = latestAttempt(directory);
155 + if (!latest) return null;
156 + const file = path.join(attempts, latest, 'receipt.json');
157 + return fs.existsSync(file)
158 + ? { file, ...readJSON<Receipt>(file) }
159 + : { file, id: latest, status: 'interrupted' };
160 +}
161 +
162 +function taskIds() {
163 + return fs.existsSync(tasksRoot)
164 + ? fs
165 + .readdirSync(tasksRoot)
166 + .filter(
167 + name =>
168 + taskIdPattern.test(name) &&
169 + fs.statSync(path.join(tasksRoot, name), { throwIfNoEntry: false })?.isDirectory()
170 + )
171 + .sort()
172 + : [];
173 +}
174 +
175 +function readLockOwner(file: string): LockOwner | null {
176 + let owner: Partial<LockOwner> | null;
177 + try {
178 + owner = readJSON<Partial<LockOwner> | null>(file);
179 + } catch (error) {
180 + if (error instanceof SyntaxError) return null;
181 + throw error;
182 + }
183 + return typeof owner?.pid === 'number' &&
184 + Number.isSafeInteger(owner.pid) &&
185 + owner.pid > 0 &&
186 + typeof owner.task === 'string' &&
187 + taskIdPattern.test(owner.task)
188 + ? { pid: owner.pid, task: owner.task }
189 + : null;
190 +}
191 +
192 +// A command saved without a pid may have started before its runner died, so its group is unknown.
193 +function commandGroupBlocker(receipt: StoredReceipt | null) {
194 + const command = receipt?.commands?.at(-1);
195 + if (!command) return null;
196 + if (command.pid === undefined && !command.finishedAt)
197 + return `command \`${command.argv.join(' ')}\` has no recorded process group and may still be running. Stop any process it started, then remove the lock`;
198 + return alive(command.pid ? processGroup(command.pid) : undefined)
199 + ? `command group ${command.pid} remains. Inspect and stop that group before retrying`
200 + : null;
201 +}
202 +
203 +const groupAlive = (pid: number) => alive(processGroup(pid));
204 +const signalGroup = (pid: number, signal: NodeJS.Signals) => {
205 + try {
206 + process.kill(processGroup(pid), signal);
207 + } catch (error) {
208 + if (errorCode(error) !== 'ESRCH') throw error;
209 + }
210 +};
211 +
212 +// A command can exit while workers it started keep running and writing shared outputs.
213 +async function endGroup(pid: number) {
214 + for (const [signal, grace] of [
215 + ['SIGTERM', 1500],
216 + ['SIGKILL', 5000],
217 + ] as const) {
218 + if (!groupAlive(pid)) return;
219 + signalGroup(pid, signal);
220 + for (let waited = 0; waited < grace && groupAlive(pid); waited += 25) await Bun.sleep(25);
221 + }
222 + if (groupAlive(pid)) throw new Error(`Command group ${pid} survived SIGKILL.`);
223 +}
224 +
225 +// Without a readable owner, only the attempts themselves can show that verification is still live.
226 +function assertNoLiveAttempt(file: string) {
227 + for (const id of taskIds()) {
228 + let receipt: StoredReceipt | null;
229 + try {
230 + receipt = latestReceipt(taskDirectory(id));
231 + } catch {
232 + continue;
233 + }
234 + if (receipt?.status !== 'running') continue;
235 + if (alive(receipt.runnerPid) || commandGroupBlocker(receipt))
236 + throw new Error(
237 + `${file} is unreadable and task ${id} still has a running attempt (runner ${receipt.runnerPid}); inspect ${receipt.file} before retrying.`
238 + );
239 + }
240 +}
241 +
242 +function lockTask(task: string) {
243 + const file = path.join(locksRoot, 'verify.lock');
244 + const acquisitionFile = path.join(locksRoot, 'acquire.lock');
245 + fs.mkdirSync(locksRoot, { recursive: true, mode: 0o700 });
246 + const acquisition = fs.openSync(acquisitionFile, 'wx', 0o600);
247 + try {
248 + fs.writeFileSync(acquisition, String(process.pid));
249 + const owner = fs.existsSync(file) ? readLockOwner(file) : undefined;
250 + if (owner === null) {
251 + assertNoLiveAttempt(file);
252 + console.error(`Replacing unreadable lock ${file}; no task attempt is still running.`);
253 + fs.unlinkSync(file);
254 + } else if (owner) {
255 + if (alive(owner.pid))
256 + throw new Error(
257 + `Worktree verification is owned by task ${owner.task} (runner ${owner.pid}); inspect ${file} before retrying.`
258 + );
259 + const previous = latestReceipt(taskDirectory(owner.task));
260 + const blocker = commandGroupBlocker(previous);
261 + if (blocker)
262 + throw new Error(
263 + `Runner ${owner.pid} ended but its ${blocker}. Lock: ${file}. Receipt: ${previous?.file}.`
264 + );
265 + fs.unlinkSync(file);
266 + }
267 + const handle = fs.openSync(file, 'wx', 0o600);
268 + try {
269 + fs.writeFileSync(handle, JSON.stringify({ pid: process.pid, task }));
270 + } catch (error) {
271 + fs.unlinkSync(file);
272 + throw error;
273 + } finally {
274 + fs.closeSync(handle);
275 + }
276 + return file;
277 + } finally {
278 + fs.closeSync(acquisition);
279 + fs.unlinkSync(acquisitionFile);
280 + }
281 +}
282 +
283 +// Claude Code rewrites this tracked file while sessions run; it is not a verification input.
284 +const volatilePaths = ['.claude/scheduled_tasks.lock'];
285 +
286 +function subject() {
287 + const hash = createHash('sha256');
288 + const addTree = (cwd: string, prefix = '') => {
289 + const pathspec = prefix ? [] : ['--', '.', ...volatilePaths.map(file => `:(exclude)${file}`)];
290 + hash.update(git(['ls-files', '--stage', '-z', ...pathspec], cwd));
291 + const names = new Set(
292 + git(['ls-files', '--cached', '--others', '--exclude-standard', '-z', ...pathspec], cwd)
293 + .toString()
294 + .split('\0')
295 + .filter(Boolean)
296 + );
297 + for (const name of [...names].sort()) {
298 + hash.update(`${prefix}${name}\0`);
299 + const file = path.join(cwd, name);
300 + try {
301 + const stat = fs.lstatSync(file);
302 + hash.update(`${stat.mode}\0`);
303 + if (stat.isSymbolicLink()) hash.update(fs.readlinkSync(file));
304 + else if (stat.isFile()) hash.update(fs.readFileSync(file));
305 + else if (stat.isDirectory() && fs.existsSync(path.join(file, '.git')))
306 + addTree(file, `${prefix}${name}/`);
307 + } catch (error) {
308 + if (errorCode(error) !== 'ENOENT') throw error;
309 + hash.update('deleted');
310 + }
311 + hash.update('\0');
312 + }
313 + };
314 + addTree(root);
315 + return { head: git(['rev-parse', 'HEAD']).toString().trim(), fingerprint: hash.digest('hex') };
316 +}
317 +
318 +function status(id: string) {
319 + const directory = taskDirectory(id);
320 + const task = readJSON<Task>(path.join(directory, 'task.json'));
321 + const receipt = latestReceipt(directory);
322 + let state = receipt?.status ?? 'pending';
323 + if (state === 'running' && receipt && !alive(receipt.runnerPid)) state = 'interrupted';
324 + if (state === 'passed' && receipt) {
325 + const profile = profileFor(task);
326 + const complete =
327 + receipt.commands?.length === profile.commands.length &&
328 + receipt.commands.every(command => command.exitCode === 0 && command.finishedAt);
329 + state =
330 + complete &&
331 + receipt.taskDigest === taskDigest(task) &&
332 + receipt.profileDigest === digest(profile) &&
333 + digest(receipt.subject) === digest(subject())
334 + ? 'passed'
335 + : 'stale';
336 + }
337 + return {
338 + task,
339 + status: state,
340 + boundary: receipt?.boundary ?? profileFor(task).boundary,
341 + receipt: receipt ? path.relative(root, receipt.file) : null,
342 + };
343 +}
344 +
345 +async function verify(id: string) {
346 + const directory = taskDirectory(id);
347 + const task = readJSON<Task>(path.join(directory, 'task.json'));
348 + const profile = profileFor(task);
349 + const lock = lockTask(id);
350 + const attempt = String(attemptNumber(latestAttempt(directory) ?? '0') + 1);
351 + const attemptDirectory = path.join(directory, 'attempts', attempt);
352 + const receiptFile = path.join(attemptDirectory, 'receipt.json');
353 + const receipt: Receipt = {
354 + version: 1,
355 + id: attempt,
356 + runnerPid: process.pid,
357 + startedAt: now(),
358 + finishedAt: null,
359 + status: 'running',
360 + taskDigest: taskDigest(task),
361 + profileDigest: digest(profile),
362 + subject: null,
363 + boundary: profile.boundary,
364 + commands: [],
365 + bunVersion: Bun.version,
366 + };
367 + let child: ChildProcessByStdio<null, Readable, Readable> | undefined;
368 + let cancellation: Promise<void> | undefined;
369 + let interrupted: NodeJS.Signals | undefined;
370 + let logBytes = 0;
371 + // The log keeps the start and end of combined output; late failures matter most.
372 + let logHead = 0;
373 + const logTail: Buffer[] = [];
374 + let logTailBytes = 0;
375 + let flushTimer: ReturnType<typeof setTimeout> | undefined;
376 + const flushLog = () => {
377 + clearTimeout(flushTimer);
378 + flushTimer = undefined;
379 + if (logBytes === logHead) return;
380 + const tail = Buffer.concat(logTail).subarray(-logTailLimit);
381 + const omitted = logBytes - logHead - tail.length;
382 + const head = fs.readFileSync(logFile).subarray(0, logHead);
383 + const marker = omitted > 0 ? `\n[agent-task: ${omitted} bytes of output omitted]\n` : '';
384 + fs.writeFileSync(logFile, Buffer.concat([head, Buffer.from(marker), tail]));
385 + };
386 + const stop = (signal: NodeJS.Signals) => {
387 + interrupted = signal;
388 + if (!child?.pid || cancellation) return;
389 + const pid = child.pid;
390 + signalGroup(pid, signal);
391 + cancellation = new Promise<void>(resolve =>
392 + setTimeout(() => {
393 + // Once the command closes, its group id may be reused; endGroup owns any stragglers.
394 + if (child?.pid === pid) signalGroup(pid, 'SIGKILL');
395 + resolve();
396 + }, 1500)
397 + );
398 + };
399 + // A closed terminal sends SIGHUP; detached command groups would otherwise outlive the runner.
400 + const stopSignals = ['SIGINT', 'SIGTERM', 'SIGHUP'] as const;
401 + for (const signal of stopSignals) process.on(signal, stop);
402 + const logFile = path.join(attemptDirectory, 'output.log');
403 + try {
404 + fs.mkdirSync(attemptDirectory, { recursive: true, mode: 0o700 });
405 + fs.writeFileSync(logFile, '', { mode: 0o600 });
406 + writeJSON(receiptFile, receipt);
407 + receipt.subject = subject();
408 + writeJSON(receiptFile, receipt);
409 + for (const command of profile.commands) {
410 + if (interrupted) break;
411 + const result: CommandResult = {
412 + ...command,
413 + startedAt: now(),
414 + finishedAt: null,
415 + exitCode: null,
416 + signal: null,
417 + };
418 + receipt.commands.push(result);
419 + writeJSON(receiptFile, receipt);
420 + await new Promise<void>(resolve => {
421 + child = spawn(command.argv[0], command.argv.slice(1), {
422 + cwd: path.resolve(root, command.cwd),
423 + shell: false,
424 + detached: process.platform !== 'win32',
425 + stdio: ['ignore', 'pipe', 'pipe'],
426 + });
427 + result.pid = child.pid ?? null;
428 + writeJSON(receiptFile, receipt);
429 + const output = (chunk: Uint8Array) => {
430 + process.stderr.write(chunk);
431 + logBytes += chunk.length;
432 + const head = chunk.subarray(0, logHeadLimit - logHead);
433 + if (head.length > 0) {
434 + fs.appendFileSync(logFile, head);
435 + logHead += head.length;
436 + }
437 + if (head.length === chunk.length) return;
438 + logTail.push(Buffer.from(chunk.subarray(head.length)));
439 + logTailBytes += chunk.length - head.length;
440 + while (logTailBytes - logTail[0]!.length >= logTailLimit)
441 + logTailBytes -= logTail.shift()!.length;
442 + // Save the tail while the command runs so a killed runner keeps recent diagnostics.
443 + flushTimer ??= setTimeout(() => {
444 + try {
445 + flushLog();
446 + } catch {}
447 + }, 250);
448 + };
449 + child.stdout.on('data', output);
450 + child.stderr.on('data', output);
451 + child.on('error', error => {
452 + result.error = error.message;
453 + });
454 + child.on('close', (code, signal) => {
455 + result.exitCode = code;
456 + result.signal = signal;
457 + resolve();
458 + });
459 + });
460 + // Interrupts stop targeting the group here; endGroup signals it only while it still has members.
461 + child = undefined;
462 + if (result.pid) await endGroup(result.pid);
463 + result.finishedAt = now();
464 + flushLog();
465 + writeJSON(receiptFile, receipt);
466 + if (result.exitCode !== 0 || result.error) break;
467 + }
468 + receipt.status = interrupted
469 + ? 'interrupted'
470 + : receipt.commands.length === profile.commands.length &&
471 + receipt.commands.every(command => command.exitCode === 0 && !command.error)
472 + ? 'passed'
473 + : 'failed';
474 + if (
475 + receipt.status === 'passed' &&
476 + (digest(receipt.subject) !== digest(subject()) ||
477 + receipt.taskDigest !== taskDigest(readJSON<Task>(path.join(directory, 'task.json'))) ||
478 + receipt.profileDigest !== digest(profileFor(task)))
479 + )
480 + receipt.status = 'stale';
481 + } catch (error) {
482 + receipt.status = 'failed';
483 + receipt.error = errorMessage(error);
484 + } finally {
485 + await cancellation;
486 + // Release the worktree only once the last command group is gone. A finished command's group
487 + // was already ended, and its id may since belong to an unrelated process group.
488 + let groupEnded = true;
489 + const last = receipt.commands.at(-1);
490 + if (last?.pid && !last.finishedAt)
491 + await endGroup(last.pid).catch((error: unknown) => {
492 + groupEnded = false;
493 + receipt.status = 'failed';
494 + receipt.error ??= errorMessage(error);
495 + });
496 + receipt.finishedAt = now();
497 + receipt.logTruncated = logBytes > logHeadLimit + logTailLimit;
498 + try {
499 + flushLog();
500 + } catch (error) {
501 + receipt.error ??= `Could not write the output log: ${errorMessage(error)}`;
502 + }
503 + try {
504 + if (fs.existsSync(attemptDirectory)) writeJSON(receiptFile, receipt);
505 + } finally {
506 + if (groupEnded) fs.unlinkSync(lock);
507 + for (const signal of stopSignals) process.off(signal, stop);
508 + }
509 + }
510 + print(status(id));
511 + if (receipt.status !== 'passed') process.exitCode = 1;
512 +}
513 +
514 +const taskId = Argument.String('id').pipe(Argument.withDescription('Local task identifier'));
515 +const operation = (run: () => void | Promise<void>) =>
516 + Effect.tryPromise({ try: async () => run(), catch: (error: unknown) => error });
517 +const stringFlag = (name: string) =>
518 + Flag.String(name).pipe(
519 + Flag.between(1, 1),
520 + Flag.map(values => values[0]!)
521 + );
522 +
523 +const initFlags = {
524 + goal: stringFlag('goal').pipe(Flag.withDescription('Observable outcome to implement and verify')),
525 + profile: stringFlag('profile').pipe(
526 + Flag.withDescription('Check group from scripts/agent-profiles.json')
527 + ),
528 + source: stringFlag('source').pipe(
529 + Flag.withDescription('Original issue or PR URL'),
530 + Flag.optional
531 + ),
532 + authority: stringFlag('authority').pipe(
533 + Flag.withDescription('User-authorized scope to preserve as context'),
534 + Flag.optional
535 + ),
536 +};
537 +const initCommand = CliCommand.make(
538 + 'init',
539 + { id: taskId, ...initFlags },
540 + ({ id, goal, profile, source, authority }) =>
541 + operation(() => {
542 + if (!goal.trim() || !profile)
543 + throw new Error('Task initialization requires --goal and --profile.');
544 + const directory = taskDirectory(id);
545 + const task: Task = {
546 + version: 1,
547 + id,
548 + goal,
549 + profile,
550 + source: Option.getOrNull(source),
551 + authority: Option.getOrNull(authority),
552 + nextStep: null,
553 + blocker: null,
554 + createdAt: now(),
555 + };
556 + profileFor(task);
557 + // Publish the task folder in one rename so an interrupted init leaves no partial task.
558 + // An empty folder left by an older interrupted init is replaced; a real task is kept.
559 + const staging = path.join(tasksRoot, `.init-${randomUUID()}`);
560 + fs.mkdirSync(staging, { recursive: true, mode: 0o700 });
561 + try {
562 + writeJSON(path.join(staging, 'task.json'), task);
563 + fs.renameSync(staging, directory);
564 + } catch (error) {
565 + fs.rmSync(staging, { recursive: true, force: true });
566 + if (errorCode(error) === 'ENOTEMPTY' || errorCode(error) === 'EEXIST')
567 + throw new Error(`Task ${id} already exists.`);
568 + throw error;
569 + }
570 + print(status(id));
571 + })
572 +).pipe(
573 + CliCommand.withDescription('Save a task goal and select its verification checks'),
574 + CliCommand.withExamples([
575 + {
576 + command:
577 + 'bun run agent:task init cli-fix --goal "Fix the reported CLI behavior" --profile cli',
578 + description: 'Start a CLI task',
579 + },
580 + ])
581 +);
582 +const statusCommand = CliCommand.make('status', { id: taskId }, ({ id }) =>
583 + operation(() => print(status(id)))
584 +).pipe(
585 + CliCommand.withDescription('Read saved context and check whether verification is still current')
586 +);
587 +const verifyCommand = CliCommand.make('verify', { id: taskId }, ({ id }) =>
588 + operation(() => verify(id))
589 +).pipe(CliCommand.withDescription('Run the selected checks and retain a revision-bound receipt'));
590 +const listCommand = CliCommand.make('list', {}, () =>
591 + operation(() => {
592 + print(
593 + taskIds().map(id => {
594 + try {
595 + return status(id);
596 + } catch (error) {
597 + return { id, status: 'unreadable', error: errorMessage(error) };
598 + }
599 + })
600 + );
601 + })
602 +).pipe(CliCommand.withDescription('List saved tasks and their verification status'));
603 +const rootCommand = CliCommand.make('agent:task').pipe(
604 + CliCommand.withDescription('Preserve task context and run repository verification checks'),
605 + CliCommand.withSubcommands([initCommand, statusCommand, verifyCommand, listCommand]),
606 + CliCommand.withExamples([
607 + { command: 'bun run agent:task status cli-fix', description: 'Resume a saved task' },
608 + {
609 + command: 'bun run agent:task verify cli-fix',
610 + description: 'Verify the current revision after implementing the fix',
611 + },
612 + ])
613 +);
614 +
615 +function taskArguments(args: readonly string[]): string[] {
616 + const valueFlags = new Set(Object.keys(initFlags).map(name => `--${name}`));
617 + const result: string[] = [];
618 + const tokens = args[Symbol.iterator]();
619 + for (const token of tokens) {
620 + if (valueFlags.has(token)) {
621 + const value = tokens.next();
622 + result.push(value.done ? token : `${token}=${value.value}`);
623 + } else result.push(token);
624 + }
625 + return result;
626 +}
627 +
628 +// Bind literal values before Effect scans built-in flags, so task text cannot activate help.
629 +// Verification owns signal cleanup so its command groups stop before the worktree lock is released.
630 +Effect.runPromise(
631 + CliCommand.runWith(rootCommand, { version: '1' })(taskArguments(process.argv.slice(2))).pipe(
632 + Effect.provide(BunServices.layer),
633 + Effect.catchTags({
634 + ShowHelp: (error: unknown) =>
635 + Effect.sync(() => {
636 + process.exitCode = Runtime.getErrorExitCode(error);
637 + }),
638 + UserError: (error: unknown) =>
639 + Effect.sync(() => {
640 + process.exitCode = Runtime.getErrorExitCode(error);
641 + }),
642 + }),
643 + Effect.catch(error =>
644 + Effect.sync(() => {
645 + console.error(errorMessage(error));
646 + process.exitCode = Runtime.getErrorExitCode(error);
647 + })
648 + )
649 + )
650 +).catch((error: unknown) => {
651 + console.error(errorMessage(error));
652 + process.exitCode = 1;
653 +});
test/agent-task.test.ts new
+648
@@ -0,0 +1,648 @@
1 +import assert from 'node:assert/strict';
2 +import { execFileSync, spawn, spawnSync } from 'node:child_process';
3 +import {
4 + copyFileSync,
5 + existsSync,
6 + mkdirSync,
7 + mkdtempSync,
8 + readFileSync,
9 + readdirSync,
10 + rmSync,
11 + symlinkSync,
12 + writeFileSync,
13 +} from 'node:fs';
14 +import os from 'node:os';
15 +import path from 'node:path';
16 +import { afterEach, test } from 'bun:test';
17 +import { setTimeout as delay } from 'node:timers/promises';
18 +
19 +interface TaskStatus {
20 + task: {
21 + id: string;
22 + goal: string;
23 + nextStep: string | null;
24 + blocker: string | null;
25 + authority: string;
26 + };
27 + status: string;
28 + receipt: string;
29 +}
30 +type Fixture = ReturnType<typeof fixture>;
31 +
32 +const script = new URL('../scripts/agent-task.ts', import.meta.url);
33 +const cleanups: Array<() => void> = [];
34 +afterEach(() => {
35 + for (const cleanup of cleanups.splice(0).reverse()) cleanup();
36 +});
37 +
38 +const lingeringWorker =
39 + 'process.on("SIGTERM",()=>{});require("fs").writeFileSync(".artifacts/child.pid",String(process.pid));setInterval(()=>{},1000)';
40 +
41 +const fixture = () => {
42 + const root = mkdtempSync(path.join(os.tmpdir(), 'composio-agent-task-'));
43 + cleanups.push(() => rmSync(root, { recursive: true, force: true }));
44 + mkdirSync(path.join(root, 'scripts'));
45 + copyFileSync(script, path.join(root, 'scripts/agent-task.ts'));
46 + symlinkSync(new URL('../node_modules', import.meta.url), path.join(root, 'node_modules'), 'dir');
47 + writeFileSync(path.join(root, '.gitignore'), '.artifacts/\nnode_modules\n');
48 + writeFileSync(path.join(root, 'source.txt'), 'original');
49 + mkdirSync(path.join(root, '.claude'));
50 + writeFileSync(path.join(root, '.claude/scheduled_tasks.lock'), 'session one');
51 + const profiles = {
52 + offline: {
53 + boundary: 'offline',
54 + commands: [{ cwd: '.', argv: [process.execPath, '-e', 'console.log("verified")'] }],
55 + },
56 + failure: {
57 + boundary: 'offline',
58 + commands: [
59 + {
60 + cwd: '.',
61 + argv: [
62 + process.execPath,
63 + '-e',
64 + 'process.exit(require("fs").existsSync(".artifacts/succeed") ? 0 : 7)',
65 + ],
66 + },
67 + { cwd: '.', argv: [process.execPath, '-e', 'console.log("second")'] },
68 + ],
69 + },
70 + missing: {
71 + boundary: 'offline',
72 + commands: [{ cwd: '.', argv: ['composio-test-missing-executable'] }],
73 + },
74 + mutating: {
75 + boundary: 'offline',
76 + commands: [
77 + {
78 + cwd: '.',
79 + argv: [
80 + process.execPath,
81 + '-e',
82 + 'require("fs").writeFileSync("source.txt", "changed during verification")',
83 + ],
84 + },
85 + ],
86 + },
87 + noisy: {
88 + boundary: 'offline',
89 + commands: [
90 + {
91 + cwd: '.',
92 + argv: [
93 + process.execPath,
94 + '-e',
95 + 'console.log("first line");console.log("x".repeat(200000));console.error("final type error");process.exitCode=2',
96 + ],
97 + },
98 + ],
99 + },
100 + interrupt: {
101 + boundary: 'offline',
102 + commands: [
103 + {
104 + cwd: '.',
105 + argv: [
106 + process.execPath,
107 + '-e',
108 + 'const child=require("child_process").spawn(process.execPath,["-e","setInterval(()=>{},1000)"],{stdio:"inherit"});require("fs").writeFileSync(".artifacts/child.pid",String(child.pid));setInterval(()=>{},1000)',
109 + ],
110 + },
111 + ],
112 + },
113 + lingering: {
114 + boundary: 'offline',
115 + commands: [
116 + {
117 + cwd: '.',
118 + argv: [
119 + process.execPath,
120 + '-e',
121 + `require("child_process").spawn(process.execPath,["-e",${JSON.stringify(lingeringWorker)}],{stdio:"ignore"});setInterval(()=>{if(require("fs").existsSync(".artifacts/child.pid"))process.exit(0)},10)`,
122 + ],
123 + },
124 + ],
125 + },
126 + };
127 + writeFileSync(path.join(root, 'scripts/agent-profiles.json'), JSON.stringify(profiles));
128 + for (const args of [
129 + ['init', '-q'],
130 + ['add', '.'],
131 + ['-c', 'user.name=Test', '-c', 'user.email=test@example.invalid', 'commit', '-qm', 'baseline'],
132 + ]) {
133 + execFileSync('git', args, { cwd: root });
134 + }
135 + const run = (...args: string[]) =>
136 + spawnSync(process.execPath, ['scripts/agent-task.ts', ...args], {
137 + cwd: root,
138 + encoding: 'utf8',
139 + });
140 + const state = (id: string): TaskStatus => JSON.parse(run('status', id).stdout);
141 + const init = (id = 'case', profile = 'offline') =>
142 + run(
143 + 'init',
144 + id,
145 + '--goal',
146 + 'Verify the changed behavior',
147 + '--profile',
148 + profile,
149 + '--authority',
150 + 'local verification only'
151 + );
152 + return { root, run, state, init };
153 +};
154 +
155 +async function runningFixture(f: Fixture) {
156 + assert.equal(f.init('case', 'interrupt').status, 0);
157 + const runner = spawn(process.execPath, ['scripts/agent-task.ts', 'verify', 'case'], {
158 + cwd: f.root,
159 + stdio: 'ignore',
160 + });
161 + const exit = new Promise(resolve =>
162 + runner.on('exit', (code, signal) => resolve({ code, signal }))
163 + );
164 + cleanups.push(() => runner.kill('SIGKILL'));
165 + const file = path.join(f.root, '.artifacts/child.pid');
166 + for (let i = 0; i < 250 && !existsSync(file); i++) await delay(20);
167 + const childPid = Number(readFileSync(file, 'utf8'));
168 + const receipt = JSON.parse(readFileSync(path.join(f.root, f.state('case').receipt), 'utf8'));
169 + const group = receipt.commands[0].pid;
170 + cleanups.push(() => {
171 + try {
172 + process.kill(-group, 'SIGKILL');
173 + } catch {}
174 + });
175 + return { runner, exit, childPid, group };
176 +}
177 +
178 +async function waitGone(pid: number) {
179 + for (let i = 0; i < 250; i++) {
180 + try {
181 + process.kill(pid, 0);
182 + } catch (error) {
183 + const code = error instanceof Error && 'code' in error ? error.code : undefined;
184 + if (code === 'ESRCH') return;
185 + assert.equal(code, 'EPERM');
186 + }
187 + await delay(20);
188 + }
189 + assert.fail(`Process ${pid} did not stop`);
190 +}
191 +
192 +function useOfflineProfile(f: Fixture) {
193 + const file = path.join(f.root, 'scripts/agent-profiles.json');
194 + const profiles = JSON.parse(readFileSync(file, 'utf8'));
195 + profiles.interrupt.commands = profiles.offline.commands;
196 + writeFileSync(file, JSON.stringify(profiles));
197 +}
198 +
199 +test('CLI help describes commands and required initialization inputs without creating a task', () => {
200 + const f = fixture();
201 + const rootHelp = f.run('--help');
202 + assert.equal(rootHelp.status, 0, rootHelp.stderr);
203 + assert.match(rootHelp.stdout, /init/);
204 + assert.match(rootHelp.stdout, /verify/);
205 + const initHelp = f.run('init', '--help');
206 + assert.equal(initHelp.status, 0, initHelp.stderr);
207 + assert.match(initHelp.stdout, /--goal/);
208 + assert.match(initHelp.stdout, /--profile/);
209 + assert.equal(existsSync(path.join(f.root, '.artifacts/agent-tasks')), false);
210 +});
211 +
212 +test('invalid CLI inputs fail before creating task state', () => {
213 + const f = fixture();
214 + const missing = f.run('init', 'case', '--profile', 'offline');
215 + assert.equal(missing.status, 1);
216 + assert.match(missing.stderr, /goal/i);
217 + const unknown = f.run(
218 + 'init',
219 + 'case',
220 + '--goal',
221 + 'Check behavior',
222 + '--profile',
223 + 'offline',
224 + '--bogus'
225 + );
226 + assert.equal(unknown.status, 1);
227 + assert.match(unknown.stderr, /bogus/i);
228 + const completions = f.run('--completions', 'nope');
229 + assert.equal(completions.status, 1);
230 + assert.match(completions.stderr, /nope/);
231 + const duplicate = f.run(
232 + 'init',
233 + 'case',
234 + '--goal',
235 + 'A goal',
236 + '--profile',
237 + 'offline',
238 + '--profile',
239 + 'failure'
240 + );
241 + assert.equal(duplicate.status, 1);
242 + assert.match(duplicate.stderr, /profile/);
243 + const authority = f.run(
244 + 'init',
245 + 'case',
246 + '--goal',
247 + 'A goal',
248 + '--profile',
249 + 'offline',
250 + '--authority',
251 + 'First scope',
252 + '--authority',
253 + 'Other scope'
254 + );
255 + assert.equal(authority.status, 1);
256 + assert.match(authority.stderr, /authority/);
257 + assert.equal(existsSync(path.join(f.root, '.artifacts/agent-tasks')), false);
258 +});
259 +
260 +test('dash-leading task text is saved literally instead of activating built-in flags', () => {
261 + const f = fixture();
262 + assert.equal(
263 + f.run(
264 + 'init',
265 + 'case',
266 + '--goal',
267 + '- Fix the CLI',
268 + '--profile',
269 + 'offline',
270 + '--authority',
271 + '- No merge'
272 + ).status,
273 + 0
274 + );
275 + assert.equal(existsSync(path.join(f.root, '.artifacts/agent-tasks/case/task.json')), true);
276 + assert.equal(f.state('case').task.goal, '- Fix the CLI');
277 + assert.equal(f.state('case').task.authority, '- No merge');
278 +});
279 +
280 +test('task context survives a fresh invocation and duplicate init preserves it', () => {
281 + const f = fixture();
282 + assert.equal(f.init().status, 0);
283 + const task = f.state('case').task;
284 + assert.equal(task.goal, 'Verify the changed behavior');
285 + assert.equal(task.authority, 'local verification only');
286 + assert.equal(f.init().status, 1);
287 + assert.deepEqual(f.state('case').task, task);
288 + assert.equal(f.state('case').status, 'pending');
289 + assert.equal(JSON.parse(f.run('list').stdout)[0].task.id, 'case');
290 +});
291 +
292 +test('list reports an unreadable task folder without hiding the others', () => {
293 + const f = fixture();
294 + assert.equal(f.init().status, 0);
295 + mkdirSync(path.join(f.root, '.artifacts/agent-tasks/broken'));
296 + writeFileSync(path.join(f.root, '.artifacts/agent-tasks/broken/task.json'), '{');
297 + const result = f.run('list');
298 + assert.equal(result.status, 0, result.stderr);
299 + const [broken, valid] = JSON.parse(result.stdout);
300 + assert.equal(broken.id, 'broken');
301 + assert.equal(broken.status, 'unreadable');
302 + assert.equal(valid.task.id, 'case');
303 + assert.equal(valid.status, 'pending');
304 +});
305 +
306 +test('successful profile evidence expires after dirty edits, untracked sources, or task changes', () => {
307 + const f = fixture();
308 + assert.equal(f.init().status, 0);
309 + assert.equal(f.run('verify', 'case').status, 0);
310 + assert.equal(f.state('case').status, 'passed');
311 + writeFileSync(path.join(f.root, 'source.txt'), 'changed without a commit');
312 + assert.equal(f.state('case').status, 'stale');
313 + assert.equal(f.run('verify', 'case').status, 0);
314 + assert.equal(f.state('case').status, 'passed');
315 + writeFileSync(path.join(f.root, 'new-source.txt'), 'new input');
316 + assert.equal(f.state('case').status, 'stale');
317 + assert.equal(f.run('verify', 'case').status, 0);
318 + const taskPath = path.join(f.root, '.artifacts/agent-tasks/case/task.json');
319 + const task = JSON.parse(readFileSync(taskPath, 'utf8'));
320 + task.goal = 'A different acceptance outcome';
321 + writeFileSync(taskPath, JSON.stringify(task));
322 + assert.equal(f.state('case').status, 'stale');
323 +});
324 +
325 +test('the Claude Code scheduler lock does not invalidate evidence', () => {
326 + const f = fixture();
327 + assert.equal(f.init().status, 0);
328 + assert.equal(f.run('verify', 'case').status, 0);
329 + writeFileSync(path.join(f.root, '.claude/scheduled_tasks.lock'), 'session two');
330 + assert.equal(f.state('case').status, 'passed');
331 + writeFileSync(path.join(f.root, 'source.txt'), 'changed without a commit');
332 + assert.equal(f.state('case').status, 'stale');
333 +});
334 +
335 +test('changed profiles invalidate receipts and failed attempts survive later success', () => {
336 + const f = fixture();
337 + assert.equal(f.init('case', 'failure').status, 0);
338 + assert.equal(f.run('verify', 'case').status, 1);
339 + assert.equal(f.state('case').status, 'failed');
340 + const attempts = path.join(f.root, '.artifacts/agent-tasks/case/attempts');
341 + const first = readdirSync(attempts)[0];
342 + const failed = readFileSync(path.join(attempts, first, 'receipt.json'), 'utf8');
343 + writeFileSync(path.join(f.root, '.artifacts/succeed'), 'ready');
344 + assert.equal(f.run('verify', 'case').status, 0);
345 + assert.equal(f.state('case').status, 'passed');
346 + assert.equal(readdirSync(attempts).length, 2);
347 + assert.equal(readFileSync(path.join(attempts, first, 'receipt.json'), 'utf8'), failed);
348 + const profilesPath = path.join(f.root, 'scripts/agent-profiles.json');
349 + const profiles = JSON.parse(readFileSync(profilesPath, 'utf8'));
350 + profiles.failure.boundary = 'packaged';
351 + writeFileSync(profilesPath, JSON.stringify(profiles));
352 + assert.equal(f.state('case').status, 'stale');
353 +});
354 +
355 +test('handoff progress notes preserve proof without changing its acceptance contract', () => {
356 + const f = fixture();
357 + assert.equal(f.init().status, 0);
358 + assert.equal(f.run('verify', 'case').status, 0);
359 + const taskPath = path.join(f.root, '.artifacts/agent-tasks/case/task.json');
360 + const task = JSON.parse(readFileSync(taskPath, 'utf8'));
361 + task.nextStep = 'Review the verified diff';
362 + task.blocker = 'Waiting for an owner decision';
363 + writeFileSync(taskPath, JSON.stringify(task));
364 + assert.equal(f.state('case').status, 'passed');
365 + assert.equal(f.state('case').task.nextStep, task.nextStep);
366 +});
367 +
368 +test('an unfinished newer attempt cannot expose an older passing receipt', () => {
369 + const f = fixture();
370 + assert.equal(f.init().status, 0);
371 + assert.equal(f.run('verify', 'case').status, 0);
372 + mkdirSync(path.join(f.root, '.artifacts/agent-tasks/case/attempts/9999999999999-incomplete'));
373 + assert.equal(f.state('case').status, 'interrupted');
374 +});
375 +
376 +test('a killed runner exposes interrupted evidence and a recoverable lock', async () => {
377 + const f = fixture();
378 + const { runner, exit, group } = await runningFixture(f);
379 + runner.kill('SIGKILL');
380 + assert.deepEqual(await Promise.race([exit, delay(5000, 'timed out', { ref: false })]), {
381 + code: null,
382 + signal: 'SIGKILL',
383 + });
384 + assert.equal(f.state('case').status, 'interrupted');
385 + const blocked = f.run('verify', 'case');
386 + assert.equal(blocked.status, 1);
387 + assert.match(blocked.stderr, /Inspect and stop that group/);
388 + process.kill(-group, 'SIGKILL');
389 + await waitGone(-group);
390 + useOfflineProfile(f);
391 + assert.equal(f.run('verify', 'case').status, 0);
392 + assert.equal(f.state('case').status, 'passed');
393 +});
394 +
395 +test('missing tools fail and preserve an attempted command receipt', () => {
396 + const f = fixture();
397 + assert.equal(f.init('case', 'missing').status, 0);
398 + assert.equal(f.run('verify', 'case').status, 1);
399 + assert.equal(f.state('case').status, 'failed');
400 + const attempts = path.join(f.root, '.artifacts/agent-tasks/case/attempts');
401 + const receipt = JSON.parse(
402 + readFileSync(path.join(attempts, readdirSync(attempts)[0], 'receipt.json'), 'utf8')
403 + );
404 + assert.equal(receipt.commands.length, 1);
405 + assert.notEqual(receipt.commands[0].exitCode, 0);
406 +});
407 +
408 +test('long output keeps its start and end in the capped log', () => {
409 + const f = fixture();
410 + assert.equal(f.init('case', 'noisy').status, 0);
411 + assert.equal(f.run('verify', 'case').status, 1);
412 + const receiptPath = path.join(f.root, f.state('case').receipt);
413 + const receipt = JSON.parse(readFileSync(receiptPath, 'utf8'));
414 + assert.equal(receipt.logTruncated, true);
415 + const log = readFileSync(path.join(path.dirname(receiptPath), 'output.log'), 'utf8');
416 + assert.ok(log.length <= 64 * 1024 + 100, `log has ${log.length} bytes`);
417 + assert.match(log, /^first line\n/);
418 + assert.match(log, /bytes of output omitted/);
419 + assert.match(log, /final type error\n$/);
420 +});
421 +
422 +test('a successful command that edits its verification subject cannot establish fresh proof', () => {
423 + const f = fixture();
424 + assert.equal(f.init('case', 'mutating').status, 0);
425 + assert.equal(f.run('verify', 'case').status, 1);
426 + assert.equal(f.state('case').status, 'stale');
427 +});
428 +
429 +for (const signal of ['SIGTERM', 'SIGHUP'] as const) {
430 + test(`${signal} terminates the command group, preserves evidence and releases the task for retry`, async () => {
431 + const f = fixture();
432 + const { runner, exit, childPid } = await runningFixture(f);
433 + runner.kill(signal);
434 + assert.deepEqual(await Promise.race([exit, delay(5000, 'timed out', { ref: false })]), {
435 + code: 1,
436 + signal: null,
437 + });
438 + assert.equal(f.state('case').status, 'interrupted');
439 + await waitGone(childPid);
440 + useOfflineProfile(f);
441 + assert.equal(f.run('verify', 'case').status, 0);
442 + assert.equal(f.state('case').status, 'passed');
443 + });
444 +}
445 +
446 +test('verification remains locked until independent-stdio descendants stop', async () => {
447 + const f = fixture();
448 + const file = path.join(f.root, 'scripts/agent-profiles.json');
449 + const profiles = JSON.parse(readFileSync(file, 'utf8'));
450 + const descendant = 'process.on("SIGTERM",()=>{});console.log("ready");setInterval(()=>{},1000)';
451 + profiles.interrupt.commands[0].argv[2] = `
452 + const child = require('child_process').spawn(process.execPath, ['-e', ${JSON.stringify(descendant)}], { stdio: ['ignore', 'pipe', 'ignore'] });
453 + child.stdout.once('data', () => require('fs').writeFileSync('.artifacts/child.pid', String(child.pid)));
454 + process.on('SIGTERM', () => process.exit(0));
455 + setInterval(() => {}, 1000);
456 + `;
457 + writeFileSync(file, JSON.stringify(profiles));
458 + const { runner, exit, childPid, group } = await runningFixture(f);
459 + assert.equal(f.init('other').status, 0);
460 + runner.kill('SIGTERM');
461 + await waitGone(group);
462 + process.kill(childPid, 0);
463 + assert.equal(existsSync(path.join(f.root, '.artifacts/agent-locks/verify.lock')), true);
464 + const overlap = f.run('verify', 'other');
465 + assert.equal(
466 + overlap.status,
467 + 1,
468 + 'the lock must outlive the direct command until its descendants stop'
469 + );
470 + assert.match(overlap.stderr, /case/);
471 + assert.deepEqual(await Promise.race([exit, delay(5000, 'timed out', { ref: false })]), {
472 + code: 1,
473 + signal: null,
474 + });
475 + await waitGone(childPid);
476 + assert.equal(f.state('case').status, 'interrupted');
477 +});
478 +
479 +test('different task profiles serialize shared worktree verification', async () => {
480 + const f = fixture();
481 + const { runner, exit } = await runningFixture(f);
482 + assert.equal(f.init('other').status, 0);
483 + try {
484 + const overlap = f.run('verify', 'other');
485 + assert.equal(overlap.status, 1, 'a different task must not overlap worktree verification');
486 + assert.match(overlap.stderr, /case/);
487 + } finally {
488 + runner.kill('SIGTERM');
489 + await exit;
490 + }
491 + assert.equal(f.run('verify', 'other').status, 0);
492 +});
493 +
494 +test('an unreadable lock is replaced once no attempt is running', () => {
495 + const f = fixture();
496 + assert.equal(f.init().status, 0);
497 + mkdirSync(path.join(f.root, '.artifacts/agent-locks'), { recursive: true });
498 + writeFileSync(path.join(f.root, '.artifacts/agent-locks/verify.lock'), '');
499 + const result = f.run('verify', 'case');
500 + assert.equal(result.status, 0, result.stderr);
501 + assert.match(result.stderr, /Replacing unreadable lock/);
502 + assert.equal(f.state('case').status, 'passed');
503 +});
504 +
505 +test('an unreadable lock still blocks verification while another attempt runs', async () => {
506 + const f = fixture();
507 + const { runner, exit } = await runningFixture(f);
508 + assert.equal(f.init('other').status, 0);
509 + try {
510 + mkdirSync(path.join(f.root, '.artifacts/agent-locks'), { recursive: true });
511 + writeFileSync(path.join(f.root, '.artifacts/agent-locks/verify.lock'), '{');
512 + const overlap = f.run('verify', 'other');
513 + assert.equal(overlap.status, 1);
514 + assert.match(overlap.stderr, /unreadable and task case still has a running attempt/);
515 + } finally {
516 + runner.kill('SIGTERM');
517 + await exit;
518 + }
519 +});
520 +
521 +test('a revision-only change invalidates otherwise identical source evidence', () => {
522 + const f = fixture();
523 + assert.equal(f.init().status, 0);
524 + assert.equal(f.run('verify', 'case').status, 0);
525 + execFileSync(
526 + 'git',
527 + [
528 + '-c',
529 + 'user.name=Test',
530 + '-c',
531 + 'user.email=test@example.invalid',
532 + 'commit',
533 + '--allow-empty',
534 + '-qm',
535 + 'new revision',
536 + ],
537 + { cwd: f.root }
538 + );
539 + assert.equal(f.state('case').status, 'stale');
540 +});
541 +
542 +test('task ids cannot collide with worktree lock files', () => {
543 + const f = fixture();
544 + assert.equal(f.init('verify.lock').status, 0);
545 + assert.equal(f.init('acquire.lock').status, 0);
546 + assert.equal(f.init().status, 0);
547 + const result = f.run('verify', 'case');
548 + assert.equal(result.status, 0, result.stderr);
549 + assert.equal(f.run('verify', 'verify.lock').status, 0);
550 + assert.equal(f.state('verify.lock').status, 'passed');
551 +});
552 +
553 +test('a command that exits 0 cannot leave workers running after verification', () => {
554 + const f = fixture();
555 + assert.equal(f.init('case', 'lingering').status, 0);
556 + const result = f.run('verify', 'case');
557 + const childPid = Number(readFileSync(path.join(f.root, '.artifacts/child.pid'), 'utf8'));
558 + cleanups.push(() => {
559 + try {
560 + process.kill(childPid, 'SIGKILL');
561 + } catch {}
562 + });
563 + assert.equal(result.status, 0, result.stderr);
564 + assert.throws(() => process.kill(childPid, 0), { code: 'ESRCH' });
565 + assert.equal(f.state('case').status, 'passed');
566 +});
567 +
568 +test('a runner killed before saving its command pid keeps the lock', () => {
569 + const f = fixture();
570 + assert.equal(f.init().status, 0);
571 + const deadRunner = spawnSync(process.execPath, ['-e', '']).pid;
572 + const attempt = path.join(f.root, '.artifacts/agent-tasks/case/attempts/1-unsaved-pid');
573 + mkdirSync(attempt, { recursive: true });
574 + writeFileSync(
575 + path.join(attempt, 'receipt.json'),
576 + JSON.stringify({
577 + version: 1,
578 + id: '1-unsaved-pid',
579 + runnerPid: deadRunner,
580 + status: 'running',
581 + commands: [
582 + {
583 + cwd: '.',
584 + argv: ['long-check'],
585 + startedAt: new Date().toISOString(),
586 + finishedAt: null,
587 + exitCode: null,
588 + signal: null,
589 + },
590 + ],
591 + })
592 + );
593 + const lock = path.join(f.root, '.artifacts/agent-locks/verify.lock');
594 + mkdirSync(path.dirname(lock), { recursive: true });
595 + writeFileSync(lock, JSON.stringify({ pid: deadRunner, task: 'case' }));
596 + const blocked = f.run('verify', 'case');
597 + assert.equal(blocked.status, 1);
598 + assert.match(blocked.stderr, /no recorded process group/);
599 + assert.equal(existsSync(lock), true);
600 +});
601 +
602 +test('an interrupted init can be retried and staging folders stay out of the list', () => {
603 + const f = fixture();
604 + mkdirSync(path.join(f.root, '.artifacts/agent-tasks/case'), { recursive: true });
605 + mkdirSync(path.join(f.root, '.artifacts/agent-tasks/.init-leftover'));
606 + const result = f.init();
607 + assert.equal(result.status, 0, result.stderr);
608 + assert.equal(f.state('case').task.goal, 'Verify the changed behavior');
609 + assert.deepEqual(
610 + JSON.parse(f.run('list').stdout).map((entry: TaskStatus) => entry.task.id),
611 + ['case']
612 + );
613 +});
614 +
615 +test('a newer attempt stays latest when the clock moves backwards', () => {
616 + const f = fixture();
617 + assert.equal(f.init('case', 'failure').status, 0);
618 + writeFileSync(path.join(f.root, '.artifacts/succeed'), 'ready');
619 + assert.equal(f.run('verify', 'case').status, 0);
620 + rmSync(path.join(f.root, '.artifacts/succeed'));
621 + const clock = path.join(f.root, '.artifacts/clock.ts');
622 + writeFileSync(clock, 'Date.now = () => 1;');
623 + const earlier = spawnSync(
624 + process.execPath,
625 + ['--preload', clock, 'scripts/agent-task.ts', 'verify', 'case'],
626 + { cwd: f.root, encoding: 'utf8' }
627 + );
628 + assert.equal(earlier.status, 1, earlier.stderr);
629 + assert.equal(f.state('case').status, 'failed');
630 +});
631 +
632 +test('output written before a runner is killed survives in the log', async () => {
633 + const f = fixture();
634 + const file = path.join(f.root, 'scripts/agent-profiles.json');
635 + const profiles = JSON.parse(readFileSync(file, 'utf8'));
636 + profiles.interrupt.commands[0].argv[2] =
637 + 'process.stdout.write("x".repeat(20000)+"\\nlate diagnostic\\n",()=>{require("fs").writeFileSync(".artifacts/child.pid",String(process.pid))});setInterval(()=>{},1000)';
638 + writeFileSync(file, JSON.stringify(profiles));
639 + const { runner, exit } = await runningFixture(f);
640 + const log = path.join(f.root, path.dirname(f.state('case').receipt), 'output.log');
641 + for (let i = 0; i < 250 && !readFileSync(log, 'utf8').includes('late diagnostic'); i++)
642 + await delay(20);
643 + runner.kill('SIGKILL');
644 + await exit;
645 + const saved = readFileSync(log, 'utf8');
646 + assert.match(saved, /^x{100}/);
647 + assert.match(saved, /late diagnostic\n$/);
648 +});
ts/scripts/test-skill-routing.ts renamed
+17 -8
@@ -1,4 +1,4 @@
1 -#!/usr/bin/env node
1 +#!/usr/bin/env bun
2 // Skill-routing smoke test: a lightweight, deterministic eval that guards against
3 // SKILL.md description edits silently breaking which skill a task routes to.
4 //
@@ -9,7 +9,7 @@
9 // obvious match, or another skill grows ambiguous overlap. It also fails if a
10 // skill has no probe, so routing coverage tracks the taxonomy.
11 //
12 -// Run: pnpm validate:skill-routing (enforced in CI by
12 +// Run: bun run validate:skill-routing (enforced in CI by
13 // .github/workflows/agent-substrate.yml).
14
15 import fs from 'node:fs';
@@ -19,7 +19,7 @@ import process from 'node:process';
19 const repoRoot = process.cwd();
20 const skillsRoot = path.join(repoRoot, '.agents/skills');
21
22 -const readDescription = skillName => {
22 +const readDescription = (skillName: string) => {
23 const file = path.join(skillsRoot, skillName, 'SKILL.md');
24 const content = fs.readFileSync(file, 'utf8');
25 const match = content.match(/^description:\s*(.*)$/m);
@@ -33,6 +33,16 @@ const readDescription = skillName => {
33 // should make `expect` the obvious match. Phrases are matched case-insensitively
34 // as substrings against the skill description.
35 const probes = [
36 + {
37 + task: 'resume a task in another agent and verify revision-bound evidence',
38 + expect: 'verify-change',
39 + terms: [
40 + 'resumable task records',
41 + 'verification profiles',
42 + 'revision-bound receipts',
43 + 'carrying work between coding agents',
44 + ],
45 + },
46 {
47 task: 'reproduce a reported bug, find the root cause, and add a regression test',
48 expect: 'bug-fixing',
@@ -158,13 +168,13 @@ const skillNames = fs
168
169 const descriptions = new Map(skillNames.map(name => [name, readDescription(name).toLowerCase()]));
170
161 -const failures = [];
171 +const failures: string[] = [];
172
173 // Coverage: every skill must have at least one probe so routing tracks the taxonomy.
174 const covered = new Set(probes.map(probe => probe.expect));
175 for (const name of skillNames) {
176 if (!covered.has(name)) {
167 - failures.push(`skill "${name}" has no routing probe; add one to test-skill-routing.mjs`);
177 + failures.push(`skill "${name}" has no routing probe; add one to test-skill-routing.ts`);
178 }
179 }
180
@@ -174,8 +184,7 @@ for (const probe of probes) {
184 continue;
185 }
186
177 - const scores = skillNames.map(name => {
178 - const description = descriptions.get(name);
187 + const scores = [...descriptions].map(([name, description]) => {
188 const score = probe.terms.reduce(
189 (total, term) => total + (description.includes(term.toLowerCase()) ? 1 : 0),
190 0
@@ -185,7 +194,7 @@ for (const probe of probes) {
194
195 const maxScore = Math.max(...scores.map(entry => entry.score));
196 const winners = scores.filter(entry => entry.score === maxScore).map(entry => entry.name);
188 - const expectScore = scores.find(entry => entry.name === probe.expect).score;
197 + const expectScore = scores.find(entry => entry.name === probe.expect)?.score ?? 0;
198
199 if (expectScore === 0) {
200 failures.push(
ts/scripts/validate-agent-skills.ts renamed
+33 -32
@@ -1,8 +1,9 @@
1 -#!/usr/bin/env node
1 +#!/usr/bin/env bun
2
3 import fs from 'node:fs';
4 import path from 'node:path';
5 import process from 'node:process';
6 +import { z } from 'zod';
7
8 const repoRoot = process.cwd();
9
@@ -30,6 +31,7 @@ const expectedSkills = [
31 'typescript-providers',
32 'typescript-sdk',
33 'typescript-testing',
34 + 'verify-change',
35 ];
36
37 const requiredAgentFiles = [
@@ -44,11 +46,12 @@ const requiredAgentFiles = [
46 'python/providers/AGENTS.md',
47 ];
48
47 -const errors = [];
49 +const errors: string[] = [];
50
49 -const fail = message => errors.push(message);
50 -const readJson = relativePath =>
51 - JSON.parse(fs.readFileSync(path.join(repoRoot, relativePath), 'utf8'));
51 +const fail = (message: string) => errors.push(message);
52 +const manifestSchema = z.object({ scripts: z.record(z.string(), z.string()).optional() });
53 +const readJson = (relativePath: string) =>
54 + manifestSchema.parse(JSON.parse(fs.readFileSync(path.join(repoRoot, relativePath), 'utf8')));
55
56 const rootScripts = new Set(Object.keys(readJson('package.json').scripts ?? {}));
57 const docsScripts = new Set(Object.keys(readJson('docs/package.json').scripts ?? {}));
@@ -68,14 +71,15 @@ for (const match of makefile.matchAll(/\bnox\s+-s\s+([A-Za-z0-9_]+)/g)) {
71 }
72 }
73
71 -const parseFrontmatter = (content, relativePath) => {
74 +const frontmatterSchema = z.object({ name: z.string(), description: z.string() });
75 +const parseFrontmatter = (content: string, relativePath: string) => {
76 const match = content.match(/^---\n([\s\S]*?)\n---\n?/);
77 if (!match) {
78 fail(`${relativePath}: missing YAML frontmatter`);
79 return null;
80 }
81
78 - const data = {};
82 + const data: Record<string, string> = {};
83 for (const line of match[1].split('\n')) {
84 const keyMatch = line.match(/^([A-Za-z0-9_-]+):\s*(.*)$/);
85 if (!keyMatch) {
@@ -98,18 +102,22 @@ const parseFrontmatter = (content, relativePath) => {
102 }
103 }
104
101 - return { data };
105 + const parsed = frontmatterSchema.safeParse(data);
106 + return parsed.success ? { data: parsed.data } : null;
107 };
108
109 const skillsRoot = path.join(repoRoot, '.agents/skills');
110 +const skillDirNames = fs.existsSync(skillsRoot)
111 + ? fs
112 + .readdirSync(skillsRoot, { withFileTypes: true })
113 + .filter(entry => entry.isDirectory())
114 + .map(entry => entry.name)
115 + : [];
116 +
117 if (!fs.existsSync(skillsRoot)) {
118 fail('.agents/skills does not exist');
119 } else {
108 - const actualSkills = fs
109 - .readdirSync(skillsRoot, { withFileTypes: true })
110 - .filter(entry => entry.isDirectory())
111 - .map(entry => entry.name)
112 - .sort();
120 + const actualSkills = [...skillDirNames].sort();
121
122 const expected = [...expectedSkills].sort();
123 if (actualSkills.join('\n') !== expected.join('\n')) {
@@ -228,6 +236,7 @@ for (const relativePath of ['docs/decisions/README.md', 'docs/agent-guidance/REA
236
237 const ignoredDirs = new Set([
238 '.git',
239 + '.artifacts',
240 '.generated',
241 'node_modules',
242 '.turbo',
@@ -273,7 +282,7 @@ const stalePatterns = [
282 'CLI' + '.md',
283 ];
284
276 -const walk = dir => {
285 +const walk = (dir: string): void => {
286 const entries = fs.readdirSync(dir, { withFileTypes: true });
287 for (const entry of entries) {
288 const absolutePath = path.join(dir, entry.name);
@@ -306,7 +315,7 @@ const walk = dir => {
315
316 walk(repoRoot);
317
309 -const validatePnpmCommands = (relativePath, text) => {
318 +const validatePnpmCommands = (relativePath: string, text: string) => {
319 const commandLines = text
320 .split('\n')
321 .map(line => line.trim().replace(/^\$\s+/, ''))
@@ -333,7 +342,7 @@ const validatePnpmCommands = (relativePath, text) => {
342 }
343 };
344
336 -const validateBunRunCommands = (relativePath, text) => {
345 +const validateBunRunCommands = (relativePath: string, text: string) => {
346 const commandLines = text
347 .split('\n')
348 .map(line => line.trim().replace(/^\$\s+/, ''))
@@ -345,13 +354,15 @@ const validateBunRunCommands = (relativePath, text) => {
354 continue;
355 }
356 const command = match[1];
348 - if (!docsScripts.has(command)) {
349 - fail(`${relativePath}: bun run command "${command}" is not a docs package.json script`);
357 + if (!docsScripts.has(command) && !rootScripts.has(command)) {
358 + fail(
359 + `${relativePath}: bun run command "${command}" is not a root or docs package.json script`
360 + );
361 }
362 }
363 };
364
354 -const validateMakeCommands = (relativePath, text) => {
365 +const validateMakeCommands = (relativePath: string, text: string) => {
366 const commandLines = text
367 .split('\n')
368 .map(line => line.trim().replace(/^\$\s+/, ''))
@@ -369,7 +380,7 @@ const validateMakeCommands = (relativePath, text) => {
380 }
381 };
382
372 -const validateNoxCommands = (relativePath, text) => {
383 +const validateNoxCommands = (relativePath: string, text: string) => {
384 const commandLines = text
385 .split('\n')
386 .map(line => line.trim().replace(/^\$\s+/, ''))
@@ -389,21 +400,11 @@ const validateNoxCommands = (relativePath, text) => {
400
401 // Derive the command-scan file set from the skills actually on disk so it can never
402 // drift from `expectedSkills` (which serves only as the taxonomy gate above).
392 -const skillDirNames = fs.existsSync(skillsRoot)
393 - ? fs
394 - .readdirSync(skillsRoot, { withFileTypes: true })
395 - .filter(entry => entry.isDirectory())
396 - .map(entry => entry.name)
397 - : [];
398 -
403 const commandFiles = [
404 ...skillDirNames.flatMap(skillName => {
405 const skillDir = path.join(skillsRoot, skillName);
402 - if (!fs.existsSync(skillDir)) {
403 - return [];
404 - }
405 - const files = [];
406 - const collect = dir => {
406 + const files: string[] = [];
407 + const collect = (dir: string): void => {
408 for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
409 const absolutePath = path.join(dir, entry.name);
410 if (entry.isDirectory()) {
tsconfig.agent-workflow.json new
+18
@@ -0,0 +1,18 @@
1 +{
2 + "compilerOptions": {
3 + "target": "ESNext",
4 + "module": "ESNext",
5 + "moduleResolution": "Bundler",
6 + "types": ["bun"],
7 + "strict": true,
8 + "skipLibCheck": true,
9 + "noEmit": true,
10 + "esModuleInterop": true
11 + },
12 + "include": [
13 + "scripts/agent-task.ts",
14 + "test/agent-task.test.ts",
15 + "ts/scripts/validate-agent-skills.ts",
16 + "ts/scripts/test-skill-routing.ts"
17 + ]
18 +}