@devries-cameron20 / composio-1 / commits / d268e4ab5
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Add full-cli-test agent skill with Slack integration tests (#3081)

## Summary - Adds new `/full-cli-test` agent skill that orchestrates a 3-phase CLI validation pipeline: poll CI for type/lint → local binary test → CI-bundled binary test - Includes a Slack integration test (`#buzz-skill-based-cli-testing`) that validates `execute()`, `experimental_subAgent()`, `z` (Zod), and `result.prompt()` in both local and bundled builds - Updates `/cli-test-with-bundling` to always use `-beta.<timestamp>` versioning, preventing accidental production releases from test runs ## Test plan - [ ] Verify `/full-cli-test` skill appears in skill list - [ ] Run `/cli-test-with-bundling` and confirm the workflow dispatch uses a beta version - [ ] Confirm the Slack test script runs against `#buzz-skill-based-cli-testing` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rahul Tarak committed Mar 31, 2026 at 00:58 UTC d268e4ab58524966e96f8934de1cf0a2743fd1c2
2 files changed +183 -5
.claude/skills/cli-test-with-bundling/SKILL.md
+12 -5
@@ -13,21 +13,28 @@ Trigger the `Build CLI Binaries` workflow on GitHub Actions, wait for it to comp
13
14 ## Step 1: Determine the version
15
16 -Read the version from `ts/packages/cli/package.json`:
16 +Read the version from `ts/packages/cli/package.json` and append a beta prerelease suffix so the build is always treated as a beta release. This prevents accidental production releases and marks the GitHub release as a prerelease.
17
18 ```bash
19 -jq -r .version ts/packages/cli/package.json
19 +BASE_VERSION=$(jq -r .version ts/packages/cli/package.json)
20 +BETA_VERSION="${BASE_VERSION}-beta.$(date +%Y%m%d%H%M%S)"
21 ```
22
23 +For example, if `package.json` has `1.2.3`, the version becomes `1.2.3-beta.20260331143022`.
24 +
25 ## Step 2: Trigger the workflow
26
27 +**Always use the beta version** — never pass the bare version from `package.json`:
28 +
29 ```bash
30 gh workflow run build-cli-binaries.yml \
31 --repo ComposioHQ/composio \
32 --ref "$(git rev-parse --abbrev-ref HEAD)" \
28 - --field version="<version-from-step-1>"
33 + --field version="${BETA_VERSION}"
34 ```
35
36 +This ensures the CI workflow creates a **prerelease** GitHub release (the workflow auto-detects `beta` in the version string).
37 +
38 ## Step 3: Find and monitor the run
39
40 After triggering, wait a few seconds then find the run:
@@ -71,10 +78,10 @@ esac
78 ARTIFACT="composio-${OS}-${ARCH}"
79 ```
80
74 -Then download from the release. The workflow creates a GitHub release tagged `@composio/cli@<version>`:
81 +Then download from the release. The workflow creates a GitHub release tagged `@composio/cli@<beta-version>`:
82
83 ```bash
77 -VERSION="<version-from-step-1>"
84 +VERSION="${BETA_VERSION}" # the beta version from Step 1
85 TAG="@composio/cli@${VERSION}"
86 ENCODED_TAG=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${TAG}', safe=''))")
87
.claude/skills/full-cli-test/SKILL.md new
+171
@@ -0,0 +1,171 @@
1 +---
2 +description: Full CLI test pipeline — monitor CI for types/lint, then run local binary test, then run bundled binary test via CI.
3 +---
4 +
5 +# Full CLI Test
6 +
7 +End-to-end CLI validation pipeline that ensures types/lint pass in CI, then tests the CLI binary built from source, and finally tests the CI-bundled binary.
8 +
9 +## Overview
10 +
11 +This skill runs three phases sequentially:
12 +
13 +1. **Phase 1: CI Lint/Type Check** — Poll CI until the CLI type-check and lint jobs pass.
14 +2. **Phase 2: Local Binary Test** — Build and test the CLI binary from source (`/cli-test`).
15 +3. **Phase 3: Bundled Binary Test** — Trigger CI binary build, download, and test (`/cli-test-with-bundling`).
16 +
17 +## Phase 1: Monitor CI for Types/Lint
18 +
19 +Use `/loop 5m` to poll the CI status for the current branch every 5 minutes. Check that **all type-check and lint jobs pass** for the CLI package before proceeding.
20 +
21 +### How to check
22 +
23 +```bash
24 +# Get the latest CI run for the current branch
25 +gh run list \
26 + --repo ComposioHQ/composio \
27 + --branch "$(git rev-parse --abbrev-ref HEAD)" \
28 + --limit 5 \
29 + --json databaseId,status,conclusion,name,headBranch
30 +```
31 +
32 +Look for workflow runs that cover type-checking and linting (e.g., the main CI workflow). If none are running or completed yet, wait for one to appear.
33 +
34 +### What to verify
35 +
36 +- All type-check jobs: **pass**
37 +- All lint jobs: **pass**
38 +- If any fail, report the failure to the user and stop — do NOT proceed to Phase 2.
39 +
40 +Use `/loop 5m` to poll: run the `gh run list` / `gh run view` command every 5 minutes until the relevant jobs succeed. Once all type/lint checks are green, move on.
41 +
42 +## Phase 2: Local Binary Test (`/cli-test`)
43 +
44 +Once CI lint/types pass, run the `/cli-test` skill:
45 +
46 +1. Install dependencies: `pnpm install`
47 +2. Build all packages: `pnpm turbo build`
48 +3. Build the standalone binary: `pnpm --dir ts/packages/cli build:binary`
49 +4. Test the binary:
50 + ```bash
51 + ./ts/packages/cli/dist/composio version
52 + ./ts/packages/cli/dist/composio whoami
53 + ./ts/packages/cli/dist/composio --help
54 + ```
55 +5. Run the Slack integration test (see [Slack Integration Test](#slack-integration-test) below):
56 + ```bash
57 + ./ts/packages/cli/dist/composio run '<SLACK_TEST_SCRIPT>'
58 + ```
59 +
60 +If any command fails, report to the user and stop — do NOT proceed to Phase 3.
61 +
62 +## Phase 3: Bundled Binary Test (`/cli-test-with-bundling`)
63 +
64 +Once the local binary test passes, run the `/cli-test-with-bundling` skill:
65 +
66 +1. Read the version from `ts/packages/cli/package.json` and append a `-beta.<timestamp>` suffix (e.g. `1.2.3-beta.20260331143022`) — **always trigger as a beta release**
67 +2. Trigger the `build-cli-binaries.yml` workflow via `gh workflow run` with the beta version
68 +3. Monitor the workflow run (use `/loop 5m` to poll)
69 +4. Download the platform-appropriate binary artifact
70 +5. Test the binary:
71 + ```bash
72 + $BINARY version
73 + $BINARY whoami
74 + $BINARY --help
75 + $BINARY run 'console.log("hello from composio run")'
76 + $BINARY run 'const result = await experimental_subAgent({ goal: "What is 2+2?", toolNames: [] }); console.log(result)'
77 + ```
78 +6. Run the Slack integration test (see [Slack Integration Test](#slack-integration-test) below):
79 + ```bash
80 + $BINARY run '<SLACK_TEST_SCRIPT>'
81 + ```
82 +7. If on a PR, post results as a PR comment
83 +
84 +## Slack Integration Test
85 +
86 +This test validates `execute()`, `experimental_subAgent()`, and end-to-end Slack connectivity. It must be run in **both Phase 2 and Phase 3** to ensure it works with both the locally-built and CI-bundled binaries.
87 +
88 +### Channel
89 +
90 +**`#buzz-skill-based-cli-testing`** — dedicated Slack channel for automated CLI test runs.
91 +
92 +### Test Script
93 +
94 +Use `composio run` (or `$BINARY run` in Phase 3) with the following script. Replace `$BINARY` with the appropriate binary path for the phase.
95 +
96 +```bash
97 +$BINARY run '
98 + // Step 1: Find the #buzz-skill-based-cli-testing channel
99 + const channels = await execute("SLACK_LIST_CHANNELS", {
100 + types: "public_channel",
101 + limit: 200,
102 + });
103 + const channel = channels.data?.channels?.find(
104 + (c) => c.name === "buzz-skill-based-cli-testing"
105 + );
106 + if (!channel) throw new Error("Channel #buzz-skill-based-cli-testing not found");
107 +
108 + // Step 2: Send an initial message tagging @cryogenicplanet
109 + const buildType = "local"; // use "bundled" in Phase 3
110 + await execute("SLACK_SEND_A_MESSAGE_TO_A_SLACK_CHANNEL", {
111 + channel: channel.id,
112 + text: `<@cryogenicplanet> CLI test run started (${buildType} build) at ${new Date().toISOString()}`,
113 + });
114 +
115 + // Step 3: Fetch recent channel history for the subAgent to summarize
116 + const history = await execute("SLACK_GET_CHANNEL_HISTORY", {
117 + channel: channel.id,
118 + limit: 20,
119 + });
120 +
121 + // Step 4: Use experimental_subAgent to summarize what happened in the channel
122 + const summary = await experimental_subAgent(
123 + `Summarize the recent activity in this Slack channel in 2-3 sentences. Focus on what tests were run and their outcomes.\n\n${history.prompt()}`,
124 + {
125 + schema: z.object({
126 + summary: z.string(),
127 + messageCount: z.number(),
128 + }),
129 + }
130 + );
131 +
132 + // Step 5: Post the summary back to the channel
133 + await execute("SLACK_SEND_A_MESSAGE_TO_A_SLACK_CHANNEL", {
134 + channel: channel.id,
135 + text: `CLI Test Summary (${buildType} build):\n${summary.structuredOutput.summary}\n(${summary.structuredOutput.messageCount} messages analyzed)`,
136 + });
137 +
138 + console.log("Slack integration test passed:", summary.structuredOutput);
139 +'
140 +```
141 +
142 +### What this validates
143 +
144 +| Capability | How it's tested |
145 +|---|---|
146 +| `execute()` with Slack tools | List channels, send messages, fetch history |
147 +| `experimental_subAgent()` | Summarizes channel history with structured output via `z` schema |
148 +| `z` (Zod) global | Used in the subAgent schema definition |
149 +| `result.prompt()` | Feeds channel history into the subAgent |
150 +| End-to-end Slack connectivity | Reads from and writes to a real Slack channel |
151 +
152 +### Important Notes
153 +
154 +- Change `buildType` to `"bundled"` when running in Phase 3.
155 +- The Slack user tag `<@cryogenicplanet>` should resolve to the correct user in your workspace. If the mention doesn't resolve, find the Slack user ID first and use `<@U_XXXXX>` format.
156 +- This test requires an active Slack connection in Composio. If not connected, run `composio connect slack` first.
157 +
158 +## Failure Handling
159 +
160 +- **Phase 1 failure (lint/types):** Report which jobs failed with links. Do not continue.
161 +- **Phase 2 failure (local build/test):** Report the error output. Do not continue.
162 +- **Phase 3 failure (bundled build/test):** Report which commands failed. Post results to PR if applicable.
163 +
164 +## Reference Files
165 +
166 +| File | Purpose |
167 +|---|---|
168 +| `.github/workflows/build-cli-binaries.yml` | CI workflow that builds binaries |
169 +| `ts/packages/cli/package.json` | Source of CLI version |
170 +| `ts/packages/cli/scripts/build-binary.ts` | Local binary build script |
171 +| `ts/packages/cli/dist/composio` | Built binary output |