@setoelkahfi / homebrew-tap
Code Commits Pulls CI/CD 205
main
main
md 22.9 KB
View source Raw

siGit Code Cloud Agent — product plan and roadmap

Status: draft v2 (2026-06-26). Owner: product/eng. Audience: internal (private repo). Naming and scope per product-overview.md; this is the detailed spec for the siGit Code Cloud Agent product defined there.

A GitHub-Copilot-coding-agent-style product: you delegate a task against a Git repository, an isolated AWS sandbox runs siGit Code autonomously against that repo, and it returns a branch plus a proposed pull request. The first repo target is sigit.si; the agent is Git-host agnostic behind a Git-host adapter so the same agent later runs against GitHub, GitLab, and other Git hosts. Scope is Git only (no SVN, Mercurial, or other version control systems). This document is the strategy, the architecture, and the phased roadmap.

1. What we are building (and what it is not)

siGit Code Cloud Agent is an asynchronous, server-side coding agent. The user delegates a task or issue ("add pagination to the repos list", "fix the failing auth test", "upgrade Rails to 8.1"), points it at one of their repos and a base branch, and walks away. The platform provisions an ephemeral AWS sandbox, clones the repo into it, runs siGit Code headless against the task with hosted inference, lets it edit files and run build/test commands, then pushes a head branch and opens a pull request for human review.

It is the autonomous sibling of the other two things siGit ships (see product-overview.md):

  • siGit Code (Rust CLI / ACP agent) runs locally / on the desktop, interactively, driven by the developer at their keyboard.
  • siGit Code Cloud is the hosted chat (interactive, synchronous). Its work unit is a chat session.
  • siGit Code Cloud Agent runs in our cloud, asynchronously, driven by a task and reviewed afterward through a PR. Its work unit is an agent run (never a "session", which belongs to the chat product).

MVP trigger (decided): from a repo on sigit.si, the user describes a task and the agent produces a PR. Formal issue-assignment (assign a tracked issue to the agent, the Copilot headline flow) is a fast-follow that depends on an Issues feature, and on the Git-host adapters for GitHub/GitLab issues.

Reference points in the market: GitHub Copilot coding agent (assign an issue, it opens a PR), OpenAI Codex cloud, Cursor background agents, Devin. The differentiator for us is that we already own the whole vertical: the git host, the agent, the inference, the account, and the billing all live inside siGit. We are not bolting an agent onto someone else's platform; we are completing a platform we already run.

What it is not, in v1: it is not an autonomous merger (a human reviews and merges), not a long-lived persistent dev environment (sandboxes are ephemeral per run), and not a chat product (the chat tier already exists; this is the task-to-PR product on top of it).

2. Why siGit is unusually well positioned

The expensive parts of a cloud-agent product already exist in this repo and the sibling repos. The new work is mostly orchestration and isolation, not net-new agent or inference plumbing.

Capability the product needs Already exists Where
A git host the agent can clone from and push to yes git_http_controller (smart-HTTP, bare repos on disk, Repository#disk_path)
The coding agent itself (edit/run/test loop, tool calling) yes sigit (public Rust CLI/ACP agent), onde-cloud tool-call mapping
Hosted inference with auth, identity masking, metering yes Api::V1::ChatCompletionsController → OndeCloudService → Onde Cloud
Accounts and a trust boundary that mints scoped tokens yes smbCloud auth, Api::V1 sessions/me, git_token exchange
Subscription gating + monthly metering yes Subscription, CloudUsage, User#entitled_to_cloud?

What is missing and must be built (see roadmap):

  1. An execution plane on AWS (the sandbox machine) and the orchestration to drive it. This is the heart of the project.
  2. A control plane in Rails: an AgentRun resource, its lifecycle, log streaming, and web UI.
  3. Pull requests (and ideally issues). sigit.si has repos, blobs, commits, and stars, but no PullRequest model today. The agent's output is a PR, so a minimal PR/diff/review surface is a hard dependency. This can be scoped down to a "compare and open PR" view for v1.
  4. Per-run scoped credentials: short-lived git-push and inference tokens minted server-side, budget-bounded, never long-lived in the sandbox.
  5. A new metering/pricing dimension: agent runs consume sandbox compute and inference tokens, so COGS has two drivers, not one.

3. Architecture

Two planes plus the existing inference path. Control plane is Rails (sigit-si); execution plane is AWS; inference reuses the existing /api/v1/chat/completions proxy unchanged.

┌─ Control plane (Rails, sigit-si) ───────────────────────────────────────────┐
│  Web UI: repo "Agent" tab → run form, live transcript, diff, "Open PR"       │
│  AgentRunsController + Api::V1::AgentRunsController (create/show/cancel/log)  │
│  AgentRun model (lifecycle state machine)                                    │
│  AgentRunJob → provisions sandbox, monitors, collects result                 │
│  Mints per-run scoped tokens (git push + inference), enforces caps/metering  │
└──────────────────────────────────────────────────────────────────────────────┘
                │  RunTask (aws-sdk)            ▲  SSE/webhook: logs, status, diff
                ▼                               │
┌─ Execution plane (AWS) ─────────────────────────────────────────────────────┐
│  Ephemeral sandbox (Fargate task v1 → Firecracker microVM at scale)          │
│   ├─ clones repo from sigit.si over smart-HTTP (scoped git token)            │
│   ├─ runs siGit Code headless against the task                              │
│   │     OPENAI_BASE_URL=https://sigit.si/api/v1  (per-run inference token)   │
│   ├─ edits files, runs build/test in restricted shell                       │
│   └─ pushes head branch back to sigit.si (scoped git token)                 │
│  Private subnet, egress allowlist (sigit.si + package registries only)       │
│  Hard caps: wall-clock, CPU/mem, token budget, max tool calls                │
└──────────────────────────────────────────────────────────────────────────────┘
                │  /v1/chat/completions (per-run token)
                ▼
┌─ Inference (unchanged) ─────────────────────────────────────────────────────┐
│  Api::V1::ChatCompletionsController → OndeCloudService → Onde Cloud → upstream│
│  Existing entitlement gate, allowance metering, and identity masking apply.  │
└──────────────────────────────────────────────────────────────────────────────┘

3.1 Control plane (Rails)

AgentRun model (new table). Belongs to user and repository. Fields:

  • status: queued, provisioning, running, pushing, needs_input, completed, failed, canceled (state machine; one-way transitions logged).
  • task_prompt (text), base_branch, head_branch (generated, e.g. agent/<run-id>-<slug>), pull_request_id (nullable until pushed).
  • sandbox_ref (ECS task ARN / microVM id), region.
  • Budgets and accounting: token_budget, tokens_used, wall_clock_limit_s, started_at, finished_at, exit_reason.
  • transcript_url (S3 pointer for the full log), plus a tail kept in Postgres for the live view.

Controllers: a web AgentRunsController (HTML, Turbo) under the repo, and an Api::V1::AgentRunsController so the CLI and desktop can trigger and follow runs. Actions: create, index, show, cancel, messages#create (steer a running agent), and a logs SSE endpoint that relays the live transcript (reuse the ActionController::Live pattern already used for chat streaming).

AgentRunJob (Active Job, on the existing DB-backed queue): transitions the run to provisioning, calls AWS to start the sandbox, persists the sandbox ref, then hands off to monitoring. Cancellation and timeout both tear the sandbox down.

Web UI: a new "Agent" (or "Tasks") tab on the repository page. A run form (task prompt, base branch, optional model tier). A live transcript panel (Turbo Streams fed by the SSE relay). On completion, a diff view and an "Open pull request" action.

3.2 Execution plane (AWS) — the sandbox machine

This is the core AWS decision and the riskiest surface, because the sandbox runs build and test commands over user code.

Runtime choice.

  • v1: AWS Fargate (ECS) ephemeral tasks. One task per run. Scales to zero, pay per second, decent container isolation, no servers to manage, and RunTask is a single SDK call from a Rails job. Fast to ship. This is the recommendation for v1.
  • At scale: Firecracker microVMs. For stronger isolation of untrusted code and for snapshot/restore warm pools (sub-second starts), move the sandbox to Firecracker microVMs on bare-metal EC2 (the model E2B / Modal / Codex-style sandboxes use). More operational weight; defer until run volume and the threat model justify it. gVisor or Kata on EC2 is a middle option if Fargate isolation proves insufficient before we are ready for Firecracker.

The sandbox image. A container that bundles headless siGit Code plus a base toolchain (git, common language runtimes). The agent boots, reads the run spec from an injected env/file, clones, works, and pushes. Per-language base images (or a .sigit/agent.yml setup step, see Phase 3) keep cold builds fast.

Orchestration. v1 keeps it simple: the Rails AgentRunJob calls ECS RunTask directly via aws-sdk-ecs, passing the run spec as container overrides, and polls task status (or receives EventBridge task-state-change events into a webhook). If the lifecycle grows (retries, multi-step, fan-out), promote to Step Functions. Avoid Step Functions on day one; it is premature.

Networking and isolation (load-bearing for safety).

  • Sandbox runs in a private subnet. Egress through a NAT restricted by an allowlist: sigit.si (git + inference) and an explicit set of package registries (rubygems, npm, pypi, crates, etc.). Everything else is denied. This is the primary control against data exfiltration, SSRF against internal services, and crypto-mining abuse.
  • No inbound. The sandbox is not reachable from the internet.
  • Per-run IAM role scoped to only what the task needs; no broad account access inside the sandbox.

Credentials (mint short-lived, never long-lived). The sandbox receives:

  • a git token scoped to the single repo and ideally the single head branch, valid for the run only (extends the existing git_token exchange);
  • an inference token minted per run, carrying the user's entitlement and a hard token budget, pointed at https://sigit.si/api/v1.

Both expire when the run ends. The sandbox never holds app_secret or any long-lived credential, mirroring the existing public-client rule.

Logs and artifacts. The agent streams transcript chunks back to Rails (the SSE relay) for the live view and writes the full transcript and build logs to S3 (pointer stored on AgentRun). CloudWatch captures infra-level logs.

3.3 Inference path (reused as-is)

The sandboxed agent sets OPENAI_BASE_URL=https://sigit.si/api/v1 and uses the per-run inference token as its bearer. That flows through the existing Api::V1::ChatCompletionsController: entitlement gate, allowance metering, and the SIGIT_IDENTITY_PROMPT identity masking all apply with no change. This is a major reason the project is tractable: the autonomous agent is just another client of an inference endpoint we already operate and protect.

This also gives a clean answer to the existing "inference token ↔ Onde Cloud auth" open item: for cloud-agent runs the token is minted server-side with a budget, so there is no public client holding credentials at all.

3.4 Git and PR flow

The agent pushes the head branch via the existing git-receive-pack endpoint. On push, the platform creates (or links) the PR record. Because no PR model exists yet, scope for v1:

  • minimal PullRequest model (base/head branch, repo, author, status, title, body), a diff/compare view (we already render blobs and commits, so the diff renderer is incremental), and "open / close / merge" actions for the reviewer;
  • the agent fills in title and body from its summary. PR prose must stay neutral and must never name the upstream model or provider (same rule as chat).

Issues (assign-an-issue-to-the-agent) are a Phase 3 surface and depend on an issue model that also does not exist yet.

4. Safety, guardrails, and abuse

Principal-engineer non-negotiables, because this executes code on our infra on behalf of users:

  • Hard caps per run: wall-clock timeout, CPU/memory limits, inference token budget (tied to CloudUsage/a new AgentUsage), max tool calls, max sandbox lifetime. A run that blows any cap is killed and marked failed with a reason.
  • Network egress allowlist (section 3.2). The single most important control.
  • Ephemeral, scoped credentials only. Nothing long-lived in the sandbox.
  • Human-in-the-loop by default: the agent proposes a PR; it does not merge. No auto-merge in v1.
  • Concurrency limits per plan: caps simultaneous runs per user to bound spend and abuse.
  • Identity hygiene: transcripts, PR titles/bodies, and error messages stay neutral; never disclose the upstream model or provider (existing rule extends to agent output).
  • Cancellation is real: cancel tears down the sandbox and revokes the run's tokens.
  • Idempotency: run creation and sandbox start are idempotent so retries cannot double-spend.

A short threat-model doc is a Phase 0 deliverable (exfiltration, SSRF to internal metadata endpoints, resource abuse / mining, secret leakage from the user's own repo, prompt injection from repo contents steering the agent).

5. Pricing and packaging

Agent runs cost us sandbox compute (Fargate per-second) + inference tokens, so metering must capture both and price above blended COGS. The numbers below are a working model with the inputs we cannot yet read from this repo marked [FILL FROM PHASE 0]. The metering schema and billing copy should be designed against this formula now; the dollar figures get populated once the Phase 0 dogfood produces measured per-run token counts.

5.1 Unit economics: cost per run (COGS)

cost_per_run  =  sandbox_cost  +  inference_cost

sandbox_cost  =  vcpu_count * $0.04048/vcpu-hr * hours
              +  mem_gb     * $0.004445/gb-hr  * hours          (Fargate, us-east-1)

inference_cost = tokens_per_run / 1_000_000 * blended_upstream_cost_per_mtok

Worked example, sandbox side (known, fixed): 1 vCPU + 2 GB for a 20-minute run:

(1 * 0.04048 + 2 * 0.004445) * (20/60)  =  $0.0165 per run

Sandbox compute is rounding error. Inference dominates, and it is the unknown:

tokens_per_run                  = [FILL FROM PHASE 0]   (expect 200K – 2M+)
blended_upstream_cost_per_mtok  = [FILL: Onde/upstream blended $/Mtok]

inference_cost_per_run          = (tokens_per_run / 1e6) * blended_cost_per_mtok
cost_per_run                    = 0.0165 + inference_cost_per_run

Conclusion that holds regardless of the blanks: price on metered inference per run; treat sandbox-minutes as a kill-switch guardrail, not a billing axis.

5.2 Packaging (allowance model, mirrors Subscription::CLOUD_ALLOWANCE)

Agent runs are an included monthly bundle per plan, with overage or an add-on for heavy use. Suggested Subscription constant shape:

# Included siGit Code Cloud Agent runs per billing period. Tunable pricing knob.
AGENT_RUNS = { "pro" => [FILL], "team" => [FILL] }.freeze   # e.g. pro 20, team 75
TRIAL_AGENT_RUNS = [FILL]                                    # e.g. 2
AGENT_OVERAGE_PER_RUN = [FILL]                               # $/run past the bundle
Plan Price (today) Included agent runs/mo Overage Notes
Free $0 0 n/a local siGit Code only, no cloud agent
Trial $0 (14 days) [FILL: ~2] n/a felt-value, hard run cap
Pro $20/mo (existing) [FILL: ~20] [FILL: $/run] bundle sized so COGS < ~X% of $20
Team (existing) [FILL: ~75] [FILL: $/run] higher bundle + concurrency

Sizing rule for the bundle: pick included-runs so that bundle COGS stays under a target fraction of plan price (e.g. included-runs * cost_per_run ≤ 40% of MRR), leaving margin for the chat allowance those plans already include. With cost_per_run from 5.1 unknown, the bundle count is the lever set last.

5.3 Metering

Add AgentUsage keyed by (user, billing_period_key) (parallel to CloudUsage), recording per period: runs_count, agent_seconds, tokens_used. Runs decrement the plan's run bundle; tokens already flow through the existing cloud allowance and its 429 cap, so a single run can never escape the token budget. GET /api/v1/billing gains agent_runs_used + agent_runs_allowance alongside the existing cloud fields.

5.4 Guardrails (worst-case cost containment)

  • Per-run token budget minted into the run token, independent of the monthly allowance, so one pathological run is bounded.
  • Per-run wall-clock + sandbox-lifetime cap (kills runaway compute).
  • Per-plan concurrency cap bounds simultaneous spend.
  • Trial run cap (TRIAL_AGENT_RUNS) prevents trial-driven upstream bills, mirroring TRIAL_ALLOWANCE.

5.5 What Phase 0 must measure to finalize this

  1. tokens_per_run distribution (p50/p90/p99) over 20–30 real dogfood runs.
  2. blended_upstream_cost_per_mtok (from the Onde/upstream cost, internal).
  3. Resulting cost_per_run distribution, then back-solve included-run bundles per tier against the target-margin rule in 5.2.

Until 1 and 2 are real, every dollar above is a placeholder by design.

6. Roadmap (phased)

Estimates are calendar weeks for a small team; adjust to staffing. Each phase ends with a demoable, dogfoodable increment.

Phase 0 — Foundations and spikes (2–3 weeks)

  • Decide sandbox runtime: Fargate for v1 (documented path to Firecracker).
  • Build the headless siGit Code container image; prove end to end manually: a container clones a sigit.si repo, runs the agent against a task with inference via /api/v1, edits files, runs a test, and pushes a branch.
  • Per-run scoped token minting (git + inference) in Rails.
  • Threat model + egress allowlist design.
  • Add aws-sdk-ecs/aws-sdk-core to the Gemfile; stand up the VPC/subnet/NAT and the task definition in IaC.
  • Exit: one task goes from prompt to pushed branch by hand.

Phase 1 — Private alpha, single happy path (3–4 weeks)

  • AgentRun model + state machine; web AgentRunsController; AgentRunJob calling ECS RunTask.
  • Repo "Agent" tab: run form, live transcript via SSE/Turbo, final diff view.
  • Branch push + a minimal compare view (full PR model can lag one phase).
  • Caps: wall-clock timeout, concurrency = 1, token budget tied to metering.
  • Internal-only feature flag; dogfood on our own repos.
  • Exit: team members trigger runs from the web and review the diff.

Phase 2 — Beta, productized (4–6 weeks)

  • Minimal PullRequest model + diff/review UI so agent output is a real PR.
  • Steerability: follow-up messages to a running agent, cancel, re-run.
  • Metering + pricing surface: AgentUsage, plan caps, billing page copy.
  • Hardening: enforce egress allowlist, isolation review, token scoping, per-plan concurrency, abuse controls.
  • Trigger surfaces: sigit cloud run in the CLI and a desktop entry point, both hitting Api::V1::AgentRunsController.
  • Cold-start work (prebaked base images; warm pool if needed).
  • Exit: invite-only beta with real external users and real billing.

Phase 3 — GA and advanced (ongoing)

  • Issues + assign-an-issue-to-the-agent (needs an issue model).
  • Custom environments: repo-level .sigit/agent.yml (setup steps, allowed commands, language matrix), the analog of Copilot's environment customization.
  • Firecracker microVM sandboxes + snapshot warm pools for isolation and sub-second starts at volume.
  • Dependency/repo caching (S3/EFS layers) for faster, cheaper runs.
  • Parallel runs, plan-then-execute, multi-file refactors at scale.
  • Observability and an eval harness (task success rate, PR acceptance rate, cost per accepted PR) to drive model-tier and prompt tuning.

7. Key decisions and open dependencies

  • PRs/issues do not exist on sigit.si yet. A minimal PR surface is on the critical path (Phase 2); issues gate the issue-to-PR flow (Phase 3). Confirm scope early.
  • Sandbox isolation level. Fargate for v1; reassess against the threat model before opening to untrusted external users at volume; Firecracker is the scale answer.
  • Inference token model. Mint per-run, server-side, budget-bounded. This also closes the standing "inference token ↔ Onde Cloud auth" gap for this path.
  • COGS visibility. Must meter sandbox time and tokens from day one, or pricing flies blind.
  • No AWS SDK in the app yet. Adds a new infra dependency and IAM/VPC footprint to own and secure.
  • Naming (resolved). The product is siGit Code Cloud Agent (see product-overview.md). "session" is reserved for siGit Code Cloud chat; the agent's work unit is a run. Do not name the agent or its runs "sessions".
  • Git-host scope. Git only (no SVN, Mercurial, or other VCS). sigit.si is the first Git host, not the only one. Keep the git clone / branch-push / PR-open behind a GitHostAdapter seam from day one (sigit.si now; GitHub, GitLab, other Git hosts later) so the agent is not welded to our own forge.

8. North-star and success metrics

  • North star: accepted agent PRs per active paying user per month.
  • Quality: task success rate (run produces a mergeable PR without human fixes), PR acceptance rate, median time-to-PR.
  • Economics: cost per accepted PR (sandbox + tokens), gross margin per plan.
  • Reliability/safety: zero egress-allowlist escapes, zero credential leaks, sandbox p95 cold start, run failure rate by cause. ```