@jhustle44 / jcr-guide-pro
Code Commits Issues Pulls Cloud Agent CI/CD
copilot/change-app-name-to-jcrguru
copilot/add-firebase copilot/change-app-name-to-jcrguru copilot/create-zip-file copilot/create-zip-file-again copilot/merge-any-updates main master
md 253 lines 17 KB
Rendered Raw
1 # Deployment Failure Debugging Reference
2
3 Diagnose and fix deployment build failures by inspecting build history, logs, configuration, and environment.
4
5 ## When to Read This Reference
6
7 Read this when:
8
9 - The user says their app failed to publish
10 - The user reports their deployed app crashes or shows errors during the publishing step
11 - After a deployment attempt that the agent initiated fails
12 - The user asks for help debugging a deployment error
13
14 Do NOT read this for:
15
16 - First-time deployment setup (use the main deployment skill instructions above)
17 - Checking runtime logs for a working production deployment (use `references/deployment-logs.md`)
18 - Issues about development-mode behavior, not production
19
20 ## Available Functions
21
22 ### listDeploymentBuilds(limit?)
23
24 List recent deployment builds for the current repl.
25
26 **Parameters:**
27
28 - `limit` (int, optional): Max builds to return (1-100, default 10)
29
30 **Returns:** `{ success, builds }` where each build has:
31
32 - `id` — build UUID
33 - `status` — e.g. `"success"`, `"failed"`, `"building"`
34 - `provider` — e.g. `"autoscale"`, `"vm"`
35 - `timeCreated` — ISO timestamp
36 - `timeUpdated` — ISO timestamp (may be absent)
37 - `user` — `{ id, username, displayName }` (may be absent)
38
39 **Example:**
40
41 ```javascript
42 const result = await listDeploymentBuilds({ limit: 5 });
43 if (result.success) {
44 console.log(JSON.stringify(result.builds, null, 2));
45 }
46 ```
47
48 ### getDeploymentBuild(buildId, skipLogs?)
49
50 Get detailed information about a specific deployment build, including build logs.
51
52 **Parameters:**
53
54 - `buildId` (string, required): The build ID to inspect
55 - `skipLogs` (bool, optional): If true, omit build log lines (default false)
56
57 **Returns:** `{ success, build }` where build has:
58
59 - `id`, `status`, `provider`, `timeCreated`, `timeUpdated` — same as list summary
60 - `suspendedReason` — reason if deployment was suspended (may be absent)
61 - `user` — `{ id, username, displayName }` (may be absent)
62 - `machineConfiguration` — `{ slug, vcpu, memory }` (may be absent)
63 - `logs` — array of build log line strings (may contain ANSI escape codes)
64
65 **Example:**
66
67 ```javascript
68 const result = await getDeploymentBuild({ buildId: "b222be9b-63d4-405a-a6ba-11948a8a0df1" });
69 if (result.success) {
70 console.log("Status:", result.build.status);
71 const logs = result.build.logs || [];
72 console.log(`Total log lines: ${logs.length}`);
73 // Print only the last 80 lines — build errors are near the end
74 // and full logs may exceed the 4K stdout observation limit.
75 const tail = logs.slice(-80);
76 for (const line of tail) {
77 console.log(line);
78 }
79 }
80 ```
81
82 ### Related functions from other skills
83
84 These existing callbacks are useful during debugging:
85
86 - `fetchDeploymentLogs({ afterTimestamp?, beforeTimestamp?, message? })` — fetch runtime/production logs (see `references/deployment-logs.md`). Use to check what happens after the app starts in production.
87 - `viewEnvVars({ environment: "development" | "production" })` — compare env var names between dev and prod (from `environment-secrets` skill). Use to find missing production secrets.
88 - `deployConfig({ deploymentTarget, run?, build?, publicDir? })` — reconfigure deployment settings (from the main deployment skill). Use to fix run commands and build commands.
89 - `suggestDeploy()` — prompt the user to click Publish after making fixes. **Only works in the main repl context** — in task-agent/subrepl sessions this callback returns `success: false`. If you are in a task agent, skip this call and instead remind the user to publish from the main version after merging.
90
91 ## Diagnostic Procedure
92
93 Follow these steps systematically. Do not skip ahead.
94
95 ### Step 1: Get the build history
96
97 ```javascript
98 const result = await listDeploymentBuilds({ limit: 5 });
99 if (!result.success) {
100 console.log("Error fetching builds:", result.error);
101 } else {
102 console.log(JSON.stringify(result.builds, null, 2));
103 }
104 ```
105
106 If the callback fails (e.g. no deployment configured, transient error), report the error to the user before proceeding.
107
108 Identify the most recent failed build. If all recent builds succeeded, the issue may be a runtime problem rather than a build failure — check runtime logs with `fetchDeploymentLogs()` instead.
109
110 ### Step 2: Get the failed build details
111
112 ```javascript
113 const detail = await getDeploymentBuild({ buildId: "<failed-build-id>" });
114 if (!detail.success) {
115 console.log("Error fetching build details:", detail.error);
116 } else {
117 console.log("Status:", detail.build.status);
118 const logs = detail.build.logs || [];
119 console.log(`Total log lines: ${logs.length}`);
120 // Only print the last 80 lines — errors appear near the end
121 // and full logs may exceed the 4K stdout observation limit.
122 const tail = logs.slice(-80);
123 for (const line of tail) {
124 console.log(line);
125 }
126 }
127 ```
128
129 If the callback fails (e.g. stale build ID, transient error), report the error and retry or fall back to runtime logs.
130
131 Read the build logs carefully. The failure cause is almost always near the end of the log output. If the tail doesn't contain enough context, fetch an earlier window with `logs.slice(-160, -80)`.
132
133 A deployment goes through these phases in order: **build** → **promote** → **serve**. The build logs show which phase failed:
134
135 - **Build phase failures** look like compilation errors, `npm install` failures, missing dependencies, or build commands exiting non-zero.
136 - **Promote phase failures** look like the build succeeding (e.g. image pushed, container started) followed by the container failing to become ready. What "ready" means depends on the deployment target:
137 - **`autoscale` and HTTP-serving `vm`**: the deployer sends a startup probe and waits for an HTTP 200 response. Build logs mention health checks, startup probes, or readiness probes timing out. The default probe path is `GET /`. The probe fails if the app crash-loops, returns non-200 on the probe path, binds to the wrong host/port, or takes too long to start.
138 - **non-HTTP `vm` (Discord bots, workers, etc.) and `scheduled` jobs**: there is no HTTP probe. The promote step fails when the run command exits non-zero or crashes during startup. Build logs typically just show the process exiting with a stack trace or non-zero exit code. Do **not** look for an HTTP route in this case — the failure is the process itself.
139 - **`static`**: there is no run command, so promote-step failures here are rare and usually mean `publicDir` is empty or misconfigured.
140 - **Serve phase failures** happen after a successful promote: the app is live but throws errors at runtime. These show up in `fetchDeploymentLogs()`, not in build logs.
141
142 ### Step 3: Branch on what you find
143
144 **Build logs show a clear error in the build phase** (compilation error, missing dependency, command not found):
145 → Fix the code or configuration directly. See Common Failure Modes below.
146
147 **Build logs show an HTTP 403 `Blocked by Security Policy` from `package-firewall.replit.local`** (typically during an `npm ci`/`npm install`/`pip install` step):
148 → This is the supply-chain firewall blocking a flagged package — not a transient error. Do not retry the same publish. See "Supply-chain firewall block" under Common Failure Modes below, and the `package-management` and `security_scan` skills.
149
150 **Build phase succeeded but the promote step failed** (build logs mention health check, startup probe, container failing to start, the process exiting, or the deployment never became ready):
151 → First check the deployment target in `.replit`'s `[deployment]` section — the right diagnosis depends on it.
152 → For **`autoscale` and HTTP-serving `vm`**: the app is failing the startup probe. The default probe is `GET /`. The probe can fail because the app crash-loops on startup, the probe path returns non-200, or the app binds to the wrong host/port. See "Health check / promote step failure" and "Application crashes on startup" below.
153 → For **non-HTTP `vm` (bots, workers) and `scheduled` jobs**: the run command exited non-zero or crashed before becoming healthy. There is no HTTP probe — do not chase a `/` route. See "Application crashes on startup" below.
154 → Either way, check runtime logs to see why the process is failing: `fetchDeploymentLogs({ message: "(?i)(error|exception|failed|crash|traceback)" })`.
155
156 **No obvious cause from build logs:**
157 → Check runtime logs for post-deploy errors.
158 → Check environment variables: compare `viewEnvVars({ environment: "development" })` with `viewEnvVars({ environment: "production" })`.
159
160 **Build status is `suspended`:**
161 → Check `suspendedReason`. This is usually a billing or quota issue, not a code problem. Inform the user.
162
163 ### Step 4: Check deployment configuration
164
165 Read `.replit`'s `[deployment]` section. Verify run command, build command, and deployment target are correct. Use `deployConfig()` to fix any misconfigured settings.
166
167 ### Step 5: After fixing
168
169 After making code or configuration fixes:
170
171 - **Main repl context:** Call `suggestDeploy()` to prompt the user to re-publish.
172 - **Task-agent / subrepl context:** Do **not** call `suggestDeploy()` — it will fail. Instead, remind the user to publish from the main version of the project after the task branch is merged.
173
174 ## Common Failure Modes
175
176 ### Application crashes on startup
177
178 - **Indicators:** Build succeeds, but the deployment fails at the **promote step**. How this surfaces depends on the deployment target:
179 - **`autoscale` / HTTP-serving `vm`**: build logs mention the container failing health checks, the startup probe timing out, or the deployment never becoming ready. A crashing or crash-looping app cannot respond to the probe, so the promote step fails.
180 - **non-HTTP `vm` (bots, workers) / `scheduled` jobs**: build logs show the run command exiting with a stack trace or non-zero exit code, or the process restarting repeatedly. There is no HTTP probe — the failure is just the process dying.
181 - **Look for:** Uncaught exceptions, import errors, missing modules, missing environment variables (e.g. `DISCORD_TOKEN`), syntax errors, or repeated restarts in runtime logs (`fetchDeploymentLogs`).
182 - **Fix:** Read the stack trace and fix the code error so the app starts cleanly and stays up. Confirm by running the production `run` command locally:
183 - For HTTP services: it must start without crashing **and** respond 200 on the probe path (`GET /` by default).
184 - For bots/workers/jobs: it must start without crashing and stay running (or, for `scheduled`, run to completion with exit code 0). If the crash is from a missing secret, use `requestEnvVar()` to ask the user to add it to production.
185
186 ### Build command failure
187
188 - **Indicators:** Build status `failed`, build logs show non-zero exit from build command.
189 - **Look for:** TypeScript compilation errors, missing dependencies not in package.json/requirements.txt, build scripts that reference dev-only paths.
190 - **Fix:** Ensure the build command succeeds locally before deploying. Check that all dependencies are listed in the manifest.
191
192 ### Supply-chain firewall block (HTTP 403 from `package-firewall.replit.local`)
193
194 - **Indicators:** Build status `failed` during the **build phase**, with a build log line like `npm error 403 Forbidden - GET http://package-firewall.replit.local/npm/<pkg>/-/<pkg>-<ver>.tgz - Blocked by Security Policy` (or the equivalent for another package manager). This usually comes from a build/install step — `npm ci`, `npm install`, `npm update`, `pip install`, etc. — that the build runs before promoting the app.
195 - **Why it happens:** Replit runs a supply-chain security firewall that blocks packages flagged as malicious or otherwise risky. The build's install step routes package fetches through this firewall, so a flagged dependency that installs fine in dev can still fail at publish time. The block is on the **package**, not the build — retrying the same publish will fail again.
196 - **Look up the package:** Identify the blocked package and version from the firewall log line, then check its public security advisories and reputation (e.g. the package's page on a vulnerability/supply-chain database for its ecosystem) to understand why it was flagged. See the `security_scan` skill for the broader scanning workflow — both it and the `package-management` skill document that a 403 from `package-firewall.replit.local` is a security block that must not be blindly retried.
197 - **Fix:** Remediate the flagged dependency rather than retrying the publish. Upgrade it to a version that isn't flagged, or replace/remove it if it's unused. Use the `package-management` skill (`installLanguagePackages` / `uninstallLanguagePackages`) to update dependencies so the manifest and lockfile change. If the install is driven by a custom build/run command, fix that command. Then re-publish (`suggestDeploy()` in the main repl).
198
199 ### Run command failure or misconfiguration
200
201 - **Indicators:** Build logs show command not found or immediate exit.
202 - **Common causes:** Run command references a file that doesn't exist, uses a dev server instead of production server, wrong binary name.
203 - **Fix:** Verify the run command in `.replit` `[deployment]` section. Use production servers: `gunicorn` not `flask run`, `node server.js` not `npm run dev`.
204
205 ### Port/binding issues
206
207 - **Indicators:** Build succeeds, but app fails health check or port check.
208 - **Common causes:**
209 - App binds to `localhost`/`127.0.0.1` instead of `0.0.0.0` — the container can't reach it.
210 - App binds to a different port than configured.
211 - App uses a `PORT` env var that isn't set in production.
212 - **Fix:** Ensure the app binds to `0.0.0.0` on the correct port.
213
214 ### Health check / promote step failure
215
216 - **Indicators:** The **build phase succeeds** but the deployment fails at the **promote step**. Build logs mention health check failures, startup probe timeouts, the container failing to become ready, or the deployment never reaching the live state. This is one of the most common deployment failure modes and it is **not** a build problem — the artifact built fine, the app just isn't responding to the probe.
217 - **Details:** Before promoting a deployment to live, the deployer sends a startup probe to `GET /` and requires an HTTP 200 response. If the probe doesn't get a 200 within the timeout, the promote step fails and the previous version (if any) keeps serving traffic.
218 - **Common causes:**
219 - **App crashes or crash-loops on startup** — a crashing app obviously can't respond to the probe. Check runtime logs for stack traces. See "Application crashes on startup" above.
220 - App returns 404, 3xx redirect, or 5xx on `/`.
221 - App requires authentication on all routes including `/`.
222 - App binds to `localhost`/`127.0.0.1` instead of `0.0.0.0`, or to the wrong port.
223 - App takes too long to start (slow imports, blocking startup work, large model downloads).
224 - **Fix:** Ensure the root route `/` returns a 200 response. Use `fetchDeploymentLogs()` to see whether the app is crashing or just responding incorrectly. Optimize startup time if needed.
225
226 ### Missing or different environment variables
227
228 - **Indicators:** App works in dev but fails in prod with connection errors, auth failures, or undefined config.
229 - **Fix:** Use `viewEnvVars({ environment: "development" })` and `viewEnvVars({ environment: "production" })` to compare. Use `requestEnvVar()` to ask the user to provide missing production secrets.
230
231 ### Autoscale-specific issues (stateless footguns)
232
233 Autoscale deployments are **stateless Cloud Run services**. Each request may hit a different instance. Instances spin down when idle.
234
235 **Things that will NOT work on autoscale:**
236
237 - In-memory state (sessions, caches, counters) — use a database or Redis instead
238 - SQLite — filesystem is ephemeral, use PostgreSQL
239 - WebSocket connections that need to persist — instances can be killed at any time
240 - Background workers / long-running processes — these get killed when the request ends
241 - Cron jobs or scheduled tasks running inside the app — use the `scheduled` deployment type
242 - File uploads stored on the local filesystem — use object storage
243
244
245 If the app needs any of the above, inform the user that they should switch to Reserved VM (`vm`) deployment type in the Deployments pane. The deployment type is not something you can change programmatically — the user must change it themselves.
246
247 ## Important Notes
248
249 1. **Don't diagnose infrastructure problems.** If the error appears to be on Replit's side (e.g., transient cloud provider issues, `UnknownError` from the deployer), suggest the user retry the deployment or contact Replit support.
250 2. **Changing deployment type.** The deployment type (autoscale, vm, scheduled, static) is set in the Deployments pane, not in code. If it needs to change, tell the user to update it themselves.
251 3. **Use production terminology.** In user-facing messages, say "publish" not "deploy" — that's the Replit product terminology.
252 4. **You cannot trigger a deployment.** In the main repl, call `suggestDeploy()` to prompt the user to click the Publish button. In task-agent/subrepl sessions, `suggestDeploy()` is unavailable — remind the user to publish from the main version after merging.
253 5. **Production config lives in `.replit`.** Deployment settings (run command, build command, deployment target) are in the `.replit` file's `[deployment]` section.