| 1 | # Finding fatals (deliberate exits) |
| 2 | |
| 3 | A "fatal" = the agent **chose** to exit because of a |
| 4 | condition it could not recover from (OOM, disk full, |
| 5 | assertion failure, listen-socket conflict, fatal() call). |
| 6 | DIFFERENT from a signal crash (the kernel didn't kill us; we |
| 7 | called `exit()` or `_exit()` ourselves). |
| 8 | |
| 9 | ## Quick recipe |
| 10 | |
| 11 | ```bash |
| 12 | .agents/skills/query-agent-events/scripts/get-events.sh \ |
| 13 | --exit-cause fatal \ |
| 14 | --since '24h ago' \ |
| 15 | --versions auto |
| 16 | ``` |
| 17 | |
| 18 | `--exit-cause fatal` is shorthand for the deliberate-fatal |
| 19 | class of `AE_EXIT_CAUSE` values (see below). |
| 20 | |
| 21 | ## Distinguishing predicates |
| 22 | |
| 23 | - `AE_FATAL_SIGNAL_CODE` is **EMPTY** (this is what separates |
| 24 | fatals from signal crashes). |
| 25 | - `AE_EXIT_CAUSE` is one of: |
| 26 | - `no last status` |
| 27 | - `out of memory` (NOT `cannot allocate` -- the .local draft |
| 28 | was wrong) |
| 29 | - `disk full` / `disk almost full` / `disk read-only` |
| 30 | - `already running` |
| 31 | - `fatal on start` / `fatal on exit` / `fatal and exit` |
| 32 | - `exit timeout` |
| 33 | - `abnormal power off` |
| 34 | - `AE_AGENT_EXIT_REASON_*` may contain `out-of-memory`, |
| 35 | `already-running`, `fatal`, `shutdown-timeout`. |
| 36 | - `AE_FATAL_MESSAGE`, `AE_FATAL_FUNCTION`, `AE_FATAL_FILENAME`, |
| 37 | `AE_FATAL_LINE` -- populated from the `fatal()` call site. |
| 38 | |
| 39 | ## Index-friendly query |
| 40 | |
| 41 | ```json |
| 42 | { |
| 43 | "after": -86400, |
| 44 | "before": 0, |
| 45 | "last": 500, |
| 46 | "__logs_sources": "agent-events", |
| 47 | "selections": { |
| 48 | "AE_EXIT_CAUSE": [ |
| 49 | "no last status", |
| 50 | "out of memory", |
| 51 | "disk full", |
| 52 | "disk almost full", |
| 53 | "disk read-only", |
| 54 | "already running", |
| 55 | "fatal on start", |
| 56 | "fatal on exit", |
| 57 | "fatal and exit" |
| 58 | ], |
| 59 | "AE_AGENT_VERSION": ["v2.10.0", "v2.10.0-135-nightly", "v2.10.0-130-nightly"] |
| 60 | }, |
| 61 | "facets": ["AE_EXIT_CAUSE", "AE_FATAL_FUNCTION", "AE_HOST_ARCHITECTURE", "AE_OS_FAMILY"] |
| 62 | } |
| 63 | ``` |
| 64 | |
| 65 | ## Per-cause triage |
| 66 | |
| 67 | ### `out of memory` |
| 68 | |
| 69 | Agent panicked when an allocation failed. Look at |
| 70 | `AE_FATAL_FUNCTION` to localize -- often `mallocz`, |
| 71 | `reallocz`, dbengine page allocators, or memory-pool |
| 72 | constructors. Cross-correlate with |
| 73 | `AE_HOST_MEMORY_FREE_PERCENT` (low at POST time = host was |
| 74 | under pressure) and `AE_HOST_MEMORY_NETDATA` (how much |
| 75 | netdata was using). |
| 76 | |
| 77 | Most useful slicers: |
| 78 | - `AE_AGENT_DB_MODE` -- dbengine memory mode. |
| 79 | - `AE_AGENT_DB_TIERS` -- number of tiers. |
| 80 | - `AE_HOST_MEMORY_TOTAL` -- absolute host RAM. OOM on a 1GB |
| 81 | host is different from OOM on a 64GB host. |
| 82 | |
| 83 | ### `disk full` / `disk almost full` / `disk read-only` |
| 84 | |
| 85 | Storage failure. Look at: |
| 86 | - `AE_HOST_DISK_DB_FREE` / `AE_HOST_DISK_DB_INODES_FREE` -- |
| 87 | what state the disk was in. |
| 88 | - `AE_HOST_DISK_DB_READ_ONLY` -- bool, set on read-only. |
| 89 | - `AE_AGENT_DB_MODE` -- whether dbengine is in `dbengine`, |
| 90 | `ram`, `ram-cache`, `none`, `alloc`, `save`, ... |
| 91 | |
| 92 | ### `already running` |
| 93 | |
| 94 | Listen-socket conflict at init. Look at `AE_AGENT_VERSION` |
| 95 | and `AE_AGENT_RESTARTS` -- if restarts > 0 and the same |
| 96 | agent_id keeps hitting this, there's a stale lock or a |
| 97 | concurrent agent on the same host. |
| 98 | |
| 99 | ### `fatal on start` / `fatal on exit` / `fatal and exit` |
| 100 | |
| 101 | Generic `fatal()` calls. Look at `AE_FATAL_MESSAGE` to find |
| 102 | the panic message and `AE_FATAL_FUNCTION` to localize. Group |
| 103 | by message to find the most common patterns. |
| 104 | |
| 105 | ### `no last status` |
| 106 | |
| 107 | First-ever start, or the prior status file was corrupted / |
| 108 | unreadable. Usually transient. If recurring on the same |
| 109 | agent (`AE_AGENT_RESTARTS` going up), the status file is |
| 110 | being deleted or the disk path is broken. |
| 111 | |
| 112 | ### `abnormal power off` |
| 113 | |
| 114 | Power loss between "agent was running" and "agent restarted". |
| 115 | Useful to see whether power events correlate with crashes |
| 116 | elsewhere -- usually filter these OUT of crash analysis. |
| 117 | |
| 118 | ### `exit timeout` |
| 119 | |
| 120 | Shutdown didn't complete in time. Look at |
| 121 | `AE_AGENT_TIMINGS_EXIT` for how long shutdown ran before the |
| 122 | timeout fired. Long exit timings + worker_job_id may localize |
| 123 | the stuck thread. |
| 124 | |
| 125 | ## Triage flow |
| 126 | |
| 127 | 1. **Get the dump** -- `get-events.sh --exit-cause fatal`. |
| 128 | 2. **Group by exit_cause** -- `analyze-events.sh --by exit_cause` |
| 129 | to see the distribution. |
| 130 | 3. **For the dominant cause**, group by function or message -- |
| 131 | `analyze-events.sh --by fatal_function --filter "exit_cause=out of memory"`. |
| 132 | 4. **Cross-check environment** -- is this OOM specific to |
| 133 | small-RAM hosts? Specific dbengine mode? |
| 134 | 5. **Read source at the panic site**, fix the bug or improve |
| 135 | the error path. |
| 136 | |
| 137 | ## Pitfalls |
| 138 | |
| 139 | - **`abnormal power off` is not a bug** -- it's environmental. |
| 140 | Filter out of crash analysis unless you specifically want |
| 141 | power events. |
| 142 | |
| 143 | - **`fatal_errno`** vs `fatal.errno` -- the producer emits |
| 144 | `fatal.errno` (nested), not `fatal_errno` (top-level). The |
| 145 | journal field is `AE_FATAL_ERRNO` in both cases (the |
| 146 | underscore comes from the dot transliteration). |
| 147 | |
| 148 | - **`exit timeout` records have NO `AE_FATAL_*` context** -- |
| 149 | the shutdown timer fired without a panic site. Use |
| 150 | `AE_AGENT_TIMINGS_EXIT` and `AE_FATAL_WORKER_JOB_ID` instead. |
| 151 | |
| 152 | - **Disk-related causes propagate** -- a `disk full` event |
| 153 | may show up as `cannot allocate` in subsequent attempts. |
| 154 | The first record is the meaningful one. |
| 155 | |
| 156 | ## Related recipes |
| 157 | |
| 158 | - `finding-crashes.md` -- the OTHER class (signal-delivered). |
| 159 | - `recipes/find-by-function.md` -- localize by function name. |
| 160 | - `recipes/find-by-version.md` -- regression spotter. |