| 1 | # Finding crashes (signal) |
| 2 | |
| 3 | A "crash" = the kernel delivered a fatal signal to the agent |
| 4 | process. The signature is `AE_FATAL_SIGNAL_CODE` non-empty. |
| 5 | |
| 6 | ## Quick recipe |
| 7 | |
| 8 | Get recent signal crashes on stable + recent nightlies: |
| 9 | |
| 10 | ```bash |
| 11 | .agents/skills/query-agent-events/scripts/get-events.sh \ |
| 12 | --health crash \ |
| 13 | --since '24h ago' \ |
| 14 | --versions auto |
| 15 | ``` |
| 16 | |
| 17 | Output is a JSON dump under |
| 18 | `<repo>/.local/audits/query-agent-events/<timestamp>.json`. |
| 19 | |
| 20 | Then aggregate: |
| 21 | |
| 22 | ```bash |
| 23 | .agents/skills/query-agent-events/scripts/analyze-events.sh \ |
| 24 | --input <bundle.json> \ |
| 25 | --by signal |
| 26 | ``` |
| 27 | |
| 28 | ## What "signal crash" means |
| 29 | |
| 30 | The agent received a fatal signal (SIGSEGV, SIGBUS, SIGFPE, |
| 31 | SIGABRT, SIGILL, etc.) and could not gracefully recover. The |
| 32 | deadly-signal handler tried to capture context (signal_code, |
| 33 | fault_address, stack_trace) before exiting. |
| 34 | |
| 35 | Distinguishing predicates: |
| 36 | |
| 37 | - `AE_FATAL_SIGNAL_CODE` non-empty -- definitive marker. |
| 38 | - `AE_AGENT_HEALTH` IN crash-first / crash-loop / crash-repeated |
| 39 | / crash-entered -- the agent classifies the result. |
| 40 | - `AE_AGENT_EXIT_REASON_*` typically contains |
| 41 | `signal-segmentation-fault`, `signal-bus-error`, |
| 42 | `signal-floating-point-exception`, `signal-illegal-instruction`, |
| 43 | `signal-abort`, etc. |
| 44 | |
| 45 | ## Index-friendly query for crashes |
| 46 | |
| 47 | ```json |
| 48 | { |
| 49 | "after": -86400, |
| 50 | "before": 0, |
| 51 | "last": 500, |
| 52 | "__logs_sources": "agent-events", |
| 53 | "selections": { |
| 54 | "AE_AGENT_HEALTH": ["crash-first", "crash-loop", "crash-repeated", "crash-entered"], |
| 55 | "AE_AGENT_VERSION": ["v2.10.0", "v2.10.0-135-nightly", "v2.10.0-130-nightly"] |
| 56 | }, |
| 57 | "facets": ["AE_FATAL_SIGNAL_CODE", "AE_FATAL_FUNCTION", "AE_HOST_ARCHITECTURE", "AE_OS_FAMILY"] |
| 58 | } |
| 59 | ``` |
| 60 | |
| 61 | The auto version filter computes the version list dynamically |
| 62 | (see `update-cadence.md`). |
| 63 | |
| 64 | ## Triage flow |
| 65 | |
| 66 | 1. **Get the dump** -- `get-events.sh --health crash`. |
| 67 | 2. **Group by signal** -- `analyze-events.sh --by signal` to |
| 68 | see SIGSEGV vs SIGBUS vs SIGABRT distribution. |
| 69 | 3. **Group by function** for the dominant signal -- |
| 70 | `analyze-events.sh --by fatal_function --filter "signal=SIGSEGV/SEGV_MAPERR"` |
| 71 | (or pre-filter the dump with `jq`). |
| 72 | 4. **Pick the dominant function**, look at one |
| 73 | representative event's stack trace |
| 74 | (`AE_FATAL_STACK_TRACE`), correlate with source. |
| 75 | 5. **Cross-check by version** -- is this on stable only? Just |
| 76 | nightlies? When did it appear? |
| 77 | 6. **Cross-check by environment** -- arch? distro? kubernetes? |
| 78 | parent vs child? Is the crash environment-specific? |
| 79 | 7. **Read source, fix bug.** |
| 80 | |
| 81 | ## Common signal-code values |
| 82 | |
| 83 | (from `<repo>/src/libnetdata/signals/signal-code.c:97-184`, |
| 84 | see `AE_FIELDS.md` for the full table) |
| 85 | |
| 86 | | Signal code | What it means | |
| 87 | |---|---| |
| 88 | | `SIGSEGV/SEGV_MAPERR` | NULL pointer / freed memory / unmapped page. | |
| 89 | | `SIGSEGV/SEGV_ACCERR` | Write to read-only / executable page. | |
| 90 | | `SIGBUS/BUS_ADRALN` | Misaligned access (mostly ARM / mmap). | |
| 91 | | `SIGBUS/BUS_OBJERR` | Object-level fault (often disk I/O). | |
| 92 | | `SIGFPE/FPE_INTDIV` | Integer divide by zero. | |
| 93 | | `SIGABRT/SI_TKILL` | abort() / assertion failure. | |
| 94 | | `SIGILL/ILL_ILLOPC` | Illegal instruction (often binary corruption). | |
| 95 | | `SIGTRAP/TRAP_BRKPT` | Breakpoint trap. Usually a debugger; sometimes a deliberate `__builtin_trap()`. | |
| 96 | |
| 97 | ## Pitfalls |
| 98 | |
| 99 | - **Empty stack trace**: `AE_FATAL_STACK_TRACE` may be the |
| 100 | string `info: will now attempt to get stack trace` or `info: stack trace is not available, libbacktrace reports no frames` |
| 101 | when the fault was instantaneous (e.g. NULL deref at low |
| 102 | address). Without the stack, fall back to |
| 103 | `AE_FATAL_FUNCTION` + `AE_FATAL_FILENAME` + `AE_FATAL_LINE` |
| 104 | (these are populated from `__FILE__` / `__LINE__` of the |
| 105 | most recent `fatal()` call -- not the crash site, but |
| 106 | often nearby). |
| 107 | |
| 108 | - **Aborted dumps**: events where `AE_FATAL_SIGNAL_CODE` is |
| 109 | set but other `AE_FATAL_*` fields are empty -- the signal |
| 110 | arrived before context capture completed. |
| 111 | |
| 112 | - **Shutdown races**: a crash during shutdown reports |
| 113 | `AE_EXIT_CAUSE = 'killed hard on exit'` or `'killed hard on shutdown'` |
| 114 | with a stack trace that may show shutdown timing rather |
| 115 | than the actual crash site. Interpret with caution. |
| 116 | |
| 117 | - **Sentry-suppressed**: when `AE_FATAL_SENTRY = true`, the |
| 118 | agent attempted a Sentry submission. Sentry breakdowns have |
| 119 | more information; cross-reference if available. |
| 120 | |
| 121 | ## Filtering out noise |
| 122 | |
| 123 | Many crashes from old / unsupported versions are already |
| 124 | fixed. The default `--versions auto` filter handles this. |
| 125 | For wider investigations, scope to stable releases only: |
| 126 | |
| 127 | ```bash |
| 128 | get-events.sh --health crash --versions '^v2\.\d+\.\d+$' |
| 129 | ``` |
| 130 | |
| 131 | ## Related recipes |
| 132 | |
| 133 | - `recipes/find-by-function.md` -- when you have a function |
| 134 | name in mind. |
| 135 | - `recipes/find-by-version.md` -- regression spotter. |
| 136 | - `finding-fatals.md` -- the OTHER class (deliberate exits, |
| 137 | not signal crashes). |