| 1 | # Update cadence |
| 2 | |
| 3 | agent-events is **not real-time**. Understanding when events |
| 4 | arrive is essential for query design. |
| 5 | |
| 6 | ## The "after-the-fact" model |
| 7 | |
| 8 | When a Netdata agent crashes, the crash itself does NOT post |
| 9 | to the ingestion server. The agent has to start again, read |
| 10 | its on-disk status file (the previous session's state), and |
| 11 | POST that to the ingestion server. |
| 12 | |
| 13 | Concretely (`src/daemon/status-file.c`): |
| 14 | |
| 15 | 1. **During a session** -- the agent maintains its current |
| 16 | status in memory and writes a snapshot to |
| 17 | `/var/lib/netdata/status-netdata.dat` periodically (see |
| 18 | "Disk save cadence" below). |
| 19 | 2. **On exit / crash** -- a final snapshot is written if |
| 20 | possible (signal-async-safe writer at |
| 21 | `<repo>/src/daemon/status-file-io.c`). For SIGKILL / OOM-kill, |
| 22 | no final snapshot is written -- the prior periodic snapshot |
| 23 | is what gets reported. |
| 24 | 3. **On NEXT start** -- the new agent reads the on-disk |
| 25 | status, computes `agent_health`, classifies `exit_cause`, |
| 26 | then POSTs to the ingestion server (`status-file.c:988`). |
| 27 | |
| 28 | So a crash at 14:00 produces a journal record on the ingestion |
| 29 | server only after the agent restarts -- which might happen |
| 30 | seconds, minutes, hours, or days later (depending on |
| 31 | operator policy and whether the agent loops). |
| 32 | |
| 33 | ### Implication for queries |
| 34 | |
| 35 | - **"Last hour"** is misleading -- it misses crashes from |
| 36 | agents that crashed in the last hour but haven't restarted |
| 37 | yet. |
| 38 | - **"Last 24 hours"** is the natural unit -- accommodates |
| 39 | typical restart latency AND aligns with the dedup window. |
| 40 | - For rare crashes (1-per-few-days class), expand to 7+ days. |
| 41 | |
| 42 | ## Disk save cadence |
| 43 | |
| 44 | `<repo>/src/daemon/status-file.c:835`: |
| 45 | |
| 46 | > "Update disk footprint at most once every 10 minutes (600 |
| 47 | > seconds)" |
| 48 | |
| 49 | The in-memory snapshot is refreshed at most every 10 minutes. |
| 50 | Each refresh triggers a save to |
| 51 | `/var/lib/netdata/status-netdata.dat` (atomic temp+rename via |
| 52 | `<repo>/src/daemon/status-file-io.c`). Saves also happen on: |
| 53 | |
| 54 | - start-up (before claiming the new session); |
| 55 | - exit (graceful shutdown writes the final state); |
| 56 | - inside a deadly-signal handler (best-effort write before |
| 57 | the process dies). |
| 58 | |
| 59 | So the on-disk snapshot a restarting agent reads is at most |
| 60 | ~10 minutes old, plus any signal-handler-time updates. |
| 61 | |
| 62 | ## 23h client-side dedup |
| 63 | |
| 64 | `<repo>/src/daemon/status-file-dedup.c:11`: |
| 65 | |
| 66 | ```c |
| 67 | #define REPORT_EVENTS_EVERY (86400 - 3600) // -1 hour to tolerate cron randomness |
| 68 | ``` |
| 69 | |
| 70 | = 82800 seconds = 23 hours. |
| 71 | |
| 72 | Each agent maintains a small dedup table at |
| 73 | `/var/lib/netdata/dedup-netdata.dat` (binary; see `:13-22`): |
| 74 | 50 slots, each storing `(hash, sentry-flag, timestamp)`. |
| 75 | |
| 76 | The hash includes (`status-file-dedup.c:42-95`): |
| 77 | |
| 78 | - `version` (schema), `status`, `signal_code`, `profile`, |
| 79 | `exit_reason`, `db_mode`, `db_tiers`, `kubernetes`, |
| 80 | `sentry_available`, `sentry_fatal`, |
| 81 | - `host_id` (Netdata machine GUID), `machine_id` (OS), |
| 82 | `worker_job_id`, `line` (in source), |
| 83 | - `version` string, `filename`, `function`, `stack_trace` |
| 84 | (with hex addresses zeroed for hashing only), |
| 85 | `thread`, |
| 86 | - `msg`, `cause`. |
| 87 | |
| 88 | So **each unique combination of these fields** has its own |
| 89 | dedup slot. If the agent saw the same combination within 23h, |
| 90 | the new POST is suppressed (`dedup_already_posted` returns |
| 91 | `true` -> producer skips the POST). |
| 92 | |
| 93 | ### Implication: 1 record per agent per event-class per day |
| 94 | |
| 95 | A given agent will report the same crash signature at most |
| 96 | once per 23h window. So at the journal level: |
| 97 | |
| 98 | - 1 distinct crash class per restart, per agent, per day. |
| 99 | - Different agents posting the same crash signature both |
| 100 | arrive (the dedup is **client-side**, not server-side). |
| 101 | - Different crash signatures from the same agent within 23h |
| 102 | all arrive (different hashes). |
| 103 | |
| 104 | This is why "last 24 hours" is the natural query window: the |
| 105 | dedup makes daily counts meaningful (each agent contributes |
| 106 | at most one record per signature). |
| 107 | |
| 108 | ### Stack-trace anonymization is dedup-only |
| 109 | |
| 110 | `status-file-dedup.c:26-36`: |
| 111 | |
| 112 | ```c |
| 113 | static void stack_trace_anonymize(char *s) { |
| 114 | char *p = s; |
| 115 | while (*p && (p = strstr(p, "0x"))) { |
| 116 | p[1] = '0'; |
| 117 | p += 2; |
| 118 | while(isxdigit((uint8_t)*p)) *p++ = '0'; |
| 119 | } |
| 120 | } |
| 121 | ``` |
| 122 | |
| 123 | Hex addresses in the stack trace are zeroed ONLY when |
| 124 | computing the dedup hash. The journal-emitted |
| 125 | `AE_FATAL_STACK_TRACE` retains real addresses (useful for |
| 126 | debugging; sensitive when sharing externally -- see |
| 127 | `redact-events.sh`). |
| 128 | |
| 129 | ## Dataset volume |
| 130 | |
| 131 | 40k-200k events / day on stable releases. Total fleet ~1.5M |
| 132 | agents (not all restart daily). The dataset is large and |
| 133 | spread across days. |
| 134 | |
| 135 | ### Implication: index-friendly queries are mandatory |
| 136 | |
| 137 | A 7-day FTS-only query over the agent-events namespace will |
| 138 | scan ~1M records on a high-volume week. With structured |
| 139 | `selections` filters (e.g. `AE_AGENT_HEALTH`, `AE_AGENT_VERSION`), |
| 140 | the same query slices to thousands or tens of thousands of |
| 141 | records before any FTS -- orders of magnitude cheaper. |
| 142 | |
| 143 | See `query-discipline.md` for the rule and worked examples. |
| 144 | |
| 145 | ## Default time windows for the skill |
| 146 | |
| 147 | | Use case | Default window | Why | |
| 148 | |---|---|---| |
| 149 | | Routine crash triage | 24h | Aligns with 23h dedup; one record per agent per signature. | |
| 150 | | Regression spotting | 24h | Same as above; group-by version. | |
| 151 | | Rare-crash hunting | 7 days | Catches 1-per-few-days classes. | |
| 152 | | "When did this start / get fixed?" | 14-30 days | Wide window with strong structured filters. Use sparingly. | |
| 153 | |
| 154 | ## Default version filter |
| 155 | |
| 156 | Latest stable + latest 2-3 nightlies. The dataset is noisy |
| 157 | because many unupdated agents report crashes that have been |
| 158 | fixed. Filtering to recent versions focuses triage on bugs |
| 159 | that still matter. |
| 160 | |
| 161 | `get-events.sh --versions auto` (the default) computes: |
| 162 | 1. Quick discovery query: 24h window, no version filter, |
| 163 | `AE_AGENT_VERSION` as a facet. |
| 164 | 2. From the facet result, pick the latest stable |
| 165 | (`v\d+\.\d+\.\d+` matching `^v2\.([89]|\d\d)\.\d+$` or the |
| 166 | newest by version sort) plus the top 3 nightlies |
| 167 | (`v\d+\.\d+\.\d+-\d+-nightly`). |
| 168 | 3. Re-run the main query with `selections.AE_AGENT_VERSION` |
| 169 | set to that list. |
| 170 | |
| 171 | `--all-versions` skips the auto filter for "when did X |
| 172 | start?" investigations. |
| 173 | `--versions <regex>` overrides with a custom pattern. |
| 174 | |
| 175 | ## What this means in practice |
| 176 | |
| 177 | - A crash you just got reported in chat will appear on |
| 178 | agent-events anywhere from minutes to days later. |
| 179 | - "Why don't I see this crash?" probably means "the agent |
| 180 | hasn't restarted yet, OR the dedup suppressed a duplicate |
| 181 | within 23h, OR the version filter excluded it". |
| 182 | - High-cardinality crashes from old versions are mostly |
| 183 | noise; default-version-filtering keeps triage focused. |
| 184 | - For "is this fixed in v2.X?", run `--versions auto` and |
| 185 | compare crash counts across the auto-selected versions. |