| 1 | # Query discipline |
| 2 | |
| 3 | How to compose efficient agent-events queries. Hard rule: |
| 4 | **structured filters first, FTS only as residual.** |
| 5 | |
| 6 | For the JSON shape of the Function payload (including the |
| 7 | `selections` multi-value filter mechanism), see: |
| 8 | |
| 9 | - `<repo>/docs/netdata-ai/skills/query-netdata-cloud/query-logs.md` |
| 10 | ("Multi-value field selections" section) |
| 11 | |
| 12 | This doc covers agent-events-specific guidance: which AE_* |
| 13 | fields to put in `selections`, how to compose them, and the |
| 14 | anti-patterns to avoid. |
| 15 | |
| 16 | ## The hard rule |
| 17 | |
| 18 | ``` |
| 19 | selections (structured, indexable) --> query (FTS, residual narrower) |
| 20 | ``` |
| 21 | |
| 22 | NEVER FTS-first on agent-events. The namespace can hold 40k-200k |
| 23 | records / day; a bare `query` over a wide window full-scans the |
| 24 | dataset. |
| 25 | |
| 26 | ## What to put in `selections` |
| 27 | |
| 28 | Always include at least one of these fields (always present on |
| 29 | every record per `<repo>/src/daemon/status-file.c:967-976`): |
| 30 | |
| 31 | - `AE_AGENT_HEALTH` -- crash class. Filter to `crash-*` values |
| 32 | to drop healthy records. |
| 33 | - `AE_EXIT_CAUSE` -- exit reason (26 distinct values; pick the |
| 34 | ones you want). |
| 35 | - `AE_AGENT_VERSION` -- producing agent version (regression |
| 36 | slicing; pair with the auto-version filter). |
| 37 | - `AE_FATAL_SIGNAL_CODE` -- empty / non-empty discriminates |
| 38 | signal crashes from deliberate fatals / graceful exits. |
| 39 | |
| 40 | For more specific triage: |
| 41 | |
| 42 | - `AE_FATAL_FUNCTION` / `AE_FATAL_FILENAME` -- localize to a |
| 43 | function or source file. |
| 44 | - `AE_HOST_ARCHITECTURE` -- arch-specific bugs. |
| 45 | - `AE_OS_FAMILY` / `AE_OS_TYPE` -- distro / OS-specific. |
| 46 | - `AE_AGENT_PROFILE_0` -- standalone / parent / child / iot. |
| 47 | - `AE_AGENT_KUBERNETES` -- k8s-specific. |
| 48 | - `AE_AGENT_INSTALL_TYPE` -- packaging-specific. |
| 49 | - `AE_AGENT_ACLK` -- cloud-claimed vs not. |
| 50 | |
| 51 | See `AE_FIELDS.md` for the full field map and which values |
| 52 | each enum supports. |
| 53 | |
| 54 | ## Worked examples |
| 55 | |
| 56 | ### Example 1: index-friendly crash slice |
| 57 | |
| 58 | Find recent signal crashes on stable v2.10.x and the latest |
| 59 | 2 nightlies: |
| 60 | |
| 61 | ```json |
| 62 | { |
| 63 | "after": -86400, |
| 64 | "before": 0, |
| 65 | "last": 500, |
| 66 | "__logs_sources": "agent-events", |
| 67 | "selections": { |
| 68 | "AE_AGENT_HEALTH": ["crash-first", "crash-loop", "crash-repeated", "crash-entered"], |
| 69 | "AE_AGENT_VERSION": ["v2.10.0", "v2.10.0-135-nightly", "v2.10.0-130-nightly"] |
| 70 | }, |
| 71 | "facets": ["AE_FATAL_SIGNAL_CODE", "AE_FATAL_FUNCTION", "AE_HOST_ARCHITECTURE"] |
| 72 | } |
| 73 | ``` |
| 74 | |
| 75 | The `selections` cuts to ~hundreds-to-low-thousands of records |
| 76 | via the facet index; `facets` on top groups by the dimensions |
| 77 | of interest. No FTS needed. |
| 78 | |
| 79 | ### Example 2: index-friendly + FTS narrower |
| 80 | |
| 81 | Find recent crashes whose stack trace mentions a specific |
| 82 | function: |
| 83 | |
| 84 | ```json |
| 85 | { |
| 86 | "after": -86400, |
| 87 | "before": 0, |
| 88 | "last": 200, |
| 89 | "__logs_sources": "agent-events", |
| 90 | "selections": { |
| 91 | "AE_AGENT_HEALTH": ["crash-first", "crash-loop", "crash-repeated", "crash-entered"] |
| 92 | }, |
| 93 | "query": "rrdcontext_release" |
| 94 | } |
| 95 | ``` |
| 96 | |
| 97 | `selections` cuts to crashes only; `query` does FTS over the |
| 98 | already-sliced subset for the substring match. This is the |
| 99 | right composition: structured first, FTS narrows. |
| 100 | |
| 101 | If you can express the function name as a `selections` value |
| 102 | on `AE_FATAL_FUNCTION`, prefer that: |
| 103 | |
| 104 | ```json |
| 105 | { |
| 106 | "after": -86400, |
| 107 | "before": 0, |
| 108 | "last": 200, |
| 109 | "__logs_sources": "agent-events", |
| 110 | "selections": { |
| 111 | "AE_FATAL_FUNCTION": ["rrdcontext_release"] |
| 112 | } |
| 113 | } |
| 114 | ``` |
| 115 | |
| 116 | This is even faster (pure indexed, no FTS). |
| 117 | |
| 118 | ### Example 3: regression spotter |
| 119 | |
| 120 | Compare crash counts across versions: |
| 121 | |
| 122 | ```json |
| 123 | { |
| 124 | "after": -86400, |
| 125 | "before": 0, |
| 126 | "last": 1, |
| 127 | "__logs_sources": "agent-events", |
| 128 | "selections": { |
| 129 | "AE_AGENT_HEALTH": ["crash-first", "crash-loop", "crash-repeated", "crash-entered"] |
| 130 | }, |
| 131 | "facets": ["AE_AGENT_VERSION"], |
| 132 | "histogram": "AE_AGENT_VERSION" |
| 133 | } |
| 134 | ``` |
| 135 | |
| 136 | `last: 1` because we only need the facet counts, not the rows. |
| 137 | The histogram + facets give the per-version distribution. |
| 138 | |
| 139 | ### Example 4: rare-crash investigation (wider window) |
| 140 | |
| 141 | Looking for a known rare crash signature (1-per-few-days): |
| 142 | |
| 143 | ```json |
| 144 | { |
| 145 | "after": -604800, |
| 146 | "before": 0, |
| 147 | "last": 500, |
| 148 | "__logs_sources": "agent-events", |
| 149 | "selections": { |
| 150 | "AE_FATAL_SIGNAL_CODE": ["SIGSEGV/SEGV_MAPERR"], |
| 151 | "AE_FATAL_FUNCTION": ["specific_function"] |
| 152 | } |
| 153 | } |
| 154 | ``` |
| 155 | |
| 156 | 7-day window is acceptable here because the `selections` |
| 157 | cuts to specific signal+function -- a sharp index-resolved |
| 158 | slice that returns small results regardless of window width. |
| 159 | |
| 160 | ## Anti-patterns |
| 161 | |
| 162 | ### Bare FTS over wide window (BAD) |
| 163 | |
| 164 | ```json |
| 165 | { |
| 166 | "after": -604800, |
| 167 | "before": 0, |
| 168 | "__logs_sources": "agent-events", |
| 169 | "query": "SIGSEGV" |
| 170 | } |
| 171 | ``` |
| 172 | |
| 173 | A 7-day FTS over the full namespace. Slow. ALWAYS pair with |
| 174 | at least one structured `selections` field. |
| 175 | |
| 176 | ### FTS for things that should be `selections` (BAD) |
| 177 | |
| 178 | ```json |
| 179 | { |
| 180 | "query": "SIGSEGV" |
| 181 | } |
| 182 | ``` |
| 183 | |
| 184 | `SIGSEGV/...` is a value in the `AE_FATAL_SIGNAL_CODE` field. |
| 185 | Use: |
| 186 | |
| 187 | ```json |
| 188 | { |
| 189 | "selections": { |
| 190 | "AE_FATAL_SIGNAL_CODE": ["SIGSEGV/SEGV_MAPERR", "SIGSEGV/SEGV_ACCERR", "SIGSEGV/SEGV_BNDERR", "SIGSEGV/SEGV_PKUERR"] |
| 191 | } |
| 192 | } |
| 193 | ``` |
| 194 | |
| 195 | ### Naive equality on `=` only (BAD when you want OR) |
| 196 | |
| 197 | The `selections` mechanism gives you OR-of-values for free. |
| 198 | Don't run multiple queries to OR results client-side; put the |
| 199 | values in the array. |
| 200 | |
| 201 | ### Wide window without version filter (BAD when noisy) |
| 202 | |
| 203 | If 40-200k events / day are reported on stable releases, a |
| 204 | 30-day window without `AE_AGENT_VERSION` filter returns |
| 205 | millions of records, most from old versions whose bugs are |
| 206 | already fixed. Always pair wide windows with a version slice. |
| 207 | |
| 208 | ## Discovery first, then query |
| 209 | |
| 210 | Use `info=true` to discover what fields and values the agent |
| 211 | currently exposes: |
| 212 | |
| 213 | ```json |
| 214 | { "info": true, "__logs_sources": "agent-events" } |
| 215 | ``` |
| 216 | |
| 217 | Or use `last: 1` + a facet to enumerate values: |
| 218 | |
| 219 | ```json |
| 220 | { |
| 221 | "after": -86400, "before": 0, "last": 1, |
| 222 | "__logs_sources": "agent-events", |
| 223 | "facets": ["AE_AGENT_VERSION"] |
| 224 | } |
| 225 | ``` |
| 226 | |
| 227 | Then build the real query using the discovered values in |
| 228 | `selections`. |
| 229 | |
| 230 | ## Combining facets + selections |
| 231 | |
| 232 | `facets` (the array of field names you want grouped in the |
| 233 | response) and `selections` (the structured filter) are |
| 234 | independent. Common pattern: filter narrowly with |
| 235 | `selections`, then `facets` over the narrow result to see |
| 236 | per-field breakdowns. |
| 237 | |
| 238 | ```json |
| 239 | { |
| 240 | "selections": { |
| 241 | "AE_AGENT_HEALTH": ["crash-loop"], |
| 242 | "AE_AGENT_VERSION": ["v2.10.0"] |
| 243 | }, |
| 244 | "facets": ["AE_FATAL_FUNCTION", "AE_HOST_ARCHITECTURE", "AE_OS_FAMILY"] |
| 245 | } |
| 246 | ``` |
| 247 | |
| 248 | This filters to "crash-loop on v2.10.0", then shows the |
| 249 | distribution by function, architecture, and OS in the result |
| 250 | facets. |
| 251 | |
| 252 | ## Quick checklist |
| 253 | |
| 254 | Before sending any query, verify: |
| 255 | |
| 256 | 1. `__logs_sources` is set to `"agent-events"` (the journal namespace -- hardcoded constant, NOT the value of `${AGENT_EVENTS_HOSTNAME}`). |
| 257 | 2. `selections` contains at least ONE always-present field |
| 258 | (`AE_AGENT_HEALTH`, `AE_EXIT_CAUSE`, `AE_AGENT_VERSION`). |
| 259 | 3. Time window matches the use case (24h default, 7d for rare |
| 260 | crashes). |
| 261 | 4. If you used `query` for FTS, you have a structured |
| 262 | `selections` slice in front of it. |
| 263 | 5. `last` is set sensibly (200-500 for triage; 1 if you only |
| 264 | want facet counts). |