@cryptotaxi247 / netdata
Code Commits Issues Pulls Sessions CI/CD
master
master
md 109 lines 3.25 KB
Rendered Raw
1 # Recipe: find events by fatal function
2
3 Use case: "Is anyone hitting a crash in `function_name` /
4 `source/file.c`?"
5
6 ## Quick path
7
8 ```bash
9 .agents/skills/query-agent-events/scripts/get-events.sh \
10 --function 'rrdcontext_release,rrdcontext_dispatch_updates_to_main' \
11 --since '7d ago' \
12 --versions auto \
13 --last 200 \
14 --output /tmp/by-function.json
15 ```
16
17 `--function` is a comma-separated list -- multiple values are
18 OR'd via `selections.AE_FATAL_FUNCTION`. The query is fully
19 indexed (no FTS), so 7d windows are cheap.
20
21 Then:
22
23 ```bash
24 .agents/skills/query-agent-events/scripts/analyze-events.sh \
25 --input /tmp/by-function.json \
26 --by version
27 # ...and...
28 .agents/skills/query-agent-events/scripts/analyze-events.sh \
29 --input /tmp/by-function.json \
30 --by signal
31 ```
32
33 ## By filename instead of function
34
35 If you know the file but not the exact function:
36
37 ```bash
38 # AE_FATAL_FILENAME is a `selections` field too.
39 # get-events.sh doesn't have a --filename flag; use jq to
40 # filter, OR construct the payload manually:
41
42 payload=$(jq -nc '{
43 "after": -604800,
44 "before": 0,
45 "last": 500,
46 "__logs_sources": "agent-events",
47 "selections": {
48 "AE_FATAL_FILENAME": ["src/database/rrdcontext/rrdcontext-cleanup.c"]
49 }
50 }')
51 agentevents_query_function cloud "$payload" > /tmp/by-filename.json
52 ```
53
54 ## By symbol via FTS narrower
55
56 When the symbol may be in the stack trace but not directly
57 matched by `AE_FATAL_FUNCTION` (i.e. an inlined or downstream
58 callee):
59
60 ```bash
61 .agents/skills/query-agent-events/scripts/get-events.sh \
62 --health crash \
63 --query 'inlined_callee_name' \
64 --since '7d ago' \
65 --versions auto \
66 --output /tmp/by-symbol.json
67 ```
68
69 `--health crash` is the structured slice (index-friendly);
70 `--query` is the FTS narrower over the resulting subset. This
71 is the right composition.
72
73 ## Triage flow
74
75 1. **Get the dump** (as above).
76 2. **Group by version** -- is this on stable, nightlies, or
77 both? Was it new in a recent version?
78 3. **Group by signal** -- is it always SIGSEGV, or mixed?
79 4. **Open one event** -- look at `AE_FATAL_STACK_TRACE`,
80 `AE_FATAL_LINE`, `AE_FATAL_MESSAGE`, `AE_FATAL_THREAD`.
81 5. **Cross-reference source** -- read the function in this
82 repo at the cited line.
83 6. **Group by environment** -- arch / os_family / kubernetes /
84 profile -- is it environment-specific?
85 7. **Hypothesize, fix, ship.**
86
87 ## Common patterns
88
89 - Multiple distinct `fatal_function` values that all crash in
90 the same source file -> the file has a structural issue
91 (state corruption, race condition, invariant violation).
92 - One `fatal_function` value but mixed signals (SIGSEGV +
93 SIGBUS + SIGABRT) -> the function is a chokepoint hit by
94 many upstream paths.
95 - Crashes in one function on parent profile only
96 (`AE_AGENT_PROFILE_0=parent`) -> the function is in the
97 streaming-receiver path.
98
99 ## Pitfalls
100
101 - **`AE_FATAL_FUNCTION` is the function where `fatal()` was
102 called**, not necessarily the function where the crash
103 occurred. For signal crashes, it's the function name
104 recorded by the deadly-signal handler.
105 - **Stack trace empty** for instantaneous crashes -- use line
106 / filename / function instead.
107 - **Demangled symbols** -- the function name may have C++
108 decorations (e.g. `MyClass::method`). Try the demangled
109 form in `--function`.