| 1 | # Recipe: find events related to current work |
| 2 | |
| 3 | Use case: "We're working on dbengine cleanup. Are there |
| 4 | crashes in agent-events that match this area? What does the |
| 5 | distribution look like?" |
| 6 | |
| 7 | This is the "is anyone hitting this?" template. Useful for |
| 8 | prioritizing work and validating fixes. |
| 9 | |
| 10 | ## Pattern |
| 11 | |
| 12 | You have a candidate area defined by some combination of: |
| 13 | |
| 14 | - a function name or symbol -> use `--function` or FTS |
| 15 | - a source file path -> use selections on `AE_FATAL_FILENAME` |
| 16 | - a code-path keyword that would appear in the stack trace |
| 17 | -> use `--query` (FTS narrower) |
| 18 | - a thread name (e.g. `STREAM:N`, `CTXLOAD`, `APP-COLLECT`) |
| 19 | -> use selections on `AE_FATAL_THREAD` |
| 20 | |
| 21 | Combine the most-specific predicates first, then widen. |
| 22 | |
| 23 | ## Worked example |
| 24 | |
| 25 | Working on dbengine page eviction: |
| 26 | |
| 27 | ```bash |
| 28 | # 1. Try by function -- the obvious symbols. |
| 29 | .agents/skills/query-agent-events/scripts/get-events.sh \ |
| 30 | --function 'rrdeng_page_descr_t,rrdeng_evict_pages,evict_main' \ |
| 31 | --since '14d ago' \ |
| 32 | --versions auto \ |
| 33 | --output /tmp/related-pass1.json |
| 34 | |
| 35 | # 2. If pass 1 is sparse, widen to filename. |
| 36 | payload=$(jq -nc '{ |
| 37 | "after": -1209600, "before": 0, "last": 200, |
| 38 | "__logs_sources": "agent-events", |
| 39 | "selections": { |
| 40 | "AE_FATAL_FILENAME": [ |
| 41 | "src/database/engine/cache.c", |
| 42 | "src/database/engine/pdc.c", |
| 43 | "src/database/engine/rrdengine.c" |
| 44 | ] |
| 45 | } |
| 46 | }') |
| 47 | agentevents_query_function cloud "$payload" > /tmp/related-pass2.json |
| 48 | |
| 49 | # 3. If still sparse, FTS over crash class. |
| 50 | .agents/skills/query-agent-events/scripts/get-events.sh \ |
| 51 | --health crash \ |
| 52 | --query 'page_descr OR cache_evict OR rrdeng' \ |
| 53 | --since '14d ago' \ |
| 54 | --versions auto \ |
| 55 | --output /tmp/related-pass3.json |
| 56 | ``` |
| 57 | |
| 58 | Pass 1 (function) is the cheapest and most precise; pass 2 |
| 59 | widens to files; pass 3 is the FTS fallback. |
| 60 | |
| 61 | ## Aggregation |
| 62 | |
| 63 | After collecting candidate events, look at the distribution: |
| 64 | |
| 65 | ```bash |
| 66 | .agents/skills/query-agent-events/scripts/analyze-events.sh \ |
| 67 | --input /tmp/related-pass2.json \ |
| 68 | --by signal |
| 69 | |
| 70 | .agents/skills/query-agent-events/scripts/analyze-events.sh \ |
| 71 | --input /tmp/related-pass2.json \ |
| 72 | --by version |
| 73 | |
| 74 | .agents/skills/query-agent-events/scripts/analyze-events.sh \ |
| 75 | --input /tmp/related-pass2.json \ |
| 76 | --by fatal_function |
| 77 | |
| 78 | .agents/skills/query-agent-events/scripts/analyze-events.sh \ |
| 79 | --input /tmp/related-pass2.json \ |
| 80 | --by db_mode |
| 81 | ``` |
| 82 | |
| 83 | Together this tells you: |
| 84 | - how often we crash in this area; |
| 85 | - whether it's all one signal or mixed; |
| 86 | - which versions are affected; |
| 87 | - which functions in the area are the dominant crash sites; |
| 88 | - whether dbengine memory mode correlates. |
| 89 | |
| 90 | ## Sample one event |
| 91 | |
| 92 | ```bash |
| 93 | jq '.data[0] as $row | .columns as $cols |
| 94 | | $cols | to_entries | sort_by(.value.index) |
| 95 | | map({(.key): $row[.value.index]}) | add' /tmp/related-pass2.json |
| 96 | ``` |
| 97 | |
| 98 | Examine a representative `AE_FATAL_STACK_TRACE`, |
| 99 | `AE_FATAL_MESSAGE`, `AE_FATAL_THREAD`. Cross-reference the |
| 100 | specific commit that introduced the path you suspect. |
| 101 | |
| 102 | ## Building a "before/after" comparison |
| 103 | |
| 104 | If you've landed a fix: |
| 105 | |
| 106 | ```bash |
| 107 | # Before-fix nightly counts. |
| 108 | .agents/skills/query-agent-events/scripts/get-events.sh \ |
| 109 | --function 'fixed_function' \ |
| 110 | --versions 'v2.10.0,v2.10.0-100-nightly' \ |
| 111 | --since '14d ago' \ |
| 112 | --last 1 --facets 'AE_AGENT_VERSION' \ |
| 113 | --output /tmp/before-fix.json |
| 114 | |
| 115 | # After-fix nightly counts. |
| 116 | .agents/skills/query-agent-events/scripts/get-events.sh \ |
| 117 | --function 'fixed_function' \ |
| 118 | --versions 'v2.10.0-130-nightly,v2.10.0-160-nightly' \ |
| 119 | --since '14d ago' \ |
| 120 | --last 1 --facets 'AE_AGENT_VERSION' \ |
| 121 | --output /tmp/after-fix.json |
| 122 | ``` |
| 123 | |
| 124 | Compare counts. Significant drop -> fix is working. |
| 125 | |
| 126 | ## Pitfalls |
| 127 | |
| 128 | - **Empty result for new code**: a function that just landed |
| 129 | in a nightly may not have crashed in the wild yet. Wide the |
| 130 | time window to 30 days OR wait for more agents to update. |
| 131 | - **Old fixes that haven't propagated**: if your fix is in |
| 132 | v2.10.1 but most of the fleet runs v2.10.0, you'll still |
| 133 | see the old crashes for weeks. |
| 134 | - **Recipe scoping**: don't forget to set |
| 135 | `"__logs_sources": "agent-events"` when constructing |
| 136 | payloads manually -- without it, the query targets |
| 137 | all-local-logs (huge and unrelated). |