| 1 | # Transports |
| 2 | |
| 3 | Three ways to query agent-events. The first two are scripted |
| 4 | in this skill; the third is operator-only. |
| 5 | |
| 6 | ## Priority order |
| 7 | |
| 8 | 1. **Cloud API** -- proxied through Netdata Cloud at the |
| 9 | space hosting agent-events. **Default for the team.** |
| 10 | 2. **Direct agent API** -- against the agent-events node's |
| 11 | own HTTP. Used when bypassing the Cloud is acceptable |
| 12 | (latency, debugging the proxy). |
| 13 | 3. **ssh to the host** -- operator-only path. Costa-only. |
| 14 | Mentioned here for completeness; this skill does NOT |
| 15 | ship a scripted ssh transport. |
| 16 | |
| 17 | ## What each transport calls |
| 18 | |
| 19 | All three speak the same `systemd-journal` Function. The |
| 20 | payload shape (`after`, `before`, `last`, `query`, `facets`, |
| 21 | `histogram`, `__logs_sources`, `selections`, ...) is identical |
| 22 | across transports. The Function payload is documented once at: |
| 23 | |
| 24 | - `<repo>/docs/netdata-ai/skills/query-netdata-cloud/query-logs.md` |
| 25 | |
| 26 | That doc is the canonical reference for: |
| 27 | - payload keys + types, |
| 28 | - the **`selections` multi-value field-filter** (AND across |
| 29 | fields, OR across values) -- this skill leans on it heavily, |
| 30 | - the response envelope (top-level `data` rows, `columns` map, |
| 31 | `facets`, `histogram`, etc.). |
| 32 | |
| 33 | This skill EXTENDS that doc with agent-events specifics: which |
| 34 | AE_* fields are best as facets, what default `selections` |
| 35 | predicate to use, what `__logs_sources` value to set. |
| 36 | |
| 37 | ## Cloud API (transport 1) |
| 38 | |
| 39 | ### Endpoint |
| 40 | |
| 41 | `POST https://${NETDATA_CLOUD_HOSTNAME}/api/v2/nodes/${AGENT_EVENTS_NODE_ID}/function?function=systemd-journal` |
| 42 | |
| 43 | Auth: `Authorization: Bearer ${NETDATA_CLOUD_TOKEN}`. |
| 44 | |
| 45 | ### Helper (from query-netdata-cloud) |
| 46 | |
| 47 | ```bash |
| 48 | source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-cloud/scripts/_lib.sh" |
| 49 | cloud_load_env |
| 50 | cloud_query \ |
| 51 | "/api/v2/nodes/${AGENT_EVENTS_NODE_ID}/function?function=systemd-journal" \ |
| 52 | "$PAYLOAD" |
| 53 | ``` |
| 54 | |
| 55 | Or the `query-netdata-agents` skill's wrapper, which works |
| 56 | identically and routes through Cloud when configured: |
| 57 | |
| 58 | ```bash |
| 59 | source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh" |
| 60 | agents_load_env |
| 61 | agents_call_function "$AGENT_EVENTS_NODE_ID" systemd-journal "$PAYLOAD" |
| 62 | ``` |
| 63 | |
| 64 | ### Pros / cons |
| 65 | |
| 66 | - **Pro:** team-accessible (no per-host SSH); central auth via |
| 67 | `NETDATA_CLOUD_TOKEN`; works from any network. |
| 68 | - **Con:** slight latency vs direct agent; rate-limited at the |
| 69 | Cloud edge; subject to Cloud-side query timeout. |
| 70 | |
| 71 | ### When to use |
| 72 | |
| 73 | - The default for the team. |
| 74 | - Anything you want to share later (Cloud requests are |
| 75 | loggable / repeatable). |
| 76 | |
| 77 | ## Direct agent API (transport 2) |
| 78 | |
| 79 | ### Endpoint |
| 80 | |
| 81 | `POST http://${AGENT_EVENTS_HOSTNAME}:19999/api/v3/function?function=systemd-journal` |
| 82 | |
| 83 | Auth: bearer token minted from the Cloud token. The |
| 84 | `agents_query_agent` helper handles minting + caching |
| 85 | transparently. |
| 86 | |
| 87 | ### Helper |
| 88 | |
| 89 | ```bash |
| 90 | source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh" |
| 91 | agents_load_env |
| 92 | |
| 93 | agents_query_agent \ |
| 94 | --node "$AGENT_EVENTS_NODE_ID" \ |
| 95 | --host "$AGENT_EVENTS_HOSTNAME:19999" \ |
| 96 | --machine-guid "$AGENT_EVENTS_MACHINE_GUID" \ |
| 97 | POST '/api/v3/function?function=systemd-journal' "$PAYLOAD" |
| 98 | ``` |
| 99 | |
| 100 | Output is the response body only. The bearer never reaches |
| 101 | the assistant's captured stdout. |
| 102 | |
| 103 | ### Pros / cons |
| 104 | |
| 105 | - **Pro:** no Cloud-edge round-trip; lower latency; agent's |
| 106 | own timeout (you set it in the body). |
| 107 | - **Con:** only reachable from inside the network; requires |
| 108 | the agent to be reachable on port 19999. |
| 109 | |
| 110 | ### When to use |
| 111 | |
| 112 | - Tight loops during local debugging (sub-second iteration). |
| 113 | - When the Cloud edge is the bottleneck. |
| 114 | |
| 115 | ## ssh to the host (transport 3 -- operator-only) |
| 116 | |
| 117 | This skill does NOT ship a scripted ssh transport. The |
| 118 | operator (Costa) sometimes runs `journalctl` directly on the |
| 119 | host: |
| 120 | |
| 121 | ```bash |
| 122 | ssh "$AGENT_EVENTS_HOSTNAME" \ |
| 123 | sudo /usr/bin/journalctl --namespace=agent-events \ |
| 124 | --since '24 hours ago' -o json |
| 125 | ``` |
| 126 | |
| 127 | Notes: |
| 128 | - The ssh host is `${AGENT_EVENTS_HOSTNAME}` (env-keyed; can be |
| 129 | an IP or DNS name). The journal namespace is `agent-events` |
| 130 | (hardcoded constant, set on the ingestion server's log2journal |
| 131 | invocation, NOT a function of the hostname). |
| 132 | - Raw `journalctl` does NOT support multi-value field filters |
| 133 | (they are a Netdata-engine feature, not journald). If you |
| 134 | need AND-of-OR filtering, use transport 1 or 2. |
| 135 | - This path requires sudo + a member of the `systemd-journal` |
| 136 | group on the ingestion host. Most team members do not have |
| 137 | this. Use transports 1 or 2 instead. |
| 138 | |
| 139 | ## Default `__logs_sources` value |
| 140 | |
| 141 | Always set `__logs_sources` to the agent-events namespace name |
| 142 | (`"agent-events"` -- a hardcoded constant set on the ingestion |
| 143 | server's log2journal invocation; NOT derived from |
| 144 | `${AGENT_EVENTS_HOSTNAME}`): |
| 145 | |
| 146 | ```json |
| 147 | { "__logs_sources": "agent-events" } |
| 148 | ``` |
| 149 | |
| 150 | Without this, the Function defaults to all-local-logs on the |
| 151 | ingestion-server agent -- which is huge and unrelated. |
| 152 | |
| 153 | ## What goes in `selections` for agent-events |
| 154 | |
| 155 | For the agent-events namespace, the most-useful index-friendly |
| 156 | predicates (always present on every record): |
| 157 | |
| 158 | - `AE_VERSION` -- schema version anchor (always 28+). |
| 159 | - `AE_AGENT_HEALTH` -- crash class (filter to `crash-*` for crashes). |
| 160 | - `AE_EXIT_CAUSE` -- exit reason (filter to specific causes). |
| 161 | - `AE_AGENT_VERSION` -- producing agent version (regression slicing). |
| 162 | - `AE_FATAL_SIGNAL_CODE` -- non-empty for signal crashes. |
| 163 | - `AE_FATAL_FUNCTION` / `AE_FATAL_FILENAME` -- localize to a |
| 164 | function or file. |
| 165 | - `AE_HOST_ARCHITECTURE` / `AE_OS_FAMILY` / `AE_AGENT_INSTALL_TYPE` |
| 166 | -- arch / distro / packaging slicers. |
| 167 | - `AE_AGENT_PROFILE_0` -- standalone / parent / child / iot. |
| 168 | - `AE_AGENT_KUBERNETES` -- k8s-specific. |
| 169 | - `AE_AGENT_ACLK` -- cloud-claimed vs not. |
| 170 | |
| 171 | See `AE_FIELDS.md` for the full field map and enum meanings. |
| 172 | |
| 173 | ## See also |
| 174 | |
| 175 | - `<repo>/docs/netdata-ai/skills/query-netdata-cloud/query-logs.md` |
| 176 | -- canonical Function payload shape and the `selections` |
| 177 | multi-value filter section. |
| 178 | - `<repo>/docs/netdata-ai/skills/query-netdata-agents/query-logs.md` |
| 179 | -- direct-agent transport details. |
| 180 | - `query-discipline.md` (this skill) -- how to compose |
| 181 | index-friendly queries against agent-events. |
| 182 | - `update-cadence.md` (this skill) -- when events arrive and why. |