| 1 | # Netdata Log Viewer Plugin |
| 2 | |
| 3 | A Netdata external plugin for querying and visualizing systemd journal entries with histogram analysis and faceted search. |
| 4 | |
| 5 | ## Overview |
| 6 | |
| 7 | This plugin provides a `systemd-journal` function that Netdata can call to query journal entries, compute histograms, and return faceted data for visualization in the Netdata dashboard. |
| 8 | |
| 9 | ## Architecture |
| 10 | |
| 11 | ``` |
| 12 | ┌─────────────┐ |
| 13 | │ Netdata │ |
| 14 | │ Agent │ |
| 15 | └──────┬──────┘ |
| 16 | │ stdin/stdout |
| 17 | │ (plugin protocol) |
| 18 | ↓ |
| 19 | ┌──────────────────────────┐ |
| 20 | │ otel-signal-viewer-plugin │ |
| 21 | │ │ |
| 22 | │ ┌───────────────┐ │ |
| 23 | │ │ Journal │ │ |
| 24 | │ │ Handler │ │ |
| 25 | │ └───────────────┘ │ |
| 26 | │ ↓ │ |
| 27 | │ ┌───────────────┐ │ |
| 28 | │ │ Shared State │ │ |
| 29 | │ │ (AppState) │ │ |
| 30 | │ └───────────────┘ │ |
| 31 | │ ↓ │ |
| 32 | │ ┌───────────────┐ │ |
| 33 | │ │ histogram- │ │ |
| 34 | │ │ service │ │ |
| 35 | │ └───────────────┘ │ |
| 36 | │ ↓ │ |
| 37 | │ ┌───────────────┐ │ |
| 38 | │ │ journal │ │ |
| 39 | │ │ (indexing) │ │ |
| 40 | │ └───────────────┘ │ |
| 41 | └──────────────────────────┘ |
| 42 | ↓ |
| 43 | ┌─────────┐ |
| 44 | │ Jaeger │ (tracing) |
| 45 | └─────────┘ |
| 46 | ``` |
| 47 | |
| 48 | ## Key Features |
| 49 | |
| 50 | - **Fast histogram computation** using pre-built indexes |
| 51 | - **Faceted search** across journal fields (PRIORITY, HOSTNAME, etc.) |
| 52 | - **Caching** with memory + disk tiers for performance |
| 53 | - **Distributed tracing** via OpenTelemetry/Jaeger |
| 54 | - **Metrics tracking** for function call success/failure rates |
| 55 | |
| 56 | ## Quick Start |
| 57 | |
| 58 | ### Prerequisites |
| 59 | |
| 60 | 1. **Jaeger** (optional, for development visibility): |
| 61 | ```bash |
| 62 | # NOTE: Port 4318 avoids conflict with Netdata's otel-plugin on 4317 |
| 63 | docker run -d --name jaeger \ |
| 64 | -p 16686:16686 \ |
| 65 | -p 4318:4317 \ |
| 66 | jaegertracing/all-in-one:latest |
| 67 | ``` |
| 68 | |
| 69 | ### Building |
| 70 | |
| 71 | ```bash |
| 72 | cargo build --bin otel-signal-viewer-plugin --release |
| 73 | ``` |
| 74 | |
| 75 | ### Running |
| 76 | |
| 77 | The plugin is designed to be spawned by Netdata: |
| 78 | |
| 79 | ```bash |
| 80 | # Netdata spawns the plugin automatically when configured |
| 81 | # Configure in /etc/netdata/netdata.conf: |
| 82 | |
| 83 | [plugins] |
| 84 | otel-signal-viewer-plugin = yes |
| 85 | ``` |
| 86 | |
| 87 | For development/testing: |
| 88 | |
| 89 | ```bash |
| 90 | # Set log level |
| 91 | export RUST_LOG="debug" |
| 92 | |
| 93 | # Run Netdata in foreground |
| 94 | sudo netdata -D |
| 95 | |
| 96 | # View traces |
| 97 | open http://localhost:16686 |
| 98 | ``` |
| 99 | |
| 100 | ## Development |
| 101 | |
| 102 | See [QUICKSTART.md](./QUICKSTART.md) for the fast development loop. |
| 103 | |
| 104 | See [DEVELOPMENT.md](./DEVELOPMENT.md) for comprehensive documentation. |
| 105 | |
| 106 | ### Development Workflow |
| 107 | |
| 108 | 1. **Make changes** to the plugin code |
| 109 | 2. **Rebuild** (fast, seconds): `cargo build --bin otel-signal-viewer-plugin` |
| 110 | 3. **Restart** Netdata: `sudo systemctl restart netdata` |
| 111 | 4. **View traces** in Jaeger: http://localhost:16686 |
| 112 | 5. **Check logs**: `sudo journalctl -u netdata -f` |
| 113 | |
| 114 | ### Key Benefits of Current Architecture |
| 115 | |
| 116 | ✅ **Simple** - Single binary, single mode (production-only) |
| 117 | ✅ **Fast iteration** - Rebuild only the plugin, not all of Netdata |
| 118 | ✅ **Observable** - Rich tracing and logging via Jaeger + stderr |
| 119 | ✅ **Production parity** - Develop with exact production setup |
| 120 | ✅ **No mock infrastructure** - No TCP bridges or test harnesses needed |
| 121 | |
| 122 | ## Project Structure |
| 123 | |
| 124 | ``` |
| 125 | netdata-log-viewer/ |
| 126 | ├── histogram-service/ # Core business logic (library) |
| 127 | ├── otel-signal-viewer-plugin/ # Netdata plugin (binary) |
| 128 | ├── types/ # Shared request/response types |
| 129 | ├── watcher-plugin/ # DEPRECATED - no longer needed |
| 130 | ├── lv/ # DEPRECATED - no longer needed |
| 131 | ├── DEVELOPMENT.md # Detailed development guide |
| 132 | ├── QUICKSTART.md # Fast reference guide |
| 133 | └── README.md # This file |
| 134 | ``` |
| 135 | |
| 136 | ## Configuration |
| 137 | |
| 138 | The plugin is configured at compile time with sensible defaults: |
| 139 | |
| 140 | - **Journal path**: `/var/log/journal` |
| 141 | - **Cache directory**: `/mnt/ramfs/foyer-storage` |
| 142 | - **Memory cache**: 10,000 entries |
| 143 | - **Disk cache**: 64 MiB |
| 144 | |
| 145 | To customize, edit `create_shared_state()` in `otel-signal-viewer-plugin/src/main.rs`. |
| 146 | |
| 147 | ## Observability |
| 148 | |
| 149 | ### Tracing (Jaeger) |
| 150 | |
| 151 | View request traces at http://localhost:16686: |
| 152 | - Function call timelines |
| 153 | - Histogram computation duration |
| 154 | - Lock acquisition times |
| 155 | - Error traces |
| 156 | |
| 157 | ### Logging (Stderr) |
| 158 | |
| 159 | Control log verbosity with `RUST_LOG`: |
| 160 | ```bash |
| 161 | # Debug everything |
| 162 | export RUST_LOG="debug" |
| 163 | |
| 164 | # Selective logging |
| 165 | export RUST_LOG="otel_signal_viewer=trace,histogram_service=debug,journal=info" |
| 166 | ``` |
| 167 | |
| 168 | ### Metrics (Netdata) |
| 169 | |
| 170 | The plugin reports its own metrics: |
| 171 | - `otel_signal_viewer.journal_calls` - Successful/failed/cancelled function calls |
| 172 | |
| 173 | ## Function Interface |
| 174 | |
| 175 | The plugin exposes the `systemd-journal` function: |
| 176 | |
| 177 | **Request**: |
| 178 | ```json |
| 179 | { |
| 180 | "after": 1699000000, |
| 181 | "before": 1699100000, |
| 182 | "selections": { |
| 183 | "PRIORITY": ["3", "4"], |
| 184 | "_HOSTNAME": ["server1"] |
| 185 | } |
| 186 | } |
| 187 | ``` |
| 188 | |
| 189 | **Response**: |
| 190 | ```json |
| 191 | { |
| 192 | "status": 200, |
| 193 | "facets": [...], |
| 194 | "histogram": [...], |
| 195 | "available_histograms": [...], |
| 196 | "columns": {...}, |
| 197 | "data": [...] |
| 198 | } |
| 199 | ``` |
| 200 | |
| 201 | ## Dependencies |
| 202 | |
| 203 | ### Core |
| 204 | - `rt` - Netdata plugin runtime |
| 205 | - `histogram-service` - Histogram computation |
| 206 | - `journal` - Journal file indexing |
| 207 | |
| 208 | ### Tracing |
| 209 | - `tracing` - Structured logging |
| 210 | - `opentelemetry` - Distributed tracing |
| 211 | - `opentelemetry-otlp` - OTLP exporter for Jaeger |
| 212 | |
| 213 | ### Async Runtime |
| 214 | - `tokio` - Async runtime |
| 215 | |
| 216 | ## Performance |
| 217 | |
| 218 | - **Index caching**: Avoids re-reading journal files |
| 219 | - **Parallel processing**: Uses Rayon for CPU-bound work |
| 220 | - **Lock-free where possible**: RwLock allows concurrent reads |
| 221 | - **Efficient filtering**: Pre-built indexes for fast queries |
| 222 | |
| 223 | ## Troubleshooting |
| 224 | |
| 225 | ### Plugin not starting |
| 226 | |
| 227 | ```bash |
| 228 | # Check logs |
| 229 | sudo tail -f /var/log/netdata/error.log |
| 230 | |
| 231 | # Run manually |
| 232 | sudo -u netdata /path/to/otel-signal-viewer-plugin |
| 233 | ``` |
| 234 | |
| 235 | ### No traces in Jaeger |
| 236 | |
| 237 | ```bash |
| 238 | # Verify Jaeger is running |
| 239 | docker ps | grep jaeger |
| 240 | |
| 241 | # Check connectivity |
| 242 | nc -zv localhost 4317 |
| 243 | ``` |
| 244 | |
| 245 | ### Slow queries |
| 246 | |
| 247 | Check Jaeger traces for: |
| 248 | - Lock contention on shared state |
| 249 | - Cache misses in IndexCache |
| 250 | - Large time ranges |
| 251 | |
| 252 | ## License |
| 253 | |
| 254 | [Your license here] |
| 255 | |
| 256 | ## Contributing |
| 257 | |
| 258 | [Contributing guidelines here] |