@cryptotaxi247 / CoPilot / commits / a1972662

feat(graylog): support Graylog 7.x while keeping 6.x compatibility (#882) (#898)

Graylog 7.0 wraps entity-creation POST bodies in a CreateEntityRequest ({"entity": {...}, "share_request": null}) where 6.x used a flat object, and 7.0 strictly rejects unknown properties — so neither version accepts the other's shape. It also removed GET /api/system/urlwhitelist in favor of /api/system/urlallowlist. SOCFortress still has clients on Graylog 6, so CoPilot must support both. Add version-aware helpers in graylog/utils/universal.py: - get_graylog_major_version(): probes GET /api/system, cached 5 min, defaults to 6 on failure so 6.x behavior is byte-for-byte unchanged. - send_post_request_create_entity(): wraps the body for >= 7.x, sends flat for 6.x. Route the four (and only four) CreateEntityRequest-wrapped endpoints CoPilot uses through the helper — verified against the live 7.1.2 server and its OpenAPI spec: - POST /api/streams (9 provisioning call sites) - POST /api/events/definitions - POST /api/events/notifications - content-pack install (POST .../content_packs/{id}/{rev}/installations) Flat endpoints (index_sets, content-pack upload, all pipelines/*, events/search) and all PUTs are deliberately left on send_post_request — wrapping them would break 7.x under strict-property rejection. Switch collector.get_url_whitelist_entries to try urlallowlist first and fall back to urlwhitelist for 6.x. Document the whole split in CLAUDE.md. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

taylor_socfortress committed Jun 1, 2026 at 12:05 UTC a197266288502e233fe81b8a63e54a7e640a180a

14 files changed +137 -26

CLAUDE.md

@@ -253,3 +253,4 @@ The `customer-portal/` mirrors this structure but is a leaner standalone app, se

253 - **Cross-branch local-DB alembic divergence is invisible until startup.** If you run a migration from a feature branch against your local DB and then switch to a different branch that lacks the migration file, `apply_migrations()` at startup crashes with `Can't locate revision identified by <hash>` — the DB's `alembic_version` table records the missing revision, alembic has no file to walk. **Safe recovery**: `git checkout <other-branch-commit> -- backend/alembic/versions/<missing-revision>_*.py` (plus any predecessors in the chain). That brings the migration *file* into the working tree without changing the DB — the schema the file describes is already applied, alembic just needs the file to recognize the state. Don't `UPDATE alembic_version` manually unless you're prepared to fight it later; don't `alembic downgrade` unless you genuinely want to drop the columns.

254 - **Don't assume Wazuh alerts live only in `wazuh-alerts-*`.** Real SOCFortress deployments spread alerts across vendor- and customer-prefixed indices: `office365-<customer_code>`, `crowdstrike-<customer_code>`, `carbonblack-<customer_code>`, `huntress_<customer_code>`, ad-hoc `newest-*` test indices, and so on — easily 1500+ on lab clusters. A query targeting only `wazuh-alerts-*` will miss the bulk of real traffic. Equally, `wazuh-*` is *too broad* and includes Wazuh's own internal indices (`wazuh-monitoring-*`, `wazuh-statistics-*`, `wazuh-states-*`, `wazuh-vulnerabilities-*`) — those carry `rule.id` for system/control events with Wazuh's parent-template rule IDs (2 = firewall template, 3 = ids template, 4 = web-log template) that never fire on real analyst alerts but produce massive fake hit counts. **Use ES's native wildcard+exclusion pattern**, not client-side index enumeration: `*,-.*,-_*,-wazuh-monitoring-*,-wazuh-statistics-*,-wazuh-states-*,-wazuh-vulnerabilities-*,-security-auditlog-*` passes as one short string and ES resolves it server-side. `wazuh_firing_stats_cache.py:ALERT_INDEX_PATTERN` is the canonical implementation. Integer coercion on the bucket keys naturally drops vendor-native rule IDs that aren't Wazuh-compatible, so over-including is safe.

255 - **Don't pass a long index list directly to `client.search(index=[...])`.** The elasticsearch7 client serializes the list as a comma-joined URL path segment. At ~1500 indices that crosses the 4096-byte HTTP line limit and you get `too_long_http_line_exception` from the server. The fix is server-side pattern resolution — pass a short string like `*,-foo-*,-bar-*` to `client.search(index=pattern)` and let ES expand it. Combine with `ignore_unavailable=True, allow_no_indices=True` so per-index permission errors and empty matches don't fail the whole call.

256 +- **Graylog 7.0 wraps entity-creation POSTs in a `CreateEntityRequest` — CoPilot must support both 6.x and 7.x.** SOCFortress still has clients on Graylog 6. In 7.0 the create-entity endpoints changed their request body from a flat object (`{"title": ..., ...}`) to `{"entity": {...}, "share_request": null}`; **neither version accepts the other's shape**, and 7.0 *strictly rejects* unknown properties, so you cannot just always-send the wrapper. The symptom on a flat body against 7.x is a 400 `RequestError` with `"entity cannot be null"` and `reference_path: org.graylog.security.shares.CreateEntityRequest`. **Use `app/connectors/graylog/utils/universal.py:send_post_request_create_entity(endpoint, entity=...)` for any entity-creation POST** — it probes `GET /api/system` for the server's major version (cached 5 min, defaults to 6 on failure so 6.x stays byte-for-byte unchanged) and wraps only when major ≥ 7. **Exactly four endpoints CoPilot uses are wrapped** (verified against the live 7.1.2 server's OpenAPI spec): `POST /api/streams`, `POST /api/events/definitions`, `POST /api/events/notifications`, and content-pack *install* (`POST /api/system/content_packs/{id}/{rev}/installations`). **Everything else is FLAT and must keep using plain `send_post_request`** — `POST /api/system/indices/index_sets` (index-set create), content-pack *upload* (`POST /api/system/content_packs`), all `/api/system/pipelines/*` (rule, pipeline, connections/to_stream), and `/api/events/search`. **No PUT endpoint is wrapped** (`UpdateStreamRequest`, `IndexSetUpdateRequest`, `PipelineSource` are all flat), so don't touch PUTs. Wrapping a flat endpoint breaks it under 7.x's strict-property rejection just as surely as not-wrapping a wrapped one — the four-vs-rest split is load-bearing, not cosmetic. Separately, **7.0 dropped `GET /api/system/urlwhitelist` entirely** (404, not aliased) in favor of `GET /api/system/urlallowlist`; `collector.py:get_url_whitelist_entries` tries the 7.x path first and falls back to the legacy path for 6.x.

backend/app/connectors/graylog/services/collector.py

+8 -1

@@ -235,7 +235,14 @@ async def get_url_whitelist_entries() -> UrlWhitelistEntryResponse:

235 UrlWhitelistEntryResponse: The response object containing the URL whitelist entries.

236 """

237 logger.info("Getting URL whitelist entries from Graylog")

238 - response = await send_get_request(endpoint="/api/system/urlwhitelist")

238 + # Graylog 7.0 renamed urlwhitelist -> urlallowlist. Try the 7.x path first and

239 + # fall back to the 6.x path so both server versions are supported. GET is

240 + # idempotent, so the fallback is safe.

241 + try:

242 + response = await send_get_request(endpoint="/api/system/urlallowlist")

243 + except HTTPException:

244 + logger.info("urlallowlist endpoint unavailable, falling back to legacy urlwhitelist (Graylog < 7.0)")

245 + response = await send_get_request(endpoint="/api/system/urlwhitelist")

246 logger.info(f"URL whitelist entries response: {response}")

247 if response["success"]:

248 try:

backend/app/connectors/graylog/services/content_packs.py

+3 -2

@@ -8,6 +8,7 @@ from app.connectors.graylog.schema.content_packs import ContentPack

8 from app.connectors.graylog.schema.content_packs import ContentPackList

9 from app.connectors.graylog.utils.universal import send_get_request

10 from app.connectors.graylog.utils.universal import send_post_request

11 +from app.connectors.graylog.utils.universal import send_post_request_create_entity

14 async def get_content_packs() -> List[ContentPack]:

@@ -97,9 +98,9 @@ async def install_content_pack(content_pack_id: str, revision: int) -> bool:

98 """

99 logger.info(f"Installing content pack {content_pack_id} in Graylog")

100 try:

100 - content_pack_installed = await send_post_request(

101 + content_pack_installed = await send_post_request_create_entity(

102 endpoint=f"/api/system/content_packs/{content_pack_id}/{revision}/installations",

102 - data={

103 + entity={

104 "comment": "Installed by SOCFortress CoPilot",

105 },

106 )

backend/app/connectors/graylog/utils/universal.py

+93

@@ -1,6 +1,8 @@

1 +import time

2 from typing import Any

3 from typing import Dict

4 from typing import Optional

5 +from typing import Tuple

7 import requests

8 from fastapi import HTTPException

@@ -12,6 +14,14 @@ from app.db.db_session import get_db_session

15 HEADERS = {"X-Requested-By": "CoPilot"}

17 +# Cache of the detected Graylog major version, keyed by connector name.

18 +# Graylog 7.0 introduced breaking changes to entity-creation POSTs (the

19 +# CreateEntityRequest wrapper) and renamed urlwhitelist -> urlallowlist, so we

20 +# branch on the server's major version. The short TTL lets an in-place Graylog

21 +# upgrade be picked up without restarting CoPilot.

22 +_GRAYLOG_VERSION_CACHE: Dict[str, Tuple[int, float]] = {}

23 +_GRAYLOG_VERSION_TTL_SECONDS = 300

24 +

26 async def verify_graylog_credentials(attributes: Dict[str, Any]) -> Dict[str, Any]:

27 """

@@ -334,3 +344,86 @@ async def send_put_request(

344 "success": False,

345 "message": f"Failed to send PUT request to {endpoint} with error: {e}",

346 }

347 +

348 +

349 +async def get_graylog_major_version(connector_name: Optional[str] = None) -> int:

350 + """

351 + Detects the major version of the configured Graylog server.

352 +

353 + Graylog 7.0 changed several entity-creation requests in a backwards-incompatible

354 + way (see ``send_post_request_create_entity``). Callers use this to build the

355 + correct request shape for the running server. The result is cached per connector

356 + for a short TTL.

357 +

358 + Falls back to major version ``6`` when the version cannot be determined, so

359 + existing Graylog 6.x deployments keep working unchanged.

360 +

361 + Args:

362 + connector_name (Optional[str]): The connector to probe. Falls back to the

363 + current context when not provided.

364 +

365 + Returns:

366 + int: The detected Graylog major version (e.g. ``6`` or ``7``).

367 + """

368 + if connector_name is None:

369 + connector_name = get_current_graylog_connector()

370 +

371 + cached = _GRAYLOG_VERSION_CACHE.get(connector_name)

372 + now = time.monotonic()

373 + if cached is not None and now - cached[1] < _GRAYLOG_VERSION_TTL_SECONDS:

374 + return cached[0]

375 +

376 + major = 6 # safe default — preserves pre-7.x behavior on detection failure

377 + try:

378 + response = await send_get_request(endpoint="/api/system", connector_name=connector_name)

379 + # version looks like "7.0.1+abc123" or "6.1.4"

380 + version_str = str(response["data"]["version"])

381 + major = int(version_str.split("+")[0].split(".")[0].strip())

382 + except Exception as e:

383 + logger.warning(

384 + f"Could not determine Graylog version for connector '{connector_name}', " f"defaulting to major version {major}: {e}",

385 + )

386 +

387 + _GRAYLOG_VERSION_CACHE[connector_name] = (major, now)

388 + logger.info(f"Detected Graylog major version {major} for connector '{connector_name}'")

389 + return major

390 +

391 +

392 +async def send_post_request_create_entity(

393 + endpoint: str,

394 + entity: Dict[str, Any],

395 + share_request: Optional[Dict[str, Any]] = None,

396 + connector_name: Optional[str] = None,

397 +) -> Dict[str, Any]:

398 + """

399 + Sends an entity-creation POST request, adapting the body to the Graylog version.

400 +

401 + Graylog 7.0 wraps entity-creation payloads in a ``CreateEntityRequest``

402 + (``{"entity": {...}, "share_request": {...}}``); Graylog 6.x expects the entity

403 + fields at the top level. Neither version accepts the other's shape, so this

404 + helper builds the correct body based on the detected server version, letting a

405 + single call site support both Graylog 6 and Graylog 7.

406 +

407 + Confirmed affected endpoints: ``POST /api/streams`` and content-pack installation

408 + (``POST /api/system/content_packs/{id}/{revision}/installations``). Index-set

409 + creation is NOT wrapped in either version and must keep using

410 + ``send_post_request`` directly.

411 +

412 + Args:

413 + endpoint (str): The entity-creation endpoint.

414 + entity (Dict[str, Any]): The entity fields (the flat Graylog 6.x body).

415 + share_request (Optional[Dict[str, Any]]): Optional Graylog 7.x sharing

416 + settings; omitted when ``None`` (it is nullable server-side).

417 + connector_name (Optional[str]): The connector to use. Falls back to context.

418 +

419 + Returns:

420 + Dict[str, Any]: The response from the POST request.

421 + """

422 + major = await get_graylog_major_version(connector_name)

423 + if major >= 7:

424 + body: Dict[str, Any] = {"entity": entity}

425 + if share_request is not None:

426 + body["share_request"] = share_request

427 + else:

428 + body = entity

429 + return await send_post_request(endpoint=endpoint, data=body, connector_name=connector_name)

backend/app/customer_provisioning/services/graylog.py

+3 -2

@@ -8,6 +8,7 @@ from app.connectors.graylog.services.pipelines import get_pipelines

8 from app.connectors.graylog.utils.universal import send_delete_request

9 from app.connectors.graylog.utils.universal import send_get_request

10 from app.connectors.graylog.utils.universal import send_post_request

11 +from app.connectors.graylog.utils.universal import send_post_request_create_entity

12 from app.customer_provisioning.schema.graylog import GraylogIndexSetCreationResponse

13 from app.customer_provisioning.schema.graylog import StreamConnectionToPipelineRequest

14 from app.customer_provisioning.schema.graylog import StreamConnectionToPipelineResponse

@@ -160,9 +161,9 @@ async def send_event_stream_creation_request(

161 """

162 json_event_stream = json.dumps(event_stream.model_dump())

163 logger.info(f"json_event_stream set: {json_event_stream}")

163 - response_json = await send_post_request(

164 + response_json = await send_post_request_create_entity(

165 endpoint="/api/streams",

165 - data=event_stream.model_dump(),

166 + entity=event_stream.model_dump(),

167 )

168 return StreamCreationResponse(**response_json)

169

backend/app/integrations/carbonblack/services/provision.py

+3 -2

@@ -12,6 +12,7 @@ from app.connectors.grafana.services.dashboards import provision_dashboards

12 from app.connectors.grafana.utils.universal import create_grafana_client

13 from app.connectors.graylog.services.management import start_stream

14 from app.connectors.graylog.utils.universal import send_post_request

15 +from app.connectors.graylog.utils.universal import send_post_request_create_entity

16 from app.connectors.wazuh_indexer.services.monitoring import (

17 output_shard_number_to_be_set_based_on_nodes,

18 )

@@ -170,9 +171,9 @@ async def send_event_stream_creation_request(

171 """

172 json_event_stream = json.dumps(event_stream.model_dump())

173 logger.info(f"json_event_stream set: {json_event_stream}")

173 - response_json = await send_post_request(

174 + response_json = await send_post_request_create_entity(

175 endpoint="/api/streams",

175 - data=event_stream.model_dump(),

176 + entity=event_stream.model_dump(),

177 )

178 return StreamCreationResponse(**response_json)

179

backend/app/integrations/cato/services/provision.py

+3 -2

@@ -12,6 +12,7 @@ from app.connectors.grafana.services.dashboards import provision_dashboards

12 from app.connectors.grafana.utils.universal import create_grafana_client

13 from app.connectors.graylog.services.management import start_stream

14 from app.connectors.graylog.utils.universal import send_post_request

15 +from app.connectors.graylog.utils.universal import send_post_request_create_entity

16 from app.connectors.wazuh_indexer.services.monitoring import (

17 output_shard_number_to_be_set_based_on_nodes,

18 )

@@ -170,9 +171,9 @@ async def send_event_stream_creation_request(

171 """

172 json_event_stream = json.dumps(event_stream.model_dump())

173 logger.info(f"json_event_stream set: {json_event_stream}")

173 - response_json = await send_post_request(

174 + response_json = await send_post_request_create_entity(

175 endpoint="/api/streams",

175 - data=event_stream.model_dump(),

176 + entity=event_stream.model_dump(),

177 )

178 return StreamCreationResponse(**response_json)

179

backend/app/integrations/darktrace/services/provision.py

+3 -2

@@ -12,6 +12,7 @@ from app.connectors.grafana.services.dashboards import provision_dashboards

12 from app.connectors.grafana.utils.universal import create_grafana_client

13 from app.connectors.graylog.services.management import start_stream

14 from app.connectors.graylog.utils.universal import send_post_request

15 +from app.connectors.graylog.utils.universal import send_post_request_create_entity

16 from app.connectors.wazuh_indexer.services.monitoring import (

17 output_shard_number_to_be_set_based_on_nodes,

18 )

@@ -170,9 +171,9 @@ async def send_event_stream_creation_request(

171 """

172 json_event_stream = json.dumps(event_stream.model_dump())

173 logger.info(f"json_event_stream set: {json_event_stream}")

173 - response_json = await send_post_request(

174 + response_json = await send_post_request_create_entity(

175 endpoint="/api/streams",

175 - data=event_stream.model_dump(),

176 + entity=event_stream.model_dump(),

177 )

178 return StreamCreationResponse(**response_json)

179

backend/app/integrations/duo/services/provision.py

+3 -2

@@ -12,6 +12,7 @@ from app.connectors.grafana.services.dashboards import provision_dashboards

12 from app.connectors.grafana.utils.universal import create_grafana_client

13 from app.connectors.graylog.services.management import start_stream

14 from app.connectors.graylog.utils.universal import send_post_request

15 +from app.connectors.graylog.utils.universal import send_post_request_create_entity

16 from app.connectors.wazuh_indexer.services.monitoring import (

17 output_shard_number_to_be_set_based_on_nodes,

18 )

@@ -170,9 +171,9 @@ async def send_event_stream_creation_request(

171 """

172 json_event_stream = json.dumps(event_stream.model_dump())

173 logger.info(f"json_event_stream set: {json_event_stream}")

173 - response_json = await send_post_request(

174 + response_json = await send_post_request_create_entity(

175 endpoint="/api/streams",

175 - data=event_stream.model_dump(),

176 + entity=event_stream.model_dump(),

177 )

178 return StreamCreationResponse(**response_json)

179

backend/app/integrations/huntress/services/provision.py

+3 -2

@@ -12,6 +12,7 @@ from app.connectors.grafana.services.dashboards import provision_dashboards

12 from app.connectors.grafana.utils.universal import create_grafana_client

13 from app.connectors.graylog.services.management import start_stream

14 from app.connectors.graylog.utils.universal import send_post_request

15 +from app.connectors.graylog.utils.universal import send_post_request_create_entity

16 from app.connectors.wazuh_indexer.services.monitoring import (

17 output_shard_number_to_be_set_based_on_nodes,

18 )

@@ -170,9 +171,9 @@ async def send_event_stream_creation_request(

171 """

172 json_event_stream = json.dumps(event_stream.model_dump())

173 logger.info(f"json_event_stream set: {json_event_stream}")

173 - response_json = await send_post_request(

174 + response_json = await send_post_request_create_entity(

175 endpoint="/api/streams",

175 - data=event_stream.model_dump(),

176 + entity=event_stream.model_dump(),

177 )

178 return StreamCreationResponse(**response_json)

179

backend/app/integrations/mimecast/services/provision.py

+3 -2

@@ -12,6 +12,7 @@ from app.connectors.grafana.services.dashboards import provision_dashboards

12 from app.connectors.grafana.utils.universal import create_grafana_client

13 from app.connectors.graylog.services.management import start_stream

14 from app.connectors.graylog.utils.universal import send_post_request

15 +from app.connectors.graylog.utils.universal import send_post_request_create_entity

16 from app.customer_provisioning.schema.grafana import GrafanaDatasource

17 from app.customer_provisioning.schema.grafana import GrafanaDataSourceCreationResponse

18 from app.customer_provisioning.schema.graylog import GraylogIndexSetCreationResponse

@@ -167,9 +168,9 @@ async def send_event_stream_creation_request(

168 """

169 json_event_stream = json.dumps(event_stream.model_dump())

170 logger.info(f"json_event_stream set: {json_event_stream}")

170 - response_json = await send_post_request(

171 + response_json = await send_post_request_create_entity(

172 endpoint="/api/streams",

172 - data=event_stream.model_dump(),

173 + entity=event_stream.model_dump(),

174 )

175 return StreamCreationResponse(**response_json)

176

backend/app/integrations/monitoring_alert/services/provision.py

+5 -5

@@ -8,7 +8,7 @@ from app.connectors.graylog.routes.monitoring import get_all_event_notifications

8 from app.connectors.graylog.schema.management import UrlWhitelistEntryResponse

9 from app.connectors.graylog.schema.monitoring import GraylogEventNotificationsResponse

10 from app.connectors.graylog.services.collector import get_url_whitelist_entries

11 -from app.connectors.graylog.utils.universal import send_post_request

11 +from app.connectors.graylog.utils.universal import send_post_request_create_entity

12 from app.connectors.graylog.utils.universal import send_put_request

13 from app.integrations.monitoring_alert.schema.provision import (

14 CustomMonitoringAlertProvisionModel,

@@ -219,9 +219,9 @@ async def provision_webhook(

219 Returns:

220 bool: True if the webhook was provisioned successfully, False otherwise.

221 """

222 - response = await send_post_request(

222 + response = await send_post_request_create_entity(

223 endpoint="/api/events/notifications",

224 - data=webhook_model.model_dump(),

224 + entity=webhook_model.model_dump(),

225 )

226 if response["success"]:

227 logger.info(f"response: {response}")

@@ -249,9 +249,9 @@ async def provision_alert_definition(

249 if hasattr(alert_definition_model.config, "event_limit"):

250 delattr(alert_definition_model.config, "event_limit")

251

252 - response = await send_post_request(

252 + response = await send_post_request_create_entity(

253 endpoint="/api/events/definitions",

254 - data=alert_definition_model.model_dump(),

254 + entity=alert_definition_model.model_dump(),

255 )

256 logger.info(f"Graylog alert definition provisioned response: {response}")

257 if response["success"]:

backend/app/integrations/office365/services/provision.py

+3 -2

@@ -33,6 +33,7 @@ from app.connectors.graylog.services.pipelines import get_pipeline_id

33 from app.connectors.graylog.services.pipelines import get_pipeline_rules

34 from app.connectors.graylog.services.pipelines import get_pipelines

35 from app.connectors.graylog.utils.universal import send_post_request

36 +from app.connectors.graylog.utils.universal import send_post_request_create_entity

37 from app.connectors.wazuh_indexer.services.monitoring import (

38 output_shard_number_to_be_set_based_on_nodes,

39 )

@@ -514,9 +515,9 @@ async def send_event_stream_creation_request(

515 """

516 json_event_stream = json.dumps(event_stream.model_dump())

517 logger.info(f"json_event_stream set: {json_event_stream}")

517 - response_json = await send_post_request(

518 + response_json = await send_post_request_create_entity(

519 endpoint="/api/streams",

519 - data=event_stream.model_dump(),

520 + entity=event_stream.model_dump(),

521 )

522 return StreamCreationResponse(**response_json)

523

backend/app/integrations/sap_siem/services/provision.py