@cryptotaxi247 / CoPilot
Code Commits Issues Pulls Sessions CI/CD
main
main
md 173 lines 9.15 KB
Rendered Raw
1 ---
2 title: Graylog Threshold Alerts
3 description: How to configure Graylog threshold-based event definitions and send them to CoPilot via webhook.
4 ---
5
6 # Graylog Threshold Alerts
7
8 ## Overview
9
10 Not all detections are triggered by a single event. Some security-relevant scenarios — such as excessive login failures, brute-force attempts, or repeated policy violations — only become meaningful when a **count of events exceeds a threshold** within a given time window.
11
12 Graylog handles these through **threshold-based Event Definitions**, which are fundamentally different from standard (single-event) alerts:
13
14 | Alert type | Trigger | Example |
15 |---|---|---|
16 | **Standard** | A single matching event | Wazuh rule fires for malware detection |
17 | **Threshold** | Count of matching events ≥ N in a time window | 10+ failed logins in 5 minutes |
18
19 Because threshold alerts aggregate multiple events, there is no single underlying event `_id` to reference. CoPilot therefore exposes a **dedicated webhook endpoint** (`/create/threshold`) that accepts the aggregated alert metadata and converts it into an Incident Management alert.
20
21 ---
22
23 ## When to use threshold alerts
24
25 Threshold alerts are the right choice when you need to detect:
26
27 - **Brute-force / credential stuffing** — e.g., ≥ 10 failed authentication attempts within 5 minutes for the same user or source IP.
28 - **Excessive privilege escalation attempts** — e.g., ≥ 5 `sudo` failures on a single host in 10 minutes.
29 - **Anomalous volume** — e.g., ≥ 100 Office 365 `FileDeleted` events from the same user in 15 minutes.
30 - **Repeated policy violations** — e.g., ≥ 3 DLP policy hits from the same endpoint in 1 hour.
31 - **Scan / enumeration detection** — e.g., ≥ 50 connection attempts to different ports on the same destination in 1 minute.
32
33 > **Rule of thumb:** If the detection logic includes words like *"more than,"* *"at least,"* or *"within X minutes,"* you likely need a threshold alert.
34
35 ---
36
37 ## Step-by-step configuration
38
39 ### 1. Create the Event Definition in Graylog
40
41 1. Navigate to **Alerts → Event Definitions** in the Graylog web UI.
42 2. Click **Create Event Definition**.
43 3. Fill in the **Title** and **Description** — this title will become the alert name inside CoPilot.
44
45 #### Condition Configuration
46
47 4. Under **Condition Type**, select **Filter & Aggregation**.
48 5. Define your **search query** (filter) to match the relevant log events (e.g., `source:office365 AND event_type:login_failure`).
49 6. Set the **Search within** time window (e.g., `5 minutes`).
50 7. Set the **Execute search every** interval (e.g., `1 minute`).
51 8. Under **Aggregation**, configure:
52 - **Group By** — the field(s) to group on (e.g., `source_ip`, `username`, `agent_name`).
53 - **Condition** — e.g., `count() >= 10`.
54
55 > ⚠️ Make sure the aggregation condition accurately reflects the threshold you want. Test with **Preview** before saving.
56
57 ### 2. Define the required Custom Fields
58
59 CoPilot's `/create/threshold` endpoint requires **four custom fields** to be set on the Event Definition. These fields map the Graylog event into a CoPilot alert.
60
61 Navigate to the **Fields** tab of the Event Definition and add the following:
62
63 | Field name | Type | Required | Description |
64 |---|---|---|---|
65 | `CUSTOMER_CODE` | `string` | ✅ | The customer code this alert belongs to (must match a customer in CoPilot). |
66 | `SOURCE` | `string` | ✅ | The source/integration name (e.g., `office365`, `wazuh`, `custom`). |
67 | `ALERT_DESCRIPTION` | `string` | ✅ | A human-readable description of what was detected. |
68 | `ASSET_NAME` | `string` | ✅ | The name of the affected asset (hostname, username, IP — whatever is most relevant). |
69
70 For each field, you can either:
71
72 - Set a **static value** (e.g., `CUSTOMER_CODE` = `ACME`) if this event definition is customer-specific, or
73 - Use a **template** that references Graylog event fields (e.g., `ASSET_NAME` = `${source.agent_name}`).
74
75 **Example field configuration:**
76
77 ```
78 CUSTOMER_CODE → ACME
79 SOURCE → office365
80 ALERT_DESCRIPTION → Excessive login failures detected (threshold: 10 in 5 min)
81 ASSET_NAME → ${source.agent_name}
82 ```
83
84 > 🚨 **IMPORTANT:** Do **NOT** add a field called `COPILOT_ALERT_ID` with a value of `NONE`. This will break the auto-alert creation functionality for standard (non-threshold) alerts.
85
86 ### 3. Create the HTTP Notification
87
88 1. Navigate to the **Notifications** tab of your Event Definition (or go to **Alerts → Notifications → Create Notification**).
89 2. Select **HTTP Notification** as the notification type.
90
91 > 🚨 **IMPORTANT:** Use the standard **HTTP Notification** type — do **NOT** use the *Custom HTTP Notification* type. The custom type sends a different payload format that CoPilot cannot parse.
92
93 3. Configure the notification:
94
95 | Setting | Value |
96 |---|---|
97 | **Title** | Give it a descriptive name (e.g., `CoPilot Threshold Webhook`) |
98 | **URL** | `http://<your-copilot-url>:5000/api/incidents/alerts/create/threshold` |
99
100 4. **Save** the notification.
101
102 ### 4. Attach the Notification to the Event Definition
103
104 1. Go back to your Event Definition → **Notifications** tab.
105 2. Click **Add Notification** and select the HTTP Notification you just created.
106 3. **Save** the Event Definition.
107
108 ### 5. Verify the Graylog API header
109
110 CoPilot validates inbound Graylog webhook requests using a shared header. Make sure your CoPilot deployment has the `GRAYLOG_API_HEADER_VALUE` environment variable set, and that the Graylog HTTP Notification sends the matching header.
111
112 Check with your CoPilot administrator that the Graylog header verification is correctly configured. This is typically set during the initial CoPilot ↔ Graylog integration setup.
113
114 ---
115
116 ## How it works end-to-end
117
118 ```
119 ┌─────────────┐ threshold met ┌─────────────────────┐
120 │ Graylog │ ─────────────────────▶ │ HTTP Notification │
121 │ Event Def │ (count ≥ N in T) │ (standard type) │
122 └─────────────┘ └─────────┬───────────┘
123 │
124 POST /api/incidents/alerts/create/threshold
125 │
126 ▼
127 ┌─────────────────────┐
128 │ CoPilot │
129 │ Incident Mgmt │
130 │ (new alert created) │
131 └─────────────────────┘
132 ```
133
134 1. Graylog evaluates the aggregation query on schedule (e.g., every minute).
135 2. When the threshold condition is met, Graylog fires the Event Definition.
136 3. The HTTP Notification sends a POST request to CoPilot's `/create/threshold` endpoint with the event payload (including your custom fields).
137 4. CoPilot validates the Graylog header, extracts the required fields (`CUSTOMER_CODE`, `SOURCE`, `ALERT_DESCRIPTION`, `ASSET_NAME`), and creates a new alert in Incident Management.
138 5. The alert appears in the CoPilot **Incident Management → Alerts** view and can be triaged, assigned to a case, or trigger downstream automation (e.g., via Shuffle).
139
140 ---
141
142 ## Differences from standard alerts
143
144 | | Standard alerts | Threshold alerts |
145 |---|---|---|
146 | **Trigger** | Single event match | Aggregation count over time window |
147 | **CoPilot ingestion** | Auto-collected from `gl-events-*` indices | Sent via webhook to `/create/threshold` |
148 | **Graylog notification type** | Not required (CoPilot polls) | **HTTP Notification** (standard) required |
149 | **Custom fields** | `COPILOT_ALERT_ID: NONE` added by default | `CUSTOMER_CODE`, `SOURCE`, `ALERT_DESCRIPTION`, `ASSET_NAME` |
150 | **Underlying event** | Links back to a specific indexed event | No single event — aggregated context only |
151
152 ---
153
154 ## Troubleshooting
155
156 | Symptom | Likely cause | Fix |
157 |---|---|---|
158 | Alert not appearing in CoPilot | Notification not attached to Event Definition | Attach the HTTP Notification in the Event Definition's Notifications tab |
159 | `403 Forbidden` from CoPilot | Graylog header mismatch | Verify `GRAYLOG_API_HEADER_VALUE` matches between CoPilot and Graylog |
160 | `422 Unprocessable Entity` | Missing required custom fields | Ensure all four fields (`CUSTOMER_CODE`, `SOURCE`, `ALERT_DESCRIPTION`, `ASSET_NAME`) are defined |
161 | Alert created but missing context | Template variable not resolving | Check that your field templates reference valid event fields (use Graylog's Preview) |
162 | Used *Custom HTTP Notification* type | Wrong notification type | Delete and recreate using the standard **HTTP Notification** type |
163
164 ---
165
166 ## Related pages
167
168 - [Alerting → Shuffle (notifications & automation)](./alerting-shuffle.md)
169 - [Alerts — SIEM view](./alerts-siem.md)
170 - [Graylog Management (detections)](./graylog-management.md)
171 - [Incident Management](./incident-management.md)
172 - [Incident Alerts](./incident-alerts.md)
173 - [Incident Sources (mapping context)](./incident-sources.md)