| 1 | <!--startmeta |
| 2 | custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/exporting/aws_kinesis/README.md" |
| 3 | meta_yaml: "https://github.com/netdata/netdata/edit/master/src/exporting/aws_kinesis/metadata.yaml" |
| 4 | sidebar_label: "AWS Kinesis" |
| 5 | learn_status: "Published" |
| 6 | learn_rel_path: "Exporting Metrics/Connectors" |
| 7 | keywords: ['exporter', 'AWS', 'Kinesis'] |
| 8 | message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE EXPORTER'S metadata.yaml FILE" |
| 9 | endmeta--> |
| 10 | |
| 11 | # AWS Kinesis |
| 12 | |
| 13 | |
| 14 | <img src="https://netdata.cloud/img/aws-kinesis.svg" width="150"/> |
| 15 | |
| 16 | |
| 17 | Export metrics to AWS Kinesis Data Streams |
| 18 | |
| 19 | |
| 20 | <img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> |
| 21 | |
| 22 | ## Setup |
| 23 | |
| 24 | |
| 25 | ### Prerequisites |
| 26 | |
| 27 | #### |
| 28 | |
| 29 | - First [install](https://docs.aws.amazon.com/en_us/sdk-for-cpp/v1/developer-guide/setup.html) AWS SDK for C++ |
| 30 | - Here are the instructions when building from source, to ensure 3rd party dependencies are installed: |
| 31 | ```bash |
| 32 | git clone --recursive https://github.com/aws/aws-sdk-cpp.git |
| 33 | cd aws-sdk-cpp/ |
| 34 | git submodule update --init --recursive |
| 35 | mkdir BUILT |
| 36 | cd BUILT |
| 37 | cmake -DCMAKE_INSTALL_PREFIX=/usr -DBUILD_ONLY=kinesis .. |
| 38 | make |
| 39 | make install |
| 40 | ``` |
| 41 | - `libcrypto`, `libssl`, and `libcurl` are also required to compile Netdata with Kinesis support enabled. |
| 42 | - Next, Netdata should be re-installed from the source. The installer will detect that the required libraries are now available. |
| 43 | |
| 44 | |
| 45 | |
| 46 | ### Configuration |
| 47 | |
| 48 | #### Options |
| 49 | |
| 50 | Netdata automatically computes a partition key for every record with the purpose to distribute records across available shards evenly. |
| 51 | The following options can be defined for this exporter. |
| 52 | |
| 53 | |
| 54 | <details open><summary>Config options</summary> |
| 55 | |
| 56 | |
| 57 | |
| 58 | | Option | Description | Default | Required | |
| 59 | |:-----|:------------|:--------|:---------:| |
| 60 | | enabled | Enables or disables an exporting connector instance (yes/no). | no | yes | |
| 61 | | [destination](#option-destination) | Accepts a space separated list of hostnames, IPs (IPv4 and IPv6) and ports to connect to. Netdata will use the first available to send the metrics. | no | yes | |
| 62 | | username | Username for HTTP authentication | my_username | no | |
| 63 | | password | Password for HTTP authentication | my_password | no | |
| 64 | | data source | Selects the kind of data that will be sent to the external database. (as collected/average/sum) | | no | |
| 65 | | hostname | The hostname to be used for sending data to the external database server. | [global].hostname | no | |
| 66 | | prefix | The prefix to add to all metrics. | Netdata | no | |
| 67 | | [update every](#option-update-every) | Frequency of sending sending data to the external database, in seconds. | 10 | no | |
| 68 | | [buffer on failures](#option-buffer-on-failures) | The number of iterations (`update every` seconds) to buffer data, when the external database server is not available. | 10 | no | |
| 69 | | timeout ms | The timeout in milliseconds to wait for the external database server to process the data. | 2 * update_every * 1000 | no | |
| 70 | | [send hosts matching](#option-send-hosts-matching) | Hosts filter. Determines which hosts will be sent to the external database. The syntax is [simple patterns](https://github.com/netdata/netdata/tree/master/src/libnetdata/simple_pattern#simple-patterns). | localhost * | no | |
| 71 | | [send charts matching](#option-send-charts-matching) | One or more space separated patterns (use * as wildcard) checked against both chart id and chart name. | * | no | |
| 72 | | [send names instead of ids](#option-send-names-instead-of-ids) | Controls the metric names Netdata should send to the external database (yes/no). | | no | |
| 73 | | send configured labels | Controls if host labels defined in the `[host labels]` section in `netdata.conf` should be sent to the external database (yes/no). | | no | |
| 74 | | send automatic labels | Controls if automatically created labels, like `_os_name` or `_architecture` should be sent to the external database (yes/no). | | no | |
| 75 | |
| 76 | <a id="option-destination"></a> |
| 77 | ##### destination |
| 78 | |
| 79 | The format of each item in this list, is: [PROTOCOL:]IP[:PORT]. |
| 80 | - PROTOCOL can be udp or tcp. tcp is the default and only supported by the current exporting engine. |
| 81 | - IP can be XX.XX.XX.XX (IPv4), or [XX:XX...XX:XX] (IPv6). For IPv6 you can to enclose the IP in [] to separate it from the port. |
| 82 | - PORT can be a number of a service name. If omitted, the default port for the exporting connector will be used. |
| 83 | |
| 84 | Example IPv4: |
| 85 | ```yaml |
| 86 | destination = 10.11.14.2:4242 10.11.14.3:4242 10.11.14.4:4242 |
| 87 | ``` |
| 88 | Example IPv6 and IPv4 together: |
| 89 | ```yaml |
| 90 | destination = [ffff:...:0001]:2003 10.11.12.1:2003 |
| 91 | ``` |
| 92 | When multiple servers are defined, Netdata will try the next one when the previous one fails. |
| 93 | |
| 94 | |
| 95 | <a id="option-update-every"></a> |
| 96 | ##### update every |
| 97 | |
| 98 | Netdata will add some randomness to this number, to prevent stressing the external server when many Netdata servers |
| 99 | send data to the same database. This randomness does not affect the quality of the data, only the time they are sent. |
| 100 | |
| 101 | |
| 102 | <a id="option-buffer-on-failures"></a> |
| 103 | ##### buffer on failures |
| 104 | |
| 105 | If the server fails to receive the data after that many failures, data loss on the connector instance is expected (Netdata will also log it). |
| 106 | |
| 107 | |
| 108 | <a id="option-send-hosts-matching"></a> |
| 109 | ##### send hosts matching |
| 110 | |
| 111 | Includes one or more space separated patterns, using * as wildcard (any number of times within each pattern). |
| 112 | The patterns are checked against the hostname (the localhost is always checked as localhost), allowing us to |
| 113 | filter which hosts will be sent to the external database when this Netdata is a central Netdata aggregating multiple hosts. |
| 114 | |
| 115 | A pattern starting with `!` gives a negative match. So to match all hosts named `*db*` except hosts containing `*child*`, |
| 116 | use `!*child* *db*` (so, the order is important: the first pattern matching the hostname will be used - positive or negative). |
| 117 | |
| 118 | |
| 119 | <a id="option-send-charts-matching"></a> |
| 120 | ##### send charts matching |
| 121 | |
| 122 | A pattern starting with ! gives a negative match. So to match all charts named apps.* except charts ending in *reads, |
| 123 | use !*reads apps.* (so, the order is important: the first pattern matching the chart id or the chart name will be used, |
| 124 | positive or negative). There is also a URL parameter filter that can be used while querying allmetrics. The URL parameter |
| 125 | has a higher priority than the configuration option. |
| 126 | |
| 127 | |
| 128 | <a id="option-send-names-instead-of-ids"></a> |
| 129 | ##### send names instead of ids |
| 130 | |
| 131 | Netdata supports names and IDs for charts and dimensions. Usually IDs are unique identifiers as read by the system and names |
| 132 | are human friendly labels (also unique). Most charts and metrics have the same ID and name, but in several cases they are |
| 133 | different : disks with device-mapper, interrupts, QoS classes, statsd synthetic charts, etc. |
| 134 | |
| 135 | |
| 136 | |
| 137 | </details> |
| 138 | |
| 139 | |
| 140 | |
| 141 | #### via File |
| 142 | |
| 143 | The configuration file name for this integration is `exporting.conf`. |
| 144 | |
| 145 | |
| 146 | You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the |
| 147 | Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory). |
| 148 | |
| 149 | ```bash |
| 150 | cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata |
| 151 | sudo ./edit-config exporting.conf |
| 152 | ``` |
| 153 | |
| 154 | ##### Examples |
| 155 | |
| 156 | ###### Example configuration |
| 157 | |
| 158 | Basic configuration |
| 159 | |
| 160 | ```yaml |
| 161 | [kinesis:my_instance] |
| 162 | enabled = yes |
| 163 | destination = us-east-1 |
| 164 | |
| 165 | ``` |
| 166 | ###### Configuration with AWS credentials |
| 167 | |
| 168 | Add `:https` modifier to the connector type if you need to use the TLS/SSL protocol. For example: `remote_write:https:my_instance`. |
| 169 | |
| 170 | ```yaml |
| 171 | [kinesis:my_instance] |
| 172 | enabled = yes |
| 173 | destination = us-east-1 |
| 174 | # AWS credentials |
| 175 | aws_access_key_id = your_access_key_id |
| 176 | aws_secret_access_key = your_secret_access_key |
| 177 | # destination stream |
| 178 | stream name = your_stream_name |
| 179 | |
| 180 | ``` |
| 181 |