@cryptotaxi247 / netdata-1
Code Commits Issues Pulls Sessions CI/CD
master
master
yaml 173 lines 8.55 KB
Raw
1 # yamllint disable rule:line-length
2 ---
3 id: 'export-aws-kinesis'
4 meta:
5 name: 'AWS Kinesis'
6 link: 'https://aws.amazon.com/kinesis/'
7 categories:
8 - export
9 icon_filename: 'aws-kinesis.svg'
10 keywords:
11 - exporter
12 - AWS
13 - Kinesis
14 overview:
15 exporter_description: |
16 Export metrics to AWS Kinesis Data Streams
17 exporter_limitations: ''
18 setup:
19 prerequisites:
20 list:
21 - title: ''
22 description: |
23 - First [install](https://docs.aws.amazon.com/en_us/sdk-for-cpp/v1/developer-guide/setup.html) AWS SDK for C++
24 - Here are the instructions when building from source, to ensure 3rd party dependencies are installed:
25 ```bash
26 git clone --recursive https://github.com/aws/aws-sdk-cpp.git
27 cd aws-sdk-cpp/
28 git submodule update --init --recursive
29 mkdir BUILT
30 cd BUILT
31 cmake -DCMAKE_INSTALL_PREFIX=/usr -DBUILD_ONLY=kinesis ..
32 make
33 make install
34 ```
35 - `libcrypto`, `libssl`, and `libcurl` are also required to compile Netdata with Kinesis support enabled.
36 - Next, Netdata should be re-installed from the source. The installer will detect that the required libraries are now available.
37 configuration:
38 file:
39 name: 'exporting.conf'
40 options:
41 description: |
42 Netdata automatically computes a partition key for every record with the purpose to distribute records across available shards evenly.
43 The following options can be defined for this exporter.
44 folding:
45 title: 'Config options'
46 enabled: true
47 list:
48 - name: 'enabled'
49 default_value: 'no'
50 description: 'Enables or disables an exporting connector instance (yes|no).'
51 required: true
52 - name: 'destination'
53 default_value: 'no'
54 description: 'Accepts a space separated list of hostnames, IPs (IPv4 and IPv6) and ports to connect to. Netdata will use the first available to send the metrics.'
55 required: true
56 detailed_description: |
57 The format of each item in this list, is: [PROTOCOL:]IP[:PORT].
58 - PROTOCOL can be udp or tcp. tcp is the default and only supported by the current exporting engine.
59 - IP can be XX.XX.XX.XX (IPv4), or [XX:XX...XX:XX] (IPv6). For IPv6 you can to enclose the IP in [] to separate it from the port.
60 - PORT can be a number of a service name. If omitted, the default port for the exporting connector will be used.
61
62 Example IPv4:
63 ```yaml
64 destination = 10.11.14.2:4242 10.11.14.3:4242 10.11.14.4:4242
65 ```
66 Example IPv6 and IPv4 together:
67 ```yaml
68 destination = [ffff:...:0001]:2003 10.11.12.1:2003
69 ```
70 When multiple servers are defined, Netdata will try the next one when the previous one fails.
71 - name: 'username'
72 default_value: 'my_username'
73 description: 'Username for HTTP authentication'
74 required: false
75 - name: 'password'
76 default_value: 'my_password'
77 description: 'Password for HTTP authentication'
78 required: false
79 - name: 'data source'
80 default_value: ''
81 description: 'Selects the kind of data that will be sent to the external database. (as collected|average|sum)'
82 required: false
83 - name: 'hostname'
84 default_value: '[global].hostname'
85 description: 'The hostname to be used for sending data to the external database server.'
86 required: false
87 - name: 'prefix'
88 default_value: 'Netdata'
89 description: 'The prefix to add to all metrics.'
90 required: false
91 - name: 'update every'
92 default_value: '10'
93 description: |
94 Frequency of sending sending data to the external database, in seconds.
95 required: false
96 detailed_description: |
97 Netdata will add some randomness to this number, to prevent stressing the external server when many Netdata servers
98 send data to the same database. This randomness does not affect the quality of the data, only the time they are sent.
99 - name: 'buffer on failures'
100 default_value: '10'
101 description: |
102 The number of iterations (`update every` seconds) to buffer data, when the external database server is not available.
103 required: false
104 detailed_description: |
105 If the server fails to receive the data after that many failures, data loss on the connector instance is expected (Netdata will also log it).
106 - name: 'timeout ms'
107 default_value: '2 * update_every * 1000'
108 description: 'The timeout in milliseconds to wait for the external database server to process the data.'
109 required: false
110 - name: 'send hosts matching'
111 default_value: 'localhost *'
112 description: |
113 Hosts filter. Determines which hosts will be sent to the external database. The syntax is [simple patterns](https://github.com/netdata/netdata/tree/master/src/libnetdata/simple_pattern#simple-patterns).
114 required: false
115 detailed_description: |
116 Includes one or more space separated patterns, using * as wildcard (any number of times within each pattern).
117 The patterns are checked against the hostname (the localhost is always checked as localhost), allowing us to
118 filter which hosts will be sent to the external database when this Netdata is a central Netdata aggregating multiple hosts.
119
120 A pattern starting with `!` gives a negative match. So to match all hosts named `*db*` except hosts containing `*child*`,
121 use `!*child* *db*` (so, the order is important: the first pattern matching the hostname will be used - positive or negative).
122 - name: 'send charts matching'
123 default_value: '*'
124 description: |
125 One or more space separated patterns (use * as wildcard) checked against both chart id and chart name.
126 required: false
127 detailed_description: |
128 A pattern starting with ! gives a negative match. So to match all charts named apps.* except charts ending in *reads,
129 use !*reads apps.* (so, the order is important: the first pattern matching the chart id or the chart name will be used,
130 positive or negative). There is also a URL parameter filter that can be used while querying allmetrics. The URL parameter
131 has a higher priority than the configuration option.
132 - name: 'send names instead of ids'
133 default_value: ''
134 description: 'Controls the metric names Netdata should send to the external database (yes|no).'
135 required: false
136 detailed_description: |
137 Netdata supports names and IDs for charts and dimensions. Usually IDs are unique identifiers as read by the system and names
138 are human friendly labels (also unique). Most charts and metrics have the same ID and name, but in several cases they are
139 different : disks with device-mapper, interrupts, QoS classes, statsd synthetic charts, etc.
140 - name: 'send configured labels'
141 default_value: ''
142 description: 'Controls if host labels defined in the `[host labels]` section in `netdata.conf` should be sent to the external database (yes|no).'
143 required: false
144 - name: 'send automatic labels'
145 default_value: ''
146 description: 'Controls if automatically created labels, like `_os_name` or `_architecture` should be sent to the external database (yes|no).'
147 required: false
148 examples:
149 folding:
150 enabled: true
151 title: ''
152 list:
153 - name: 'Example configuration'
154 folding:
155 enabled: false
156 description: 'Basic configuration'
157 config: |
158 [kinesis:my_instance]
159 enabled = yes
160 destination = us-east-1
161 - name: 'Configuration with AWS credentials'
162 folding:
163 enabled: false
164 description: 'Add `:https` modifier to the connector type if you need to use the TLS/SSL protocol. For example: `remote_write:https:my_instance`.'
165 config: |
166 [kinesis:my_instance]
167 enabled = yes
168 destination = us-east-1
169 # AWS credentials
170 aws_access_key_id = your_access_key_id
171 aws_secret_access_key = your_secret_access_key
172 # destination stream
173 stream name = your_stream_name