fc51d16c6 · cryptotaxi247/netdata-1

@@ -0,0 +1,173 @@

1 +# yamllint disable rule:line-length

2 +---

3 +id: 'export-aws-kinesis'

4 +meta:

5 + name: 'AWS Kinesis'

6 + link: 'https://aws.amazon.com/kinesis/'

7 + categories:

8 + - export

9 + icon_filename: 'kinesis.png'

10 +keywords:

11 + - exporter

12 + - AWS

13 + - Kinesis

14 +overview:

15 + exporter_description: |

16 + Export metrics to AWS Kinesis Data Streams

17 + exporter_limitations: ''

18 +setup:

19 + prerequisites:

20 + list:

21 + - title: ''

22 + description: |

23 + - First [install](https://docs.aws.amazon.com/en_us/sdk-for-cpp/v1/developer-guide/setup.html) AWS SDK for C++

24 + - Here are the instructions when building from source, to ensure 3rd party dependencies are installed:

25 + ```bash

26 + git clone --recursive https://github.com/aws/aws-sdk-cpp.git

27 + cd aws-sdk-cpp/

28 + git submodule update --init --recursive

29 + mkdir BUILT

30 + cd BUILT

31 + cmake -DCMAKE_INSTALL_PREFIX=/usr -DBUILD_ONLY=kinesis ..

32 + make

33 + make install

34 + ```

35 + - `libcrypto`, `libssl`, and `libcurl` are also required to compile Netdata with Kinesis support enabled.

36 + - Next, Netdata should be re-installed from the source. The installer will detect that the required libraries are now available.

37 + configuration:

38 + file:

39 + name: 'exporting.conf'

40 + options:

41 + description: |

42 + Netdata automatically computes a partition key for every record with the purpose to distribute records across available shards evenly.

43 + The following options can be defined for this exporter.

44 + folding:

45 + title: 'Config options'

46 + enabled: true

47 + list:

48 + - name: 'enabled'

49 + default_value: 'no'

50 + description: 'Enables or disables an exporting connector instance (yes|no).'

51 + required: true

52 + - name: 'destination'

53 + default_value: 'no'

54 + description: 'Accepts a space separated list of hostnames, IPs (IPv4 and IPv6) and ports to connect to. Netdata will use the first available to send the metrics.'

55 + required: true

56 + detailed_description: |

57 + The format of each item in this list, is: [PROTOCOL:]IP[:PORT].

58 + - PROTOCOL can be udp or tcp. tcp is the default and only supported by the current exporting engine.

59 + - IP can be XX.XX.XX.XX (IPv4), or [XX:XX...XX:XX] (IPv6). For IPv6 you can to enclose the IP in [] to separate it from the port.

60 + - PORT can be a number of a service name. If omitted, the default port for the exporting connector will be used.

61 +

62 + Example IPv4:

63 + ```yaml

64 + destination = 10.11.14.2:4242 10.11.14.3:4242 10.11.14.4:4242

65 + ```

66 + Example IPv6 and IPv4 together:

67 + ```yaml

68 + destination = [ffff:...:0001]:2003 10.11.12.1:2003

69 + ```

70 + When multiple servers are defined, Netdata will try the next one when the previous one fails.

71 + - name: 'username'

72 + default_value: 'my_username'

73 + description: 'Username for HTTP authentication'

74 + required: false

75 + - name: 'password'

76 + default_value: 'my_password'

77 + description: 'Password for HTTP authentication'

78 + required: false

79 + - name: 'data source'

80 + default_value: ''

81 + description: 'Selects the kind of data that will be sent to the external database. (as collected|average|sum)'

82 + required: false

83 + - name: 'hostname'

84 + default_value: '[global].hostname'

85 + description: 'The hostname to be used for sending data to the external database server.'

86 + required: false

87 + - name: 'prefix'

88 + default_value: 'Netdata'

89 + description: 'The prefix to add to all metrics.'

90 + required: false

91 + - name: 'update every'

92 + default_value: '10'

93 + description: |

94 + Frequency of sending sending data to the external database, in seconds.

95 + required: false

96 + detailed_description: |

97 + Netdata will add some randomness to this number, to prevent stressing the external server when many Netdata servers

98 + send data to the same database. This randomness does not affect the quality of the data, only the time they are sent.

99 + - name: 'buffer on failures'

100 + default_value: '10'

101 + description: |

102 + The number of iterations (`update every` seconds) to buffer data, when the external database server is not available.

103 + required: false

104 + detailed_description: |

105 + If the server fails to receive the data after that many failures, data loss on the connector instance is expected (Netdata will also log it).

106 + - name: 'timeout ms'

107 + default_value: '2 * update_every * 1000'

108 + description: 'The timeout in milliseconds to wait for the external database server to process the data.'

109 + required: false

110 + - name: 'send hosts matching'

111 + default_value: 'localhost *'

112 + description: |

113 + Hosts filter. Determines which hosts will be sent to the external database. The syntax is [simple patterns](https://github.com/netdata/netdata/tree/master/libnetdata/simple_pattern#simple-patterns).

114 + required: false

115 + detailed_description: |

116 + Includes one or more space separated patterns, using * as wildcard (any number of times within each pattern).

117 + The patterns are checked against the hostname (the localhost is always checked as localhost), allowing us to

118 + filter which hosts will be sent to the external database when this Netdata is a central Netdata aggregating multiple hosts.

119 +

120 + A pattern starting with `!` gives a negative match. So to match all hosts named `*db*` except hosts containing `*child*`,

121 + use `!*child* *db*` (so, the order is important: the first pattern matching the hostname will be used - positive or negative).

122 + - name: 'send charts matching'

123 + default_value: '*'

124 + description: |

125 + One or more space separated patterns (use * as wildcard) checked against both chart id and chart name.

126 + required: false

127 + detailed_description: |

128 + A pattern starting with ! gives a negative match. So to match all charts named apps.* except charts ending in *reads,

129 + use !*reads apps.* (so, the order is important: the first pattern matching the chart id or the chart name will be used,

130 + positive or negative). There is also a URL parameter filter that can be used while querying allmetrics. The URL parameter

131 + has a higher priority than the configuration option.

132 + - name: 'send names instead of ids'

133 + default_value: ''

134 + description: 'Controls the metric names Netdata should send to the external database (yes|no).'

135 + required: false

136 + detailed_description: |

137 + Netdata supports names and IDs for charts and dimensions. Usually IDs are unique identifiers as read by the system and names

138 + are human friendly labels (also unique). Most charts and metrics have the same ID and name, but in several cases they are

139 + different : disks with device-mapper, interrupts, QoS classes, statsd synthetic charts, etc.

140 + - name: 'send configured labels'

141 + default_value: ''

142 + description: 'Controls if host labels defined in the `[host labels]` section in `netdata.conf` should be sent to the external database (yes|no).'

143 + required: false

144 + - name: 'send automatic labels'

145 + default_value: ''

146 + description: 'Controls if automatically created labels, like `_os_name` or `_architecture` should be sent to the external database (yes|no).'

147 + required: false

148 + examples:

149 + folding:

150 + enabled: true

151 + title: ''

152 + list:

153 + - name: 'Example configuration'

154 + folding:

155 + enabled: false

156 + description: 'Basic configuration'

157 + config: |

158 + [kinesis:my_instance]

159 + enabled = yes

160 + destination = us-east-1

161 + - name: 'Configuration with AWS credentials'

162 + folding:

163 + enabled: false

164 + description: 'Add `:https` modifier to the connector type if you need to use the TLS/SSL protocol. For example: `remote_write:https:my_instance`.'

165 + config: |

166 + [kinesis:my_instance]

167 + enabled = yes

168 + destination = us-east-1

169 + # AWS credentials

170 + aws_access_key_id = your_access_key_id

171 + aws_secret_access_key = your_secret_access_key

172 + # destination stream

173 + stream name = your_stream_name

Create metadata.yaml for AWS kinesis exporter (#15577)