Update stream documentation bringing explanation of some errors (#6995)
* stream_doc: Update documentation Update documentation for stream to clarify the users some errors * stream_doc: Update documentation 2 Explain more errors that can happen on master * Fix errors reported This commit fixes errors reported by @cosmix * Fix and more errors This commit brings more fixes for the text and description for more errors * stream_doc: Gramatical fixes This commit brings all the fixes suggested by Joel * stream_doc: Gramatical fixes This commit brings the last fix suggested by Joel * stream_doc: Message format I missed a message format in the previous sprint * Fix doc: This commit fix the errors reported by Joel * Fix test This commit brings part of the fixes requested from Christopher * New explanation: The previous explanation for these two features were not good enough, so I rewrite it to avoid confusion * stream_doc: Fix text After to receive the text from Joel, I am bringing all the fix for the new section on documentation
thiagoftsm committed
Jan 27, 2020 at 14:14 UTC
cb7f19ba2dc40d24ec9cd68c08a7d20acd891a37
1 file changed
+96
streaming/README.md
+96
@@ -511,4 +511,100 @@ metrics, following the same pattern of the receiving side.
511
512
For a practical example see [Monitoring ephemeral nodes](#monitoring-ephemeral-nodes).
513
514
+## Troubleshooting streaming connections
515
+
516
+This section describes the most common issues you might encounter when connecting slave and master Netdata agents.
517
+
518
+### Slow connections between slave and master
519
+
520
+When you have a slow connection between master and slave, Netdata raises a few different errors. Most of the errors will
521
+appear in the slave's `error.log`.
522
+
523
+```
524
+netdata ERROR : STREAM_SENDER[SLAVE HOSTNAME] : STREAM SLAVE HOSTNAME [send to MASTER IP:MASTER PORT]: too many data pending - buffer is X bytes long,
525
+Y unsent - we have sent Z bytes in total, W on this connection. Closing connection to flush the data.
526
+```
527
+
528
+On the master side, you may see various error messages, most commonly the following:
529
+
530
+```
531
+netdata ERROR : STREAM_RECEIVER[SLAVE HOSTNAME,[SLAVE IP]:SLAVE PORT] : read failed: end of file
532
+```
533
+
534
+Another common problem in slow connections is the slave sending a partial message to the master. In this case, the
535
+master will write the following in its `error.log`:
536
+
537
+```
538
+ERROR : STREAM_RECEIVER[SLAVE HOSTNAME,[SLAVE IP]:SLAVE PORT] : sent command 'B' which is not known by netdata, for host 'HOSTNAME'. Disabling it.
539
+```
540
+
541
+In this example, `B` was part of a `BEGIN` message that was cut due to connection problems.
542
+
543
+Slow connections can also cause problems when the master misses a message and then recieves a command related to the
544
+missed message. For example, a master might miss a message containing the slave's charts, and then doesn't know what to
545
+do with the `SET` message that follows. When that happens, the master will show a message like this:
546
+
547
+```
548
+ERROR : STREAM_RECEIVER[SLAVE HOSTNAME,[SLAVE IP]:SLAVE PORT] : requested a SET on chart 'CHART NAME' of host 'HOSTNAME', without a dimension. Disabling it.
549
+```
550
+
551
+### Slave cannot connect to master
552
+
553
+When the slave can't connect to a master for any reason (misconfiguration, networking, firewalls, master down), you will
554
+see the following in the slave's `error.log`.
555
+
556
+```
557
+ERROR : STREAM_SENDER[HOSTNAME] : Failed to connect to 'MASTER IP', port 'MASTER PORT' (errno 113, No route to host)
558
+```
559
+
560
+### 'Is this a Netdata?'
561
+
562
+This question can appear when Netdata starts the stream and receives an unexpected response. This error can appear when
563
+the master is using SSL and the slave tries to connect using plain text. You will also see this message when Netdata
564
+connects to another server that isn't Netdata. The complete error message will look like this:
565
+
566
+```
567
+ERROR : STREAM_SENDER[SLAVE HOSTNAME] : STREAM SLAVE HOSTNAME [send to MASTER HOSTNAME:MASTER PORT]: server is not replying properly (is it a netdata?).
568
+```
569
+
570
+### Stream charts wrong
571
+
572
+Chart data needs to be consistent between slave and master agents. If there are differences between chart data on a
573
+master and a slave, such as gaps in metrics collection, it most often means your slave's `memory mode` does not match
574
+the master's. To learn more about the different ways Netdata can store metrics, and thus keep chart data consistent,
575
+read our [memory mode documentation](../database).
576
+
577
+### Forbidding access
578
+
579
+You may see errors about "forbidding access" for a number of reasons. It could be because of a slow connection between
580
+the master and slave nodes, but it could also be due to other failures. Look in your master's `error.log` for errors
581
+that look like this:
582
+
583
+```
584
+STREAM [receive from [SLAVE HOSTNAME]:SLAVE IP]: `MESSAGE`. Forbidding access."
585
+```
586
+
587
+`MESSAGE` will have one of the following patterns:
588
+
589
+- `request without KEY` : The message received is incomplete and the KEY value can be API, hostname, machine GUID.
590
+- `API key 'VALUE' is not valid GUID`: The UUID received from slave does not have the format defined in [RFC 4122]
591
+ (https://tools.ietf.org/html/rfc4122)
592
+- `machine GUID 'VALUE' is not GUID.`: This error with machine GUID is like the previous one.
593
+- `API key 'VALUE' is not allowed`: This stream has a wrong API key.
594
+- `API key 'VALUE' is not permitted from this IP`: The IP is not allowed to use STREAM with this master.
595
+- `machine GUID 'VALUE' is not allowed.`: The GUID that is trying to send stream is not allowed.
596
+- `Machine GUID 'VALUE' is not permitted from this IP. `: The IP does not match the pattern or IP allowed to connect
597
+ to use stream.
598
+
599
+### Netdata could not create a stream
600
+
601
+The connection between master and slave is a stream. When the master can't convert the initial connection into a stream,
602
+it will write the following message inside `error.log`:
603
+
604
+```
605
+file descriptor given is not a valid stream
606
+```
607
+
608
+After logging this error, Netdata will close the stream.
609
+
610
[](<>)