mingw: intercept isatty() to handle /dev/null as Git expects it

When Git's source code calls isatty(), it really asks whether the respective file descriptor is connected to an interactive terminal. Windows' _isatty() function, however, determines whether the file descriptor is associated with a character device. And NUL, Windows' equivalent of /dev/null, is a character device. Which means that for years, Git mistakenly detected an associated interactive terminal when being run through the test suite, which almost always redirects stdin, stdout and stderr to /dev/null. This bug only became obvious, and painfully so, when the new bisect--helper entered the `pu` branch and made the automatic build & test time out because t6030 was waiting for an answer. For details, see https://msdn.microsoft.com/en-us/library/f4s0ddew.aspx Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Johannes Schindelin committed Dec 11, 2016 at 12:16 UTC cbb3f3c9b1975c9bdd07f24fc4ef4e504507adaa
2 files changed +36
compat/mingw.h
+3
@@ -384,6 +384,9 @@ int mingw_raise(int sig);
384 * ANSI emulation wrappers
385 */
386
387 +int winansi_isatty(int fd);
388 +#define isatty winansi_isatty
389 +
390 void winansi_init(void);
391 HANDLE winansi_get_osfhandle(int fd);
392
compat/winansi.c
+33
@@ -7,6 +7,9 @@
7 #include <wingdi.h>
8 #include <winreg.h>
9
10 +/* In this file, we actually want to use Windows' own isatty(). */
11 +#undef isatty
12 +
13 /*
14 ANSI codes used by git: m, K
15
@@ -570,6 +573,36 @@ static void detect_msys_tty(int fd)
573
574 #endif
575
576 +int winansi_isatty(int fd)
577 +{
578 + int res = isatty(fd);
579 +
580 + if (res) {
581 + /*
582 + * Make sure that /dev/null is not fooling Git into believing
583 + * that we are connected to a terminal, as "_isatty() returns a
584 + * nonzero value if the descriptor is associated with a
585 + * character device."; for more information, see
586 + *
587 + * https://msdn.microsoft.com/en-us/library/f4s0ddew.aspx
588 + */
589 + HANDLE handle = (HANDLE)_get_osfhandle(fd);
590 + if (fd == STDIN_FILENO) {
591 + DWORD dummy;
592 +
593 + if (!GetConsoleMode(handle, &dummy))
594 + res = 0;
595 + } else if (fd == STDOUT_FILENO || fd == STDERR_FILENO) {
596 + CONSOLE_SCREEN_BUFFER_INFO dummy;
597 +
598 + if (!GetConsoleScreenBufferInfo(handle, &dummy))
599 + res = 0;
600 + }
601 + }
602 +
603 + return res;
604 +}
605 +
606 void winansi_init(void)
607 {
608 int con1, con2;