ci: use the new GitHub Action to download git-sdk-64-minimal

In our continuous builds, Windows is the odd cookie that requires a complete development environment to be downloaded because there is no suitable one installed by default on Windows. Side note: technically, there _is_ a development environment present in GitHub Actions' build agents: MSYS2. But it differs from Git for Windows' SDK in subtle points, unfortunately enough so to prevent Git's test suite from running without failures. Traditionally, we support downloading this environment (which we nicknamed `git-sdk-64-minimal`) via a PowerShell scriptlet that accesses the build artifacts of a dedicated Azure Pipeline (which packages a tiny subset of the full Git for Windows SDK, containing just enough to build Git and run its test suite). This PowerShell script is unfortunately not very robust and sometimes fails due to network issues. Of course, we could add code to detect that situation, wait a little, try again, if it fails again wait a little longer, lather, rinse and repeat. Instead of doing all of this in Git's own `.github/workflows/`, though, let's offload this logic to the new GitHub Action at https://github.com/marketplace/actions/setup-git-for-windows-sdk This Action not only downloads and extracts git-sdk-64-minimal _outside_ the worktree (making it no longer necessary to meddle with `.gitignore` or `.git/info/exclude`), it also adds the `bash.exe` to the `PATH` and sets the environment variable `MSYSTEM` (an implementation detail that Git's workflow should never have needed to know about). This allows us to convert all those funny PowerShell tasks that wanted to call git-sdk-64-minimal's `bash.exe`: they all are now regular `bash` scriptlets. This finally lets us get rid of the funny quoting and escaping where we had to pay attention not only to quote and escape the Bash scriptlets properly, but also to add a second level of escaping (with backslashes for double quotes and backticks for dollar signs) to stop PowerShell from doing unintended things. Further, this Action uses a fast caching strategy native to GitHub Actions that should accelerate the download across CI runs: git-sdk-64-minimal is usually updated once per 24h, and needs to be cached only once within that period. Caching it (unfortunately only on a per-branch basis) speeds up the download step, and makes it much more robust at the same time by virtue of accessing a cache location that is closer in the network topology. With this we can drop the home-rolled caching where we try to accelerate the test phase by uploading git-sdk-64-minimal as a workflow artifact after using it to build Git, and then download it as workflow artifact in the test phase. Even better: the `vs-test` job no longer needs to depend on the `windows-build` job. The only reason it depended on it was to ensure that the `git-sdk-64-minimal` workflow artifact was available. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Johannes Schindelin committed Jun 23, 2021 at 15:24 UTC 0eb6c189a3576aa8745f471c1ad23b3ca123cc75
1 file changed +19 -84
.github/workflows/main.yml
+19 -84
@@ -82,43 +82,18 @@ jobs:
82 runs-on: windows-latest
83 steps:
84 - uses: actions/checkout@v1
85 - - name: download git-sdk-64-minimal
86 - shell: bash
87 - run: |
88 - ## Get artifact
89 - urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
90 - id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
91 - jq -r ".value[] | .id")
92 - download_url="$(curl "$urlbase/$id/artifacts" |
93 - jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
94 - curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
95 - -o artifacts.zip "$download_url"
96 -
97 - ## Unzip and remove the artifact
98 - unzip artifacts.zip
99 - rm artifacts.zip
85 + - uses: git-for-windows/setup-git-for-windows-sdk@v1
86 - name: build
101 - shell: powershell
87 + shell: bash
88 env:
89 HOME: ${{runner.workspace}}
104 - MSYSTEM: MINGW64
90 NO_PERL: 1
106 - run: |
107 - & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @"
108 - printf '%s\n' /git-sdk-64-minimal/ >>.git/info/exclude
109 -
110 - ci/make-test-artifacts.sh artifacts
111 - "@
91 + run: ci/make-test-artifacts.sh artifacts
92 - name: upload build artifacts
93 uses: actions/upload-artifact@v1
94 with:
95 name: windows-artifacts
96 path: artifacts
117 - - name: upload git-sdk-64-minimal
118 - uses: actions/upload-artifact@v1
119 - with:
120 - name: git-sdk-64-minimal
121 - path: git-sdk-64-minimal
97 windows-test:
98 runs-on: windows-latest
99 needs: [windows-build]
@@ -136,25 +111,14 @@ jobs:
111 - name: extract build artifacts
112 shell: bash
113 run: tar xf artifacts.tar.gz
139 - - name: download git-sdk-64-minimal
140 - uses: actions/download-artifact@v1
141 - with:
142 - name: git-sdk-64-minimal
143 - path: ${{github.workspace}}/git-sdk-64-minimal/
114 + - uses: git-for-windows/setup-git-for-windows-sdk@v1
115 - name: test
145 - shell: powershell
146 - run: |
147 - & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @"
148 - # Let Git ignore the SDK
149 - printf '%s\n' /git-sdk-64-minimal/ >>.git/info/exclude
150 -
151 - ci/run-test-slice.sh ${{matrix.nr}} 10
152 - "@
116 + shell: bash
117 + run: ci/run-test-slice.sh ${{matrix.nr}} 10
118 - name: ci/print-test-failures.sh
119 if: failure()
155 - shell: powershell
156 - run: |
157 - & .\git-sdk-64-minimal\usr\bin\bash.exe -lc ci/print-test-failures.sh
120 + shell: bash
121 + run: ci/print-test-failures.sh
122 - name: Upload failed tests' directories
123 if: failure() && env.FAILED_TEST_ARTIFACTS != ''
124 uses: actions/upload-artifact@v1
@@ -165,27 +129,12 @@ jobs:
129 needs: ci-config
130 if: needs.ci-config.outputs.enabled == 'yes'
131 env:
168 - MSYSTEM: MINGW64
132 NO_PERL: 1
133 GIT_CONFIG_PARAMETERS: "'user.name=CI' 'user.email=ci@git'"
134 runs-on: windows-latest
135 steps:
136 - uses: actions/checkout@v1
174 - - name: download git-sdk-64-minimal
175 - shell: bash
176 - run: |
177 - ## Get artifact
178 - urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
179 - id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
180 - jq -r ".value[] | .id")
181 - download_url="$(curl "$urlbase/$id/artifacts" |
182 - jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
183 - curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
184 - -o artifacts.zip "$download_url"
185 -
186 - ## Unzip and remove the artifact
187 - unzip artifacts.zip
188 - rm artifacts.zip
137 + - uses: git-for-windows/setup-git-for-windows-sdk@v1
138 - name: initialize vcpkg
139 uses: actions/checkout@v2
140 with:
@@ -211,19 +160,17 @@ jobs:
160 shell: bash
161 run: |
162 cmake `pwd`/contrib/buildsystems/ -DCMAKE_PREFIX_PATH=`pwd`/compat/vcbuild/vcpkg/installed/x64-windows \
214 - -DMSGFMT_EXE=`pwd`/git-sdk-64-minimal/mingw64/bin/msgfmt.exe -DPERL_TESTS=OFF -DPYTHON_TESTS=OFF -DCURL_NO_CURL_CMAKE=ON
163 + -DMSGFMT_EXE=C:/git-sdk-64-minimal/mingw64/bin/msgfmt.exe -DPERL_TESTS=OFF -DPYTHON_TESTS=OFF -DCURL_NO_CURL_CMAKE=ON
164 - name: MSBuild
165 run: msbuild git.sln -property:Configuration=Release -property:Platform=x64 -maxCpuCount:4 -property:PlatformToolset=v142
166 - name: bundle artifact tar
218 - shell: powershell
167 + shell: bash
168 env:
169 MSVC: 1
170 VCPKG_ROOT: ${{github.workspace}}\compat\vcbuild\vcpkg
171 run: |
223 - & git-sdk-64-minimal\usr\bin\bash.exe -lc @"
224 - mkdir -p artifacts &&
225 - eval \"`$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts 2>&1 | grep ^tar)\"
226 - "@
172 + mkdir -p artifacts &&
173 + eval "$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts 2>&1 | grep ^tar)"
174 - name: upload build artifacts
175 uses: actions/upload-artifact@v1
176 with:
@@ -231,18 +178,14 @@ jobs:
178 path: artifacts
179 vs-test:
180 runs-on: windows-latest
234 - needs: [vs-build, windows-build]
181 + needs: vs-build
182 strategy:
183 fail-fast: false
184 matrix:
185 nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
186 steps:
187 - uses: actions/checkout@v1
241 - - name: download git-sdk-64-minimal
242 - uses: actions/download-artifact@v1
243 - with:
244 - name: git-sdk-64-minimal
245 - path: ${{github.workspace}}/git-sdk-64-minimal/
188 + - uses: git-for-windows/setup-git-for-windows-sdk@v1
189 - name: download build artifacts
190 uses: actions/download-artifact@v1
191 with:
@@ -252,23 +195,15 @@ jobs:
195 shell: bash
196 run: tar xf artifacts.tar.gz
197 - name: test
255 - shell: powershell
198 + shell: bash
199 env:
257 - MSYSTEM: MINGW64
200 NO_SVN_TESTS: 1
201 GIT_TEST_SKIP_REBASE_P: 1
260 - run: |
261 - & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @"
262 - # Let Git ignore the SDK and the test-cache
263 - printf '%s\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude
264 -
265 - ci/run-test-slice.sh ${{matrix.nr}} 10
266 - "@
202 + run: ci/run-test-slice.sh ${{matrix.nr}} 10
203 - name: ci/print-test-failures.sh
204 if: failure()
269 - shell: powershell
270 - run: |
271 - & .\git-sdk-64-minimal\usr\bin\bash.exe -lc ci/print-test-failures.sh
205 + shell: bash
206 + run: ci/print-test-failures.sh
207 - name: Upload failed tests' directories
208 if: failure() && env.FAILED_TEST_ARTIFACTS != ''
209 uses: actions/upload-artifact@v1