| 1 | Content-type: text/asciidoc |
| 2 | Abstract: When a vulnerability is reported, we follow these guidelines to |
| 3 | assess the vulnerability, create and review a fix, and coordinate embargoed |
| 4 | security releases. |
| 5 | |
| 6 | How we coordinate embargoed releases |
| 7 | ------------------------------------ |
| 8 | |
| 9 | To protect Git users from critical vulnerabilities, we do not just release |
| 10 | fixed versions like regular maintenance releases. Instead, we coordinate |
| 11 | releases with packagers, keeping the fixes under an embargo until the release |
| 12 | date. That way, users will have a chance to upgrade on that date, no matter |
| 13 | what Operating System or distribution they run. |
| 14 | |
| 15 | The `git-security` mailing list |
| 16 | ------------------------------- |
| 17 | |
| 18 | Responsible disclosures of vulnerabilities, analysis, proposed fixes as |
| 19 | well as the orchestration of coordinated embargoed releases all happen on the |
| 20 | `git-security` mailing list at <git-security@googlegroups.com>. |
| 21 | |
| 22 | In this context, the term "embargo" refers to the time period that information |
| 23 | about a vulnerability is kept under wraps and only shared on a need-to-know |
| 24 | basis. This is necessary to protect Git's users from bad actors who would |
| 25 | otherwise be made aware of attack vectors that could be exploited. "Lifting the |
| 26 | embargo" refers to publishing the version that fixes the vulnerabilities. |
| 27 | |
| 28 | Audience of the `git-security` mailing list |
| 29 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 30 | |
| 31 | Anybody may contact the `git-security` mailing list by sending an email |
| 32 | to <git-security@googlegroups.com>, though the archive is closed to the |
| 33 | public and only accessible to subscribed members. |
| 34 | |
| 35 | There are a few dozen subscribed members: core Git developers who are trusted |
| 36 | with addressing vulnerabilities, and stakeholders (i.e. owners of products |
| 37 | affected by security vulnerabilities in Git). |
| 38 | |
| 39 | Most of the discussions revolve around assessing the severity of the reported |
| 40 | issue (including the decision whether the report is security-relevant or can be |
| 41 | redirected to the public mailing list), how to remediate the issue, determining |
| 42 | the timeline of the disclosure as well as aligning priorities and |
| 43 | requirements. |
| 44 | |
| 45 | Communications |
| 46 | ~~~~~~~~~~~~~~ |
| 47 | |
| 48 | If you are a stakeholder, it is a good idea to pay close attention to the |
| 49 | discussions, as pertinent information may be buried in the middle of a lively |
| 50 | conversation that might not look relevant to your interests. For example, the |
| 51 | tentative timeline might be agreed upon in the middle of discussing code |
| 52 | comment formatting in one of the patches and whether or not to combine fixes |
| 53 | for multiple, separate vulnerabilities into the same embargoed release. Most |
| 54 | mail threads are not usually structured specifically to communicate |
| 55 | agreements, assessments or timelines. |
| 56 | |
| 57 | Typical timeline |
| 58 | ---------------- |
| 59 | |
| 60 | - A potential vulnerability is reported to the `git-security` mailing list. |
| 61 | |
| 62 | - The members of the git-security list start a discussion to give an initial |
| 63 | assessment of the severity of the reported potential vulnerability. |
| 64 | We aspire to do so within a few days. |
| 65 | |
| 66 | - After discussion, if consensus is reached that it is not critical enough |
| 67 | to warrant any embargo, the reporter is redirected to the public Git mailing |
| 68 | list. This ends the reporter's interaction with the `git-security` list. |
| 69 | |
| 70 | - If it is deemed critical enough for an embargo, ideas are presented on how to |
| 71 | address the vulnerability. |
| 72 | |
| 73 | - Usually around that time, the Git maintainer or their delegate(s) open a draft |
| 74 | security advisory in the `git/git` repository on GitHub (see below for more |
| 75 | details). |
| 76 | |
| 77 | - Code review can take place in a variety of different locations, |
| 78 | depending on context. These are: patches sent inline on the git-security list, |
| 79 | a private fork on GitHub associated with the draft security advisory, or the |
| 80 | git/cabal repository. |
| 81 | |
| 82 | - Contributors working on a fix should consider beginning by sending |
| 83 | patches to the git-security list (inline with the original thread), since they |
| 84 | are accessible to all subscribers, along with the original reporter. |
| 85 | |
| 86 | - Once the review has settled and everyone involved in the review agrees that |
| 87 | the patches are nearing the finish line, the Git maintainer, and others |
| 88 | determine a release date as well as the release trains that are serviced. The |
| 89 | decision regarding which versions need a backported fix is based on input from |
| 90 | the reporter, the contributor who worked on the patches, and from |
| 91 | stakeholders. Operators of hosting sites who may want to analyze whether the |
| 92 | given issue is exploited via any of the repositories they host, and binary |
| 93 | packagers who want to make sure their product gets patched adequately against |
| 94 | the vulnerability, for example, may want to give their input at this stage. |
| 95 | |
| 96 | - While the Git community does its best to accommodate the specific timeline |
| 97 | requests of the various binary packagers, the nature of the issue may preclude |
| 98 | a prolonged release schedule. For fixes deemed urgent, it may be in the best |
| 99 | interest of the Git users community to shorten the disclosure and release |
| 100 | timeline, and packagers may need to adapt accordingly. |
| 101 | |
| 102 | - Subsequently, branches with the fixes are pushed to the git/cabal repository. |
| 103 | |
| 104 | - The tags are created by the Git maintainer and pushed to the same repository. |
| 105 | |
| 106 | - The Git for Windows, Git for macOS, BSD, Debian, etc. maintainers prepare the |
| 107 | corresponding release artifacts, based on the tags created that have been |
| 108 | prepared by the Git maintainer. |
| 109 | |
| 110 | - The release artifacts prepared by various binary packagers can be |
| 111 | made available to stakeholders under embargo via a mail to the |
| 112 | `git-security` list. |
| 113 | |
| 114 | - Less than a week before the release, a mail with the relevant information is |
| 115 | sent to <distros@vs.openwall.org> (see below), a list used to pre-announce |
| 116 | embargoed releases of open source projects to the stakeholders of all major |
| 117 | distributions of Linux as well as other OSes. |
| 118 | |
| 119 | - Public communication is then prepared in advance of the release date. This |
| 120 | includes blog posts and mails to the Git and Git for Windows mailing lists. |
| 121 | |
| 122 | - On the day of the release, at around 10am Pacific Time, the Git maintainer |
| 123 | pushes the tag and the `master` branch to the public repository, then sends |
| 124 | out an announcement mail. |
| 125 | |
| 126 | - Once the tag is pushed, the Git for Windows maintainer publishes the |
| 127 | corresponding tag and creates a GitHub Release with the associated release |
| 128 | artifacts (Git for Windows installer, Portable Git, MinGit, etc). |
| 129 | |
| 130 | - Git for Windows release is then announced via a mail to the public Git and |
| 131 | Git for Windows mailing lists as well as via a tweet. |
| 132 | |
| 133 | - Ditto for distribution packagers for Linux and other platforms: |
| 134 | their releases are announced via their preferred channels. |
| 135 | |
| 136 | - A mail to <oss-security@lists.openwall.org> (see below for details) is sent |
| 137 | as a follow-up to the <distros@vs.openwall.org> one, describing the |
| 138 | vulnerability in detail, often including a proof of concept of an exploit. |
| 139 | |
| 140 | Note: The Git project makes no guarantees about timelines, but aims to keep |
| 141 | embargoes reasonably short in the interest of keeping Git's users safe. |
| 142 | |
| 143 | Opening a Security Advisory draft |
| 144 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 145 | |
| 146 | The first step is to https://github.com/git/git/security/advisories/new[open |
| 147 | an advisory]. Technically, this is not necessary. However, it is the most |
| 148 | convenient way to obtain the CVE number and it gives us a private repository |
| 149 | associated with it that can be used to collaborate on a fix. |
| 150 | |
| 151 | Notifying the Linux distributions |
| 152 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 153 | |
| 154 | At most two weeks before release date, we need to send a notification to |
| 155 | <distros@vs.openwall.org>, preferably less than 7 days before the release date. |
| 156 | This will reach most (all?) Linux distributions. See an example below, and the |
| 157 | guidelines for this mailing list at |
| 158 | https://oss-security.openwall.org/wiki/mailing-lists/distros#how-to-use-the-lists[here]. |
| 159 | |
| 160 | Once the version has been published, we send a note about that to oss-security. |
| 161 | As an example, see https://www.openwall.com/lists/oss-security/2019/12/13/1[the |
| 162 | v2.24.1 mail]; |
| 163 | https://oss-security.openwall.org/wiki/mailing-lists/oss-security[Here] are |
| 164 | their guidelines. |
| 165 | |
| 166 | The mail to oss-security should also describe the exploit, and give credit to |
| 167 | the reporter(s): security researchers still receive too little respect for the |
| 168 | invaluable service they provide, and public credit goes a long way to keep them |
| 169 | paid by their respective organizations. |
| 170 | |
| 171 | Technically, describing any exploit can be delayed up to 7 days, but we usually |
| 172 | refrain from doing that, including it right away. |
| 173 | |
| 174 | As a courtesy we typically attach a Git bundle (as `.tar.xz` because the list |
| 175 | will drop `.bundle` attachments) in the mail to distros@ so that the involved |
| 176 | parties can take care of integrating/backporting them. This bundle is typically |
| 177 | created using a command like this: |
| 178 | |
| 179 | git bundle create cve-xxx.bundle ^origin/master vA.B.C vD.E.F |
| 180 | tar cJvf cve-xxx.bundle.tar.xz cve-xxx.bundle |
| 181 | |
| 182 | Example mail to distros@vs.openwall.org |
| 183 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 184 | |
| 185 | .... |
| 186 | To: distros@vs.openwall.org |
| 187 | Cc: git-security@googlegroups.com, <other people involved in the report/fix> |
| 188 | Subject: [vs] Upcoming Git security fix release |
| 189 | |
| 190 | Team, |
| 191 | |
| 192 | The Git project will release new versions on <date> at 10am Pacific Time or |
| 193 | soon thereafter. I have attached a Git bundle (embedded in a `.tar.xz` to avoid |
| 194 | it being dropped) which you can fetch into a clone of |
| 195 | https://github.com/git/git via `git fetch --tags /path/to/cve-xxx.bundle`, |
| 196 | containing the tags for versions <versions>. |
| 197 | |
| 198 | You can verify with `git tag -v <tag>` that the versions were signed by |
| 199 | the Git maintainer, using the same GPG key as e.g. v2.24.0. |
| 200 | |
| 201 | Please use these tags to prepare `git` packages for your various |
| 202 | distributions, using the appropriate tagged versions. The added test cases |
| 203 | help verify the correctness. |
| 204 | |
| 205 | The addressed issues are: |
| 206 | |
| 207 | <list of CVEs with a short description, typically copy/pasted from Git's |
| 208 | release notes, usually demo exploit(s), too> |
| 209 | |
| 210 | Credit for finding the vulnerability goes to <reporter>, credit for fixing |
| 211 | it goes to <developer>. |
| 212 | |
| 213 | Thanks, |
| 214 | <name> |
| 215 | |
| 216 | .... |
| 217 | |
| 218 | Example mail to oss-security@lists.openwall.com |
| 219 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 220 | |
| 221 | .... |
| 222 | To: oss-security@lists.openwall.com |
| 223 | Cc: git-security@googlegroups.com, <other people involved in the report/fix> |
| 224 | Subject: git: <copy from security advisory> |
| 225 | |
| 226 | Team, |
| 227 | |
| 228 | The Git project released new versions on <date>, addressing <CVE>. |
| 229 | |
| 230 | All supported platforms are affected in one way or another, and all Git |
| 231 | versions all the way back to <version> are affected. The fixed versions are: |
| 232 | <versions>. |
| 233 | |
| 234 | Link to the announcement: <link to lore.kernel.org/git> |
| 235 | |
| 236 | We highly recommend to upgrade. |
| 237 | |
| 238 | The addressed issues are: |
| 239 | * <list of CVEs and their explanations, along with demo exploits> |
| 240 | |
| 241 | Credit for finding the vulnerability goes to <reporter>, credit for fixing |
| 242 | it goes to <developer>. |
| 243 | |
| 244 | Thanks, |
| 245 | <name> |
| 246 | .... |