docs: add Thunderdome docs for releases (#9872)
Gus Eggert committed
May 14, 2023 at 23:22 UTC
4acadd49ebc445d86afb678477abad880fec36cf
2 files changed
+63
docs/RELEASE_ISSUE_TEMPLATE.md
+3
@@ -63,6 +63,7 @@ This section covers tasks to be done ahead of the release.
63
- open an access request in the [pldw](https://github.com/protocol/pldw/issues/new/choose)
64
- [example](https://github.com/protocol/pldw/issues/158)
65
- [ ] [kuboreleaser](https://github.com/ipfs/kuboreleaser) checked out on your system (_only if you're using [kuboreleaser](https://github.com/ipfs/kuboreleaser)_)
66
+ - [ ] [Thunderdome](https://github.com/ipfs-shipyard/thunderdome) checked out on your system and configured (see the [Thunderdome release docs](./releases_thunderdome.md) for setup)
67
- [ ] [docker](https://docs.docker.com/get-docker/) installed on your system (_only if you're using [kuboreleaser](https://github.com/ipfs/kuboreleaser)_)
68
- [ ] [npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) installed on your system (_only if you're **NOT** using [kuboreleaser](https://github.com/ipfs/kuboreleaser)_)
69
- [ ] [zsh](https://github.com/ohmyzsh/ohmyzsh/wiki/Installing-ZSH#install-and-set-up-zsh-as-default) installed on your system
@@ -102,6 +103,8 @@ This section covers tasks to be done during each release.
103
- do **NOT** use `Squash and merge` nor `Rebase and merge` because we need to be able to sign the merge commit
104
- do **NOT** delete the `release-vX.Y` branch
105
</details>
106
+- [ ] Run Thunderdome testing, see the [Thunderdome release docs](./releases_thunderdome.md) for details
107
+ - [ ] create a PR and merge the experiment config into Thunderdome
108
- [ ] Create the release tag <details><summary>using `kuboreleaser release --version vX.Y.Z(-rcN) tag` or ...</summary>
109
- This is a dangerous operation! Go and Docker publishing are difficult to reverse! Have the release reviewer verify all the commands marked with ⚠️!
110
- [ ] ⚠️  tag the HEAD commit using `git tag -s vX.Y.Z(-RCN) -m 'Prerelease X.Y.Z(-RCN)'`
docs/releases_thunderdome.md
new
+60
@@ -0,0 +1,60 @@
1
+# Testing Kubo releases with Thunderdome
2
+This document is for running Thunderdome tests by release engineers as part of releasing Kubo.
3
+
4
+We use Thunderdome to replay ipfs.io gateway traffic in a controlled environment against two different versions of Kubo, and we record metrics and compare them to look for logic or performance regressions before releasing a new Kubo version.
5
+
6
+For background information about how Thunderdome works, see: https://github.com/ipfs-shipyard/thunderdome
7
+
8
+## Prerequisites
9
+
10
+* Ensure you have access to the "IPFS Stewards" vault in 1Password, which contains the requisite AWS Console and API credentials
11
+* Ensure you have Docker and the Docker CLI installed
12
+* Checkout the Thunderdome repo locally (or `git pull` to ensure it's up-to-date)
13
+* Install AWS CLI v2: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
14
+* Configure the AWS CLI
15
+ * Configure the credentials as described in the [Thunderdome documentation](https://github.com/ipfs-shipyard/thunderdome/blob/main/cmd/thunderdome/README.md#credentials), using the credentials from 1Password
16
+* Make sure the `thunderdome` binary is up-to-date: `go build ./cmd/thunderdome`
17
+
18
+## Add & run an experiment
19
+
20
+Create a new release configuration JSON in the `experiments/` directory, based on the most recent `kubo-release` configuration, and tweak as necessary. Generally we setup the targets to run a commit against the tag of the last release, such as:
21
+
22
+```json
23
+ "targets": [
24
+ {
25
+ "name": "kubo190-4283b9",
26
+ "description": "kubo 0.19.0-rc1",
27
+ "build_from_git": {
28
+ "repo": "https://github.com/ipfs/kubo.git",
29
+ "commit":"4283b9d98f8438fc8751ccc840d8fc24eeae6f13"
30
+ }
31
+ },
32
+ {
33
+ "name": "kubo181",
34
+ "description": "kubo 0.18.",
35
+ "build_from_git": {
36
+ "repo": "https://github.com/ipfs/kubo.git",
37
+ "tag":"v0.18.1"
38
+ }
39
+ }
40
+ ]
41
+```
42
+
43
+Run the experiment (where `$EXPERIMENT_CONFIG_JSON` is a path to the config JSON created above):
44
+
45
+```shell
46
+AWS_PROFILE=thunderdome ./thunderdome deploy --verbose --duration 120 $EXPERIMENT_CONFIG_JSON
47
+```
48
+
49
+This will build the Docker images, upload them to ECR, and then launch the experiment in Thunderdome. Once the experiment starts, the CLI will exit and the experiment will continue to run for the duration.
50
+
51
+## Analyze Results
52
+
53
+Add a log entry in https://www.notion.so/pl-strflt/ce2d1bd56f3541028d960d3711465659 and link to it from the release issue, so that experiment results are publicly visible.
54
+
55
+The `deploy` command will output a link to the Grafana dashboard for the experiment. We don't currently have rigorous acceptance criteria, so you should look for anomalies or changes in the metrics and make sure they are tolerable and explainable. Unexplainable anomalies should be noted in the log with a screenshot, and then root caused.
56
+
57
+
58
+## Open a PR to merge the experiment config into Thunderdome
59
+
60
+This is important for both posterity, and so that someone else can sanity-check the test parameters.