@samitouri / QOS-React-1 / commits / 80efd75b8e

[ci] Add benchmark comparison

Updates the parser-benchmark script to use a benchmarking framework, and also brings in yargs to make it easier to handle parsing arguments. Non-CI usage: ``` $ cd bench/parser-benchmark $ yarn bench --help Options: --help Show help [boolean] --ci [boolean] [default: false] --sampleSize [number] [default: 1] $ yarn bench --sampleSize=3 [ISOBENCH ENDED] Parser Benchmark (3 times) OXC - 32 op/s. 3 samples in 9518 ms. 5.288x (BEST) SWC - 20 op/s. 3 samples in 9487 ms. 3.256x HermesParser - 6 op/s. 3 samples in 9124 ms. 1.000x (WORST) Forget (Rust) - 15 op/s. 3 samples in 9749 ms. 2.499x ``` In CI this outputs JSON instead of logging to stdout. The results are stored in github's action cache and used as a point of comparison when new commits are pushed to main. The comparison should be shown on workflow [summary pages](https://github.com/facebook/react-forget/actions/runs/5956421081), but nothing will be populated until we merge this PR. We intentionally only run this workflow on main as any run would override the last cached result, so a PR with multiple pushes would then start having comparisons with itself.

Lauren Tan committed Aug 23, 2023 at 17:58 UTC 80efd75b8e5f83a5473afa0dc45f49428e2fce6f
2 files changed +54 -18
compiler/.github/workflows/bench.yml new
+54
@@ -0,0 +1,54 @@
1 +# Runs the parser benchmark on every push to main, and compares it with the previous run
2 +name: Parser Benchmark (Rust)
3 +
4 +on:
5 + push:
6 + branches: ["main"]
7 + paths:
8 + - .github/workflows/**
9 + - crates/**
10 + - Cargo.*
11 + - ./*.toml
12 +
13 +env:
14 + CARGO_TERM_COLOR: always
15 + RUSTFLAGS: -Dwarnings
16 +
17 +jobs:
18 + bench:
19 + name: Parser Benchmark
20 + runs-on: ubuntu-latest
21 + steps:
22 + - uses: actions/checkout@v3
23 + - uses: Swatinem/rust-cache@v2
24 + - run: yarn install --frozen-lockfile
25 + working-directory: crates/forget_napi
26 + - run: mkdir dist -p
27 + working-directory: crates/forget_napi
28 + - run: yarn build
29 + working-directory: crates/forget_napi
30 + - run: yarn install --frozen-lockfile
31 + working-directory: bench/parser-benchmark
32 + - name: Run benchmark
33 + run: yarn bench --ci --sampleSize=3 | tee results.json
34 + working-directory: bench/parser-benchmark
35 + # Download previous benchmark result from cache (if exists)
36 + - name: Download previous benchmark data
37 + uses: actions/cache@v3
38 + with:
39 + path: ./cache
40 + key: ${{ runner.os }}-benchmark
41 + - name: Store benchmark result
42 + uses: benchmark-action/github-action-benchmark@v1
43 + with:
44 + name: "${{ runner.os }} Benchmark"
45 + tool: customBiggerIsBetter
46 + output-file-path: bench/parser-benchmark/results.json
47 + # Where the previous data file is stored
48 + external-data-json-path: ./cache/parser-benchmark-results.json
49 + # Workflow will fail when an alert happens
50 + fail-on-alert: true
51 + # Enable Job Summary for PRs
52 + summary-always: true
53 + # GitHub API token to make a commit comment
54 + github-token: ${{ secrets.GITHUB_TOKEN }}
\ No newline at end of file
compiler/.github/workflows/rust.yml
-18
@@ -66,21 +66,3 @@ jobs:
66 - uses: Swatinem/rust-cache@v2
67 - name: cargo build
68 run: cargo build --release
69 -
70 - bench:
71 - name: Parser Benchmark
72 - runs-on: ubuntu-latest
73 - steps:
74 - - uses: actions/checkout@v3
75 - - uses: Swatinem/rust-cache@v2
76 - - run: yarn install --frozen-lockfile
77 - working-directory: crates/forget_napi
78 - - run: mkdir dist -p
79 - working-directory: crates/forget_napi
80 - - run: yarn build
81 - working-directory: crates/forget_napi
82 - - run: yarn install --frozen-lockfile
83 - working-directory: bench/parser-benchmark
84 - - name: Run benchmark
85 - run: echo "$(node index.js --ci)" >> $GITHUB_STEP_SUMMARY
86 - working-directory: bench/parser-benchmark
\ No newline at end of file