Add README.md
Eelco Dolstra committed
Jan 24, 2023 at 13:53 UTC
3ce43a1fb5181a0e33b1f67d36fa0f3affa6bc6c
1 file changed
+120
metrics/fastly/README.md
new
+120
@@ -0,0 +1,120 @@
1
+# Fastly log processing
2
+
3
+This flake provides a systemd timer (`./cron.sh`) that every week:
4
+
5
+* Ingests raw Fastly logs for
6
+ {cache,channels,tarballs,releases}.nixos.org (which are very big)
7
+ and aggregates them into a smaller AWS Athena database.
8
+
9
+ This is performed by `./ingest-raw-logs.sh`.
10
+
11
+* Runs a number of SQL queries against the Athena database and stores
12
+ them in S3.
13
+
14
+ This is performed by `./run-queries.sh`.
15
+
16
+## AWS Athena database
17
+
18
+The Athena database is stored in the NixOS Foundation AWS account. To
19
+get the schema, run
20
+
21
+```
22
+# aws athena list-table-metadata --region eu-west-1 --catalog-name AwsDataCatalog --database-name default
23
+```
24
+
25
+It has the following external tables:
26
+
27
+* `requests`: An external table. These are the raw fastly logs stored
28
+ in s3://fastly-logs-20220622145016462800000001/ as compressed JSON
29
+ records. Note that this bucket has a lifecycle rule that moves logs
30
+ to Glacier after a few weeks. Logs in Glacier are not processed by
31
+ Athena.
32
+
33
+* `asn_list`: A list of ASNs. This can be updated by running
34
+ `./update-asn-list.sh`.
35
+
36
+* `hosting-asns`: A list of ASNs belonging to hosting/cloud providers.
37
+
38
+* `all_paths`: The set of all store paths known in the hydra.nixos.org
39
+ database. This is used to expand the hash part of `.narinfo` requests
40
+ (e.g. `8kbx6s9nn7060zsdms3br0mk7bjrvbij`) to store paths
41
+ (e.g. `/nix/store/8kbx6s9nn7060zsdms3br0mk7bjrvbij-coreutils-full-9.0`).
42
+
43
+ FIXME: describe how to update.
44
+
45
+* `release_paths`: All the store paths belonging to NixOS evals in
46
+ hydra.nixos.org, as `{project, jobset, eval, release_name, build,
47
+ output, path}` tuples.
48
+
49
+ FIXME: describe how to update.
50
+
51
+The ingestion script populates the following tables stored in
52
+s3://nixos-athena/fastly-logs-processed/:
53
+
54
+* `urls`: For each host/day/url, the total number of requests, bytes
55
+ and elapsed microseconds. This only includes info about successful
56
+ (2xx/3xx) requests.
57
+
58
+* `clients`: For each host/day/ASN/country/region, the total number of
59
+ requests, bytes and elapsed microseconds.
60
+
61
+* `nix_cache_info`: For each day/ASN/country/region/user-agent, the
62
+ number of requests for `nix-cache-info`.
63
+
64
+## Reports
65
+
66
+Currently the following reports are created every week:
67
+
68
+* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/traffic-per-day.csv
69
+
70
+ For each day and site, the number of requests and the number of
71
+ bytes transferred.
72
+
73
+* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/traffic-per-country.csv
74
+
75
+ For each country, the number of requests and the number of
76
+ bytes transferred.
77
+
78
+* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day.csv
79
+
80
+ For each day, the number of requests for
81
+ https://cache.nixos.org/nix-cache-info.
82
+
83
+* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day-not-hosted.csv
84
+
85
+ The same, but with requests from "hosting" ASNs (e.g. AWS and
86
+ Hetzner) filtered out. Note that Nix caches `nix-cache-info` file
87
+ for a week, so the intent of this report is to gauge the number of
88
+ active weekly users.
89
+
90
+* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day-per-ua.csv
91
+
92
+ For each day and user agent (e.g. `Nix/2.12.0`), the number of
93
+ requests for https://cache.nixos.org/nix-cache-info. This is
94
+ intended to track the adoption of Nix releases.
95
+
96
+* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/flake-registry-requests-per-day.csv
97
+
98
+ For each day, the number of requests for
99
+ https://channels.nixos.org/flake-registry.json. This is intended to
100
+ track how widely flakes are used.
101
+
102
+* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/top-store-paths.csv
103
+
104
+ For each store path listed in `all_paths`, the number of requests for its `.narinfo` file.
105
+
106
+* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/narinfo-queries-per-release.csv
107
+
108
+ For each major NixOS release (e.g. `nixos-22.05`), the number of
109
+ requests for `.narinfo` files of store paths that are part of an
110
+ eval of that release.
111
+
112
+* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/nix-installer-downloads.csv
113
+
114
+ For each day, the number of downloads of the Nix installer
115
+ (i.e. `https://releases.nixos.org/nix/nix-[^/]+/install`).
116
+
117
+* http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/nix-installer-architectures.csv
118
+
119
+ For each architecture (e.g. `x86_64-linux`), the number of downloads
120
+ of the Nix binary tarball.