| 1 | # Fastly log processing |
| 2 | |
| 3 | This flake provides a systemd timer (`./cron.sh`) that every week: |
| 4 | |
| 5 | - Ingests raw Fastly logs for {cache,channels,tarballs,releases}.nixos.org |
| 6 | (which are very big) and aggregates them into a smaller AWS Athena database. |
| 7 | |
| 8 | This is performed by `./ingest-raw-logs.sh`. |
| 9 | |
| 10 | - Runs a number of SQL queries against the Athena database and stores them in |
| 11 | S3. |
| 12 | |
| 13 | This is performed by `./run-queries.sh`. |
| 14 | |
| 15 | ## AWS Athena database |
| 16 | |
| 17 | The Athena database is stored in the NixOS Foundation AWS account. To get the |
| 18 | schema, run |
| 19 | |
| 20 | ``` |
| 21 | # aws athena list-table-metadata --region eu-west-1 --catalog-name AwsDataCatalog --database-name default |
| 22 | ``` |
| 23 | |
| 24 | It has the following external tables: |
| 25 | |
| 26 | - `requests`: An external table. These are the raw fastly logs stored in |
| 27 | s3://fastly-logs-20220622145016462800000001/ as compressed JSON records. Note |
| 28 | that this bucket has a lifecycle rule that moves logs to Glacier after a few |
| 29 | weeks. Logs in Glacier are not processed by Athena. |
| 30 | |
| 31 | - `asn_list`: A list of ASNs. This can be updated by running |
| 32 | `./update-asn-list.sh`. |
| 33 | |
| 34 | - `hosting-asns`: A list of ASNs belonging to hosting/cloud providers. |
| 35 | |
| 36 | - `all_paths`: The set of all store paths known in the hydra.nixos.org database. |
| 37 | This is used to expand the hash part of `.narinfo` requests (e.g. |
| 38 | `8kbx6s9nn7060zsdms3br0mk7bjrvbij`) to store paths (e.g. |
| 39 | `/nix/store/8kbx6s9nn7060zsdms3br0mk7bjrvbij-coreutils-full-9.0`). |
| 40 | |
| 41 | FIXME: describe how to update. |
| 42 | |
| 43 | - `release_paths`: All the store paths belonging to NixOS evals in |
| 44 | hydra.nixos.org, as |
| 45 | `{project, jobset, eval, release_name, build, |
| 46 | output, path}` tuples. |
| 47 | |
| 48 | FIXME: describe how to update. |
| 49 | |
| 50 | The ingestion script populates the following tables stored in |
| 51 | s3://nixos-athena/fastly-logs-processed/: |
| 52 | |
| 53 | - `urls`: For each host/day/url, the total number of requests, bytes and elapsed |
| 54 | microseconds. This only includes info about successful (2xx/3xx) requests. |
| 55 | |
| 56 | - `clients`: For each host/day/ASN/country/region, the total number of requests, |
| 57 | bytes and elapsed microseconds. |
| 58 | |
| 59 | - `nix_cache_info`: For each day/ASN/country/region/user-agent, the number of |
| 60 | requests for `nix-cache-info`. |
| 61 | |
| 62 | ## Reports |
| 63 | |
| 64 | Currently the following reports are created every week: |
| 65 | |
| 66 | - http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/traffic-per-day.csv |
| 67 | |
| 68 | For each day and site, the number of requests and the number of bytes |
| 69 | transferred. |
| 70 | |
| 71 | - http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/traffic-per-country.csv |
| 72 | |
| 73 | For each country, the number of requests and the number of bytes transferred. |
| 74 | |
| 75 | - http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day.csv |
| 76 | |
| 77 | For each day, the number of requests for |
| 78 | https://cache.nixos.org/nix-cache-info. |
| 79 | |
| 80 | - http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day-not-hosted.csv |
| 81 | |
| 82 | The same, but with requests from "hosting" ASNs (e.g. AWS and Hetzner) |
| 83 | filtered out. Note that Nix caches `nix-cache-info` file for a week, so the |
| 84 | intent of this report is to gauge the number of active weekly users. |
| 85 | |
| 86 | - http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/cache-info-requests-per-day-per-ua.csv |
| 87 | |
| 88 | For each day and user agent (e.g. `Nix/2.12.0`), the number of requests for |
| 89 | https://cache.nixos.org/nix-cache-info. This is intended to track the adoption |
| 90 | of Nix releases. |
| 91 | |
| 92 | - http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/flake-registry-requests-per-day.csv |
| 93 | |
| 94 | For each day, the number of requests for |
| 95 | https://channels.nixos.org/flake-registry.json. This is intended to track how |
| 96 | widely flakes are used. |
| 97 | |
| 98 | - http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/top-store-paths.csv |
| 99 | |
| 100 | For each store path listed in `all_paths`, the number of requests for its |
| 101 | `.narinfo` file. |
| 102 | |
| 103 | - http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/narinfo-queries-per-release.csv |
| 104 | |
| 105 | For each major NixOS release (e.g. `nixos-22.05`), the number of requests for |
| 106 | `.narinfo` files of store paths that are part of an eval of that release. |
| 107 | |
| 108 | - http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/nix-installer-downloads.csv |
| 109 | |
| 110 | For each day, the number of downloads of the Nix installer (i.e. |
| 111 | `https://releases.nixos.org/nix/nix-[^/]+/install`). |
| 112 | |
| 113 | - http://nixos-metrics.s3-website-eu-west-1.amazonaws.com/latest/nix-installer-architectures.csv |
| 114 | |
| 115 | For each architecture (e.g. `x86_64-linux`), the number of downloads of the |
| 116 | Nix binary tarball. |