@marcel91200 / caddy Mirror · read-only / commits / 857d5731
Fork 0

Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS

Mirrored from https://github.com/Slazee/caddy.git

Synced Nov 11, 02:15 UTC

caddyhttp: optimise header access and fix encoding negotiation (#8025)

* perf: bypass header key normalization for canonical literals http.Header's Get, Set, Add and Del run textproto.CanonicalMIMEHeaderKey on every call. When the key is a canonical literal the result is the key itself, so the scan is pure overhead. Access the map directly instead. Only hot, allocation-free paths are converted, and only where the key is a canonical literal. Call sites taking a key from config, a template or the request itself still go through the normalizing methods, since they need it for correctness. allHeaderValues and lastHeaderValue dropped their http.CanonicalHeaderKey call for the same reason: all callers pass canonical literals. benchstat, Apple M5 Pro, go1.27.1: PrepareRequest-15 659.4n ± 2% -> 457.7n ± 2% -30.58% (p=0.000 n=12) AcceptedEncodings-15 330.8n ± 5% -> 309.4n ± 7% -6.44% (p=0.019 n=10) ResponseWriterWriteHeader/event-stream-15 1007.5n ± 3% -> 732.5n ± 1% -27.30% ResponseWriterWriteHeader/informational-15 60.0n ± 3% -> 48.0n ± 2% -20.01% ResponseWriterWriteHeader/not-modified-15 214.3n ± 6% -> 175.3n ± 4% -18.20% ResponseWriterWriteHeader/connect-15 60.2n ± 3% -> 51.9n ± 4% -13.85% ResponseWriterWriteHeader/html-15 206.5n ± 4% -> 197.9n ± 2% -4.19% geomean -14.38% Allocations are unchanged. * tests: verify http.Header map keys are canonical Indexing http.Header directly skips the canonicalization Get/Set/Add/Del perform, which also puts the key outside what CI checks. canonicalheader only inspects method calls. staticcheck's SA1008 covers reads, but stops descending as soon as an assignment writes to a header, so it misses the write, the delete builtin and the read-modify-write append. A key that is not canonical in those positions is a silent lookup miss, not a compile error. Walk the loaded packages and verify every string constant used as a key in an http.Header map operation matches http.CanonicalHeaderKey. A helper that takes the key from its caller would hide the literal again, so a parameter named with a canonical prefix opts its arguments into the same check. The WebSocket headers Caddy deliberately writes with RFC 6455 casing are allowed explicitly; they are only ever written out, never looked up against a parsed header map. Covers 64 operations, runs in under a second, and promotes golang.org/x/tools from an indirect to a direct dependency at the version already in the module graph. * headercheck: resolve aliases of http.Header * headercheck: check http.Header composite literal keys * headercheck: note overlap with canonicalheader * headercheck: drop parameter-name check Helpers now take the indexed values, so every key sits in an index expression the analyzer and the linter already check. * headercheck: scope RFC 6455 exceptions to their call sites A key-based allow-list also accepted reads such as h["Sec-WebSocket-Key"], which miss because parsed headers are canonical. Honor //nolint:canonicalheader on the line instead, which the linter will use too. * encode: read every Accept-Encoding field line AcceptedEncodings read only the first field line, so a client that split the list across several lines had the rest ignored. RFC 9110 section 5.3 treats repeated list-field lines as one comma-joined list. Iterate over the items of every line instead of joining them, and skip empty items, which used to yield an empty encoding. file_server's precompressed selection uses the same function and is fixed with it. Fixes https://github.com/caddyserver/caddy/issues/8272 Co-authored-by: Kağan Küçük <kaankck@gmail.com> * encode: read every Cache-Control and Content-Encoding field line Only the first field line was read, but RFC 9110 section 5.3 makes multiple lines of a list-based field equivalent to one comma-joined value: - Cache-Control: no-transform on a later line was ignored, so the request or response was compressed anyway. - Content-Encoding: an empty first line followed by a real coding let the handler encode the body a second time. * encode: parse Cache-Control no-transform per RFC 9111 The substring match missed No-Transform, though RFC 9111 section 5.2 compares directive names case-insensitively, and fired on text that is not the directive: longer tokens such as x-no-transform-ext, quoted arguments such as community="no-transform", and semicolon-separated values, which are not valid Cache-Control syntax. Split the field on commas outside quoted strings and match only a well-formed no-transform element. * encode: keep accepted encoding preferences on the stack AcceptedEncodings counted commas to size the preference slice, and the variable capacity put it on the heap. A constant capacity covers typical Accept-Encoding headers and lets the compiler keep the slice on the stack; append still moves it to the heap for longer ones. BenchmarkAcceptedEncodings: 151.8ns -> 120.5ns (-21%), 192 -> 64 B/op, 2 -> 1 allocs/op (n=8). * encode: build the encoded ETag without fmt fmt.Sprintf boxed both arguments and parsed the format on every encoded response with a strong ETag. Concatenation: 47ns -> 16.5ns, 56 -> 24 B, 3 -> 1 allocs. * reverseproxy: access response Server and Via headers directly finalizeResponse runs on every proxied response; skip the key normalization as prepareRequest already does on the request side. * encode: access headers directly in Write, WriteHeader and Close These run once per response; convert the remaining literal-key Get, Set and Add calls like the rest of the handler. * encode: match Cache-Control no-transform with httpguts httpguts.HeaderValuesContainsToken already compares list elements case-insensitively across field lines, so the hand-written list parser is unnecessary. A no-transform inside a comma-separated quoted argument now also disables encoding, which only errs toward not transforming. * encode: inline the no-transform check isEncodeAllowed only wrapped one httpguts call. Its unit test now drives the response directive through ServeHTTP instead. --------- Co-authored-by: Kağan Küçük <kaankck@gmail.com>

Kévin Dunglas committed Oct 29, 2026 at 14:19 UTC 857d5731f381bf9e4fbae06377503fce1bf25058
8 files changed +425 -93
go.mod
+2
@@ -170,8 +170,10 @@ require (
170 go.opentelemetry.io/otel/trace v1.46.0
171 go.opentelemetry.io/proto/otlp v1.11.0 // indirect
172 go.uber.org/multierr v1.11.0 // indirect
173 + golang.org/x/mod v0.41.0 // indirect
174 golang.org/x/sys v0.48.0
175 golang.org/x/text v0.42.0 // indirect
176 + golang.org/x/tools v0.50.0
177 google.golang.org/grpc v1.83.2 // indirect
178 google.golang.org/protobuf v1.36.12 // indirect
179 howett.net/plist v1.0.1 // indirect
go.sum
+4
@@ -483,6 +483,8 @@ golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
483 golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
484 golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
485 golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
486 +golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
487 +golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
488 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
489 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
490 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
@@ -556,6 +558,8 @@ golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc
558 golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
559 golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
560 golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
561 +golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
562 +golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
563 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
564 gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
565 gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
internal/headercheck/headercheck_test.go new
+179
@@ -0,0 +1,179 @@
1 +// Copyright 2015 Matthew Holt and The Caddy Authors
2 +//
3 +// Licensed under the Apache License, Version 2.0 (the "License");
4 +// you may not use this file except in compliance with the License.
5 +// You may obtain a copy of the License at
6 +//
7 +// http://www.apache.org/licenses/LICENSE-2.0
8 +//
9 +// Unless required by applicable law or agreed to in writing, software
10 +// distributed under the License is distributed on an "AS IS" BASIS,
11 +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 +// See the License for the specific language governing permissions and
13 +// limitations under the License.
14 +
15 +// Package headercheck verifies that every http.Header key written directly
16 +// into the map uses net/http's canonical form.
17 +//
18 +// Hot paths index http.Header directly to skip the canonicalization that
19 +// Get/Set/Add/Del perform on every call. That trades a linted call for an
20 +// unlinted map operation: the canonicalheader linter only inspects method
21 +// calls, and staticcheck's SA1008 stops descending as soon as an assignment
22 +// writes to a header, so neither covers a direct write or a delete. A key
23 +// that is not canonical there is a silent lookup miss, not a compile error.
24 +//
25 +// This covers four shapes the enabled linters miss:
26 +//
27 +// h["x-key"] = []string{"v"} // direct write
28 +// http.Header{"x-key": {"v"}} // composite literal
29 +// delete(h, "x-key") // delete builtin
30 +// h["x-key"] = append(h["x-key"], "v) // read-modify-write
31 +package headercheck
32 +
33 +import (
34 + "go/ast"
35 + "go/constant"
36 + "go/token"
37 + "go/types"
38 + "net/http"
39 + "strings"
40 + "testing"
41 +
42 + "golang.org/x/tools/go/packages"
43 +)
44 +
45 +// headerType is the type whose map operations must use canonical keys.
46 +const headerType = "net/http.Header"
47 +
48 +// nolintDirective marks a deliberate non-canonical key on its line, such as
49 +// the RFC 6455 spellings written for case-sensitive upstreams.
50 +const nolintDirective = "//nolint:canonicalheader"
51 +
52 +// TODO: delete this package once golangci-lint ships https://github.com/lasiar/canonicalheader/pull/90;
53 +// the linter honors the same //nolint:canonicalheader directives.
54 +func TestHeaderKeysAreCanonical(t *testing.T) {
55 + cfg := &packages.Config{
56 + Mode: packages.NeedName | packages.NeedSyntax | packages.NeedTypes |
57 + packages.NeedTypesInfo | packages.NeedDeps | packages.NeedImports,
58 + Dir: "../..",
59 + Tests: false,
60 + }
61 +
62 + pkgs, err := packages.Load(cfg, "./...")
63 + if err != nil {
64 + t.Fatalf("loading packages: %v", err)
65 + }
66 + if packages.PrintErrors(pkgs) > 0 {
67 + t.Fatal("packages failed to load; see errors above")
68 + }
69 +
70 + var checked int
71 + for _, pkg := range pkgs {
72 + for _, file := range pkg.Syntax {
73 + checked += checkFile(t, pkg, file)
74 + }
75 + }
76 +
77 + // Guard against the walk silently covering nothing, e.g. if package
78 + // loading is reconfigured and stops yielding type information.
79 + if checked == 0 {
80 + t.Fatal("no http.Header map operations found; the check is not running")
81 + }
82 + t.Logf("checked %d direct http.Header map operations", checked)
83 +}
84 +
85 +// checkFile reports the number of direct http.Header map operations examined.
86 +func checkFile(t *testing.T, pkg *packages.Package, file *ast.File) int {
87 + var checked int
88 +
89 + exempt := make(map[int]bool)
90 + for _, group := range file.Comments {
91 + for _, c := range group.List {
92 + if strings.HasPrefix(c.Text, nolintDirective) {
93 + exempt[pkg.Fset.Position(c.Slash).Line] = true
94 + }
95 + }
96 + }
97 +
98 + report := func(pos token.Pos, key, form string) {
99 + canonical := http.CanonicalHeaderKey(key)
100 + if key == canonical || exempt[pkg.Fset.Position(pos).Line] {
101 + return
102 + }
103 + t.Errorf("%s: %s uses non-canonical key %q; want %q",
104 + pkg.Fset.Position(pos), form, key, canonical)
105 + }
106 +
107 + // keyOf returns the constant string value of an index or argument
108 + // expression, so both literals and named constants are covered.
109 + keyOf := func(expr ast.Expr) (string, bool) {
110 + tv, ok := pkg.TypesInfo.Types[expr]
111 + if !ok || tv.Value == nil || tv.Value.Kind() != constant.String {
112 + return "", false
113 + }
114 + return constant.StringVal(tv.Value), true
115 + }
116 +
117 + isHeader := func(expr ast.Expr) bool {
118 + tv, ok := pkg.TypesInfo.Types[expr]
119 + if !ok || tv.Type == nil {
120 + return false
121 + }
122 + named, ok := types.Unalias(tv.Type).(*types.Named)
123 + if !ok {
124 + return false
125 + }
126 + obj := named.Obj()
127 + return obj != nil && obj.Pkg() != nil &&
128 + obj.Pkg().Path()+"."+obj.Name() == headerType
129 + }
130 +
131 + ast.Inspect(file, func(n ast.Node) bool {
132 + switch node := n.(type) {
133 + // h["Key"], including the assignment form h["Key"] = ...
134 + // that SA1008 skips.
135 + case *ast.IndexExpr:
136 + if !isHeader(node.X) {
137 + return true
138 + }
139 + key, ok := keyOf(node.Index)
140 + if !ok {
141 + return true
142 + }
143 + checked++
144 + report(node.Index.Pos(), key, "http.Header index")
145 +
146 + // http.Header{"Key": ...}, which stores the key verbatim.
147 + case *ast.CompositeLit:
148 + if !isHeader(node) {
149 + return true
150 + }
151 + for _, elt := range node.Elts {
152 + kv, ok := elt.(*ast.KeyValueExpr)
153 + if !ok {
154 + continue
155 + }
156 + key, ok := keyOf(kv.Key)
157 + if !ok {
158 + continue
159 + }
160 + checked++
161 + report(kv.Key.Pos(), key, "http.Header literal")
162 + }
163 +
164 + // delete(h, "Key"), which no enabled linter inspects.
165 + case *ast.CallExpr:
166 + if ident, ok := node.Fun.(*ast.Ident); ok && ident.Name == "delete" && len(node.Args) == 2 {
167 + if _, ok := pkg.TypesInfo.Uses[ident].(*types.Builtin); ok && isHeader(node.Args[0]) {
168 + if key, ok := keyOf(node.Args[1]); ok {
169 + checked++
170 + report(node.Args[1].Pos(), key, "delete on http.Header")
171 + }
172 + }
173 + }
174 + }
175 + return true
176 + })
177 +
178 + return checked
179 +}
modules/caddyhttp/encode/encode.go
+63 -27
@@ -23,6 +23,7 @@ import (
23 "cmp"
24 "fmt"
25 "io"
26 + "iter"
27 "math"
28 "net/http"
29 "slices"
@@ -30,6 +31,8 @@ import (
31 "strings"
32 "sync"
33
34 + "golang.org/x/net/http/httpguts"
35 +
36 "github.com/caddyserver/caddy/v2"
37 "github.com/caddyserver/caddy/v2/modules/caddyhttp"
38 )
@@ -157,12 +160,17 @@ func (enc *Encode) Validate() error {
160 return nil
161 }
162
160 -func isEncodeAllowed(h http.Header) bool {
161 - return !strings.Contains(h.Get("Cache-Control"), "no-transform")
163 +func hasNonEmptyValue(h http.Header, key string) bool {
164 + for _, v := range h[key] {
165 + if v != "" {
166 + return true
167 + }
168 + }
169 + return false
170 }
171
172 func (enc *Encode) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error {
165 - if isEncodeAllowed(r.Header) {
173 + if !httpguts.HeaderValuesContainsToken(r.Header["Cache-Control"], "no-transform") {
174 for _, encName := range AcceptedEncodings(r, enc.Prefer) {
175 if _, ok := enc.writerPools[encName]; !ok {
176 continue // encoding not offered
@@ -176,11 +184,11 @@ func (enc *Encode) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyh
184 // the Etag in the first place don't know that we appended to their Etag! so here
185 // we have to strip our addition so the upstream handlers can still honor client
186 // caches without knowing about our changes...
179 - if etags := r.Header.Values("If-None-Match"); len(etags) > 0 {
187 + if etags := r.Header["If-None-Match"]; len(etags) > 0 {
188 etag := strings.Join(etags, ", ")
189 stripped := strings.ReplaceAll(etag, "-"+encName+`"`, `"`)
190 if stripped != etag || len(etags) > 1 {
183 - r.Header.Set("If-None-Match", stripped)
191 + r.Header["If-None-Match"] = []string{stripped}
192 }
193 }
194
@@ -277,7 +285,7 @@ func (rw *responseWriter) WriteHeader(status int) {
285 // we would miss the Vary header in this case when compression was also enabled; note that we set this
286 // header in the responseWriter.init() method but that is only called if we are writing a response body
287 if status == http.StatusNotModified && !hasVaryValue(h, "Accept-Encoding") {
280 - h.Add("Vary", "Accept-Encoding")
288 + h["Vary"] = append(h["Vary"], "Accept-Encoding")
289 }
290
291 // write status immediately if status is 2xx and the request is CONNECT
@@ -299,7 +307,7 @@ func (rw *responseWriter) WriteHeader(status int) {
307 // body may not be written for a while and the client needs the headers
308 // to establish the stream; see #6293
309 if !rw.wroteHeader && finalStatus &&
302 - (rw.incremental || isSSE(h.Get("Content-Type"))) {
310 + (rw.incremental || isSSE(firstValue(h["Content-Type"]))) {
311 rw.init()
312 rw.ResponseWriter.WriteHeader(status)
313 rw.wroteHeader = true
@@ -377,8 +385,8 @@ func (rw *responseWriter) Write(p []byte) (int, error) {
385 rw.WriteHeader(http.StatusOK)
386 }
387 if !rw.wroteHeader && rw.statusCode == 0 && caddyhttp.IsIncremental(rw.Header()) {
380 - if rw.Header().Get("Content-Type") == "" {
381 - rw.Header().Set("Content-Type", http.DetectContentType(p))
388 + if hdr := rw.Header(); firstValue(hdr["Content-Type"]) == "" {
389 + hdr["Content-Type"] = []string{http.DetectContentType(p)}
390 }
391 rw.WriteHeader(http.StatusOK)
392 }
@@ -388,13 +396,13 @@ func (rw *responseWriter) Write(p []byte) (int, error) {
396 var gtMinLength bool
397 if len(p) > rw.config.MinLength {
398 gtMinLength = true
391 - } else if cl, err := strconv.Atoi(rw.Header().Get("Content-Length")); err == nil && cl > rw.config.MinLength {
399 + } else if cl, err := strconv.Atoi(firstValue(rw.Header()["Content-Length"])); err == nil && cl > rw.config.MinLength {
400 gtMinLength = true
401 }
402
403 if gtMinLength {
396 - if rw.Header().Get("Content-Type") == "" {
397 - rw.Header().Set("Content-Type", http.DetectContentType(p))
404 + if hdr := rw.Header(); firstValue(hdr["Content-Type"]) == "" {
405 + hdr["Content-Type"] = []string{http.DetectContentType(p)}
406 }
407 rw.init()
408 }
@@ -487,7 +495,7 @@ func (rw *responseWriter) ReadFrom(r io.Reader) (int64, error) {
495 func (rw *responseWriter) Close() error {
496 // didn't write, probably head request
497 if !rw.wroteHeader {
490 - cl, err := strconv.Atoi(rw.Header().Get("Content-Length"))
498 + cl, err := strconv.Atoi(firstValue(rw.Header()["Content-Length"]))
499 if err == nil && cl > rw.config.MinLength {
500 rw.init()
501 }
@@ -526,16 +534,17 @@ func (rw *responseWriter) init() {
534
535 hdr := rw.Header()
536
529 - if hdr.Get("Content-Encoding") == "" && isEncodeAllowed(hdr) &&
537 + if !hasNonEmptyValue(hdr, "Content-Encoding") &&
538 + !httpguts.HeaderValuesContainsToken(hdr["Cache-Control"], "no-transform") &&
539 rw.config.Match(rw) {
540 rw.w = rw.config.writerPools[rw.encodingName].Get().(Encoder)
541 rw.w.Reset(rw.ResponseWriter)
533 - hdr.Del("Content-Length") // https://github.com/golang/go/issues/14975
534 - hdr.Set("Content-Encoding", rw.encodingName)
542 + delete(hdr, "Content-Length") // https://github.com/golang/go/issues/14975
543 + hdr["Content-Encoding"] = []string{rw.encodingName}
544 if !hasVaryValue(hdr, "Accept-Encoding") {
536 - hdr.Add("Vary", "Accept-Encoding")
545 + hdr["Vary"] = append(hdr["Vary"], "Accept-Encoding")
546 }
538 - hdr.Del("Accept-Ranges") // we don't know ranges for dynamically-encoded content
547 + delete(hdr, "Accept-Ranges") // we don't know ranges for dynamically-encoded content
548
549 // strong ETags need to be distinct depending on the encoding ("selected representation")
550 // see RFC 9110 section 8.8.3.3:
@@ -544,15 +553,24 @@ func (rw *responseWriter) init() {
553 // (We have to strip the value we append from If-None-Match headers before
554 // sending subsequent requests back upstream, however, since upstream handlers
555 // don't know about our appending to their Etag since they've already done their work)
547 - if etag := hdr.Get("Etag"); etag != "" && !strings.HasPrefix(etag, "W/") {
548 - etag = fmt.Sprintf(`%s-%s"`, strings.TrimSuffix(etag, `"`), rw.encodingName)
549 - hdr.Set("Etag", etag)
556 + if etag := firstValue(hdr["Etag"]); etag != "" && !strings.HasPrefix(etag, "W/") {
557 + etag = strings.TrimSuffix(etag, `"`) + "-" + rw.encodingName + `"`
558 + hdr["Etag"] = []string{etag}
559 }
560 }
561 }
562
563 +// firstValue is http.Header.Get without key normalization, for callers that
564 +// index the header with a canonical key.
565 +func firstValue(values []string) string {
566 + if len(values) > 0 {
567 + return values[0]
568 + }
569 + return ""
570 +}
571 +
572 func hasVaryValue(hdr http.Header, target string) bool {
555 - for _, vary := range hdr.Values("Vary") {
573 + for _, vary := range hdr["Vary"] {
574 for val := range strings.SplitSeq(vary, ",") {
575 if strings.EqualFold(strings.TrimSpace(val), target) {
576 return true
@@ -570,17 +588,21 @@ func hasVaryValue(hdr http.Header, target string) bool {
588 // encodings are not considered. See
589 // http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html.
590 func AcceptedEncodings(r *http.Request, preferredOrder []string) []string {
573 - acceptEncHeader := r.Header.Get("Accept-Encoding")
574 - if acceptEncHeader == "" {
591 + values := r.Header["Accept-Encoding"]
592 + if len(values) == 0 {
593 return []string{}
594 }
577 - websocketKey := r.Header.Get("Sec-Websocket-Key")
595 + websocketKey := firstValue(r.Header["Sec-Websocket-Key"])
596
579 - prefs := make([]encodingPreference, 0, strings.Count(acceptEncHeader, ",")+1)
597 + // sized for typical headers so the slice stays on the stack
598 + prefs := make([]encodingPreference, 0, 8)
599
581 - for accepted := range strings.SplitSeq(acceptEncHeader, ",") {
600 + for accepted := range listItems(values) {
601 encName, params, found := strings.Cut(accepted, ";")
602 encName = strings.ToLower(strings.TrimSpace(encName))
603 + if encName == "" {
604 + continue
605 + }
606
607 // determine q-factor
608 qFactor := 1.0
@@ -636,6 +658,20 @@ func AcceptedEncodings(r *http.Request, preferredOrder []string) []string {
658 return prefEncNames
659 }
660
661 +// listItems yields the items of a list-based field, which RFC 9110 section 5.3
662 +// allows to be split across several field lines.
663 +func listItems(lines []string) iter.Seq[string] {
664 + return func(yield func(string) bool) {
665 + for _, line := range lines {
666 + for item := range strings.SplitSeq(line, ",") {
667 + if !yield(item) {
668 + return
669 + }
670 + }
671 + }
672 + }
673 +}
674 +
675 // encodingPreference pairs an encoding with its q-factor.
676 type encodingPreference struct {
677 encoding string
modules/caddyhttp/encode/encode_conformance_test.go
+51
@@ -259,6 +259,46 @@ var encodeScenarios = []encodeScenario{
259 },
260 checkResponse: checkBypassesEncoding,
261 },
262 + {
263 + name: "request cache-control no-transform on a later line prevents encoding",
264 + method: http.MethodGet,
265 + minLength: 1,
266 + reqHeaders: func(encoderCase) http.Header {
267 + return http.Header{"Cache-Control": {"no-cache", "no-transform"}}
268 + },
269 + next: func(encoderCase) caddyhttp.Handler {
270 + return conformanceLargeBodyHandler(conformanceContentType)
271 + },
272 + checkResponse: checkBypassesEncoding,
273 + },
274 + {
275 + name: "response cache-control no-transform on a later line prevents encoding",
276 + method: http.MethodGet,
277 + minLength: 1,
278 + next: func(encoderCase) caddyhttp.Handler {
279 + return caddyhttp.HandlerFunc(func(w http.ResponseWriter, r *http.Request) error {
280 + w.Header().Set("Content-Type", conformanceContentType)
281 + w.Header()["Cache-Control"] = []string{"no-cache", "no-transform"}
282 + _, err := w.Write(conformanceLargeBody())
283 + return err
284 + })
285 + },
286 + checkResponse: checkBypassesEncoding,
287 + },
288 + {
289 + name: "content-encoding after an empty line prevents encoding",
290 + method: http.MethodGet,
291 + minLength: 1,
292 + next: func(encoderCase) caddyhttp.Handler {
293 + return caddyhttp.HandlerFunc(func(w http.ResponseWriter, r *http.Request) error {
294 + w.Header().Set("Content-Type", conformanceContentType)
295 + w.Header()["Content-Encoding"] = []string{"", "gzip"}
296 + _, err := w.Write(conformanceLargeBody())
297 + return err
298 + })
299 + },
300 + checkResponse: checkKeepsUpstreamContentEncoding,
301 + },
302 {
303 name: "content type matcher rejection prevents encoding",
304 method: http.MethodGet,
@@ -458,6 +498,17 @@ func checkBypassesEncoding(t *testing.T, w *httptest.ResponseRecorder, encCase e
498 }
499 }
500
501 +func checkKeepsUpstreamContentEncoding(t *testing.T, w *httptest.ResponseRecorder, encCase encoderCase) {
502 + t.Helper()
503 +
504 + if got, want := w.Header().Values("Content-Encoding"), []string{"", "gzip"}; !slices.Equal(got, want) {
505 + t.Fatalf("Content-Encoding = %q, want %q", got, want)
506 + }
507 + if want := conformanceLargeBody(); !bytes.Equal(w.Body.Bytes(), want) {
508 + t.Fatalf("body len = %d, want len = %d", w.Body.Len(), len(want))
509 + }
510 +}
511 +
512 // TestEncodeHeadResponseContentLength verifies that a HEAD response does not
513 // advertise the length of an empty encoded stream.
514 func TestEncodeHeadResponseContentLength(t *testing.T) {
modules/caddyhttp/encode/encode_test.go
+96 -42
@@ -129,7 +129,7 @@ func TestPreferOrder(t *testing.T) {
129 name: "PreferOrder(): 2 accept (1 empty), 3 prefer",
130 accept: "gzip,,zstd",
131 prefer: []string{"zstd", "br", "gzip"},
132 - expected: []string{"zstd", "gzip", ""},
132 + expected: []string{"zstd", "gzip"},
133 },
134 {
135 name: "PreferOrder(): 1 accept, 2 prefer",
@@ -341,50 +341,50 @@ func TestValidate(t *testing.T) {
341 }
342 }
343
344 -func TestIsEncodeAllowed(t *testing.T) {
345 - testCases := []struct {
346 - name string
347 - headers http.Header
348 - expected bool
349 - }{
350 - {
351 - name: "Without any headers",
352 - headers: http.Header{},
353 - expected: true,
354 - },
355 - {
356 - name: "Without Cache-Control HTTP header",
357 - headers: http.Header{
358 - "Accept-Encoding": {"gzip"},
359 - },
360 - expected: true,
361 - },
362 - {
363 - name: "Cache-Control HTTP header ending with no-transform directive",
364 - headers: http.Header{
365 - "Accept-Encoding": {"gzip"},
366 - "Cache-Control": {"no-cache; no-transform"},
367 - },
368 - expected: false,
369 - },
370 - {
371 - name: "With Cache-Control HTTP header no-transform as Cache-Extension value",
372 - headers: http.Header{
373 - "Accept-Encoding": {"gzip"},
374 - "Cache-Control": {`no-store; no-cache; community="no-transform"`},
375 - },
376 - expected: false,
344 +func TestServeHTTPResponseNoTransform(t *testing.T) {
345 + enc := &Encode{
346 + MinLength: 1,
347 + writerPools: map[string]*sync.Pool{
348 + "gzip": {New: func() any { return mockEncoder{} }},
349 },
350 }
351 + ctx, cancel := caddy.NewContext(caddy.Context{Context: context.Background()})
352 + defer cancel()
353 + if err := enc.Provision(ctx); err != nil {
354 + t.Fatalf("Provision failed: %v", err)
355 + }
356
380 - for _, test := range testCases {
381 - t.Run(test.name, func(t *testing.T) {
382 - if result := isEncodeAllowed(test.headers); result != test.expected {
383 - t.Errorf("The headers given to the isEncodeAllowed should return %t, %t given.",
384 - result,
385 - test.expected)
386 - }
387 - })
357 + for _, tc := range []struct {
358 + cacheControl []string
359 + encoded bool
360 + }{
361 + {cacheControl: nil, encoded: true},
362 + {cacheControl: []string{"no-cache, no-transform"}, encoded: false},
363 + {cacheControl: []string{"max-age=0, No-Transform"}, encoded: false},
364 + {cacheControl: []string{"no-cache", "no-transform"}, encoded: false},
365 + {cacheControl: []string{`private="Set-Cookie, Vary", no-transform`}, encoded: false},
366 + // extension values and longer tokens are not the directive
367 + {cacheControl: []string{`community="no-transform"`}, encoded: true},
368 + {cacheControl: []string{"community=no-transform"}, encoded: true},
369 + {cacheControl: []string{"x-no-transform-ext, no-transformation"}, encoded: true},
370 + // semicolons are not Cache-Control list separators
371 + {cacheControl: []string{"no-cache; no-transform"}, encoded: true},
372 + } {
373 + r := httptest.NewRequest(http.MethodGet, "/", nil)
374 + r.Header.Set("Accept-Encoding", "gzip")
375 + w := httptest.NewRecorder()
376 + err := enc.ServeHTTP(w, r, caddyhttp.HandlerFunc(func(w http.ResponseWriter, r *http.Request) error {
377 + w.Header()["Content-Type"] = []string{"text/plain"}
378 + w.Header()["Cache-Control"] = tc.cacheControl
379 + _, err := w.Write([]byte("hello"))
380 + return err
381 + }))
382 + if err != nil {
383 + t.Fatalf("Cache-Control %q: ServeHTTP: %v", tc.cacheControl, err)
384 + }
385 + if encoded := w.Header().Get("Content-Encoding") == "gzip"; encoded != tc.encoded {
386 + t.Errorf("Cache-Control %q: encoded = %t, want %t", tc.cacheControl, encoded, tc.encoded)
387 + }
388 }
389 }
390
@@ -456,3 +456,57 @@ func TestServeHTTPDefaultEncodingPreference(t *testing.T) {
456 t.Errorf("Expected Content-Encoding to be 'gzip' when explicitly preferred, got '%s'", contentEncoding2)
457 }
458 }
459 +
460 +// RFC 9110 section 5.3: a recipient MAY combine multiple field lines with the
461 +// same name by joining their values with commas; the semantics of a list-based
462 +// field like Accept-Encoding do not change when it is split across lines.
463 +func TestAcceptedEncodingsMultipleFieldLines(t *testing.T) {
464 + for _, tc := range []struct {
465 + lines []string
466 + expected []string
467 + }{
468 + {[]string{"gzip", "br"}, []string{"br", "gzip"}},
469 + {[]string{"identity", "br"}, []string{"br", "identity"}},
470 + {[]string{"gzip;q=0", "br"}, []string{"br"}},
471 + {[]string{"", "gzip"}, []string{"gzip"}},
472 + {[]string{"gzip;q=0.5", "zstd;q=0.8, br;q=0"}, []string{"zstd", "gzip"}},
473 + } {
474 + r, _ := http.NewRequest(http.MethodGet, "/", nil)
475 + for _, l := range tc.lines {
476 + r.Header.Add("Accept-Encoding", l)
477 + }
478 + got := AcceptedEncodings(r, []string{"zstd", "br", "gzip"})
479 + if !slices.Equal(got, tc.expected) {
480 + t.Errorf("Accept-Encoding lines %q: got %v, expected %v", tc.lines, got, tc.expected)
481 + }
482 + }
483 +}
484 +
485 +func TestServeHTTPMultipleAcceptEncodingLines(t *testing.T) {
486 + enc := new(Encode)
487 + enc.MinLength = 1
488 + enc.writerPools = map[string]*sync.Pool{
489 + "br": {New: func() any { return mockEncoder{} }},
490 + }
491 + ctx, cancel := caddy.NewContext(caddy.Context{Context: context.Background()})
492 + defer cancel()
493 + if err := enc.Provision(ctx); err != nil {
494 + t.Fatal(err)
495 + }
496 + r, _ := http.NewRequest(http.MethodGet, "/", nil)
497 + r.Header.Add("Accept-Encoding", "identity")
498 + r.Header.Add("Accept-Encoding", "br")
499 + w := httptest.NewRecorder()
500 + w.Header().Set("Content-Type", "text/plain")
501 + next := caddyhttp.HandlerFunc(func(w http.ResponseWriter, r *http.Request) error {
502 + w.WriteHeader(http.StatusOK)
503 + _, err := w.Write([]byte("hello world"))
504 + return err
505 + })
506 + if err := enc.ServeHTTP(w, r, next); err != nil {
507 + t.Fatal(err)
508 + }
509 + if ce := w.Header().Get("Content-Encoding"); ce != "br" {
510 + t.Errorf("expected Content-Encoding br, got %q", ce)
511 + }
512 +}
modules/caddyhttp/reverseproxy/reverseproxy.go
+29 -23
@@ -655,7 +655,7 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyht
655 if randErr != nil {
656 return randErr
657 }
658 - clonedReq.Header["Sec-WebSocket-Key"] = []string{base64.StdEncoding.EncodeToString(key)}
658 + clonedReq.Header["Sec-WebSocket-Key"] = []string{base64.StdEncoding.EncodeToString(key)} //nolint:canonicalheader // RFC 6455 casing, see normalizeWebsocketHeaders
659 }
660
661 // we will need the original headers and Host value if
@@ -965,7 +965,7 @@ func (h Handler) prepareRequest(req *http.Request, repl *caddy.Replacer) (*http.
965 // set Content-Length when body is fully buffered
966 if b, ok := req.Body.(bodyReadCloser); ok && b.body == nil {
967 req.ContentLength = readBytes
968 - req.Header.Set("Content-Length", strconv.FormatInt(req.ContentLength, 10))
968 + req.Header["Content-Length"] = []string{strconv.FormatInt(req.ContentLength, 10)}
969 }
970 }
971
@@ -987,7 +987,7 @@ func (h Handler) prepareRequest(req *http.Request, repl *caddy.Replacer) (*http.
987 // if User-Agent is not set by client, then explicitly
988 // disable it so it's not set to default value by std lib
989 if _, ok := req.Header["User-Agent"]; !ok {
990 - req.Header.Set("User-Agent", "")
990 + req.Header["User-Agent"] = []string{""}
991 }
992
993 // Indicate if request has been conveyed in early data.
@@ -999,7 +999,7 @@ func (h Handler) prepareRequest(req *http.Request, repl *caddy.Replacer) (*http.
999 // might already have been forwarded by it or another instance
1000 // (see Section 6.2)."
1001 if req.TLS != nil && !req.TLS.HandshakeComplete {
1002 - req.Header.Set("Early-Data", "1")
1002 + req.Header["Early-Data"] = []string{"1"}
1003 }
1004
1005 reqUpgradeType := upgradeType(req.Header)
@@ -1015,17 +1015,18 @@ func (h Handler) prepareRequest(req *http.Request, repl *caddy.Replacer) (*http.
1015 // advertise that unless the incoming client request thought it was worth
1016 // mentioning.)
1017 if h == "Te" && httpguts.HeaderValuesContainsToken(req.Header["Te"], "trailers") {
1018 - req.Header.Set("Te", "trailers")
1018 + req.Header["Te"] = []string{"trailers"}
1019 continue
1020 }
1021 - req.Header.Del(h)
1021 + // hopHeaders are already canonical, so skip the normalization Del does
1022 + delete(req.Header, h)
1023 }
1024
1025 // After stripping all the hop-by-hop connection headers above, add back any
1026 // necessary for protocol upgrades, such as for websockets.
1027 if reqUpgradeType != "" {
1027 - req.Header.Set("Connection", "Upgrade")
1028 - req.Header.Set("Upgrade", reqUpgradeType)
1028 + req.Header["Connection"] = []string{"Upgrade"}
1029 + req.Header["Upgrade"] = []string{reqUpgradeType}
1030 }
1031
1032 // Set up the PROXY protocol info
@@ -1057,7 +1058,8 @@ func (h Handler) prepareRequest(req *http.Request, repl *caddy.Replacer) (*http.
1058 }
1059
1060 // Via header(s)
1060 - req.Header.Add("Via", strconv.Itoa(req.ProtoMajor)+"."+strconv.Itoa(req.ProtoMinor)+" Caddy")
1061 + req.Header["Via"] = append(req.Header["Via"],
1062 + strconv.Itoa(req.ProtoMajor)+"."+strconv.Itoa(req.ProtoMinor)+" Caddy")
1063
1064 return req, nil
1065 }
@@ -1083,9 +1085,9 @@ func (h Handler) addForwardedHeaders(req *http.Request) error {
1085 // for security. If trusted, there is no peer IP to append to
1086 // X-Forwarded-For, so clientIP stays empty.
1087 if !trusted {
1086 - req.Header.Del("X-Forwarded-For")
1087 - req.Header.Del("X-Forwarded-Proto")
1088 - req.Header.Del("X-Forwarded-Host")
1088 + delete(req.Header, "X-Forwarded-For")
1089 + delete(req.Header, "X-Forwarded-Proto")
1090 + delete(req.Header, "X-Forwarded-Host")
1091 return nil
1092 }
1093 } else {
@@ -1099,9 +1101,9 @@ func (h Handler) addForwardedHeaders(req *http.Request) error {
1101 if err != nil {
1102 // Remove the `X-Forwarded-*` headers to avoid upstreams
1103 // potentially trusting a header that came from the client
1102 - req.Header.Del("X-Forwarded-For")
1103 - req.Header.Del("X-Forwarded-Proto")
1104 - req.Header.Del("X-Forwarded-Host")
1104 + delete(req.Header, "X-Forwarded-For")
1105 + delete(req.Header, "X-Forwarded-Proto")
1106 + delete(req.Header, "X-Forwarded-Host")
1107 return nil
1108 }
1109
@@ -1130,12 +1132,12 @@ func (h Handler) addForwardedHeaders(req *http.Request) error {
1132 if !omit {
1133 if trusted && ok && prior != "" {
1134 if clientIP != "" {
1133 - req.Header.Set("X-Forwarded-For", prior+", "+clientIP)
1135 + req.Header["X-Forwarded-For"] = []string{prior + ", " + clientIP}
1136 } else {
1135 - req.Header.Set("X-Forwarded-For", prior)
1137 + req.Header["X-Forwarded-For"] = []string{prior}
1138 }
1139 } else if clientIP != "" {
1138 - req.Header.Set("X-Forwarded-For", clientIP)
1140 + req.Header["X-Forwarded-For"] = []string{clientIP}
1141 }
1142 }
1143
@@ -1151,7 +1153,7 @@ func (h Handler) addForwardedHeaders(req *http.Request) error {
1153 proto = prior
1154 }
1155 if !omit {
1154 - req.Header.Set("X-Forwarded-Proto", proto)
1156 + req.Header["X-Forwarded-Proto"] = []string{proto}
1157 }
1158
1159 // Set X-Forwarded-Host; often this is redundant because
@@ -1164,7 +1166,7 @@ func (h Handler) addForwardedHeaders(req *http.Request) error {
1166 host = prior
1167 }
1168 if !omit {
1167 - req.Header.Set("X-Forwarded-Host", host)
1169 + req.Header["X-Forwarded-Host"] = []string{host}
1170 }
1171
1172 return nil
@@ -1424,12 +1426,13 @@ func (h *Handler) finalizeResponse(
1426 }
1427
1428 // delete our Server header and use Via instead (see #6275)
1427 - rw.Header().Del("Server")
1429 + rwHeader := rw.Header()
1430 + delete(rwHeader, "Server")
1431 var protoPrefix string
1432 if !strings.HasPrefix(strings.ToUpper(res.Proto), "HTTP/") {
1433 protoPrefix = res.Proto[:strings.Index(res.Proto, "/")+1]
1434 }
1432 - rw.Header().Add("Via", protoPrefix+strconv.Itoa(res.ProtoMajor)+"."+strconv.Itoa(res.ProtoMinor)+" Caddy")
1435 + rwHeader["Via"] = append(rwHeader["Via"], protoPrefix+strconv.Itoa(res.ProtoMajor)+"."+strconv.Itoa(res.ProtoMinor)+" Caddy")
1436
1437 // apply any response header operations
1438 if h.Headers != nil && h.Headers.Response != nil {
@@ -1872,7 +1875,10 @@ func upgradeType(h http.Header) string {
1875 if !httpguts.HeaderValuesContainsToken(h["Connection"], "Upgrade") {
1876 return ""
1877 }
1875 - return strings.ToLower(h.Get("Upgrade"))
1878 + if v := h["Upgrade"]; len(v) > 0 {
1879 + return strings.ToLower(v[0])
1880 + }
1881 + return ""
1882 }
1883
1884 // removeConnectionHeaders removes hop-by-hop headers listed in the "Connection" header of h.
modules/caddyhttp/reverseproxy/streaming.go
+1 -1
@@ -127,7 +127,7 @@ func (h *Handler) handleUpgradeResponse(logger *zap.Logger, rw http.ResponseWrit
127 req.Body = body
128 rw.Header().Del("Upgrade")
129 rw.Header().Del("Connection")
130 - delete(rw.Header(), "Sec-WebSocket-Accept")
130 + delete(rw.Header(), "Sec-WebSocket-Accept") //nolint:canonicalheader // set by normalizeWebsocketHeaders above
131 rw.WriteHeader(http.StatusOK)
132
133 if c := logger.Check(zap.DebugLevel, "upgrading connection"); c != nil {