diff --git a/docs/architecture.md b/docs/architecture.md index b6c654ff..24af2fbb 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -355,7 +355,7 @@ Prometheus metrics for cache performance, upstream latency, storage operations, ### Cooldown -Version age filtering for supply chain attack mitigation, provided by [github.com/git-pkgs/cooldown](https://github.com/git-pkgs/cooldown). Configurable at global, ecosystem, and per-package levels. Supported by npm, PyPI, pub.dev, and Composer handlers. +Version age filtering for supply chain attack mitigation, provided by [github.com/git-pkgs/cooldown](https://github.com/git-pkgs/cooldown). Configurable at global, ecosystem, and per-package levels. Supported by the npm, PyPI, pub.dev, Composer, Cargo, NuGet, Conda, RubyGems, Hex, Go, Maven and Helm handlers. ### `internal/enrichment` diff --git a/docs/configuration.md b/docs/configuration.md index 832860c2..2f8d3c18 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1124,9 +1124,9 @@ Exact `packages` entries take precedence over patterns. When several patterns ma Resolution order: exact package override, then package pattern, then ecosystem override, then global default. This lets you set a conservative default while exempting trusted package families. -The web UI's dashboard shows the default and the ecosystem overrides, and how many package overrides and patterns there are; their names aren't shown, since the UI has no authentication. A package's page shows the cooldown that applies to it and which kind of rule it comes from. +The web UI's dashboard shows the default and the ecosystem overrides, and how many package overrides and patterns there are; their names aren't shown, since the UI has no authentication. A package's page shows the cooldown that applies to it and which kind of rule it comes from; it shows none for ecosystems without cooldown support, and the dashboard marks an ecosystem override for one as not supported. -Currently supported for npm, PyPI, pub.dev, Composer, Cargo, NuGet, Conda, RubyGems, Hex, Go and Maven. Most of these ecosystems include publish timestamps in their metadata; Go and Maven are described below. +Currently supported for npm, PyPI, pub.dev, Composer, Cargo, NuGet, Conda, RubyGems, Hex, Go, Maven and Helm. Most of these ecosystems include publish timestamps in their metadata; Go and Maven are described below. Go has no publish time. The module proxy's `.info` gives the commit time of the revision a version points at, so a version tagged on an old commit counts as old. The proxy reads each listed version's `.info` once, stores the time, and leaves versions in cooldown out of `@v/list` and `@latest`. When upstream's `@latest` is in cooldown, it serves the newest release out of cooldown instead. Requests for a version in cooldown, including its `.info`, `.mod` and `.zip`, and a branch or commit query that resolves to one, get 403 rather than 404, because the go command moves on to the next `GOPROXY` entry, such as `direct`, after a 404. If a version's `.info` can't be fetched, the list or download fails with 502 instead of letting the version through. @@ -1139,6 +1139,8 @@ Maven caveats: - Maven Central lists versions in version order, not in the order they were published. A patch to an older release line published after a newer release that is out of cooldown stays listed while it is in cooldown, but its files are still refused. - SNAPSHOT versions are not held back. +Helm uses each release's `created` time from the repository's `index.yaml`. Releases in cooldown are left out of the index the proxy serves, and their archives aren't served; a release without `created` is let through. Charts pulled over OCI have no publish time and aren't held back. + RubyGems, Hex and PyPI keep their publish times outside the metadata clients read: the RubyGems versions API, the Hex HTTP API (`upstream.hex_api`) and the PyPI JSON API. The proxy fetches them alongside each compact index, package or simple page it filters, and for PyPI downloads. Cooldown fails closed: when it applies to a package and those times can't be fetched or parsed, or the metadata can't be rewritten, the request fails with 502, or with 503 and `Retry-After` while the upstream is rate limiting the proxy, instead of serving the metadata unfiltered. The same goes for npm, Composer and pub.dev metadata that can't be rewritten, and for npm tarball downloads whose publish time can't be read. A PyPI file or npm tarball that is already cached is still served when its publish time can't be read; only a download that would have to be fetched fails. A PyPI release's time is the upload time of its newest file. The proxy stores it when a download looks it up, and serves files of that release that are already cached against the stored time without reading the JSON API again. A file that isn't cached yet is always looked up, since it may have been added to the release since, and the newer time is stored. Packages exempted with a duration of `0` are served as before and don't need the times. A PyPI package the JSON API returns 404 for has no times to go by and is served unfiltered. For RubyGems and Hex, both documents go through the metadata cache. With [`cache_metadata`](#metadata-caching) they are served from it within `metadata_ttl` without asking the upstream, revalidated with their ETags after it, and filtered from the last cached copies, marked stale, while the upstream is down; without it they are still fetched once for concurrent requests. Each document has its own TTL, so the compact index or Hex package can be newer than the cached times and list a version they don't have. The times are then fetched again before filtering, and if that fails, such versions are left out until the times can be read. The filtered index or package carries an ETag of its own, so Bundler and Mix revalidate it with `304 Not Modified`. Publish times are compared with the current time on each request, but the filtered result is kept for up to a minute (see [rewritten metadata cache](#rewritten-metadata-cache)), so a version leaving cooldown can show up a minute late; a newly published version shows up within `metadata_ttl`, as with other cached metadata. Upstream errors are answered as for other cached metadata (404, or 502) rather than relayed. diff --git a/internal/handler/cooldown_supported_test.go b/internal/handler/cooldown_supported_test.go new file mode 100644 index 00000000..7187edeb --- /dev/null +++ b/internal/handler/cooldown_supported_test.go @@ -0,0 +1,47 @@ +package handler + +import ( + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +func TestCooldownSupportedCoversHandlers(t *testing.T) { + calls := regexp.MustCompile(`(?:IsAllowed|Evaluate)\("([a-z]+)"|withheld\(h\.proxy, "([a-z]+)"|IsAllowed\((helmMetadataEcosystem)`) + files, err := filepath.Glob("*.go") + if err != nil { + t.Fatal(err) + } + found := map[string]bool{} + for _, f := range files { + if strings.HasSuffix(f, "_test.go") { + continue + } + src, err := os.ReadFile(f) + if err != nil { + t.Fatal(err) + } + for _, m := range calls.FindAllStringSubmatch(string(src), -1) { + if m[3] != "" { + found[helmMetadataEcosystem] = true + continue + } + found[m[1]+m[2]] = true + } + } + if len(found) < 10 { + t.Fatalf("found only %d ecosystems checking cooldown, the pattern no longer matches the handlers", len(found)) + } + for eco := range found { + if !CooldownSupported(eco) { + t.Errorf("%s checks cooldown but is missing from cooldownEcosystems", eco) + } + } + for eco := range cooldownEcosystems { + if !found[eco] { + t.Errorf("%s is in cooldownEcosystems but no handler checks cooldown for it", eco) + } + } +} diff --git a/internal/handler/handler.go b/internal/handler/handler.go index b3ca1c43..571a4669 100644 --- a/internal/handler/handler.go +++ b/internal/handler/handler.go @@ -283,6 +283,20 @@ type Proxy struct { started time.Time } +// cooldownEcosystems are the ecosystems whose handlers apply cooldown. Others +// have no publish time to check, so a configured cooldown has no effect there. +var cooldownEcosystems = map[string]bool{ + "cargo": true, "composer": true, "conda": true, "gem": true, + "golang": true, "helm": true, "hex": true, "maven": true, + "npm": true, "nuget": true, "pub": true, "pypi": true, +} + +// CooldownSupported reports whether the proxy applies cooldown to an +// ecosystem. +func CooldownSupported(ecosystem string) bool { + return cooldownEcosystems[ecosystem] +} + // CooldownPolicy supplies version-age filtering and package-specific durations. type CooldownPolicy interface { IsAllowed(ecosystem, packagePURL string, publishedAt time.Time) bool diff --git a/internal/server/cooldown_view.go b/internal/server/cooldown_view.go index 57a64ecb..f244c851 100644 --- a/internal/server/cooldown_view.go +++ b/internal/server/cooldown_view.go @@ -7,6 +7,7 @@ import ( "github.com/git-pkgs/cooldown" "github.com/git-pkgs/proxy/internal/config" + "github.com/git-pkgs/proxy/internal/handler" "github.com/git-pkgs/proxy/internal/packageurl" ) @@ -21,10 +22,12 @@ type CooldownSummary struct { PackagePatterns int } -// CooldownOverride is one ecosystem's cooldown. +// CooldownOverride is one ecosystem's cooldown. Unsupported marks an +// ecosystem the proxy applies no cooldown to, so the setting has no effect. type CooldownOverride struct { - Ecosystem string - Duration string + Ecosystem string + Duration string + Unsupported bool } // PackageCooldown is the cooldown that applies to one package, and the kind @@ -73,15 +76,20 @@ func (v *cooldownView) summary() CooldownSummary { PackagePatterns: len(v.cfg.PackagePatterns), } for eco, d := range v.cfg.Ecosystems { - s.Ecosystems = append(s.Ecosystems, CooldownOverride{Ecosystem: eco, Duration: formatConfigDuration(d)}) + s.Ecosystems = append(s.Ecosystems, CooldownOverride{ + Ecosystem: eco, + Duration: formatConfigDuration(d), + Unsupported: !handler.CooldownSupported(eco), + }) } sort.Slice(s.Ecosystems, func(i, j int) bool { return s.Ecosystems[i].Ecosystem < s.Ecosystems[j].Ecosystem }) return s } -// forPackage returns the cooldown for a package, or nil with cooldown off. +// forPackage returns the cooldown for a package, or nil with cooldown off or +// for an ecosystem the proxy applies no cooldown to. func (v *cooldownView) forPackage(ecosystem, name string) *PackageCooldown { - if !v.enabled() { + if !v.enabled() || !handler.CooldownSupported(ecosystem) { return nil } purl := packageurl.MakeString(ecosystem, name, "") diff --git a/internal/server/cooldown_view_test.go b/internal/server/cooldown_view_test.go index 41ff789b..13e731cb 100644 --- a/internal/server/cooldown_view_test.go +++ b/internal/server/cooldown_view_test.go @@ -3,6 +3,7 @@ package server import ( "net/http" "net/http/httptest" + "slices" "strings" "testing" "time" @@ -78,6 +79,22 @@ func TestCooldownViewOnlyPattern(t *testing.T) { } } +func TestCooldownViewUnsupportedEcosystem(t *testing.T) { + v := testCooldownView(t, config.CooldownConfig{ + Default: "3d", + Ecosystems: map[string]string{"deb": "0", "npm": "7d"}, + }) + for _, eco := range []string{"deb", "alpine", "oci", "generic"} { + if got := v.forPackage(eco, "pkg"); got != nil { + t.Errorf("forPackage(%s) = %+v, want nil", eco, got) + } + } + want := []CooldownOverride{{"deb", "off", true}, {"npm", "7d", false}} + if got := v.summary().Ecosystems; !slices.Equal(got, want) { + t.Errorf("summary ecosystems = %+v, want %+v", got, want) + } +} + func TestFormatCooldown(t *testing.T) { for d, want := range map[time.Duration]string{ 0: "off", diff --git a/internal/server/templates/pages/dashboard.html b/internal/server/templates/pages/dashboard.html index eb4c786a..0a90aedf 100644 --- a/internal/server/templates/pages/dashboard.html +++ b/internal/server/templates/pages/dashboard.html @@ -32,7 +32,7 @@

Cooldown

Default:
{{.Default}}
{{range .Ecosystems}} -
{{.Ecosystem}}:
{{.Duration}}
+
{{.Ecosystem}}:
{{.Duration}}
{{if .Unsupported}} (not supported){{end}}
{{end}} {{if .PackageOverrides}}
Package overrides:
{{.PackageOverrides}}
{{end}} {{if .PackagePatterns}}
Package patterns:
{{.PackagePatterns}}
{{end}}