fetchguard

Guarded outbound HTTPS fetcher that blocks private and reserved addresses and enforces host, size and timeout limits.

import "git.golem15.com/golem15/summercms/modules/fetchguard"

Overview#

fetchguard is the framework's server-side request forgery guard for fetching URLs that come from users or third parties, such as a remote image address. Every call takes a fetchguard.Policy that either restricts the target to an allow list of hosts or permits any public host; in both modes the dial-time check refuses private, loopback, link-local, carrier-grade NAT, documentation, multicast and other reserved IPv4 and IPv6 ranges, including IPv4 embedded in NAT64 and 6to4 addresses. Failures come back as a fetchguard.Error carrying one fetchguard.Reason from a closed set, so callers can map them onto stable API error codes. WinterCMS has no dedicated counterpart; plugins there typically used Guzzle with hand-written checks.

Features#

HTTPS only: any other scheme fails with fetchguard.ReasonScheme.
Two modes: fetchguard.AllowHostsMode (exact or dotted-suffix host match against fetchguard.Policy.AllowHosts) and fetchguard.PublicOnlyMode (any public host).
The private and reserved address check runs on the resolved IP at dial time, so DNS answers that point inside the network are refused (fetchguard.ReasonPrivateIP); environment proxies are ignored so the check sees the real target.
Redirects are never followed: a 3xx response is returned as a successful fetchguard.Result, and a caller that wants to follow the Location header calls fetchguard.Fetch again, which re-runs the guard.
The response body is capped at the policy's byte limit (fetchguard.ReasonTooLarge when exceeded), with a per-call timeout.
Limits left at zero in the policy fall back to config keys, then to framework defaults of 10 MiB and 10 seconds (fetchguard.Defaults, fetchguard.DefaultsFromConfig).
Typed failure reasons: fetchguard.ReasonInvalidURL, fetchguard.ReasonScheme, fetchguard.ReasonUnresolvable, fetchguard.ReasonPrivateIP, fetchguard.ReasonNetworkError, fetchguard.ReasonTooLarge.

Usage#

policy := fetchguard.Policy{
	Mode:       fetchguard.AllowHostsMode,
	AllowHosts: []string{"images.example.com"},
	MaxBytes:   5 << 20,
	Timeout:    5 * time.Second,
}

res, err := fetchguard.Fetch(ctx, imageURL, policy, app.Config)
if err != nil {
	var fe *fetchguard.Error
	if errors.As(err, &fe) && fe.Reason == fetchguard.ReasonPrivateIP {
		return errRejectedURL
	}
	return err
}
if res.StatusCode != http.StatusOK {
	return fmt.Errorf("image fetch: status %d", res.StatusCode)
}
image := res.Body

API reference#

Identifier	Description
`fetchguard.Fetch`	Validates the URL against the policy and performs the guarded HTTPS GET; a non-nil error is always a `fetchguard.Error`.
`fetchguard.Policy`	Per-call settings: mode, allowed hosts, byte limit and timeout (zero means use the configured default).
`fetchguard.Mode`	Selects `fetchguard.AllowHostsMode` or `fetchguard.PublicOnlyMode`.
`fetchguard.Result`	Response body, Content-Type header value and status code of any completed response, including 3xx and non-2xx.
`fetchguard.Error`	Failure carrying a `fetchguard.Reason` and the underlying error for logging.
`fetchguard.Reason`	Closed set of failure reasons (`invalid_url`, `scheme`, `unresolvable`, `private_ip`, `network_error`, `too_large`).
`fetchguard.Defaults`	Framework fallback limits: 10 MiB and 10 seconds.
`fetchguard.DefaultsFromConfig`	Reads the limits from a `compass.Config`, falling back to `fetchguard.Defaults` for absent keys.

Configuration#

fetchguard.Fetch and fetchguard.DefaultsFromConfig read these keys from the compass.Config passed to them. They apply only when the policy leaves the matching limit at zero, and an explicitly configured zero or negative value is an error.

Key	Default	Controls
`http.fetch.max_bytes`	`10485760` (10 MiB)	Maximum response body size in bytes.
`http.fetch.timeout_seconds`	`10`	Dial and overall request timeout, in seconds.

http:
  fetch:
    max_bytes: 5242880
    timeout_seconds: 5

Dependencies#

SummerCMS modules: compass (config lookup).
Third-party: none.
Standard library: context, crypto/tls, errors, fmt, io, math, net, net/http, net/netip, net/url, strings, syscall, time.

Testing#

go test ./modules/fetchguard/...

The tests run against local net/http/httptest TLS servers and cover the address classifier directly; they need no external services.