A domain and HTTPS

nginx in front of the compose deployment — one script, a certificate that renews itself, and a second name for the API.

A plain VM leaves you with the stack answering plain HTTP on port 3000, which is enough to reach an IP address and not enough for a domain a browser will trust. This page is the rest: nginx in front, a Let's Encrypt certificate that renews itself, and api. as a second public name.

Everything here lives in infra/ in the repository, so the box is reproducible from a clone rather than from memory.

What you end up with

address	serves	for
`https://example.org`	the web app, which serves `/api/*` itself	everyone
`https://www.example.org`	a redirect to the apex	habit
`https://api.example.org`	the API directly	the SDK, `curl`, a scheduled Action
`http://…`	a redirect to HTTPS, except the ACME challenge	renewals

The browser never uses api.example.org

The frontend keeps calling /api/* on its own origin, exactly as it does without a domain — docker-compose.yml builds it that way, and nothing about adding a proxy changes it. The api. name is for clients that are not browsers.

This is deliberate, not tidiness. The API fetches arbitrary URLs on its caller's behalf, so its CORS allowlist is the only thing between the open internet and a proxy inside your network — which is why WEBGRAPH_ALLOWED_ORIGINS may never be *. Keeping the app's own calls same-origin means that allowlist is a second line rather than the only one.

Before you start

Point the names at the box. Three A records at your DNS provider, all to the VM's public address:

A     @      203.0.113.10
A     www    203.0.113.10
A     api    203.0.113.10

Check them before going on — the script checks too, and refuses rather than burning a certificate request against a name that does not resolve yet:

getent ahostsv4 example.org www.example.org api.example.org

Run it

On the VM, in the repository:

sudo bash infra/nginx/setup.sh example.org you@example.com

It installs nginx and certbot, obtains one certificate covering every name that resolves, renders the config from the templates in infra/nginx/, tests it, and reloads. The email is optional and only buys you expiry warnings from Let's Encrypt.

The script is idempotent. Edit a template, run it again. It will not re-request a certificate that is still valid.

Apply the settings it wrote to .env:

docker compose up -d

Four of them, and each matters:


`WEB_PORT=127.0.0.1:3000`	binds the web app to loopback
`COMPOSE_FILE=…:infra/compose.api-subdomain.yml`	publishes the API on loopback for `api.`
`WEBGRAPH_ALLOWED_ORIGINS=https://example.org`	the CORS allowlist, no longer localhost
`WEBGRAPH_PUBLIC_BASE_URL=https://api.example.org`	what a feed calls itself

This recreates both containers, because their environment changed — the API reads its settings once at process start, so there is no way to apply them without one. Any crawl in flight is lost; finished site graphs are not, they live in the graph-cache volume.

Check it, from anywhere:

curl -sI  https://example.org           # 200
curl -s   https://example.org/api/health
curl -s   https://api.example.org/api/health
curl -sI  http://example.org            # 301 to https

http://YOUR_IP:3000 stops working, on purpose

WEB_PORT=127.0.0.1:3000 is what makes nginx the only way in. Until now the container published port 3000 on every interface; after this it is reachable only from the host itself.

On a Docker host that bind address is the access control, and a firewall is not a substitute: published ports are reached through Docker's own DOCKER-USER iptables chain, which a ufw rule added later does not override. A port bound to 127.0.0.1 is not subject to that argument at all.

Why the config lives in the repository

The usual recipe is certbot --nginx, which finds your config and rewrites it. Run it once and the file on the box no longer matches the file in git — and you discover that the next time someone re-renders and silently undoes a hand edit.

setup.sh uses certbot certonly --webroot, which obtains the certificate and touches nothing else. infra/nginx/webgraph.conf.template stays the only description of how this deployment is served.

The one line to never get wrong

The port-80 block serves /.well-known/acme-challenge/ before it redirects to HTTPS. Put the redirect first and renewal fails 60 days later, over plain HTTP, on a deployment that has looked perfectly healthy the whole time — and you find out when the certificate expires at 90.

What the proxy has to get right for this API

Three settings in infra/nginx/snippets/webgraph-api.conf.template, each for a specific endpoint rather than for general good practice:

Buffering off. Five endpoints answer text/event-stream — the site report, the crawl, the text stream, the graph, and the knowledge-graph routes — and they are the live views the product is built around. With nginx's default buffering the events are held until the response ends, which for a crawl is minutes later: the progress view shows nothing, then everything, and looks broken throughout. The application already sends X-Accel-Buffering: no, which nginx honours on its own; the proxy says it again because the failure is invisible until someone watches a real crawl.

An hour of patience. nginx gives up after 60 seconds of silence by default. A crawl renders every page in a real browser and paces its requests a second apart out of politeness, so a quiet minute is ordinary rather than a fault.

32 MB bodies. /api/text accepts an html body — the bring-your-own-HTML path for a site that will not serve us directly. Real pages exceed nginx's 1 MB default routinely, and the 413 comes from the proxy, so the error blames nginx for something the API would have accepted.

Renewal

certbot's own systemd timer, installed with the package. setup.sh registers a deploy hook that reloads nginx when it fires — without it certbot writes a new certificate and nginx keeps serving the one it loaded at start.

systemctl list-timers 'certbot*'
certbot renew --dry-run