Day-0 Caddyfile bare :443 prevents tls internal cert issuance

Context

Sixth friction signal from the first-real-world cloud-deployment exercise (siblings #12014, #12015, #12016, #12017, #12019). The canonical reverse-proxy reference shipped via #10803 + the Day-0 Tutorial's "copy-paste-runnable as-is" claim about tls internal are empirically broken: a fresh boot of ai/deploy --profile ingress will accept TLS connections on :443 but never serve a self-signed cert, so every browser / curl hits tlsv1 alert internal error on handshake.

Surfaced while wiring an external ai/deploy-canonical-mirror compose for OIDC validation. The same shape would fail on a fresh neo checkout the moment anyone follows the Day-0 path.

The Problem

ai/deploy/Caddyfile:27 declares the public server block as bare :443:

:443 {
    tls internal

    route {
        ...
    }
}

The inline comment + the Day-0 Tutorial promise:

Copy-paste-runnable as-is: tls internal issues a self-signed certificate, so the stack stands up with zero operator configuration.

This is incorrect. tls internal requires Caddy to know which hostname(s) to issue the cert against. Bare :443 gives Caddy no SNI / hostname to bind a cert to. Caddy starts cleanly, listens on :443, opens its admin endpoint, installs the root CA — but never issues a leaf certificate for any identifier. Every TLS handshake fails with tlsv1 alert internal error.

Empirical evidence — Caddy logs on a stack booted with this exact :443 { tls internal } block:

{"level":"info","msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","msg":"automatic HTTPS is completely disabled for server","server_name":"srv0"}
{"level":"info","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","logger":"pki.ca.local","msg":"installing root certificate ...","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","msg":"certificate installed properly in linux trusts"}
<h1 class="neo-h1" data-record-id="4">NO &quot;obtaining certificate&quot; / &quot;certificate obtained&quot; line for ANY identifier</h1>

Resulting client behavior:

$ curl -k -v https://localhost/oauth2/sign_in
* Connected to localhost (127.0.0.1) port 443
* TLS handshake, Client hello (1):
* LibreSSL/3.3.6: error:1404B438:SSL routines:ST_CONNECT:tlsv1 alert internal error

Compare with the same Caddyfile after a one-line change to localhost:443 { tls internal }:

{"level":"info","logger":"http","msg":"enabling automatic TLS certificate management","domains":["localhost"]}
{"level":"info","logger":"tls.obtain","msg":"obtaining certificate","identifier":"localhost"}
{"level":"info","logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"localhost","issuer":"local"}

Client now succeeds:

$ curl -k -s -o /dev/null -w '%{http_code}\n' https://localhost/oauth2/sign_in
200

The Architectural Reality

Affected:

• ai/deploy/Caddyfile:27 — the :443 { opener.
• learn/agentos/cloud-deployment/Day0Tutorial.md — the "copy-paste-runnable as-is" claim is empirically false for the ingress profile until the Caddyfile is fixed.
• ai/mcp/deploy/proxy/Caddyfile:14 — the older reference Caddyfile shipped by #10803 with the same shape. Same defect class if anyone uses that one too.

Caddy's TLS issuance requires an SNI hostname per Caddy automatic HTTPS docs. With :443 alone, Caddy has no identifier to bind a cert to and silently skips issuance even when tls internal is set.

The Fix

Replace :443 { with an explicit hostname. Two reasonable shapes:

1. Single canonical hostname (simplest; matches the Day-0 "copy-paste-runnable" intent):

      localhost:443 {
       tls internal
       ...
   }

2. Env-driven hostname (most flexible for operators with non-localhost deployments):

      {$NEO_DEPLOY_HOSTNAME:localhost}:443 {
       tls internal
       ...
   }

   The operator overrides NEO_DEPLOY_HOSTNAME for production; default localhost keeps the copy-paste-runnable promise intact.

Either works. Option 1 is the smallest fix; Option 2 is the smallest fix that also documents the production-hostname extension point. Implementer's call.

Day-0 Tutorial should either be updated to reflect the chosen shape (mentioning the hostname override) or the existing "copy-paste-runnable" line stays accurate if Option 2 is chosen.

Contract Ledger

Decision Record impact

aligned-with ADR 0014 — completes the cloud-deployment ingress contract under existing authority. No ADR successor risk.

Acceptance Criteria

• ai/deploy/Caddyfile opens with an explicit hostname (either localhost:443 or {$NEO_DEPLOY_HOSTNAME:localhost}:443).
• docker compose -f ai/deploy/docker-compose.yml --profile ingress up -d --wait followed by curl -k https://localhost/ returns a non-handshake-failure HTTP response on a fresh checkout (regression guard).
• Caddy logs from that boot include a certificate obtained successfully ... issuer: local line for the chosen hostname.
• ai/mcp/deploy/proxy/Caddyfile (sibling reference) carries the same fix or is explicitly retired/superseded.
• Day-0 Tutorial's "copy-paste-runnable" claim either stays accurate (Option 2 path) or is updated to note the hostname requirement (Option 1 path).

Out of Scope

• Real TLS-cert provisioning (Let's Encrypt etc.). Stay on tls internal for the reference path.
• Production deployment scripts / Ansible / etc. — operator territory.
• IPv6 listener tuning, HTTP/3 buffer-size warnings observed in logs (unrelated to this issue).
• The auto_https off global-block question (separate latent gotcha; not in canonical neo but a known external-deployment trap — could be a doc note in DeploymentCookbook).

Avoided Traps

• Filing as enhancement — rejected; the canonical reference is currently unbuildable end-to-end for the TLS step. That's a bug.
• Recommending an absolute production hostname as default — rejected; would break the "fresh checkout boots cleanly without operator config" promise. localhost default + env override (Option 2) preserves both.
• Splitting per-file (one ticket for ai/deploy/Caddyfile, one for ai/mcp/deploy/proxy/Caddyfile, one for Day-0 doc) — rejected; they share root cause and a single PR can address all three.

Related

• #10803 — originating ticket for the reverse-proxy reference (closed); shipped the defective :443 { tls internal } shape.
• #12014, #12015, #12016, #12017, #12019 — sibling friction tickets from the same external-deployment exercise.
• ADR 0014 — cloud-deployment topology authority.
• Caddy automatic-HTTPS docs — explains the SNI / hostname requirement for tls internal.

Handoff Retrieval Hints

• Empirical: boot ai/deploy --profile cloud --profile ingress, then curl -k -v https://localhost/oauth2/sign_in → tlsv1 alert internal error. After hostname fix: 200.
• Caddy log discriminator: grep "obtaining certificate" /var/log/caddy.log returns zero lines pre-fix; at least one post-fix.
• Semantic: "Caddy tls internal no hostname", "Day-0 Tutorial tls internal cert issuance failure", "bare :443 ingress profile TLS handshake".
• File anchors: ai/deploy/Caddyfile:27, ai/mcp/deploy/proxy/Caddyfile:14, learn/agentos/cloud-deployment/Day0Tutorial.md (the copy-paste-runnable line).