Reverse Proxy Operations Master Class #

Production operations reference for NGINX, Traefik, and Caddy: configuration validation, routing, TLS, upstream health, observability, safe reloads, troubleshooting, and rollback.

Overview #

Reverse proxies terminate client connections and route traffic to application services. They often sit on the critical path for every request, so configuration changes require validation, peer review, controlled rollout, and immediate rollback capability. This guide separates read-only discovery from commands that reload services or change runtime state.

Architecture Checklist #

Before changing a proxy, identify:

• The public hostname, DNS target, listener, and certificate owner.
• The proxy instance or cluster serving the request.
• The route, middleware, and upstream service selected for the hostname and path.
• Health-check behavior, connection timeouts, retries, and load-balancing policy.
• Where access logs, error logs, metrics, and traces are collected.
• The last known-good configuration and the rollback command.

NGINX Discovery and Validation #

Show the installed version and build options:

nginx -V

Validate the active configuration without reloading:

nginx -t

Dump the complete evaluated configuration:

nginx -T

Inspect service state and recent logs:

systemctl status nginx
journalctl -u nginx --since "30 minutes ago"

Check listening sockets:

ss -lntp

Reload only after nginx -t succeeds and the diff has been approved:

sudo systemctl reload nginx

A reload is safer than a restart because existing workers can finish active requests. Use a restart only when a reload cannot apply the required change and an interruption has been approved.

NGINX Routing Pattern #

A minimal TLS virtual host can proxy to an internal service:

server {
    listen 443 ssl;
    server_name app.example.com;

    ssl_certificate     /etc/ssl/app/fullchain.pem;
    ssl_certificate_key /etc/ssl/app/privkey.pem;

    location / {
        proxy_pass http://app_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_connect_timeout 5s;
        proxy_read_timeout 60s;
    }
}

Never expose private keys in configuration reviews, tickets, terminal output, or repositories. Validate file ownership and permissions without printing key content.

Traefik Discovery and Validation #

Show the Traefik version:

traefik version

Validate a static configuration file:

traefik check-config --configFile=/etc/traefik/traefik.yml

Inspect the service and logs:

systemctl status traefik
journalctl -u traefik --since "30 minutes ago"

Query the API only when its management endpoint is securely restricted:

curl --silent --show-error http://127.0.0.1:8080/api/http/routers
curl --silent --show-error http://127.0.0.1:8080/api/http/services

Restart after an approved static-configuration change:

sudo systemctl restart traefik

Dynamic provider changes may apply automatically. Confirm the provider event, route status, and upstream health instead of assuming a successful file write changed runtime state.

Traefik Routing Pattern #

A file-provider route can bind a hostname to a service:

http:
  routers:
    app:
      rule: Host(`app.example.com`)
      entryPoints: [websecure]
      service: app
      tls: {}
  services:
    app:
      loadBalancer:
        servers:
          - url: http://192.0.2.20:8080
        healthCheck:
          path: /health
          interval: 10s
          timeout: 3s

Keep the dashboard and API off public entry points unless protected by network controls and strong authentication.

Caddy Discovery and Validation #

Show the installed version:

caddy version

Validate a Caddyfile without loading it:

caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile

Format a Caddyfile and review the diff:

caddy fmt --diff /etc/caddy/Caddyfile

Adapt a Caddyfile to JSON for inspection:

caddy adapt --config /etc/caddy/Caddyfile --adapter caddyfile --pretty

Inspect service state and logs:

systemctl status caddy
journalctl -u caddy --since "30 minutes ago"

Reload an approved configuration gracefully:

sudo caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile

Caddy Routing Pattern #

A basic Caddyfile route includes automatic HTTPS and upstream health checks:

app.example.com {
    reverse_proxy 192.0.2.20:8080 {
        health_uri /health
        health_interval 10s
        health_timeout 3s
    }
    log
}

Automatic certificate issuance requires correct DNS, reachable ACME challenge paths, and access to the configured ACME provider. Verify those dependencies before changing certificate automation.

End-to-End Verification #

Resolve the hostname using an authoritative or expected resolver:

dig +short app.example.com

Inspect the TLS handshake and certificate chain:

openssl s_client -connect app.example.com:443 -servername app.example.com

Check response headers and redirects:

curl --silent --show-error --head https://app.example.com/

Test an application health endpoint:

curl --silent --show-error --fail https://app.example.com/health

Test a specific proxy IP before DNS cutover:

curl --silent --show-error --resolve app.example.com:443:192.0.2.10 https://app.example.com/health

Check the upstream directly from the proxy host:

curl --silent --show-error --fail http://192.0.2.20:8080/health

Compare direct-upstream and proxied behavior to isolate proxy, TLS, DNS, firewall, and application failures.

Common Failure Patterns #

HTTP 502 Bad Gateway #

• Confirm the upstream process is listening.
• Test the upstream from the proxy host.
• Verify address, port, protocol, and container or service-network reachability.
• Check error logs for connection refusal, timeout, or TLS negotiation failures.

HTTP 504 Gateway Timeout #

• Measure upstream response time directly.
• Check application saturation and dependency latency.
• Review proxy connect, read, and idle timeouts.
• Fix the slow dependency before increasing timeouts broadly.

Redirect Loops #

• Verify Host and X-Forwarded-Proto handling.
• Confirm only one layer performs HTTP-to-HTTPS redirection.
• Check application trusted-proxy settings.

Certificate Errors #

• Confirm hostname coverage, validity dates, and complete chain delivery.
• Check ACME account, challenge reachability, DNS propagation, and rate limits.
• Verify the proxy loaded the renewed certificate rather than only writing it to disk.

Safe Change and Rollback Workflow #

1. Capture current config, service state, routes, upstream health, and external response.
2. Create a timestamped backup or identify the version-controlled last known-good commit.
3. Make the smallest configuration change.
4. Run the proxy-specific validation command.
5. Review the diff for hostnames, ports, certificates, management endpoints, and secrets.
6. Reload or restart only after approval.
7. Verify DNS, TLS, HTTP status, headers, health endpoint, logs, and metrics.
8. Roll back immediately if errors, latency, or availability regress.

Example NGINX rollback sequence after restoring the approved configuration:

nginx -t
sudo systemctl reload nginx
curl --silent --show-error --fail https://app.example.com/health

Example Caddy rollback validation:

caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
sudo caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
curl --silent --show-error --fail https://app.example.com/health

Production Guardrails #

• Never expose Traefik, NGINX, or Caddy management endpoints publicly without explicit controls.
• Never print or commit private keys, DNS API tokens, ACME credentials, or basic-auth source values.
• Treat reloads as writes and restarts as potentially disruptive changes.
• Require explicit approval for public DNS changes, certificate-provider changes, firewall changes, or listener removal.
• Monitor error rate, latency, certificate expiry, upstream health, and configuration reload failures.
• Keep management access on a dedicated network or loopback interface wherever possible.