Skip to main content

Nginx Zero-Downtime Reload — nginx -s reload Internals and Deployment Automation

One of Nginx's most powerful features is the ability to apply configuration changes without any service interruption. With a single nginx -s reload command, configuration changes can take effect while maintaining tens of thousands of concurrent connections. This chapter comprehensively covers everything from the internal mechanics of Nginx reload to deployment pipeline automation.

How nginx -s reload Works

Master/Worker Process Architecture

Nginx consists of one Master process and multiple Worker processes.

┌─────────────────────────────────────────────────────────────────┐
│ Nginx Process Structure │
│ │
│ ┌──────────────────────────────────────────┐ │
│ │ Master Process │ │
│ │ - Reads and validates config files │ │
│ │ - Creates and manages Worker processes │ │
│ │ - Handles signals (HUP, QUIT, TERM...) │ │
│ │ - PID: /var/run/nginx.pid │ │
│ └────────────────┬─────────────────────────┘ │
│ │ fork() │
│ ┌─────────────┼─────────────┐ │
│ ▼ ▼ ▼ │
│ Worker 1 Worker 2 Worker N │
│ (handles (handles (handles │
│ requests) requests) requests) │
└─────────────────────────────────────────────────────────────────┘

The Master process runs as root, parses configuration files, and manages Worker processes. It does not handle actual client requests.

Worker processes handle actual HTTP requests. Each Worker runs single-threaded, asynchronously handling thousands of concurrent connections based on an event loop (epoll/kqueue).

Internal Flow During Reload

nginx -s reload executed


HUP signal sent to Master process


Master: Reads and validates new configuration file

┌────┴────┐
│Validation│ → Log error, keep current Workers running (no service interruption)
│ failed │
│Validation│
│succeeded │
└────┬────┘


Master: Spawns new Worker processes with new configuration


New Workers: Begin accepting new connections


Master: Sends Graceful Shutdown signal to existing Workers


Existing Workers: Complete in-progress requests, then exit


Complete: Only new Workers running

Throughout this process, not a single client request is dropped. Existing connections are completed by the old Workers; new connections are handled by the new Workers.

Direct Signal Control

nginx -s reload is internally equivalent to kill -HUP $(cat /var/run/nginx.pid).

# Check Master PID from nginx.pid file
cat /var/run/nginx.pid
# Example: 1234

# Send signal directly (identical to nginx -s reload)
kill -HUP 1234

# Or use nginx -s command
nginx -s reload # HUP signal (zero-downtime reload)
nginx -s quit # QUIT signal (graceful shutdown)
nginx -s stop # TERM signal (immediate stop)
nginx -s reopen # USR1 signal (reopen log files)

nginx -t Configuration Validation

Always run nginx -t to validate configuration syntax before reloading. Reloading with an invalid configuration can leave Nginx in an error state.

# Basic syntax validation
nginx -t

# Example successful output:
# nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /etc/nginx/nginx.conf test is successful

# Example failed output:
# nginx: [emerg] unexpected ";" in /etc/nginx/conf.d/app.conf:15
# nginx: configuration file /etc/nginx/nginx.conf test failed

# Verbose output (shows entire validation process)
nginx -T

# Validate with a specific config file
nginx -t -c /path/to/custom/nginx.conf

Common Configuration Error Types

# 1. Missing semicolon
server_name example.com # Error: no semicolon
server_name example.com; # Correct

# 2. Mismatched braces
server {
listen 80;
# Error: missing closing brace

# 3. Invalid directive
proxiy_pass http://backend; # Error: typo (proxiy)
proxy_pass http://backend; # Correct

# 4. Undefined upstream
proxy_pass http://undefined_upstream; # Error: no upstream block defined

Dynamic Upstream Change Pattern

The safest way to change the upstream server list during deployment is to use symbolic links.

# Directory structure
/etc/nginx/
├── conf.d/
│ ├── upstream_v1.conf # v1 upstream config
│ ├── upstream_v2.conf # v2 upstream config
│ └── upstream.conf → upstream_v1.conf # symlink (currently active)
└── nginx.conf # include conf.d/upstream.conf;
# upstream_v1.conf
upstream app_backend {
server 192.168.1.20:8080;
server 192.168.1.21:8080;
}

# upstream_v2.conf
upstream app_backend {
server 192.168.1.30:8080;
server 192.168.1.31:8080;
}
# Atomic symlink swap (ln -sfn is an atomic operation)
ln -sfn /etc/nginx/conf.d/upstream_v2.conf /etc/nginx/conf.d/upstream.conf

# Validate then reload
nginx -t && nginx -s reload

ln -sfn is an atomic operation, so even if a request arrives during the swap, it always references a complete configuration file.

Deployment Automation Script

deploy-nginx.sh — Complete Deployment Pipeline

#!/bin/bash
# deploy-nginx.sh
# Usage: ./deploy-nginx.sh <new_upstream_conf>

set -euo pipefail # Exit immediately on error

NGINX_CONF_DIR="/etc/nginx/conf.d"
UPSTREAM_SYMLINK="$NGINX_CONF_DIR/upstream.conf"
NEW_CONF=$1
BACKUP_CONF="${UPSTREAM_SYMLINK}.bak"
LOG_FILE="/var/log/nginx-deploy.log"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')

log() {
echo "[$TIMESTAMP] $1" | tee -a "$LOG_FILE"
}

# Pre-checks
if [ -z "$NEW_CONF" ] || [ ! -f "$NEW_CONF" ]; then
echo "Usage: $0 <new_upstream_conf_path>"
exit 1
fi

log "Starting deployment with config: $NEW_CONF"

# 1. Backup current config (for rollback)
if [ -L "$UPSTREAM_SYMLINK" ]; then
CURRENT_CONF=$(readlink "$UPSTREAM_SYMLINK")
log "Current config: $CURRENT_CONF"
ln -sfn "$CURRENT_CONF" "$BACKUP_CONF"
fi

# 2. Replace symlink with new config
log "Updating symlink to $NEW_CONF..."
ln -sfn "$NEW_CONF" "$UPSTREAM_SYMLINK"

# 3. Validate configuration
log "Validating nginx configuration..."
if ! nginx -t 2>&1 | tee -a "$LOG_FILE"; then
log "ERROR: Nginx configuration test failed. Rolling back..."
# Rollback: restore previous config
if [ -L "$BACKUP_CONF" ]; then
ln -sfn "$(readlink $BACKUP_CONF)" "$UPSTREAM_SYMLINK"
rm -f "$BACKUP_CONF"
fi
exit 1
fi

# 4. Reload Nginx
log "Reloading Nginx..."
if nginx -s reload; then
log "Nginx reloaded successfully"
else
log "ERROR: Nginx reload failed. Rolling back..."
if [ -L "$BACKUP_CONF" ]; then
ln -sfn "$(readlink $BACKUP_CONF)" "$UPSTREAM_SYMLINK"
nginx -s reload
fi
exit 1
fi

# 5. Health check (verify service is normal after reload)
sleep 2
log "Running post-reload health check..."
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
--max-time 5 http://127.0.0.1/health 2>/dev/null)

if [ "$HTTP_CODE" = "200" ]; then
log "Health check passed (HTTP 200)"
rm -f "$BACKUP_CONF"
log "Deployment completed successfully!"
else
log "WARNING: Health check returned HTTP $HTTP_CODE"
log "Rolling back..."
if [ -L "$BACKUP_CONF" ]; then
ln -sfn "$(readlink $BACKUP_CONF)" "$UPSTREAM_SYMLINK"
nginx -s reload
log "Rolled back to previous configuration"
fi
exit 1
fi

Automatic Rollback Script on Reload Failure

#!/bin/bash
# safe-nginx-reload.sh

NGINX_CONF_DIR="/etc/nginx/conf.d"
MAX_RETRIES=3
ROLLBACK_WAIT=5

safe_reload() {
local ATTEMPT=1

while [ $ATTEMPT -le $MAX_RETRIES ]; do
echo "Reload attempt $ATTEMPT of $MAX_RETRIES..."

# Config validation
if ! nginx -t &>/dev/null; then
echo "Config validation failed on attempt $ATTEMPT"
ATTEMPT=$((ATTEMPT + 1))
sleep $ROLLBACK_WAIT
continue
fi

# Execute reload
if nginx -s reload; then
sleep 2
# Post-reload health check
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
--max-time 5 http://127.0.0.1/health)
if [ "$HTTP_CODE" = "200" ]; then
echo "Reload successful!"
return 0
else
echo "Post-reload health check failed: HTTP $HTTP_CODE"
fi
fi

ATTEMPT=$((ATTEMPT + 1))
echo "Waiting ${ROLLBACK_WAIT}s before retry..."
sleep $ROLLBACK_WAIT
done

echo "All reload attempts failed. Manual intervention required."
return 1
}

safe_reload

Nginx Reload in Docker Environments

Sending Reload Signal to Running Container

# Method 1: Using docker exec
docker exec nginx_container nginx -s reload

# Method 2: Sending HUP signal with docker kill
docker kill --signal=HUP nginx_container

# Method 3: With docker compose
docker compose exec nginx nginx -s reload

# Validate config then reload
docker exec nginx_container nginx -t && \
docker exec nginx_container nginx -s reload

Updating Config via Docker Volume

# docker-compose.yml
version: '3.8'
services:
nginx:
image: nginx:alpine
volumes:
- ./nginx/conf.d:/etc/nginx/conf.d:ro
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- "80:80"
- "443:443"
# Edit config file on host, then reload
vim ./nginx/conf.d/upstream.conf

# Validate inside container then reload
docker exec nginx_container nginx -t && \
docker exec nginx_container nginx -s reload

Nginx Reload Integration with GitHub Actions Workflow

# .github/workflows/deploy.yml
name: Deploy and Reload Nginx

on:
push:
branches: [main]

jobs:
deploy:
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Build application
run: |
docker build -t myapp:${{ github.sha }} .
docker tag myapp:${{ github.sha }} myapp:latest

- name: Deploy to server
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.SERVER_HOST }}
username: ${{ secrets.SERVER_USER }}
key: ${{ secrets.SSH_PRIVATE_KEY }}
script: |
# 1. Deploy new container
docker pull myapp:${{ github.sha }}
docker stop app-green || true
docker rm app-green || true
docker run -d --name app-green -p 8081:8080 myapp:${{ github.sha }}

# 2. Wait for health check
for i in {1..30}; do
HTTP=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8081/health)
[ "$HTTP" = "200" ] && break
echo "Waiting for app-green... attempt $i"
sleep 3
done

# 3. Update Nginx upstream
cat > /etc/nginx/conf.d/upstream_new.conf << 'EOF'
upstream app_backend {
server 127.0.0.1:8081;
}
EOF

# 4. Validate config and reload
nginx -t && nginx -s reload

# 5. Stop old container
docker stop app-blue || true
docker rm app-blue || true
docker rename app-green app-blue

echo "Deployment completed: ${{ github.sha }}"

- name: Notify on success
if: success()
run: |
curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
-H "Content-type: application/json" \
-d '{"text": "Deployment successful: ${{ github.sha }}"}'

- name: Notify on failure
if: failure()
run: |
curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
-H "Content-type: application/json" \
-d '{"text": "Deployment FAILED: ${{ github.sha }}"}'

Signal Sending via nginx_pid

# Find nginx.pid location (may differ by distribution)
nginx -V 2>&1 | grep pid # Check --pid-path option

# Common locations
/var/run/nginx.pid # Ubuntu/Debian
/run/nginx.pid # Newer systemd-based distributions
/usr/local/nginx/logs/nginx.pid # Source-compiled installation

# Direct signal via PID
NGINX_PID=$(cat /var/run/nginx.pid)
kill -HUP $NGINX_PID # Zero-downtime reload
kill -QUIT $NGINX_PID # Graceful shutdown
kill -TERM $NGINX_PID # Immediate stop
kill -USR1 $NGINX_PID # Reopen log files

# View both Master and Worker PIDs
ps aux | grep nginx
# Example output:
# root 1234 0.0 0.0 ... nginx: master process
# www-data 1235 0.1 0.2 ... nginx: worker process
# www-data 1236 0.1 0.2 ... nginx: worker process

Verifying Service Continuity Before and After Reload

#!/bin/bash
# verify-reload.sh — Verify service continuity before and after reload

TARGET_URL="http://localhost/health"
DURATION=30 # Test duration (seconds)
INTERVAL=0.5 # Request interval (seconds)
SUCCESS=0
FAILED=0

echo "Starting continuous request test for ${DURATION} seconds..."

# Continuous requests in the background
start_time=$(date +%s)
while true; do
current_time=$(date +%s)
elapsed=$((current_time - start_time))
[ $elapsed -ge $DURATION ] && break

HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
--max-time 2 "$TARGET_URL" 2>/dev/null)

if [ "$HTTP_CODE" = "200" ]; then
SUCCESS=$((SUCCESS + 1))
else
FAILED=$((FAILED + 1))
echo "[$(date)] FAILED: HTTP $HTTP_CODE"
fi

sleep $INTERVAL
done

TOTAL=$((SUCCESS + FAILED))
echo ""
echo "=== Test Results ==="
echo "Total requests: $TOTAL"
echo "Success: $SUCCESS"
echo "Failed: $FAILED"
echo "Success rate: $(echo "scale=2; $SUCCESS * 100 / $TOTAL" | bc)%"

Run this script simultaneously while performing an actual reload to precisely measure request loss during the reload.

Pro Tips

  • nginx -s reload only creates new Workers when the configuration file is valid. If there's a config error, the existing Workers continue running, so service is maintained.
  • In deployment automation scripts, always run nginx -t validation before the reload, and abort immediately on failure.
  • In high-traffic environments, use the worker_shutdown_timeout directive to set a graceful shutdown timeout for existing Workers.
  • When using with systemd, systemctl reload nginx is safer since systemd manages the process state.
  • nginx -T (uppercase) outputs the entire currently applied configuration, making it extremely useful for debugging.