Nginx Zero-Downtime Reload — nginx -s reload Internals and Deployment Automation
One of Nginx's most powerful features is the ability to apply configuration changes without any service interruption. With a single nginx -s reload command, configuration changes can take effect while maintaining tens of thousands of concurrent connections. This chapter comprehensively covers everything from the internal mechanics of Nginx reload to deployment pipeline automation.
How nginx -s reload Works
Master/Worker Process Architecture
Nginx consists of one Master process and multiple Worker processes.
┌─────────────────────────────────────────────────────────────────┐
│ Nginx Process Structure │
│ │
│ ┌──────────────────────────────────────────┐ │
│ │ Master Process │ │
│ │ - Reads and validates config files │ │
│ │ - Creates and manages Worker processes │ │
│ │ - Handles signals (HUP, QUIT, TERM...) │ │
│ │ - PID: /var/run/nginx.pid │ │
│ └────────────────┬─────────────────────────┘ │
│ │ fork() │
│ ┌─────────────┼─────────────┐ │
│ ▼ ▼ ▼ │
│ Worker 1 Worker 2 Worker N │
│ (handles (handles (handles │
│ requests) requests) requests) │
└─────────────────────────────────────────────────────────────────┘
The Master process runs as root, parses configuration files, and manages Worker processes. It does not handle actual client requests.
Worker processes handle actual HTTP requests. Each Worker runs single-threaded, asynchronously handling thousands of concurrent connections based on an event loop (epoll/kqueue).
Internal Flow During Reload
nginx -s reload executed
│
▼
HUP signal sent to Master process
│
▼
Master: Reads and validates new configuration file
│
┌────┴────┐
│Validation│ → Log error, keep current Workers running (no service interruption)
│ failed │
│Validation│
│succeeded │
└────┬────┘
│
▼
Master: Spawns new Worker processes with new configuration
│
▼
New Workers: Begin accepting new connections
│
▼
Master: Sends Graceful Shutdown signal to existing Workers
│
▼
Existing Workers: Complete in-progress requests, then exit
│
▼
Complete: Only new Workers running
Throughout this process, not a single client request is dropped. Existing connections are completed by the old Workers; new connections are handled by the new Workers.
Direct Signal Control
nginx -s reload is internally equivalent to kill -HUP $(cat /var/run/nginx.pid).
# Check Master PID from nginx.pid file
cat /var/run/nginx.pid
# Example: 1234
# Send signal directly (identical to nginx -s reload)
kill -HUP 1234
# Or use nginx -s command
nginx -s reload # HUP signal (zero-downtime reload)
nginx -s quit # QUIT signal (graceful shutdown)
nginx -s stop # TERM signal (immediate stop)
nginx -s reopen # USR1 signal (reopen log files)
nginx -t Configuration Validation
Always run nginx -t to validate configuration syntax before reloading. Reloading with an invalid configuration can leave Nginx in an error state.
# Basic syntax validation
nginx -t
# Example successful output:
# nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /etc/nginx/nginx.conf test is successful
# Example failed output:
# nginx: [emerg] unexpected ";" in /etc/nginx/conf.d/app.conf:15
# nginx: configuration file /etc/nginx/nginx.conf test failed
# Verbose output (shows entire validation process)
nginx -T
# Validate with a specific config file
nginx -t -c /path/to/custom/nginx.conf
Common Configuration Error Types
# 1. Missing semicolon
server_name example.com # Error: no semicolon
server_name example.com; # Correct
# 2. Mismatched braces
server {
listen 80;
# Error: missing closing brace
# 3. Invalid directive
proxiy_pass http://backend; # Error: typo (proxiy)
proxy_pass http://backend; # Correct
# 4. Undefined upstream
proxy_pass http://undefined_upstream; # Error: no upstream block defined
Dynamic Upstream Change Pattern
The safest way to change the upstream server list during deployment is to use symbolic links.
Atomic Replacement Using Symbolic Links
# Directory structure
/etc/nginx/
├── conf.d/
│ ├── upstream_v1.conf # v1 upstream config
│ ├── upstream_v2.conf # v2 upstream config
│ └── upstream.conf → upstream_v1.conf # symlink (currently active)
└── nginx.conf # include conf.d/upstream.conf;
# upstream_v1.conf
upstream app_backend {
server 192.168.1.20:8080;
server 192.168.1.21:8080;
}
# upstream_v2.conf
upstream app_backend {
server 192.168.1.30:8080;
server 192.168.1.31:8080;
}
# Atomic symlink swap (ln -sfn is an atomic operation)
ln -sfn /etc/nginx/conf.d/upstream_v2.conf /etc/nginx/conf.d/upstream.conf
# Validate then reload
nginx -t && nginx -s reload
ln -sfn is an atomic operation, so even if a request arrives during the swap, it always references a complete configuration file.
Deployment Automation Script
deploy-nginx.sh — Complete Deployment Pipeline
#!/bin/bash
# deploy-nginx.sh
# Usage: ./deploy-nginx.sh <new_upstream_conf>
set -euo pipefail # Exit immediately on error
NGINX_CONF_DIR="/etc/nginx/conf.d"
UPSTREAM_SYMLINK="$NGINX_CONF_DIR/upstream.conf"
NEW_CONF=$1
BACKUP_CONF="${UPSTREAM_SYMLINK}.bak"
LOG_FILE="/var/log/nginx-deploy.log"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
log() {
echo "[$TIMESTAMP] $1" | tee -a "$LOG_FILE"
}
# Pre-checks
if [ -z "$NEW_CONF" ] || [ ! -f "$NEW_CONF" ]; then
echo "Usage: $0 <new_upstream_conf_path>"
exit 1
fi
log "Starting deployment with config: $NEW_CONF"
# 1. Backup current config (for rollback)
if [ -L "$UPSTREAM_SYMLINK" ]; then
CURRENT_CONF=$(readlink "$UPSTREAM_SYMLINK")
log "Current config: $CURRENT_CONF"
ln -sfn "$CURRENT_CONF" "$BACKUP_CONF"
fi
# 2. Replace symlink with new config
log "Updating symlink to $NEW_CONF..."
ln -sfn "$NEW_CONF" "$UPSTREAM_SYMLINK"
# 3. Validate configuration
log "Validating nginx configuration..."
if ! nginx -t 2>&1 | tee -a "$LOG_FILE"; then
log "ERROR: Nginx configuration test failed. Rolling back..."
# Rollback: restore previous config
if [ -L "$BACKUP_CONF" ]; then
ln -sfn "$(readlink $BACKUP_CONF)" "$UPSTREAM_SYMLINK"
rm -f "$BACKUP_CONF"
fi
exit 1
fi
# 4. Reload Nginx
log "Reloading Nginx..."
if nginx -s reload; then
log "Nginx reloaded successfully"
else
log "ERROR: Nginx reload failed. Rolling back..."
if [ -L "$BACKUP_CONF" ]; then
ln -sfn "$(readlink $BACKUP_CONF)" "$UPSTREAM_SYMLINK"
nginx -s reload
fi
exit 1
fi
# 5. Health check (verify service is normal after reload)
sleep 2
log "Running post-reload health check..."
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
--max-time 5 http://127.0.0.1/health 2>/dev/null)
if [ "$HTTP_CODE" = "200" ]; then
log "Health check passed (HTTP 200)"
rm -f "$BACKUP_CONF"
log "Deployment completed successfully!"
else
log "WARNING: Health check returned HTTP $HTTP_CODE"
log "Rolling back..."
if [ -L "$BACKUP_CONF" ]; then
ln -sfn "$(readlink $BACKUP_CONF)" "$UPSTREAM_SYMLINK"
nginx -s reload
log "Rolled back to previous configuration"
fi
exit 1
fi
Automatic Rollback Script on Reload Failure
#!/bin/bash
# safe-nginx-reload.sh
NGINX_CONF_DIR="/etc/nginx/conf.d"
MAX_RETRIES=3
ROLLBACK_WAIT=5
safe_reload() {
local ATTEMPT=1
while [ $ATTEMPT -le $MAX_RETRIES ]; do
echo "Reload attempt $ATTEMPT of $MAX_RETRIES..."
# Config validation
if ! nginx -t &>/dev/null; then
echo "Config validation failed on attempt $ATTEMPT"
ATTEMPT=$((ATTEMPT + 1))
sleep $ROLLBACK_WAIT
continue
fi
# Execute reload
if nginx -s reload; then
sleep 2
# Post-reload health check
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
--max-time 5 http://127.0.0.1/health)
if [ "$HTTP_CODE" = "200" ]; then
echo "Reload successful!"
return 0
else
echo "Post-reload health check failed: HTTP $HTTP_CODE"
fi
fi
ATTEMPT=$((ATTEMPT + 1))
echo "Waiting ${ROLLBACK_WAIT}s before retry..."
sleep $ROLLBACK_WAIT
done
echo "All reload attempts failed. Manual intervention required."
return 1
}
safe_reload
Nginx Reload in Docker Environments
Sending Reload Signal to Running Container
# Method 1: Using docker exec
docker exec nginx_container nginx -s reload
# Method 2: Sending HUP signal with docker kill
docker kill --signal=HUP nginx_container
# Method 3: With docker compose
docker compose exec nginx nginx -s reload
# Validate config then reload
docker exec nginx_container nginx -t && \
docker exec nginx_container nginx -s reload
Updating Config via Docker Volume
# docker-compose.yml
version: '3.8'
services:
nginx:
image: nginx:alpine
volumes:
- ./nginx/conf.d:/etc/nginx/conf.d:ro
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- "80:80"
- "443:443"
# Edit config file on host, then reload
vim ./nginx/conf.d/upstream.conf
# Validate inside container then reload
docker exec nginx_container nginx -t && \
docker exec nginx_container nginx -s reload
Nginx Reload Integration with GitHub Actions Workflow
# .github/workflows/deploy.yml
name: Deploy and Reload Nginx
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Build application
run: |
docker build -t myapp:${{ github.sha }} .
docker tag myapp:${{ github.sha }} myapp:latest
- name: Deploy to server
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.SERVER_HOST }}
username: ${{ secrets.SERVER_USER }}
key: ${{ secrets.SSH_PRIVATE_KEY }}
script: |
# 1. Deploy new container
docker pull myapp:${{ github.sha }}
docker stop app-green || true
docker rm app-green || true
docker run -d --name app-green -p 8081:8080 myapp:${{ github.sha }}
# 2. Wait for health check
for i in {1..30}; do
HTTP=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8081/health)
[ "$HTTP" = "200" ] && break
echo "Waiting for app-green... attempt $i"
sleep 3
done
# 3. Update Nginx upstream
cat > /etc/nginx/conf.d/upstream_new.conf << 'EOF'
upstream app_backend {
server 127.0.0.1:8081;
}
EOF
# 4. Validate config and reload
nginx -t && nginx -s reload
# 5. Stop old container
docker stop app-blue || true
docker rm app-blue || true
docker rename app-green app-blue
echo "Deployment completed: ${{ github.sha }}"
- name: Notify on success
if: success()
run: |
curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
-H "Content-type: application/json" \
-d '{"text": "Deployment successful: ${{ github.sha }}"}'
- name: Notify on failure
if: failure()
run: |
curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
-H "Content-type: application/json" \
-d '{"text": "Deployment FAILED: ${{ github.sha }}"}'
Signal Sending via nginx_pid
# Find nginx.pid location (may differ by distribution)
nginx -V 2>&1 | grep pid # Check --pid-path option
# Common locations
/var/run/nginx.pid # Ubuntu/Debian
/run/nginx.pid # Newer systemd-based distributions
/usr/local/nginx/logs/nginx.pid # Source-compiled installation
# Direct signal via PID
NGINX_PID=$(cat /var/run/nginx.pid)
kill -HUP $NGINX_PID # Zero-downtime reload
kill -QUIT $NGINX_PID # Graceful shutdown
kill -TERM $NGINX_PID # Immediate stop
kill -USR1 $NGINX_PID # Reopen log files
# View both Master and Worker PIDs
ps aux | grep nginx
# Example output:
# root 1234 0.0 0.0 ... nginx: master process
# www-data 1235 0.1 0.2 ... nginx: worker process
# www-data 1236 0.1 0.2 ... nginx: worker process
Verifying Service Continuity Before and After Reload
#!/bin/bash
# verify-reload.sh — Verify service continuity before and after reload
TARGET_URL="http://localhost/health"
DURATION=30 # Test duration (seconds)
INTERVAL=0.5 # Request interval (seconds)
SUCCESS=0
FAILED=0
echo "Starting continuous request test for ${DURATION} seconds..."
# Continuous requests in the background
start_time=$(date +%s)
while true; do
current_time=$(date +%s)
elapsed=$((current_time - start_time))
[ $elapsed -ge $DURATION ] && break
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
--max-time 2 "$TARGET_URL" 2>/dev/null)
if [ "$HTTP_CODE" = "200" ]; then
SUCCESS=$((SUCCESS + 1))
else
FAILED=$((FAILED + 1))
echo "[$(date)] FAILED: HTTP $HTTP_CODE"
fi
sleep $INTERVAL
done
TOTAL=$((SUCCESS + FAILED))
echo ""
echo "=== Test Results ==="
echo "Total requests: $TOTAL"
echo "Success: $SUCCESS"
echo "Failed: $FAILED"
echo "Success rate: $(echo "scale=2; $SUCCESS * 100 / $TOTAL" | bc)%"
Run this script simultaneously while performing an actual reload to precisely measure request loss during the reload.
Pro Tips
nginx -s reloadonly creates new Workers when the configuration file is valid. If there's a config error, the existing Workers continue running, so service is maintained.- In deployment automation scripts, always run
nginx -tvalidation before the reload, and abort immediately on failure. - In high-traffic environments, use the
worker_shutdown_timeoutdirective to set a graceful shutdown timeout for existing Workers. - When using with systemd,
systemctl reload nginxis safer since systemd manages the process state. nginx -T(uppercase) outputs the entire currently applied configuration, making it extremely useful for debugging.