Skip to main content

Connection Pooling & Keep-Alive: Eliminating Connection Overhead

Opening a new TCP connection for every HTTP request incurs the cost of a 3-way handshake plus a TLS handshake. Keep-Alive and Connection Pooling eliminate most of this overhead.

Understanding Connection Cost​

Reuse an existing Keep-Alive connection   β†’ ~1 ms
TCP 3-way Handshake                       β†’ ~10 ms (round-trip latency)
+ TLS 1.3 Handshake (1-RTT)               β†’ ~20 ms more
+ TLS 1.2 Handshake (2-RTT)               β†’ ~40 ms more
────────────────────────────────────────────
Total cost per new connection (TLS 1.2)   β†’ ~50 ms

On a service handling 1,000 requests/s with no Keep-Alive, handshakes alone would add 50 seconds of latency per second β€” an impossible load.

Nginx Upstream keepalive (Nginx ↔ Backend)​

upstream backend {
    server app1.internal:8080;
    server app2.internal:8080;
    server app3.internal:8080;

    # Idle keepalive connections per worker to maintain
    keepalive 64;

    # Maximum requests per keepalive connection (then reconnect)
    keepalive_requests 1000;

    # How long to keep an idle connection open
    keepalive_time    1h;
    keepalive_timeout 60s;
}

server {
    location / {
        proxy_pass http://backend;

        # HTTP/1.1 is required for Keep-Alive
        proxy_http_version 1.1;

        # Remove the Connection header (HTTP/1.0 default)
        proxy_set_header Connection "";
    }
}

Client ↔ Nginx Keep-Alive​

http {
    # Client keepalive duration (default: 75s)
    keepalive_timeout 65;

    # Maximum requests per keepalive connection
    keepalive_requests 1000;
}

Apache KeepAlive​

# /etc/apache2/apache2.conf

KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 5

Tomcat Connection Pool (server.xml)​

<Connector port="8080"
           protocol="HTTP/1.1"
           acceptCount="100"
           maxConnections="10000"
           maxThreads="200"
           minSpareThreads="10"
           keepAliveTimeout="20000"
           maxKeepAliveRequests="100"
           connectionTimeout="20000"
           compression="on"
           compressionMinSize="2048"
           compressableMimeType="text/html,text/xml,text/plain,text/css,
                                  application/json,application/javascript"
           />

Timeout Matrix​

Timeouts across the connection chain must be consistent to prevent 502/504 errors.

Client ──[65s]──▢ Nginx ──[60s]──▢ Tomcat
server {
    # Client-side timeouts
    client_header_timeout  10s;
    client_body_timeout    30s;
    send_timeout           30s;

    location / {
        proxy_pass http://backend;

        proxy_connect_timeout  5s;    # time to establish TCP to backend
        proxy_send_timeout     30s;   # time to send the request to backend
        proxy_read_timeout     60s;   # time to wait for backend response

        # Tomcat connectionTimeout (20 s) < proxy_read_timeout (60 s)
        # If Tomcat closes first β†’ Nginx returns 502
        # If Nginx closes first β†’ client receives 504
    }
}

Rule: backend timeout < Nginx timeout < client timeout

Monitoring Connection Pool State​

# Nginx connection status
curl http://localhost/nginx_status
# Active connections: 45
# Reading: 0 Writing: 5 Waiting: 40  ← Waiting = idle keepalive connections

# Tomcat thread state
curl http://localhost:8080/manager/status

# OS-level socket state
ss -s
ss -tn | awk '{print $1}' | sort | uniq -c
# ESTABLISHED: active connections
# TIME_WAIT:   closing (too many = missing keepalive)
# CLOSE_WAIT:  app not closing sockets (possible leak)

Resolving Too Many TIME_WAIT Sockets​

# Count TIME_WAIT sockets (over 10,000 is a problem)
ss -tn | grep TIME_WAIT | wc -l

# Fix 1: Enable keepalive (root fix)
# Fix 2: OS TCP reuse settings

# /etc/sysctl.conf
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.ip_local_port_range = 1024 65535

sudo sysctl -p