Upgrading Nginx normally means restarting the service, and the connections being processed go down with it. The hot upgrade takes a different path: the USR2 signal runs the old and new master processes side by side, the new version takes new traffic, and the old version finishes the requests in hand before exiting — no downtime throughout. OpenResty’s hot-upgrade flow is exactly the same. This post runs the entire signal flow in a container — upgrade, verification, rollback — plus one real pitfall that makes USR2 silently fail.

Lab Environment

  • Docker container nginx:1.28-alpine (tested 2026-09): a separate instance started inside the container (-p for the prefix, an independent port); the running old version is nginx/1.28.3.
  • The new binary is a source-built nginx/1.30.4 (the objs/nginx produced by ./configure --with-http_ssl_module && make) — a genuine cross-version binary swap, not a same-version copy; the signal flow matches a production upgrade to the letter.
  • The original article’s requirement still holds: the configure arguments, dependency library versions, and install paths of the new and old binaries must match; a “new version” missing libraries only exposes problems at runtime.

The Core Mechanism in One Sentence

The essence of USR2: the old master renames nginx.pid to nginx.pid.oldbin, then forks and execs a fresh copy of the new binary — the new master inherits the old master’s listening sockets (no re-bind), while the old master keeps serving established connections. Afterwards, WINCH retires the old workers gracefully and QUIT retires the old master; if anything goes wrong, walking the signals in reverse returns you to the old version.

Step One: Back Up and Swap the Binary

1
2
cp /usr/sbin/nginx /srv/nginx1/nginx.old           # back up the old binary (1.28.3)
cp /tmp/nginx-1.30.4/objs/nginx /usr/sbin/nginx    # the new build in place (1.30.4)

Note: a source-built Nginx’s default binary lives at /usr/local/nginx/sbin/nginx — a different path, the same action: back up the old file, put the new one at the original path. The running master is unaffected by the overwrite; it executes code already loaded into memory.

Verification points: nginx -v should show the new version (measured 1.30.4); nginx.old -v shows the old one (1.28.3). Both files in place, rollback has an entry point.

Step Two: USR2, Raise the New Master

First note the old master’s PID (cat /srv/nginx1/logs/nginx.pid, 2729 in the lab environment), then:

1
kill -USR2 2729

Verification points: ps should show two master processes side by side; ls /srv/nginx1/logs/ should reveal nginx.pid.oldbin — the old master’s pid file was renamed, and the new master now occupies nginx.pid:

Figure 1

PID 2729 (old) and PID 2752 (new, ppid=2729) exist simultaneously, the .oldbin suffix has appeared — the binary swap is half done. At this point old and new masters each carry their own workers, both serving.

Step Three: WINCH, Retire the Old Workers Gracefully

1
kill -WINCH 8

Verification points: in ps, the workers under the old master (8) disappear — they exit normally after finishing the requests in hand (the log reads worker process exited with code 0); the old master itself remains, and it is the sole entry point for rollback. In error.log, the corresponding entries are the old workers exiting gracefully one by one, with no abnormal codes.

Step Four: Verify the New Version, Send the Old Master Off with QUIT

Let the new version run for a while and confirm the business is fine (curl -s http://127.0.0.1/ returns a normal page), then:

1
kill -QUIT 2729

Verification points: the old master process disappears, the new master (2752) is adopted by init (ppid becomes 1), nginx.pid.oldbin disappears, and the service keeps responding:

Figure 2

The upgrade loop is closed here. Throughout the whole process, client connections were never broken — the old workers only left after finishing their requests.

Rollback on a Failed Upgrade: Walk the Signals in Reverse

When the new version misbehaves, the rollback path is perfectly symmetric: swap the old binary back (cp nginx.old /usr/sbin/nginx), then fire another round of USR2/WINCH/QUIT — at the new master — and the master raised from the old binary takes over the traffic:

1
2
3
4
cp /srv/nginx1/nginx.old /usr/sbin/nginx   # back to 1.28.3
kill -USR2 2752   # raise an old-version master from the old binary
kill -WINCH 2752  # retire the "new version" workers
kill -QUIT 2752   # send the "new version" master off

Verification points: nginx -v returns to the old version, the page responds normally, and only one pid file remains. Measured: after rollback the service is fine and the pid file has its normal naming back:

Figure 3

Note: during rollback, if the old master (the one holding the .oldbin pid) is still alive, kill -HUP oldMasterPID to re-raise the old workers and then shut down the new master also works — both roads lead home; the premise in each case is that the old binary backup still exists.

Wrong-Way Comparison: How USR2 Quietly Does Nothing

A real environment-dependent pitfall from testing, worth putting in the manual: if the master was started with a relative path (say, a bare nginx command, or some script that cds before starting), then at USR2 it execs the new binary using a relative path too, and the exec fails outright because the file cannot be found:

Figure 4

The symptom: after USR2, the old and new masters never coexist and the pid file gains no .oldbin, while the service keeps responding normally — without reading error.log you cannot see the failure at all. Prevention: always start Nginx with an absolute path (systemd’s ExecStart=/usr/sbin/nginx satisfies this naturally), and before sending USR2, confirm nginx.pid.oldbin appears as expected.

Notes

  • Keep configure arguments consistent: the modules, dependency libraries, and install paths of the new build must align with the old version; a hot upgrade missing modules is putting a half-finished product into production.
  • Do not mistake the PID: after USR2 there are two masters; before operating, cat both pid files to confirm which is which — sending QUIT to the wrong process sends the new version straight away.
  • The old master is the rollback entry: WINCH retires workers, not the master; until you are sure, do not rush to QUIT it. The binary backup nginx.old is the same — the whole rollback flow depends on it.
  • Pre-checks before execve: start with an absolute path and run nginx -t before sending USR2 to verify the new binary can parse the current configuration; skipping both steps and firing the signal is the source of incidents.

Summary

Back to the opening question: can upgrading keep every connection alive — measured, the answer is yes, on the division of labor among three signals: USR2 raises the new master that inherits the listeners, WINCH lets the old workers finish their work and leave, QUIT sends the old master off to close out; when things break, swap the binary back to the backup and walk the signals in reverse to roll back. Remember the two details that most easily blow up — start with an absolute path, and only an appearing .oldbin means the switch happened — and this zero-downtime upgrade can go into the production manual with confidence.

This article was rebuilt from the author’s CSDN blog posts written between 2020 and 2024, originally published on CSDN.