To verify whether a DHCP server configuration is actually correct, the textbook approach is to set aside two machines — one running dhcpd, the other playing the client — and stage the whole encounter just to watch one conversation. These days, when I set out to do this kind of verification, my first step is docker run. Over the past year and more, a whole batch of server-side experiments has gradually moved into containers: the four-step DHCP negotiation for binding a fixed IP by MAC, a three-node ZooKeeper election, two machines mounting each other over NFS, a cross-version Nginx hot upgrade, troubleshooting an OpenResty source build, and installing an old Elasticsearch release on an ARM machine by emulating amd64. Each of these experiments has its process and its conclusions written up in a dedicated post of its own (the links are gathered at the end of this article), so this piece deliberately answers exactly one question: when you treat containers as an experimental lab, how do you make the work smooth, the results trustworthy, and the cleanup clean?

Let me draw the boundary first: everything below is first-hand, hands-on testing, not armchair design and not borrowed experience. For each experiment, the environment — the image, the version, the architecture — is written down at the top of its dedicated post, and every error message quoted here came out of a real run, not out of imagination. What this article distills is the methodology; the experiment details themselves are left where they already live, in those dedicated posts. If you want the command-level detail of any single experiment, follow its link in the series list at the end; this piece deliberately stays at the altitude of method.

Why Treat Containers as an Experimental Lab

The demand for these experiments came entirely out of real work, not out of testing for testing’s sake. After CentOS 7 went EOL, somebody had to walk the migration path for the old environments first and find out where it breaks. dhcpd on Rocky 9 behaves differently from CentOS 7 (the listening interface no longer has to be specified manually), and with a version difference like that, reading the documentation alone never feels quite reassuring — it is the kind of thing you have to verify with your own hands on the new system before you believe it. The Nginx hot-upgrade signal sequence was about to go into the production runbook, which meant both of its faces — the run that succeeds and the run that fails — had to be rehearsed in advance. What all of these verifications have in common is this: they need an environment that is clean, describable, and disposable.

Virtual machines can certainly provide that kind of environment, but they charge too high a price for it. Spinning up a VM is a minutes-scale operation, it occupies gigabytes of disk, and once the experiment is over you find yourself hesitating to delete it — and the moment an environment becomes precious, you start making do inside it, and making do is exactly how environments go dirty. Containers drive the unit price of an environment down to a few tens of seconds and a few hundred megabytes. An experiment is, at its core, “asking an environment a question”: the cheaper the environment, the more willing you are to keep asking — and the more willing you are to throw it away the moment the answer is in.

There is also a benefit that is much less visible at first: containers force you to write the environment down. Image tag, software version, machine architecture — these three items must go into the experiment record, because six months later, what lets you reproduce the experiment is not the container that happens to still be alive somewhere, but the environment description written in the record. Every one of my hands-on posts opens with an environment declaration (for example: a rockylinux:9 container, dhcp-server 4.4.2, on aarch64), and that very format is a habit that the container-as-lab practice hammered into me — one I have kept ever since.

The Playbook

It is the same plot of lab ground throughout, but different kinds of experiments call for different ways of working it — a play that suits a lone service check would be clumsy for a three-node cluster, and the reverse holds just as true. The six plays below are therefore laid out one by one, in the order I actually ran them, each shown together with the real experiment that exercised it.

Single-Container Closed Loop: Server and Client Talk Inside One Container

The most topology-free way to play is to let both roles run and talk to each other inside one single container. The DHCP verification was done exactly this way: dhcpd and dhclient lived together in one Rocky 9 container, the client kicked off the four-step negotiation (DHCPDISCOVER → DHCPOFFER → DHCPREQUEST → DHCPACK), each side watched the other’s packets arrive through its own kernel receive path, the server’s OFFER/ACK logs matched up one-for-one with the client’s bound to 192.168.254.150, and the MAC binding hit precisely, address for address.

The value of the single-container closed loop goes well beyond saving a machine or two — it also closes the loop on negative verification. The rule that a binding address must stay out of the dynamic pool range is one I only truly learned by running the counterexample: write the range as 150-200 (which includes the binding address 150), and a client whose MAC matches no host declaration walks off with 150 right away — dynamic allocation only looks at free slots inside the range and never consults the host table. When the device that actually holds the binding connects later, the conflict is already planted. That counterexample reproduced itself in a matter of minutes inside the very same container, and no amount of written argument is half as convincing as watching the conflict happen with your own eyes.

Multi-Container Topology: A Custom Network Plus Container Names as Hostnames

For experiments that need a genuinely multi-machine shape, build a custom network. The commands go roughly like this: first docker network create --subnet 192.168.254.0/24 lab to weave a network, then hang the containers off it one by one, with the container name doubling as the hostname. That is exactly how the three-node ZooKeeper cluster was built: three Rocky 9.3 containers on the same network, hostnames zk1/zk2/zk3, and server.1=zk1:2888:3888 in zoo.cfg written down exactly as it would be on real machines, no contortions required. The election, the cross-node reads and writes (a write on zk2 read back from zk1), and stopping a follower and restarting it to watch it rejoin the cluster on its own — every one of these checks was put through a real test and passed.

Topology-style experiments also hand you one extra piece of insight for free: distributed-system startup scripts can lie to you. On the three nodes’ first startup, the start command printed FAILED TO START, yet the processes were in fact already up — the start script only waits a few seconds before it goes probing the port, and the brand-new cluster’s very first election had not finished within that window. So take status as the source of truth, and never trust the closing verdict of the start command. Lessons of the “a tool’s word cannot be taken at face value” kind are ones you only stumble into by actually bringing three nodes up.

Two Containers in Tandem: Deliberately Doing It Wrong Is Also an Experiment

The NFS verification used two containers: a server at 192.168.254.2 and a client at 192.168.254.3, with the subnet deliberately planned to sit in the range a typical internal network would use. The flow validation followed the standard sequence — “export, check the export list, mount, then read and write across machines”. But the real yield of this experiment was three error messages — every single one of them reproduced by deliberately doing it wrong: alter the export subnet to a wrong value and let the client mount it, and access denied by server reproduces reliably; skip resetting rpcbind in the minimal container, and rpc.statd is not running reproduces; fall back to the default root_squash on the export, and the client’s root gets Permission denied on writes.

This is the most underrated play in the whole container lab. On a real machine, who would ever be willing to misconfigure permissions on purpose, or write a subnet wrong just to see what happens? Inside a container, the cost of being wrong drops effectively to zero — and yet the exact wording of a production incident’s error message is knowledge that only stays with you if you have made that mistake with your own hands at least once. One environmental discovery came along the way: the export directory must sit on a filesystem that supports NFS exports; in the container I solved it with tmpfs. Leave a detail like that out of the article, and the next person runs into the same wall.

Cross-Version Upgrade Rehearsal: Run Production Moves in a Container First

Hot upgrades — the kind of operation where getting it wrong means an incident — are the best candidates for rehearsal inside a container. I kept the old version (1.28.3) running in an nginx:1.28-alpine container, pushed the new binary built from source (1.30.4, --with-http_ssl_module) onto the original path, then used USR2 to bring up the new master, WINCH to let the old workers exit the stage gracefully, and QUIT to see the old master off. After the full signal sequence had run through end to end, I walked the whole thing backwards to verify that the rollback works as well. This was a genuine cross-version binary replacement — not a same-version copy of the same binary — and it is identical to a production upgrade, step for step and signal for signal.

Even more valuable is the reproduction of the failure mode: if the master happens to have been started with a relative path, then at USR2 time it execs the new binary with a relative path as well, fails to find the file, and fails right there — the old and new masters never coexist even for a moment, the .oldbin pid file never appears, and the service keeps responding normally the whole time. Unless you go dig through error.log, you will never even notice that the upgrade has failed. The countermeasure is to always start the master with an absolute path, and to confirm that .oldbin appears on schedule before you send USR2. Insight of the “what the error actually looks like” kind cannot be picked up from any book; you can only earn it by getting it wrong once with your own hands.

Source Build Verification: Let the Container Absorb the Compile Pollution

Build-type experiments are a natural fit for containers — the pile of -dev dependencies that a source build drags in pollutes exactly the plot of ground you throw away afterwards, pot and all. The OpenResty troubleshooting round took place inside a debian:12 container: OpenResty 1.27.1.1 paired against OpenSSL 3.3.1, with --with-openssl pointed at the already-installed directory — and the make stage failed with ./config: not found. The root cause (the option wants the source tree, not the installed directory; the build rules live in objs/Makefile) and the compile-time verification of both fixes were completed entirely inside the container. Even the fine-grained detail that “when you add only -L and no rpath, openresty -V prints the mismatched built with X (running with Y)” was nailed down only by compiling again and again inside the container until the behavior was beyond doubt.

Cross-Architecture Emulation: Filling the Gap the Official Builds Leave

The last play is one that many people have never touched: architecture emulation. The machine at hand was ARM, and the target was Elasticsearch 7.6.1 — but for that particular version, the official release shipped only a linux-x86_64 build: the aarch64 download link returned 404 when actually tested, and official ARM builds did not begin appearing until 7.8. Docker’s cross-architecture capability fills this gap precisely: pull the linux/amd64 variant of the rockylinux:9 image, and run the entire installation process through unchanged.

This emulation run also netted a whole batch of errors that are specific to container environments: the minimal image ships no ps command, so ES’s background start failed outright with command not found — solved by installing procps-ng; ES refuses to start as root, so per the usual practice a dedicated user was created; and with the installation directory under /root, the es user could not even enter it (700 permissions), so it was moved to /opt. None of these pitfalls has anything to do with Elasticsearch itself — they are all properties of “a stripped-down container environment”. Walk through them once with your own eyes, and they join your arsenal for good.

Privileged Containers and Loop Devices: When You Need Kernel Facilities

An ordinary container can handle every play described above, but one class of experiments needs to reach down and touch kernel facilities: mounting filesystems, or fabricating loop devices to serve as block disks — giving an NFS export a “real disk”, for instance. These needs call for extra container privileges — --privileged in one sweeping gesture is the least effort, or you can grant precisely what is needed with --cap-add plus --device. The stance here should be stated plainly: an experimental lab is not a lawless zone. Grant as few privileges as you can possibly get away with, and always know exactly what is running inside.

Environment Pitfalls and Countermeasures

Beyond the playbook itself, four environmental pitfalls kept recurring across these experiments — annoying in the moment, educational in hindsight — and each of them deserves a note of its own.

First, 404s from EOL software repositories. After CentOS 7 stopped service, its official repositories were taken offline, and yum installs began failing right away. The countermeasure is straightforward: comment out the mirrorlist lines in the repo file, point baseurl at http://vault.centos.org, run yum clean all, and then install again. Vault occasionally rate-limits the altarch (ARM) paths with 403 responses — the answer is to retry a little later, or to switch over to an x86_64 machine.

Second, VPN fake-ip polluting container DNS. While the host is running a proxy in fake-ip mode, external domain lookups made inside the container come back with fake addresses instead of real ones, and pulling images or installing packages fails in ways that make no obvious sense. The countermeasure: hand --add-host=domain:realIP to docker run, so that the handful of key domains are pinned straight into /etc/hosts and the polluted resolution chain is bypassed entirely. What this trick really treats is one and the same underlying illness: DNS inside the container depends on the host’s environment, and from a container’s point of view, whatever the host’s networking software does to it is an “act of God”.

Third, the log files are symbolic links. In the official nginx image, access.log and error.log are symbolic links pointing at stdout/stderr — go hunting for a log file inside the container and you will come up empty-handed; the uniform way to read the logs is through docker logs. The first time you hit this, you will suspect that the logs have been lost; in fact the image is working exactly as designed: the container world’s logging philosophy is “emit to the standard streams”, not “settle quietly into a file”.

Fourth, no systemd inside the container. Most service documentation assumes systemctl as the default interface; inside a container, that whole apparatus simply does not exist. The equivalent techniques used across these tests were these: dhcpd’s listening verification was completed with dhcpd -d -f in foreground debug mode (the equivalence is noted in the article — never passed off as an actual systemctl run); firewalld depends on systemd, so it was not run at all, and the corresponding step is honestly labeled “not actually run”; when a component depends on D-Bus, first raise the bus by hand with dbus-daemon --system, and only then start talking about the services that sit on top of it. There is also a broader version of this rule: whatever “the startup script says” inside a container should be taken at a discount — the ZK false FAILED TO START from earlier in this article is the living proof.

The Hygiene Discipline: Discard After Use

That the lab stays this highly productive over the long run comes down not to any particular technology but to discipline — the unglamorous kind, the kind that only looks obvious in hindsight. There are four rules here, and every single one of them was solidified the hard way, after stepping into the corresponding pit.

Write the environment down before you touch anything. Image, version, architecture — that trio goes into the very first paragraph of the experiment record, before any command is run. Version anchoring is not fastidiousness: installing DHCP on CentOS 7 and installing DHCP on Rocky 9 are two different articles, and writing a tutorial that jumps across versions is planting a landmine.

Give every step a checkpoint. After executing each step, spell out in the record “what you should expect to see once it is done” — exportfs -v should list the export rules, zkServer.sh status should report a Mode, nginx.pid.oldbin should appear. A step without a checkpoint does not count as complete, and this one habit gives every experiment a naturally checkable chain of evidence.

Label equivalences and non-runs honestly. Whatever cannot be done inside the container, say so plainly. dhcpd’s systemctl verification was performed as a foreground-mode equivalent, and the article carries a note saying so; firewalld was never run inside the container, and the source is stated to be common usage and the package’s own service definitions. An equivalence is an equivalence, and a non-run is a non-run — the entire value of an experiment record rests on the single word “trustworthy”, and one act of inflating it voids the whole piece.

Containers hold no state. When the experiment ends, the conclusions settle into the article, and the container itself becomes deletable — no ceremony required. The moment the thought “I might still need this environment” shows up, be on guard: as soon as an environment becomes an asset, it starts to decay. What reproduces the work is the environment description written in the record — never the corpse of a container.

Boundaries: What Does Not Belong in the Lab

To be fully honest, containers cannot cover every experiment there is. Three kinds of scenarios either do not work or work only approximately: first, real multi-machine topology — cross-machine latency, network partitions, and machine-level firewalls are beyond what container networking can emulate, so those call for virtual machines or real ones; second, anything bound up with physical hardware, since containers have no real hardware to offer; and third, service stacks that require a full systemd session — the missing init system, a weakness already covered earlier in this article. My principle is this: for any conclusion reached inside a container, re-verify the critical steps in the target environment before migrating them to real machines. The lab buys you the cheap first answer, not the final word. The ZooKeeper article states explicitly that “the bare-metal deployment steps are exactly identical to these, with only the node IPs swapped for real addresses” — and that one sentence is itself a commitment to migration verification, not a disclaimer.

Problems Still Unsolved

Two problems, at least for now, are left open. First, distributed experiments spanning multiple hosts — with real network partitions and cross-datacenter topologies — have no answer in this lab yet: however flexible container networking gets, it remains a LAN confined inside one host machine. Second, the complete scheme for putting systemd into the container to serve as its init has not been verified — if systemd truly runs, whether components like firewalld work as-is is worth a dedicated round of experiments and an article of its own.

Back to the question posed at the opening: how do you make the lab smooth to work in, trustworthy to read, and clean to leave behind? Smooth comes from the playbook, drawn on as needed; trustworthy comes from checkpoints and honest labeling; clean comes from the discipline of using and discarding. Containers press the cost of “verifying an idea” down to something you can do on a whim, and once that cost barrier falls, verification stops being a chore you postpone and becomes a reflex you act on. That, in the end, is the lab’s real output: not the six articles it produced, but the conditioned reflex it installed — “next time a new problem shows up, open a container and try it first”.

The Series Posts