Secret rotation without downtime: four phases instead of one long evening

Short answer: rotation gets postponed not because it is hard, but because the classic drill — “change the password and push it to every service” — is guaranteed to break live connections. The way out is to split the replacement into four phases, create → set → test → finish, where consumers keep reading the old value until the very last step and the new one becomes current only after a successful check. In env-master this automaton is built in: the value is durable before it starts being applied, and a failed check raises an event, a webhook and a metric instead of a silent outage.

Why rotation gets postponed until an incident

Ask a team when it last changed the database password, and the most common answer is “when that admin left.” The reason is rational: rotating by hand opens a window where some instances already read the new value and some still read the old one, and one of the two must fail. The risk is known and immediate; the benefit is deferred and abstract. So secrets live for years, until a leak turns rotation into an emergency.

The conclusion is not “discipline people” but remove the window. Once a new value becomes current atomically and only after a check, rotation stops being an event — and can run on a schedule instead of on an occasion.

The four phases

The rotation automaton in env-master is the same for any target secret:

• create — produce the new value: for the built-in PostgreSQL rotator that is a generated password, and the phase never touches the database;
• set — apply it where it is supposed to take effect;
• test — verify that it works;
• finish — move the current label so consumers start reading the new value.

A run always starts from the beginning: there is no resuming from a phase, and a crash halfway through leaves no undefined state — the next run simply starts over. The order of the phases is what makes starting over safe: the password is generated without the database, and the current label moves last. Details and limits are in the rotation documentation.

Why test cannot be skipped

Rotation without verifying the new value is worse than no rotation: you get confidence instead of working access. The password is created, stored and rolled out — and does not work: wrong grants, wrong database name, wrong host. That is why the phase is mandatory, and only after it does finish move the current label. A failed run is not hushed up: a rotation.failed event, a webhook to the administrators and a metric.

The pending label: durable before it is applied

Between create and finish the new value already sits in the store under the pending label. This is not an implementation detail. If the process dies after creating the password in the database but before writing it to the store, the password is lost — while already being in effect, with nowhere to recover it from. The value becomes durable before it starts being applied. Consumers keep reading current the whole time and notice nothing.

Rollback moves forward

A rollback is one action: the previous value is published as a new version. Pointers are not moved backwards; history stays monotonic, and it shows that a rollback happened at all — a fact that matters more during an investigation than the value change itself. A repeated rollback moves forward too, instead of swinging the value back and forth. The version and label model is described in the data model.

What ships out of the box

Today there is a PostgreSQL rotator with two strategies: change one user's password, or alternate between two users. Connections that are already open survive either one — PostgreSQL checks the password when a connection is established, not on every query. The engine's skeleton is not tied to a specific database: adding a rotator for your own system means implementing four methods — one per phase.

Rotation is one of the things the free plan does not cut: like encryption and auditing, it works from the first minute on every plan.