Taking secrets out of CI leaves exactly two lines behind

When a running service moves onto an external secrets store, the expectation is usually one: "the CI settings will be empty". They will not. Exactly two lines remain there — and understanding why those two matters more than the list itself.

Below is a breakdown of a real migration: an update-delivery service, a bun monorepo, docker compose on a single host, CI in Gitea Actions. The migration went without a downtime window and without a "big switch". The step-by-step guide is in the adoption documentation; here we look at why the order of steps is what it is, and what breaks when you violate it.

What stays in the CI settings

Two lines, and only those:

• ENV_MASTER_CLIENT_ID as a variable, not a secret. There is nothing secret in an identifier, and one visible in the interface saves time during an investigation. The workflow reads it as vars.ENV_MASTER_CLIENT_ID || secrets.ENV_MASTER_CLIENT_ID — so moving it from secrets to variables does not break a run.
• ENV_MASTER_CLIENT_SECRET as a secret only, with no fallback.

Everything else — the host address, the user, the port, the deploy path, the host fingerprint, the image registry, the deploy user's private key — is no longer a CI setting but a key in the project's environment in env-master.

"Taking secrets out of CI" does not mean "taking them all out". These two lines are not application secrets; they are the pass to the store. And what matters most is what this pass does not open.

Two identities, the heart of the design

Universal mode means a permanent client_id and client_secret: a runner comes up fresh on every run, and a one-time enrollment token does not suit it. The pipeline identity's allowed key list is narrowed (allowedKeys) to what the deployment needs.

The consequence the whole thing is for: the pipeline's identity does not see application keys at all. A compromised build does not hand over JWT_SECRET — not because it was "configured well", but because there is no way for it to read that key. This is an operating property, and half of the remaining decisions follow from it — including the check story below.

The order that cannot be reversed

The migration is three moves, and their order is not cosmetic:

1. The deployment stops keeping configuration of its own. The keys move into env-master, and the hand-rolled SSH is replaced by reads from the store.
2. The configuration gets verified. As a separate step, before the first SSH — before the deployment touches anything on the host. A check that fails halfway through a deploy leaves the service in an intermediate state.
3. The configuration stops being visible to whoever does not need it. The identity's rights are narrowed, and extra services are dropped from the stack along with their access.

Reverse the order and the deploy stops halfway. Narrow the rights before the application has learned to read its configuration by itself, and the pipeline loses keys it still needs. The cause will look like a permissions error rather than the order of the steps — and the investigation goes the wrong way. That is why narrowing is always the last, separate step.

The check that pointed the opposite way

This actually happened. When the rights were first narrowed, the pipeline failed demanding that JWT_SECRET and PUBLIC_API_URL be created — keys that were honestly present in the environment. The reason: the check compared the full list, while the pipeline identity already saw only its own half. For an invisible key it reported "the key did not arrive from env-master" — and a human went off to create a key that already existed instead of fixing the check.

Hence the rule: the pipeline verifies only its own half — the keys marked for deployment (deploy and both in the environment contract). This is not thrift: it is incapable of checking the other half, and the attempt produces a message pointing the opposite way. Runtime keys are verified by the application itself at startup — where they are needed, and by the identity that can see them. Every key in the contract says who needs it (consumer), and a key without the marker counts as "both": a forgotten marker must fail the check loudly instead of quietly exempting the key from it.

The mount the log will not show

After the migration, the only secret file on the server is the machine identity. It lives in the project's directory: /etc/envmaster/<project>, owned by root, mode 0700. Root, not the deploy user, on purpose: the container runs as root and reads the directory, while the deploy user — that is, everything arriving over SSH from the pipeline — must not see the machine's credentials. The whole directory is mounted into the container, not a single file: the SDK writes machine.json through a temporary file next to it plus rename, and rename does not work across a single-file mount boundary.

On a host with several projects there is a directory per project, and exactly one is mounted:

volumes:
  - /etc/envmaster/my-project:/run/envmaster   # like this
# - /etc/envmaster:/run/envmaster              # NOT like this

The difference is one line, and it looks like a typo. What it costs is access to every project on the host: mount the parent and you hand the container the credentials of each project, turning one compromised build into access to all of them at once. The nastiest part — nothing about it stands out in the access log: the reads come from legitimate identities, and each one looks ordinary. The log records who read; it cannot know the process should never have received that file. That is why the matching of paths is held by a test, not by attention during review.

A PEM reaches the log one line at a time

The deploy private key lives in env-master like everything else. But a CI runner masks a secret only as a whole — and a multi-line PEM reaches the output one line at a time, which whole-value masking does not catch: with an early version of the action the key would have ended up in the run log in plain view. That is why the action version the pipeline uses to fetch keys masks a PEM line by line — every line except the header. This fixes a real trap rather than decorating anything: a leak one line at a time is indistinguishable from ordinary output until you see it.

The bottom line

• The CI settings hold two lines: an identifier as a variable, and a secret. There is nothing else to look for there — and that is the answer to "so what remains?".
• The server holds one secret file: the machine identity. No application keys on disk: the .env is gone, and the service reads values itself, into process memory.
• A compromised build gives up deployment keys at most — JWT_SECRET is structurally out of its reach.
• The order of the steps is part of the design: first the configuration leaves, then it is verified, and only then hidden from whoever does not need it.

The full order — six steps, each one stoppable on its own — is in the documentation on adopting in an existing product. How a service proves it is itself is covered in the machine identity article, and the log mentioned above is the subject of the secrets access audit article.