Adopting in an existing product

How to move a service that is already running onto env-master: with no downtime window, no "big switch", and the option to stop after any step.

This page describes not an ideal scheme but the one actually carried out on a real service — ota-service: a bun monorepo, docker compose on a single host behind an external Traefik, CI in Gitea Actions. The order of the steps, the traps and the caveats come from there.

Where you end up

Two identities with different rights instead of one pile of variables:

Two lines remain in the CI settings — the ones without which there is no way to open env-master at all. Everything else lives in the project's environment.

Not one application secret is left on the server: the .env full of passwords disappears, and the only secret file on the host is the machine identity.

The consequence the whole thing is for: the pipeline's identity does not see application keys at all. A build that gets compromised does not hand over JWT_SECRET, because it cannot read it.

The order: six steps, each one stoppable on its own

The order is not cosmetic. First the deployment stops keeping configuration of its own, then that configuration gets verified, and only then it stops being visible to whoever does not need it. The reverse order breaks the deploy halfway through.

1. Create the project, the environment and the pipeline identity

Use the names as they exist in env-master, not "as is customary". The environment is prod, not production, if that is what you called it. The project slug is the one in the address, not the human-facing name. This is the first thing people trip over: the pipeline gets Secret reference is not accessible and it looks like a permissions problem.

The pipeline identity is universal mode (a permanent client_id and client_secret): a runner comes up fresh on every run, and a one-time enrollment token does not suit it.

In the CI settings:

• ENV_MASTER_CLIENT_ID as a variable, not a secret. There is nothing secret in an identifier, and one visible in the interface saves time during an investigation. Read it in the workflow as ${{ vars.ENV_MASTER_CLIENT_ID || secrets.ENV_MASTER_CLIENT_ID }} — then moving it from secrets to variables does not break a run.
• ENV_MASTER_CLIENT_SECRET as a secret only, with no fallback.

2. Move the deployment keys and drop the hand-rolled SSH

The first thing to move into env-master is the configuration of the deployment itself: host address, user, port, path, host fingerprint, image registry, the deploy user's private key.

- name: Fetch deploy env from env-master
  uses: aistastudio/env-action@v1
  with:
    base-url: https://env.example.com
    project: my-project
    environment: prod
    client-id: ${{ vars.ENV_MASTER_CLIENT_ID || secrets.ENV_MASTER_CLIENT_ID }}
    client-secret: ${{ secrets.ENV_MASTER_CLIENT_SECRET }}

A private key must be masked line by line. The runner masks a value as a whole, and a multi-line PEM reaches the log one line at a time — fully visible in the output. Use the action version that masks every line separately; an older one does not, and the key leaks into the run log.

An empty value is a failure too. A key can exist and be empty; check for non-emptiness by name right after fetching, so the error names the key instead of surfacing three steps later as a failing ssh:

test -n "${REGISTRY_HOST:-}" || { echo '::error::env-master: REGISTRY_HOST is empty'; exit 1; }

3. Put the environment contract next to the code

Add a file to the repository that lists every key: name, kind (var/secret), whether it is required, who needs it, and what it means.

version: 1
keys:
  - name: REGISTRY_HOST
    kind: var
    required: true
    consumer: deploy
    description: Image registry domain
  - name: JWT_SECRET
    kind: secret
    required: true
    consumer: runtime
    description: >-
      Session signing. When moving to another server it must stay THE SAME:
      changing it logs everyone out.

Why a file in the repository rather than the environment schema on the server. These are two different contracts, and they are allowed to diverge: the schema on the server says what the operator must hold; the file says what this code needs. The file is reviewed in a pull request and changes in the same commit as the compose file — whereas the server-side schema would have to be synchronised by hand, and one day would not be.

Three rules learned in practice:

• consumer: is mandatory, and a key without it counts as "both". A forgotten marker must fail the check loudly instead of quietly exempting the key from it.
• Do not describe formats. A format check breaks the deploy and catches little: S3_ENDPOINT as a url will not accept minio:9000, a number breaks on a legitimately empty value. Whether a value is meaningful is what the application checks at startup. The schema answers "does it exist at all".
• Bootstrap keys do not belong in the contract. The env-master address, the project slug, the environment and the credentials path are substituted by compose; they are not env-master keys, and declaring them creates a second source of truth.

4. Verify the contract before the first SSH

As a separate step, before the deployment touches anything on the host. A check that fails halfway through a deploy leaves the service in an intermediate state.

Verify only your half — the keys with consumer: deploy and both. This is not thrift: the pipeline identity is narrowed and does not see runtime keys. Checking them, you get "the key did not arrive from env-master" for a key that is there — a message pointing the opposite way. That is exactly what happened when the rights were first narrowed: JWT_SECRET was honestly present in the environment while the check demanded it be created.

Runtime keys are verified by the application itself at startup — where they are needed, and by the identity that can see them.

5. Make the application read from env-master

This is where .env disappears from the server. compose passes the container only four values — where to look, who it is, and what to authenticate with:

environment:
  ENVMASTER_URL: https://env.example.com
  ENVMASTER_PROJECT: my-project
  ENVMASTER_ENVIRONMENT: prod
  ENVMASTER_CREDENTIALS_FILE: /run/envmaster/machine.json
volumes:
  - /etc/envmaster/my-project:/run/envmaster

The application reads the rest itself through the SDK:

const client = await EnvMasterClient.connect({
  baseUrl: process.env.ENVMASTER_URL!,
  project: process.env.ENVMASTER_PROJECT!,
  environment: process.env.ENVMASTER_ENVIRONMENT!,
  credentialsFile: process.env.ENVMASTER_CREDENTIALS_FILE!,
});
await client.assertRequirements(["JWT_SECRET", "S3_ENDPOINT", "S3_ACCESS_KEY"]);

The list of required keys is not "the important ones" but the ones whose default does not fail — it quietly does the wrong thing. A missing JWT_SECRET falling back to change-me-in-production brings up a service that signs tokens anyone can forge. A key whose default honestly works does not belong on the list — otherwise the operator invents values for settings nobody needs to set.

The check goes before the configuration is read: a missing key must stop the start and name itself, instead of becoming a default that points production at localhost.

6. Take access away from whoever no longer needs it

The last step is narrowing. Now that only the service reads application keys, the pipeline identity can have them taken away (allowedKeys), and any extra services can be dropped from the stack along with their access.

Do this last, as a separate change: narrow the rights before the application can read on its own and the deploy stops, with a cause that looks like a permissions error rather than the order of the steps.

The machine identity on the host

The only secret file on the server. Created once:

sudo install -d -o root -g root -m 0700 /etc/envmaster/my-project

Owned by root, not by the deploy user, mode 0700. The reason is simple: the container runs as root and reads the directory, while the deploy user — that is, everything arriving over SSH from the pipeline — must not be able to see the machine's credentials.

Mount the directory, not the file. The SDK writes machine.json through a temporary file next to it plus rename, and rename does not work across a single-file mount boundary.

Several projects on one host

A directory per project, and exactly one is mounted:

/etc/envmaster/
├── my-project/machine.json
├── other-project/machine.json
└── <project>/machine.json

volumes:
  - /etc/envmaster/my-project:/run/envmaster   # like this
# - /etc/envmaster:/run/envmaster              # NOT like this

The difference is not cosmetic. Mounting the parent hands the container the credentials of every project on the host, and with them everything those identities are allowed to reach. And the nastiest part: nothing about it stands out in the env-master access log — the reads come from legitimate identities, and each one looks ordinary.

Inside the container the path is always the same; only the mount sources differ. That keeps one image for every project, and the container learns who it is from the directory it was given.

Hold the seam between code and contract with a test

A contract nobody checks drifts from the code on the first edit. What needs checking is not the file itself but the agreement of three lists — what the code reads, what the contract declares, what compose substitutes:

• every key the application reads is declared in the contract;
• everything the application requires at startup, the contract also marks required;
• every key compose substitutes is marked for the pipeline;
• every key the application reads is marked for the runtime;
• bootstrap keys are absent from the contract;
• the project name is the same in compose, in the deployment and in the host rules;
• the credentials file lies inside the mounted directory — not next to it.

The last one deserves its own test: the path /etc/envmaster instead of /etc/envmaster/my-project looks like a typo and costs access to every project on the host.

What changes at build time and what at runtime

Values baked into the image during the build (analytics counter ids, public addresses, anything the frontend receives statically) do not change at runtime. Changing such a key requires a rebuild, not an edit of the value. Say so in the key's description — otherwise one day you will spend half an hour looking for why the new value was not picked up.

A key can be declared optional and still have to exist: the action fetches keys by name and fails on a missing one. "Optional" in the contract and "may be absent from env-master" are different things.

How the service learns about rotation

A secret read once into a variable at startup never learns about rotation. The order in which rotation actually works:

1. the value is read through the SDK, not from an environment variable;
2. the SDK keeps entries in a cache — 15 minutes by default;
3. refresh() drops the cache immediately; calling it from the entry.rotated webhook turns rotation into an event for the service instead of a wait.

Details are in secret rotation.

Rights on the host: narrow them, and tell the truth about the boundary

If the deployment goes over SSH and runs docker, give the deploy user sudo on exactly the commands the pipeline actually runs, and verify it twice: an allowed command goes through, a deliberately extra one is refused. That second refusal is the proof that the rule is no wider than it needs to be.

And a caveat worth writing into your runbook: such a rule does not make the deployment safe in the full sense. The compose file is placed in that directory by the deployment itself, and a compose file describes what will run as root. The boundary is not "the deploy cannot become root" but "the deploy cannot run an arbitrary command outside what is in the repository and passed review".

What this scheme does not do

• It does not protect you from whoever can change the code. Whoever edits the repository edits what the service will read.
• It does not hide values from the service itself. It reads them — that is what the access is for. The difference is that it reads them into process memory instead of receiving a file that stays on disk and ends up in backups.
• It does not replace rotation. Central storage makes rotation feasible, but until you turn it on, the secret is the same one it always was.