Part 4: A Vault Agent Sidecar and Docker Compose, Secrets Without the App Knowing Vault Exists

By the end of Part 3 we had two things that hadn't met yet. Vault holds the secrets, policies, and an AppRole per environment (Part 2). The repo holds a Go app that reads its secrets from one JSON file and stops immediately if that file is wrong (Part 3).

This part connects them. Not by adding the Vault SDK to the app, but with a sidecar: a small container called vault-agent that logs in to Vault, fetches the secrets, and writes config.json to a volume it shares with the app. The app itself doesn't know Vault exists. All it knows is that there's a file at /run/config/config.json.

What you get: two compose files (dev and production) that Dokploy will use in Part 7, one override file for the laptop, and a script that starts a local Vault dev server. That last one matters. Every test in this part runs on the laptop, so the real Vault is only touched at the very end.

Why a sidecar instead of the Vault SDK in the app

I weighed two options:

For this project I went with the sidecar. The app stays small and easy to test (hand it a dummy JSON file), and everything Vault-related lives in one place that can be reviewed. I list the downside honestly as an accepted risk at the end.

The flow looks like this:

Dokploy .env: VAULT_ADDR, VAULT_ROLE_ID, VAULT_SECRET_ID        │        ▼┌──────────────────── vault-agent (UID 65532) ────────────────────┐│ entrypoint.sh: write role-id & secret-id → /vault/auth (tmpfs)  ││ vault agent: AppRole login → token → read 2 secrets             ││ template: render → /vault/out/config.json (tmpfs, 0400)         │└─────────────────────────────────────────────────────────────────┘        │ volume config_vol (shared)       volume vault_auth (NOT shared)        ▼┌──────────────────── app (UID 65532, distroless) ────────────────┐│ /run/config/config.json  (read-only mount)                      ││ read-only root filesystem, cap_drop ALL                         │└─────────────────────────────────────────────────────────────────┘

The vault-agent/ folder in the repo has four files: agent.hcl, config.json.ctmpl, entrypoint.sh, and Dockerfile. Let's go through them one at a time.

agent.hcl: AppRole auto_auth and the template

This is the core Vault Agent config:

# No vault { address } stanza: HCL does not expand env vars, so the agent reads VAULT_ADDR itself. auto_auth {  method "approle" {    mount_path = "auth/approle"    config = {      role_id_file_path                   = "/vault/auth/.role-id"      secret_id_file_path                 = "/vault/auth/.secret-id"      remove_secret_id_file_after_reading = false    }  }} # Give up instead of serving a stale or partial config when Vault keeps refusing.template_config {  exit_on_retry_failure = true} template {  source               = "/vault/config/config.json.ctmpl"  destination          = "/vault/out/config.json"  perms                = "0400"  error_on_missing_key = true}

Block by block:

• auto_auth with method "approle": the agent logs itself in with the role_id and secret_id it reads from files. The resulting token (1-hour TTL, 4-hour max, as set on the role in Part 2) is renewed automatically. Once it hits token_max_ttl, the agent logs in again.

• remove_secret_id_file_after_reading = false: by default the agent deletes the secret_id file after reading it. I turn that off because the agent needs the file again to log back in after 4 hours. If the file is gone, it can't log in and rendering stops.

• template_config { exit_on_retry_failure = true }: if Vault keeps refusing (a 403, say), the agent gives up and exits instead of sitting there pretending to be fine.

• template: render config.json.ctmpl to /vault/out/config.json with mode 0400 (only the owner can read it). error_on_missing_key = true turns a missing key into an error instead of an empty string that quietly slips through.

One thing was in the plan but didn't make it: disable_mlock = true. I assumed a non-root agent would need it to avoid IPC_LOCK. In testing, the agent ran fine without it, so I removed it. Config that isn't proven necessary shouldn't ship.

config.json.ctmpl: paths only, never values

The template is committed to a public repo, so it contains only secret paths. The values stay in Vault.

{{- /* Paths only, never values. APP_ENV is pinned in the compose file of each environment. */ -}}{{- $env := env "APP_ENV" -}}{{- $jwt := "" -}}{{- $aes := "" -}}{{- with secret (printf "static-secret/data/secure-cicd-go/%s/secrets/jwt" $env) }}{{ $jwt = .Data.data.secret_key }}{{ end -}}{{- with secret (printf "static-secret/data/secure-cicd-go/%s/secrets/aes" $env) }}{{ $aes = .Data.data.secret_key }}{{ end -}}{"env": {{ $env | toJSON }}, "jwtSecret": {{ $jwt | toJSON }}, "aesKey": {{ $aes | toJSON }}}

A few decisions here:

• The path is built from APP_ENV, which is pinned in the compose file rather than read from .env. If .env could change APP_ENV, one typo would be enough to make the dev stack try to read production paths. Vault would refuse (the dev policy only allows read on the dev folder), but it's better not to open that door at all.

• data/ in the middle of the path because the static-secret mount is KV version 2. KV v2 keeps secret contents under <mount>/data/..., and the fields live in .Data.data.

• Every value goes through toJSON. That's what keeps the output valid JSON even when a secret contains " or \. In the local test I seed a secret with exactly those characters.

• One JSON line at the end, always the full shape. My first draft put chunks of JSON (commas included) inside the with blocks. The problem: if one block doesn't run, its comma vanishes with it and the JSON breaks in a confusing way. The final version fills variables first, then prints the whole JSON once.

The JSON shape matches exactly what config.Load expects from Part 3: env, jwtSecret, aesKey. The app uses DisallowUnknownFields, rejects empty fields, and requires the AES key to be 64 hex characters. So content validation stays in the app (fail closed), and the agent only has to make sure the file exists.

entrypoint.sh: from env to file, failing fast

Dokploy passes credentials as env vars, but AppRole auto_auth reads them from files. This small script bridges the two:

#!/bin/sh# Writes the AppRole credentials from the environment to the tmpfs auth volume, then runs the agent.set -eu : "${VAULT_ADDR:?VAULT_ADDR is not set}": "${APP_ENV:?APP_ENV is not set}": "${VAULT_ROLE_ID:?VAULT_ROLE_ID is not set}": "${VAULT_SECRET_ID:?VAULT_SECRET_ID is not set}" umask 077printf '%s' "$VAULT_ROLE_ID" > /vault/auth/.role-idprintf '%s' "$VAULT_SECRET_ID" > /vault/auth/.secret-id exec vault agent -config=/vault/config/agent.hcl

• The : "${X:?message}" lines make the script exit right away with a clear message if any env var is empty. Much easier to read than a vague login error.

• umask 077 makes both files readable by their owner only.

• printf '%s' (not echo) so no trailing newline ends up in the value.

• exec replaces the shell with vault agent, so Docker's stop signal goes straight to the agent.

This file wasn't in the original plan. I added it during implementation because writing the credentials in the compose command: got long and hard to review.

The vault-agent Dockerfile: pinned, baked, non-root

# Config and template are baked in: a template change changes the image, so compose recreates# the container instead of serving a stale render.FROM hashicorp/vault:2.2.1@sha256:962a92876e168d68eaaa24b0b8b49bbce1cb63024060a67bba95cd158a366fc2COPY --chmod=0444 agent.hcl config.json.ctmpl /vault/config/COPY --chmod=0555 entrypoint.sh /usr/local/bin/agent-entrypoint.sh# Same UID as the app, so the 0400 config.json is readable by the app and nobody else.USER 65532:65532# Healthy means "a config has been rendered", not "Vault is reachable now": after an agent restart the# previous render stays on the tmpfs volume while the app holds it, so a revoked secret_id still reads healthy.HEALTHCHECK --interval=3s --timeout=3s --start-period=5s --retries=30 CMD ["test", "-s", "/vault/out/config.json"]ENTRYPOINT ["/usr/local/bin/agent-entrypoint.sh"]

Why it looks like this:

• The base is pinned by digest. The 2.2.1 tag could be overwritten, a digest can't. A rebuild on the server always uses the same image.

• Config and template are baked into the image, not mounted from the host. Change the template and the image changes, so Compose recreates the container. With a mount, the old container could keep running on a stale render and nobody would notice.

• --chmod=0444 / 0555: config files are read-only, the entrypoint is read-and-execute only.

• USER 65532:65532: the agent doesn't run as root. Without this line Trivy config reports DS002 (HIGH), and that would block the very first PR in Part 5.

• Same UID as the app (65532). This is what makes mode 0400 workable: the agent (owner) writes, the app (same UID) reads, nobody else gets anything.

• Healthcheck test -s: healthy means config.json exists and isn't empty. The app validates the contents.

Check:

trivy config --severity HIGH,CRITICAL --exit-code 1 vault-agent/

It should finish with no findings.

The compose files: two containers, two tmpfs volumes

Here's docker-compose.development.yml in full. The production file is identical except for APP_ENV: production and the :production image tag.

services:  vault-agent:    build: ./vault-agent    restart: unless-stopped    environment:      APP_ENV: development      VAULT_ADDR: ${VAULT_ADDR:?set VAULT_ADDR}      VAULT_ROLE_ID: ${VAULT_ROLE_ID:?set VAULT_ROLE_ID}      VAULT_SECRET_ID: ${VAULT_SECRET_ID:?set VAULT_SECRET_ID}    volumes:      - vault_auth:/vault/auth      - config_vol:/vault/out    read_only: true    cap_drop: [ALL]    security_opt: [no-new-privileges:true]   app:    image: ghcr.io/dimss113/secure-cicd-go:development    pull_policy: always    restart: unless-stopped    depends_on:      vault-agent:        condition: service_healthy    # Only the rendered config; the AppRole volume is never mounted here.    volumes:      - config_vol:/run/config:ro    expose:      - "8080"    read_only: true    cap_drop: [ALL]    security_opt: [no-new-privileges:true]    networks:      - dokploy-network # tmpfs: secrets live in RAM only and vanish when the stack stops.volumes:  vault_auth:    driver_opts: {type: tmpfs, device: tmpfs, o: "uid=65532,gid=65532,mode=0700"}  config_vol:    driver_opts: {type: tmpfs, device: tmpfs, o: "uid=65532,gid=65532,mode=0700"} networks:  # Created by Dokploy; Traefik routes the domain to the app over it.  dokploy-network:    external: true

(I trimmed the header comment. It says Dokploy writes the three VAULT_* variables to .env, and that APP_ENV is pinned here on purpose.)

Two volumes, two levels of secret

Both are tmpfs: the contents live in RAM, never touch the VPS disk, and disappear when the stack stops. The uid=65532,gid=65532,mode=0700 options make the volume root owned by UID 65532. Without them the volume is created owned by root, and a non-root agent can't write anything to it.

Why does the split matter? If the app ever has a hole (path traversal, for example), the most that leaks is that environment's own config.json. The secret_id, which could be used to log in to Vault again from somewhere else, never exists on the app's filesystem.

The same hardening on both containers

• read_only: true: the container's root filesystem can't be written. The only places the agent can write are the two tmpfs volumes. The app has nowhere to write at all.

• cap_drop: [ALL]: every Linux capability is dropped. Neither container needs one.

• no-new-privileges:true: processes can't gain privileges through setuid binaries.

depends_on and the healthcheck: start order

vault-agent starts  → entrypoint writes the credentials  → agent logs in, renders config.json  → healthcheck test -s passes  (every 3 seconds, up to 30 times)  → status: healthyonly then is app created and started  → config.Load reads /run/config/config.json  → valid → listen on :8080

condition: service_healthy means the app won't start until the agent is healthy. If rendering fails (a 403, a missing key), the agent never becomes healthy and the app never runs. That's what I want: a service that's down beats a service running on an empty config.

The app has its own healthcheck in its Dockerfile (Part 3): /app healthcheck, since a distroless image has no shell or curl.

Why dokploy-network

The app joins dokploy-network, which Dokploy creates, because that's the network Traefik uses to route the domain to the app on port 8080. The agent doesn't join it. It stays on the stack's default network to reach Vault. Nothing is published to the host, just expose.

Check: both files are syntactically valid.

docker compose -f docker-compose.development.yml config -qdocker compose -f docker-compose.production.yml config -q

Both should exit 0. If they complain about set VAULT_ADDR, that's expected: export dummy values first, or run this after the next step.

Testing locally against a Vault dev server

Before touching the real Vault, I want to prove everything on the laptop: the happy path, isolation, and failures. That's what scripts/local-vault.sh is for. It starts a Vault dev server in a container and seeds it with exactly the same layout as the real Vault. Here's a slightly trimmed version:

NAME=scg-vault-devNET=scg-localIMAGE=hashicorp/vault:2.2.1@sha256:962a92876e168d68eaaa24b0b8b49bbce1cb63024060a67bba95cd158a366fc2 up() {  mkdir -p .local && chmod 700 .local  openssl rand -hex 16 > .local/root-token  docker network inspect "$NET" >/dev/null 2>&1 || docker network create "$NET" >/dev/null  docker run -d --name "$NAME" --network "$NET" -p 127.0.0.1:8200:8200 --cap-add IPC_LOCK \    -e VAULT_DEV_ROOT_TOKEN_ID="$(cat .local/root-token)" "$IMAGE" \    server -dev -dev-listen-address=0.0.0.0:8200 >/dev/null  until vexec status >/dev/null 2>&1; do sleep 1; done   vexec secrets enable -path=static-secret kv-v2 >/dev/null  vexec auth enable approle >/dev/null  for env in development production; do    # The development JWT secret holds a quote and a backslash on purpose: the template must still emit valid JSON.    jwt="$(openssl rand -hex 24)"    [ "$env" = development ] && jwt="q\"b\\s-$jwt"    vexec kv put -mount=static-secret "secure-cicd-go/$env/secrets/jwt" secret_key="$jwt" >/dev/null    vexec kv put -mount=static-secret "secure-cicd-go/$env/secrets/aes" secret_key="$(openssl rand -hex 32)" >/dev/null    vexec policy write "secure-cicd-go-$env" - < "vault/policies/secure-cicd-go-$env.hcl" >/dev/null    vexec write "auth/approle/role/secure-cicd-go-$env" token_policies="secure-cicd-go-$env" \      token_ttl=1h token_max_ttl=4h secret_id_ttl=0 secret_id_num_uses=0 >/dev/null    # ... write VAULT_ADDR, VAULT_ROLE_ID, VAULT_SECRET_ID to .local/$env.env (chmod 600)  done}

What matters in this script:

• Same Vault image as the vault-agent base, pinned to the same digest.

• Policies come from vault/policies/ in the repo, so the test uses the same policies that are installed in the real Vault. If a policy is wrong, the local test fails too.

• **The dev JWT secret contains " and \ on purpose**. If the template forgot toJSON, the app would reject its config and we'd find out here, not in production.

• Every value is random and disposable. The root token is random per session, the dev server keeps everything in memory, and the AppRole credentials go to .local/, which is in .gitignore.

• The dev Vault joins the scg-local network, and the address written to .local/*.env is http://scg-vault-dev:8200, a container name that resolves on that network.

The local override file

The server compose pulls from GHCR and publishes no port. On the laptop I need the opposite, hence docker-compose.local.yml:

services:  vault-agent:    networks: [default, scg-local]  app:    build:      context: .      args:        COMMIT: ${COMMIT:-dev}        BUILD_TIME: ${BUILD_TIME:-unknown}    image: secure-cicd-go:local    pull_policy: build    ports:      - "127.0.0.1:8080:8080" networks:  scg-local:    external: true

Compose merges -f files from left to right. The file on the right overrides or adds, everything else stays from the file on the left:

Why not just edit the development file? Because Dokploy uses that file. The override keeps server settings and laptop settings apart, with no risk of an open port or a local build leaking into production. The port is also bound to 127.0.0.1 on purpose, not to every interface.

The happy path

scripts/local-vault.sh updocker network create dokploy-network 2>/dev/null   # once; on the server Dokploy creates it set -a; . .local/development.env; set +a            # export VAULT_ADDR, VAULT_ROLE_ID, VAULT_SECRET_IDexport COMPOSE_FILE=docker-compose.development.yml:docker-compose.local.ymlexport COMMIT=$(git rev-parse HEAD)export BUILD_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)docker compose up -d --build --wait

A few small things that make this test session pleasant:

• set -a exports every variable read from the file, so the local credentials land in the environment without ever being typed or shown on screen.

• COMPOSE_FILE is a colon-separated list of files. It's the same as writing -f docker-compose.development.yml -f docker-compose.local.yml on every command, in the same order.

• --wait waits until every container is healthy, and fails if they aren't.

Why do the credentials need to be exported even for ps or logs? Because the compose file uses ${VAULT_ADDR:?set VAULT_ADDR}. Compose interpolates the file on every command and refuses right away if the variable is empty.

Check:

docker compose ps                  # both healthycurl -s localhost:8080/health      # {"status":"ok"}curl -s localhost:8080/version     # commit hash and build timecurl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/v1/tokens \  -H 'Content-Type: application/json' -d '{"subject":"demo"}'   # 201

A 201 from POST /v1/tokens proves the JWT secret (the one with " and \) reached the app as valid JSON and passed the length check.

Isolation: file mode, mounts, and filesystem

docker compose exec vault-agent ls -l /vault/out/# -r--------  1 65532 65532 ... config.json docker compose exec app /app healthcheck && echo ok docker inspect "$(docker compose ps -q app)" \  --format '{{range .Mounts}}{{.Destination}} rw={{.RW}}{{"\n"}}{{end}}'# /run/config rw=false        <- only this; no /vault/auth

The app image is distroless, so there's no ls or cat inside it. That's why the file mode is checked from the agent container, and the app's mounts are checked with docker inspect from outside. You want to see: the file at 0400 owned by 65532, and exactly one mount on the app, read-only. Its root filesystem is read-only thanks to read_only: true, and the app has no reason to write anything anyway.

Then restart the agent:

docker compose restart vault-agentdocker compose ps      # vault-agent healthy again, app still running

Failure tests: the part that matters most

A happy path alone proves nothing. What needs proving is that the system refuses correctly.

1. Dev credentials used on the production stack.

docker compose down -vexport COMPOSE_FILE=docker-compose.production.yml:docker-compose.local.ymldocker compose up -d --build --wait# fails: vault-agent never becomes healthy docker compose logs vault-agent | grep -i "permission denied"docker compose down -v

The exported variables are still the dev credentials, but APP_ENV in the production file is production, so the template asks for production paths. The dev role's token only carries the dev policy, Vault answers 403 permission denied, rendering fails, the agent exits, and the app never reaches running. Exactly what we want.

2. A secret field with the wrong name.

export COMPOSE_FILE=docker-compose.development.yml:docker-compose.local.ymlscripts/local-vault.sh kv put -mount=static-secret \  secure-cicd-go/development/secrets/aes secret_keyx="$(openssl rand -hex 32)"docker compose up -d --build --wait# fails: render error because secret_key is missingdocker compose logs vault-agent | tail -n 20

Thanks to error_on_missing_key = true, the missing key makes rendering fail and the app doesn't start. Without that option the template would print an empty string and we'd rely entirely on the app's validation. Two layers beat one.

Clean up:

docker compose down -vscripts/local-vault.sh down       # removes the container, the scg-local network, and .local/unset COMPOSE_FILE VAULT_ADDR VAULT_ROLE_ID VAULT_SECRET_ID

Testing against the real Vault

Once everything passed locally, I tried the real Vault with the development AppRole from Part 2. The only difference is where the credentials come from:

export VAULT_ADDR=https://vault.example.comread -r VAULT_ROLE_ID        # paste the DEVELOPMENT role_id, Enterread -rs VAULT_SECRET_ID     # paste the DEVELOPMENT secret_id (hidden), Enterexport VAULT_ROLE_ID VAULT_SECRET_IDprintenv VAULT_ROLE_ID >/dev/null && echo "exported" docker network create scg-local 2>/dev/null   # required by the local override fileexport COMMIT=$(git rev-parse HEAD) BUILD_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)docker compose -f docker-compose.development.yml -f docker-compose.local.yml up -d --build --waitcurl -s localhost:8080/health

read -rs keeps the secret_id off the screen and out of your shell history.

I hit three problems at this step. None were bugs in the code, but they still cost time.

The third one was subtler. /version returned {"buildTime":"unknown","commit":"dev"}. Those are the ARG defaults in the app's Dockerfile, which only change when someone passes a build arg. At the time, docker-compose.local.yml just said build: ., so nothing was passed. The fix is the args block you saw above, plus exporting COMMIT and BUILD_TIME before up --build. The image has to be rebuilt, since an existing image doesn't change. In CI (Part 8) the workflow passes these build args, and that's what the /version smoke test relies on.

Clean up:

docker compose -f docker-compose.development.yml -f docker-compose.local.yml down -vdocker network rm scg-localunset VAULT_ROLE_ID VAULT_SECRET_ID

Limits I accepted on purpose

This design is deliberately simple, and that has a cost. I'm writing it down so nobody is surprised later:

The healthcheck one is the easiest to misread, and I misread it myself. When I restarted the agent in the isolation test, it showed healthy again right away. That was because the old config.json was still on the tmpfs the app was holding, not because the agent had logged in again. So if the secret_id has been revoked, a freshly restarted agent can still read as healthy as long as the old render is there. The agent logs, not the healthy status, are the source of truth.

Recap

What exists now:

• A vault-agent image based on hashicorp/vault:2.2.1, pinned by digest, running as UID 65532, with config and template baked in.

• agent.hcl with AppRole auto_auth, exit_on_retry_failure, and a template that writes config.json at mode 0400 and fails on a missing key.

• A template that contains only paths, uses a pinned APP_ENV, and handles special JSON characters thanks to toJSON.

• Dev and production compose files: two tmpfs volumes, vault_auth for the agent only, config_vol read-only for the app, read-only root filesystems, cap_drop: [ALL], no-new-privileges, and an app that only starts once the agent is healthy.

• scripts/local-vault.sh and docker-compose.local.yml to test the happy path, isolation, wrong-environment credentials (403), and a missing key, all without touching the real Vault.

• A written list of limits: rotation needs a restart, what the healthcheck really means, and the shared UID.

The stack runs on the laptop, but the code has no guard yet. In Part 5 we create the GitHub repo and a PR workflow that runs tests, Gitleaks, Semgrep, govulncheck, and Trivy on every pull request, including the Trivy config scan we just ran by hand on the vault-agent Dockerfile.