Byte-identical or bust

Migrating a secret is not "copy it over and switch the reference."

That sentence has two halves. The copy is where the bug gets in, and the switch is where it goes off. Usually not at the switch itself, either. It goes off at the next restart, days later, when somebody else is holding the pager.

Here's the rule I followed for every application we moved into the new secrets manager:

The new source has to be byte-identical to the live Secret before anything points at it. And the operator that syncs it adopts the existing Secret. It never owns it.

Nine applications went across that way. Not one pod restarted. A good secrets migration should be the most boring change you ship all quarter.

---

Why the obvious version is wrong

If you'd asked me cold, this is the plan I'd have written. I know because it's the plan everyone writes:

1. Put the values in the secrets manager.
2. Create an ExternalSecret that syncs them to a new Kubernetes Secret, api-config-v2.
3. Change the Deployment to reference api-config-v2.
4. Delete the old one.

Each step looks harmless. Put together, they're a mess.

Step 1 is unverified. Somebody pasted forty values into a web form. You're trusting the paste.

Step 3 is a rollout. Changing the pod template rolls every pod. Post 3 covered what a rollout is: a stress test that ships everything queued behind it. Now your secrets migration is also a deploy.

The failure shows up late. Environment variables from a secretKeyRef are read once, at container start. A bad value in api-config-v2 sits there quietly until a pod starts and reads it. Then it fails under real traffic, and the rollback is another rollout.

Step 4 removes the thing you'd roll back to.

The fix is to not switch the reference at all. The app keeps reading the same Secret by the same name. The only thing that changes is who writes to it.

---

The technique: check, then adopt

This is External Secrets Operator (ESO) against a self-hosted Infisical, but the shape works with any sync operator.

Step 1: import the values without ever reading them. Pipe them from the live Secret into the store, as in post 4. Nothing depends on the store yet, so undoing this means deleting the entries.

Step 2: sync them somewhere harmless. Create a throwaway ExternalSecret that writes to a temporary name nobody references:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: api-config-check
  namespace: apps
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: secrets-api          # scoped to the API service's folder in the store
  target:
    name: api-config-check     # NOT the live name
    creationPolicy: Owner      # fine here: we want this one garbage-collected
  dataFrom:
    - find:
        name:
          regexp: ".*"

kubectl -n apps wait externalsecret/api-config-check --for=condition=Ready --timeout=60s

Now two Secrets sit side by side: the live one, and what the operator will actually write. Not what you think you imported.

Step 3: compare, and print only verdicts.

jq -rs '
  .[0].data as $live | .[1].data as $new
  | ($live + $new | keys[]) as $k
  | if   $new[$k]  == null then "\($k): MISSING FROM STORE"
    elif $live[$k] == null then "\($k): ONLY IN STORE"
    elif $live[$k] == $new[$k] then "\($k): MATCH"
    else "\($k): MISMATCH (\($live[$k] | @base64d | utf8bytelength) vs \($new[$k] | @base64d | utf8bytelength) bytes)"
    end' \
  <(kubectl -n apps get secret api-config -o json) \
  <(kubectl -n apps get secret api-config-check -o json)

Comparing the base64 strings directly is a real byte comparison. The API server encodes the stored bytes the same way every time, so the strings match exactly when the bytes do. The values stay inside jq, and only key names, verdicts and lengths come out. The Secrets go in through process substitution, not as arguments, because post 4 already covered what happens to secrets you put in argv.

In the real migration every key came back MATCH. Here's what the check prints when it catches the failure it exists for, reproduced with synthetic values:

DB_PASSWORD: MATCH
LOG_LEVEL: MATCH
PROVIDER_API_KEY: MISMATCH (48 vs 49 bytes)
SMTP_PASSWORD: MATCH

One byte. You already know which byte.

Step 4: adopt. Delete the check resource (it's Owner, so its Secret gets garbage-collected along with it). Then point the real ExternalSecret at the live name:

  target:
    name: api-config
    creationPolicy: Merge

Merge means ESO writes into a Secret that already exists and never sets an ownerReference. The bytes are identical, so the data doesn't change, the pods don't restart, and the app never finds out it has a new landlord.

Rollback is kubectl delete externalsecret api-config. With no owner reference, nothing gets garbage-collected. The Secret stays exactly as it was, with a writer that simply stopped writing.

Step 5: retire the old source. Archive it, don't delete it. Rollback from here is kubectl apply on the archive.

When the ESO resources later moved under GitOps, the same idea came back one layer up. We stamped them with Helm's ownership label and release annotations (app.kubernetes.io/managed-by: Helm, meta.helm.sh/release-name, -namespace) so Helm adopted them instead of choking on resources it didn't create. Adopt, don't recreate.

---

"If the bytes can't change, how do you change anything?"

This is the objection: byte-identical sounds like a rule that bans the one thing a secrets manager is for.

It doesn't. It bans doing two things in one step.

A migration changes where a value lives. A rotation changes what the value is. Each one is easy to verify alone. Combine them and every failure has two possible causes and no clean rollback. If a MISMATCH could be either a paste error or an intended new credential, the check can't tell you anything.

So migrate first, with every key a MATCH. Once the store is the source of truth, rotate there as a separate change you can watch.

The other half of the objection is fair: "you never restarted a pod, so you never proved the app actually starts on the new source." True, and this rule argues with post 3, which tells you to restart on purpose. Here's how I square it. Right after adoption, a restart can't teach you anything about the migration, because the bytes are the same bytes. It can still teach you about everything else. So roll the pods deliberately, soon after, as a separate step, and don't treat the two results as one.

---

Trailing newlines, and friends

PROVIDER_API_KEY: MISMATCH (48 vs 49 bytes) is almost always \n. It sneaks in from all over:

• echo "$v" | secrets-cli set --stdin adds one. printf '%s' "$v" doesn't.
• kubectl create secret generic --from-file=KEY=key.txt keeps the final newline your editor quietly added.
• A web form's multi-line field keeps whatever you pasted, including a newline you swept up with the value.
• A file edited on Windows brings \r\n. That's two bytes, and it looks fine in every terminal on earth.

It cuts the other way too. $(...) strips trailing newlines, so a value that should end in one (a PEM key, a JSON service-account file) can lose it on the way in.

What makes this nasty is that a newline doesn't fail loudly. A password with a newline on the end is just a wrong password. Most HTTP clients refuse to send a header that contains one, and the error mentions a header, not your migration. Env-var consumers break at their next restart. File-mounted consumers can break whenever the kubelet refreshes the mount. Neither one happens at a moment that points back at you.

---

What goes wrong anyway

• Leaving out creationPolicy. The default is Owner. A manifest that doesn't say Merge is a manifest that says Owner.
• Pointing Owner at a live Secret. If the Secret has no owner reference, ESO treats it as orphaned and takes it over. It writes the provider's desired state every reconcile. Since v0.11 that drops any key the provider doesn't have. Delete the ExternalSecret during cleanup and Kubernetes garbage-collects your production Secret.
• Trusting deletionPolicy to save you. It controls what happens when provider-side secrets disappear, not what happens when you delete the ExternalSecret. ESO also rejects deletionPolicy: Delete together with creationPolicy: Merge. It won't delete a Secret it doesn't own.
• Assuming Merge never removes anything. Keys the ExternalSecret manages can still disappear from the Secret if they disappear from the source. After adoption, the store is the truth. Treat deleting an entry there as deleting it in production.
• ONLY IN STORE lines. A find with regexp: ".*" syncs whatever is in that folder, including a key someone parked there "temporarily."
• Forgetting the check resource. It's another full copy of every credential in the namespace. Delete it in the same step.
• last-applied-configuration. A Secret created with kubectl apply keeps a full copy of its data in that annotation. That copy doesn't rotate when you rotate the value. Remove it with kubectl -n apps annotate secret api-config kubectl.kubernetes.io/last-applied-configuration-.
• Someone re-applies the old YAML. ESO writes it back at the next refresh, and now you have a Secret that flips between two values. Archive the old source somewhere kubectl apply won't find it.

---

Making it stick

Define the evidence, not the virtue. Don't tell the agent "verify the migration." Tell it "adopt only when the comparison shows MATCH for every key and no MISSING or ONLY IN lines, and paste that output into the PR." That's a check you can grep, not a feeling you have to trust.

Make creationPolicy mandatory in review. A one-line CI check that fails on any ExternalSecret without an explicit creationPolicy takes the dangerous default off the table.

Never put a migration and a rotation in the same PR. If a diff changes where a value lives and what it is, split it.

Keep the old source until the new one has survived a deliberate restart. Then archive it, and write down where.

Rule to steal

Prove the new source is byte-identical before anything depends on it. Sync to a throwaway name, compare every key and print only the verdicts, then adopt the live Secret with Merge so the rollback is deleting one resource. Migrate the value or change the value, never both in one step.

---

Next: Rotating a leaked credential, in the right order — why the git-history rewrite is the one step you don't need, and where the key was actually being baked in.