Skip to main content

Encryption Key Rotation Runbook

Overview​

This runbook covers rotating the master encryption key (LIBSODIUM_ENCRYPTION_KEY) used for database field-level encryption. Rotation can be performed with zero downtime using the two-key transition window.

When to rotate:

  • Suspected key compromise
  • Scheduled rotation policy (annually recommended by CJIS)
  • Staff change (anyone who had access to the key)

Pre-Rotation Checklist​

  • New 32-byte key generated: openssl rand -hex 32
  • New key backed up securely (separate from the current key)
  • Maintenance window communicated (rotation itself is zero-downtime, but the migration step has a brief load spike)
  • Database backup taken and verified
  • psap:encrypt:migrate-fields --dry-run run on production data

Step-by-Step Rotation​

Step 1 — Generate new key​

openssl rand -hex 32
# Output: e.g. a3f9c2...  (64 hex chars = 32 bytes)

Store this as NEW_V2_KEY.

Step 2 — Add v2 key to environment​

In .env.local (or your secrets manager):

LIBSODIUM_ENCRYPTION_KEY=<existing-v1-key>         # keep unchanged
LIBSODIUM_ENCRYPTION_KEY_V2=<new-v2-key>           # add this
LIBSODIUM_ENCRYPTION_KEY_VERSION=v2                 # switch to v2

Step 3 — Deploy​

Redeploy the application. After this deploy:

  • New writes: encrypted with v2 key
  • Old reads: still decryptable (v1 key still in env)
  • Zero downtime: old requests continue to work

Step 4 — Verify dual-key mode is working​

# Create a new incident — it should get a v2: prefix
psql -c "SELECT title FROM incidents ORDER BY created_at DESC LIMIT 1;"
# Should return: v2:AAAAAA...

# Read an old incident — should still decrypt correctly via the API
curl -H "Authorization: Bearer $JWT" /api/v1/incidents/$OLD_INCIDENT_ID
# Should return the plaintext title

Step 5 — Migrate existing rows​

# Dry run first
php bin/console psap:encrypt:rotate-keys --from=v1 --dry-run

# Live rotation (re-encrypts v1 rows with the current v2 key)
php bin/console psap:encrypt:rotate-keys --from=v1

# Monitor progress — the command logs to symfony logger
tail -f var/log/prod.log | grep "rotate-keys"

Expected output:

Rotating 2847 rows from v1 to v2 (batch=500)...
[incidents] Batch 1/6 — rotated 500 rows
...
[incidents] Done — rotated 2847 rows
[incident_communications] Batch 1/2 — rotated 347 rows
...
Rotation complete.

Step 6 — Verify all rows are v2​

# Should return 0 if all rows are rotated
psql -c "SELECT COUNT(*) FROM incidents WHERE title LIKE 'v1:%';"
psql -c "SELECT COUNT(*) FROM cfs_incidents WHERE raw_xml LIKE 'v1:%';"

Step 7 — Remove v1 key​

In .env.local:

LIBSODIUM_ENCRYPTION_KEY=<new-v2-key>   # promote v2 to the primary slot
LIBSODIUM_ENCRYPTION_KEY_V2=<new-v2-key>  # keep in v2 slot (needed for v2: prefix routing)
LIBSODIUM_ENCRYPTION_KEY_VERSION=v2

Why keep both slots with the same value? The EncryptionService maps the first argument to the v1 slot and LIBSODIUM_ENCRYPTION_KEY_V2 to the v2 slot. After rotation, all DB ciphertext has v2: prefix, so the service must have the v2 key in the v2 slot to route decryption correctly. The v1 slot is never used for decryption after all rows are migrated.

Step 8 — Final deploy​

Redeploy. Verify:

# Old v1 key is gone — this should fail (good)
# New reads/writes work
curl -H "Authorization: Bearer $JWT" /api/v1/incidents/$ANY_INCIDENT_ID

Rollback​

If something goes wrong during rotation:

  1. Stop the rotate-keys command (Ctrl+C — it processes in batches, partial completion is fine)
  2. Revert env — restore LIBSODIUM_ENCRYPTION_KEY_VERSION=v1 and redeploy
  3. Rows that were already rotated to v2 are still readable (v2 key is still in env)
  4. Rows that were not yet rotated remain as v1 (v1 key is still in env)
  5. After rollback is stable, investigate the issue, then retry from Step 5

Emergency Key Compromise​

If the current key is suspected compromised:

  1. Immediately rotate to a new key (Steps 1–8 above, expedited)
  2. Audit all data access logs for the compromised period
  3. Notify affected agencies per your incident response policy
  4. Document in the audit log (create an AuditEvent manually if needed)

Rotation Schedule​

TriggerFrequency
CJIS compliance (recommended)Annually
Key personnel departureImmediately
Suspected compromiseImmediately
Infrastructure breachImmediately

Key Backup Requirements​

The encryption key is not recoverable. If lost, all encrypted data becomes permanently inaccessible.

Required backups:

  • Store in a hardware security module (HSM) or secrets manager (Vault, AWS Secrets Manager)
  • Maintain at least two offline copies in geographically separate locations
  • Test key restoration quarterly
  • Never store the key in the same system as the encrypted data