Every serious incident review eventually reaches the same question: which secrets were exposed, and how quickly can we revoke and replace them? Teams that cannot answer that question in minutes discover the real cost of ad hoc secrets management. Secrets scattered in environment files, CI variables, chat messages, and developer laptops cannot be rotated quickly, cannot be audited reliably, and cannot be revoked with confidence. The incident becomes longer and more expensive than the original breach warranted.
A secrets management system — typically a vault — solves the storage problem, but storage alone does not survive an incident. What matters is the full lifecycle: how secrets are created, who can read them, how they are delivered to workloads, how often they rotate, how revocation works, and what happens when the vault itself is unavailable or compromised. This article covers that lifecycle, drawing on the failure modes that show up repeatedly in incident reviews: long-lived static credentials, shared secrets with no owner, rotation that nobody has tested, and break-glass access that either does not exist or is never exercised.
The goal is not perfect secrecy. It is a system where a leaked credential is a contained, routine event — revoke, rotate, verify — rather than a crisis that requires rebuilding trust in the entire platform.
#Quick Answer: How Do You Design Secrets Management That Survives an Incident?
Centralise secrets in a vault with identity-based access, short-lived dynamic credentials where possible, and automatic rotation for everything else. Give each workload its own identity and least-privilege policy, deliver secrets at runtime rather than baking them into images or repositories, and log every read. Test rotation and revocation regularly, including a full rotation drill after a simulated leak. Keep a tested break-glass path for vault administration that does not depend on the vault itself, and define in advance who can declare a secret compromised and who approves emergency rotation. If you can rotate a leaked database credential in under an hour without downtime, your design works; if rotation requires a weekend of manual edits, it does not.
#Centralise, But Design for Vault Failure
The first decision is centralisation. A single vault — HashiCorp Vault, a cloud secrets manager, or a comparable platform — gives you one place to audit, one place to enforce policy, and one place to rotate. Multiple ad hoc stores — a mix of CI variables, cloud parameter stores, Kubernetes Secrets, and spreadsheets — give you none of those. Centralisation is worth the migration effort, even though it creates a high-value target. That target must be defended accordingly: the vault runs on hardened, dedicated infrastructure, its data is encrypted at rest with keys managed separately, access is restricted by network policy, and administrative actions require strong authentication.
Identity-based access, not shared tokens
The most common vault anti-pattern is a single shared token that every service uses. It is easy to distribute and impossible to audit: when a secret leaks, you cannot tell which workload read it, and revoking the token breaks everything. Replace shared tokens with workload identity. In Kubernetes, that means the workload's service account authenticates to the vault and receives a short-lived token scoped to its own policy. In cloud environments, use the platform's workload identity — instance roles, workload identity federation, or equivalent — so that no long-lived credential needs to be stored at all. Each policy should grant read access only to the specific secret paths that workload needs, and nothing else. Review policies quarterly; policies accumulate permissions the same way firewall rules do.
#Dynamic Secrets and Rotation That Actually Happens
Static secrets are the core problem. A database password created two years ago, shared by five services, and never rotated is a standing invitation. If it leaks, you cannot tell where it is used, and rotating it risks breaking unknown consumers. Dynamic secrets solve this by generating a unique, short-lived credential for each workload on demand. The vault creates a database user with a limited lifetime, the workload uses it, and the credential expires automatically. There is nothing long-lived to leak, and revocation is simply deleting the vault lease.
Dynamic secrets are not feasible for every system. Some legacy databases, SaaS APIs, and third-party services only support static API keys. For those, automate rotation on a schedule and on demand. Scheduled rotation — every 30 to 90 days depending on sensitivity — keeps the window of exposure bounded. On-demand rotation is the incident path, and it must be tested. A rotation procedure that has never been run under time pressure will fail when it matters: someone will discover a hardcoded copy, a service that caches the old credential, or a consumer that nobody documented.
Rotation without downtime
The safe rotation pattern is dual-credential, or versioned, rotation. First, create the new credential alongside the old one, so both are valid. Second, update consumers to use the new credential, in waves, verifying each wave. Third, after confirming no consumer still uses the old credential — check vault audit logs and database connection logs, not just deployment records — revoke the old credential. Finally, verify that revocation did not break anything, and only then close the incident or rotation task. Rotating by deleting the old credential first and hoping consumers pick up the new one is how incidents get worse.
Automation makes this reliable for common systems. For custom systems, write a rotation runbook and script, and run it in staging monthly. Track rotation coverage: what percentage of production secrets rotated in the last 90 days, and which are exempt and why? Exemptions should be few, documented, and approved.
Example: Vault database role with short TTL (illustrative)
Each workload gets a unique, expiring database user
path "database/creds/app-readonly" { capabilities = ["read"] }
Role configuration: default_ttl 1h, max_ttl 24h
Application renews or re-requests; expiry revokes access automatically
```
#Delivery: Keep Secrets Out of Images, Repos, and Logs
How a secret reaches a workload matters as much as where it is stored. Secrets baked into container images are exposed to anyone with registry access and persist in image layers even after deletion from the running container. Secrets committed to git, even briefly, must be treated as compromised: git history preserves them, and scanning tools can only tell you after the fact. Secrets passed as plain environment variables are visible in process listings, crash dumps, and some monitoring tools. None of these are acceptable for production.
Prefer runtime injection: a vault agent or sidecar that renders secrets to a temporary file in memory, or injects them into the process environment at startup from the vault, with the file or environment scoped to that process. In Kubernetes, the Secrets Store CSI driver or a vault agent injector follows this pattern. The secret never appears in the image, the repository, or the deployment manifest. For CI pipelines, use short-lived, job-scoped tokens from an OIDC identity provider rather than long-lived repository secrets, and mask secret values in logs. For background on pipeline gates that enforce these rules, see the discussion of DevSecOps pipeline security and secrets gates.
#Incident Response: Revoke, Rotate, Verify
When a secret is suspected compromised, speed and clarity matter more than perfect forensics. The incident plan should answer, in advance: who can declare a secret compromised, who approves emergency rotation, which systems are affected, and how you verify that the old credential no longer works. If those answers require a meeting to determine, the plan is incomplete.
The response sequence is straightforward. First, scope: identify every system that used the secret, using vault audit logs and dependency records. Second, revoke or rotate: issue new credentials via the dual-credential pattern, or revoke leases immediately if downtime is acceptable and the risk justifies it. For dynamic secrets, revoking the lease is usually sufficient. For static secrets, rotate everywhere, including copies in CI, staging, and disaster recovery environments that teams forget. Third, verify: attempt to authenticate with the old credential and confirm failure, check logs for use of the old credential after revocation, and monitor for anomalous access with the new credential. Fourth, investigate: determine how the leak happened and fix that path, whether it was a committed secret, an over-permissive policy, or a compromised workstation.
Lessons from real failures
Three failure modes recur in incident reviews. The first is unknown consumers: a secret is rotated, and a forgotten cron job, reporting tool, or partner integration breaks days later, because it was never inventoried. Prevent this with a secret inventory that maps each secret to its consumers, maintained automatically from vault policies and connection logs where possible. The second is partial rotation: the production secret is rotated but the staging copy, the backup copy, or the CI copy is not, and the attacker uses the stale copy. Rotate all copies, or delete copies that should not exist. The third is slow detection: a secret leaks and is used for weeks before anyone notices, because vault audit logs were not monitored. Forward vault logs to the SIEM on day one, alert on bulk secret reads, reads outside business hours for sensitive paths, and reads from unusual identities. Detection is part of secrets management, not an afterthought.
#Break-Glass Access and Vault Recovery
Break-glass access is the paradox of vault design: you need a path that works when normal access fails, but that path must not become a backdoor. Design it explicitly. Vault administration should require multiple approvers for sensitive operations, using unseal keys or recovery keys split among trusted operators, so no single person can unseal or reconfigure the vault alone. Break-glass credentials for the vault itself should be stored offline or in a separate, heavily monitored store — not in the vault they are meant to recover. Their use should trigger immediate alerts to security leadership and a post-use review.
Test break-glass procedures twice a year, including a full restore from backup in an isolated environment. A backup never restored is a hope, not a plan. Test vault unavailability, identity provider outage, and administrator absence. Document the procedure offline, and rotate break-glass credentials after each test or use. The vault's own keys and root token need their own lifecycle; a root token left active is the master key to every secret.
#Frequently Asked Questions
Should we use HashiCorp Vault, a cloud secrets manager, or both?
Choose based on your platform. If you are multi-cloud or on-premises, a self-managed vault gives portability and unified policy. If you are primarily on one cloud, that cloud's secrets manager is simpler, integrates with workload identity, and removes vault operations burden. Many teams use both: a central vault for cross-platform secrets, and cloud managers for cloud-native services. Avoid three or more stores; each additional store multiplies audit and rotation work.
How often should secrets rotate?
Dynamic secrets should expire in hours, not days. Static secrets should rotate every 30 to 90 days depending on sensitivity, and immediately on suspected compromise, staff departure from a sensitive role, or detection in logs or repositories. The right interval is the one you can actually sustain with automation. A 30-day policy that nobody follows is worse than a 90-day policy that is automated and verified.
What is break-glass access, and why does it need testing?
Break-glass access is an emergency administrative path that works when normal identity or vault access fails. It needs testing because untested emergency paths fail in predictable ways: stored credentials expire, the procedure references decommissioned systems, or the only person who knows the procedure is unavailable. Test twice a year, including a restore from backup, and rotate the credentials after each test.
How do we handle secrets in CI/CD pipelines?
Use OIDC-based, job-scoped tokens so pipelines authenticate as a workload identity without long-lived repository secrets. Inject secrets at runtime, mask them in logs, and scope each job to only the secrets it needs. Audit pipeline secret access like production access, because a compromised pipeline can exfiltrate every secret it can read. Never commit secrets to repositories; treat any committed secret as compromised and rotate it.
What should we do first if a production secret leaks?
Scope, revoke, rotate, and verify — in that order, without waiting for full forensics. Identify affected systems from vault audit logs, rotate via the dual-credential pattern or revoke leases, verify the old credential no longer works, and monitor for anomalous use. Then investigate the leak path and fix it. Communicate the timeline and affected systems to stakeholders early; a secret leak handled transparently and quickly does less damage than one discovered later through external reporting.
#Sources and Further Reading
- HashiCorp Vault documentation, Concepts and Secrets Engines — https://developer.hashicorp.com/vault/docs — dynamic secrets, policies, and audit devices.
- NIST SP 800-57, Recommendation for Key Management — https://csrc.nist.gov/pubs/sp/800/57/pt1/final — key lifecycle, rotation, and destruction guidance.
- OWASP, Secrets Management Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html — storage, transmission, and rotation practices.
- AWS Secrets Manager documentation, Rotation — https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html — automated rotation patterns.
- Kubernetes documentation, Secrets and Secrets Store CSI Driver — https://kubernetes.io/docs/concepts/configuration/secret/ — delivery patterns for workloads.
- Cloud Security Alliance, Secrets Management guidance — https://cloudsecurityalliance.org/ — governance and lifecycle recommendations.