Slightly on topic but we've been talking about vaults and keys for years now and I work in CI/CD but I still am no wiser to how I'm supposed to practically use this in my pipelines.
Can someone link something that explains it like I have 20 years in IT but I'm clueless.
I can't get past the fact that a key has to exist somewhere, a key that will give you some sort of access to a secret. So how is it any better if the key already exists in the CI/CD pipeline variables?
Another thing I'm curious about is rotation, which on paper is amazing but in practice would require your vault to have sysadmin access to all your systems, in order to do rotation. It just seems like a tall order to integrate.
I'm going to talk in rather vague terms here as I'm not talking about one specific implementation, but the concept here:
The idea is that there is some system that is capable of provisioning access credentials and automatically providing them to resources based on their identity. This could be your CI runner, scping auth credentials into known locations when the system is deployed. This could be your cloud provider, using something like IAM roles and the metadata service. This could be your kubernetes cluster, providing a mount inside the container. Ideally this system is also capable of rotating these credentials too.
Then you have another system which is able to verify those credentials to give access. If you're fancy, you're doing something like mTLS on all your internal network calls so every endpoint on the other hand is verifying every other endpoint, but it could also be something like secrets manager or vault just dispensing out API keys based on your systems credentials provided in the previous step.
There is definitely a tradeoff here that the system that manages the credentials effectively has the keys to the kingdom and could if compromised provide identity for anything. Some companies respond to this by delegating the whole lot to their cloud provider, which has its own risks of course.
The advantages of doing this are basically:
- You only have to manage one set of credentials manually (whatever your system managing the instance/service identity is)
- You can then easily make credentials for things accessing each other short lived if you know they're getting replaced periodically, which helps reduce the impact of breaches/leaks
- Depending on your specific implementation, you can avoid secrets hitting disk entirely, which further helps reduce the impact of breaches.
The basic concept is that the vault will hand out a secret to an app based on some other form of identification. Let’s say the app platform can assert that App A is allowed to use Secret S. then the vault will hand it out. This turns “something you are” into “something you have”
By centralising the secrets storage, you make it easier to change the secrets in one place rather than having to change a bunch of variables everywhere.
As for how you change the secrets? You should have some automated way of pulling the current secret out of the vault, storing a future secret in the vault, performing an upgrade of the secrets, verifying the new secrets are deployed everywhere, and then moving the "future secret" into "current secret" and keeping a historical copy of the "previous secret".
Whatever configures your infrastructure automatically will already need to have root access to everything and so that's a good place to implement the rotation.
The vault doesn’t need to access each system to rotate the keys. It just needs to sign a new token with a short expiry date.
The system reading the token can verify its integrity based on the signature.
Now the vault obviously must have a master key to do the signature. It’s a very powerful privilege, that can impersonate, yet not quite as privileged as being sysadmin to all systems.
The main advantage of this is that tokens that the users use can quickly be revoked. They always need to go back to the vault to get new tokens, here you can add more powerful protections and logs, always MFA.
So what actually should happen here is your pipeline mints a JWT token with some near term expiry per run.
You send that token to vaults JWT endpoint, and it validates that it knows the issuer and the signature matches the provided keys.
When configuring the vault side, you can further validate the signed token data which will include things like the scm org, the repository name, the actor ( user who made the commit ), whether or not it's a PR, what branch it's against, whether the repository is private.
From there you can set roles within vault that allows different policy per risk, i.e. random PRs from the public shouldn't get deployment secrets.
If you use an app role you can define policies that limit the usage of said app role to specific ip addresses, ec2 instance ids etc so that even if someone steals the key it will be useless unless they somehow root the server hosting your application (and if they do that they're getting the secret no matter what)
Even with a regular token the benefit is that if it is leaked (say, by a git commit etc) this by itself doesn't grant you access to the actual secret because it is stored in vault, and you'd need to have connectivity to the vault server(s) to make use of it.
Workload identity federation is the direction the industry is taking. All the hyperscalers have their own version, if you’re on-premise look into SPIFFE.
Couldn't they use something that made more sense like OpenKeystore or OpenKeyvault? I know Hashicorp might be sensitive about them using the word Vault though, I just feel that it should be descriptive more than a game.
At least to me, both of those imply a key-value storage system, not a secrets manager. `OpenSecrets` maybe implies less security to some, and developers tend to want to give products they work on a creative name.
I'm always looking for the secret management solution for my self hosted apps. This looks very cool, but still unable to solve my biggest problem - how should I manage secrets to access the secret manager?
In my day job, we use AWS SSM. It works great. For my home network, I just put secrets on my docker-compose.yaml. Obviously I shouldn't but I can't find a better solution.
If you run Docker in Swarm mode instead (docker stack deploy instead of docker compose up), then you can make use of Swarm secrets. You don't have to make full use of Swarm; it can be a single node with a single instance of the service(s). As I manage the host with Ansible, I'm able to use Ansible's Docker module with a play to communicate the current secrets to Swarm, so that they are available to the services. On the Ansible side, the secrets are encrypted at rest using ansible-vault.
I’ve written a small Python app to deploy to Swarm mode from CI pipelines, and configure secrets and configs from CI environment variables, taking care of rotation and recreating Swarm secrets if the variable content changes. This delegates the whole secret management to the CI tool (BitBucket in my case).
The default implementation in Swarm has the problem that you cannot update secrets, so you’ll need to reconfigure and redeploy the service with a secret with a new name if that changes. That was quite a pain!
I encountered the secret update problem too. I have a secret rotation playbook that stops the Docker services stack, removes the secrets, recreates under the same name, and restarts the Docker services stack. The community.docker Ansible module does all the lifting there.
My CI runs as a container in that stack too, so in Jenkins I have an init.d Groovy script to establish Jenkins Credentials from the current Swarm secrets.
I'm not sure if this would be a solution for you, and I've never used it myself, but I found this interesting secrets management project a few years ago that I always find myself thinking "is that what I want?":
Like I said, I haven't used it, I can't vouch for it, but it looked interesting for my own use, which is personal/small team, with an emphasis on simplicity.
I like sealed secrets (https://github.com/bitnami-labs/sealed-secrets) a lot. It's like 1Password, but for apps in kubernetes. You only need to secure a private key, and can throw encrypted secrets in a public github repo or anywhere you want.
It's owned by VMware (Broadcom) now, so you have to decide which company you hate less.
For your home network, you might like Mozilla SOPS for secrets storage. You can keep your secrets in an encrypted file and have a container that loads those secrets and puts it into each service's environment variables. At least that way you aren't committing plaintext secrets to source or having them laying around in the clear, plus it's a breeze to manage and edit them.
I hold my vault seal key in my KeePass database. It's set to start and prompt for the master password when I login and it integrates into the FreeDesktop/DBus secrets API (and ssh-agent). Obviously I only need the seal/root tokens when the Vault server reboots. Once it's running it hands out secrets and certificates to everything else.
Provide a TPM encrypted credential (made by systemd-cred) and it will be decrypted and placed in a memory backed file within a private namespace mount.
Bitwarden Secrets Manager might fit your needs as an open source alternative to HashiCorp Vault fyi. I am biased (I work at Bitwarden) but still a great option for those looking to switch from HashiCorp! https://bitwarden.com/products/secrets-manager/
I always thought that Bitwarden, the password manager, was open source, but the Secret Manager was a paid add-on that was not fully open source. Is that not the case?
It isn't clear how to build this without the enterprise proprietary components, except maybe to fork it and tear out all the code for the ee folders and everything that references it.
I looked at vault, but I opted for a simpler, less flexible solution: rrsync (restricted rsync) to a tree available only to an account with its .ssh/authorized_keys populated with the host public keys with forced rrsync commands restricted to that host's secrets. Root is the only account that can read a host's corresponding private key, so that means an attacker must crack root to get this extra access - but why bother when the secret (e.g. private certificate) is already on the host for root to read? Code to translate the known_hosts into the .ssh/authorized_keys file is a dozen lines more than a perl one-liner only because of triple checking to prevent damage to the result, e.g., in file system full circumstances. Chicken/egg: other means must manage host private keys and ssh_known_hosts. But you had to do that anyways.
Can someone link something that explains it like I have 20 years in IT but I'm clueless.
I can't get past the fact that a key has to exist somewhere, a key that will give you some sort of access to a secret. So how is it any better if the key already exists in the CI/CD pipeline variables?
Another thing I'm curious about is rotation, which on paper is amazing but in practice would require your vault to have sysadmin access to all your systems, in order to do rotation. It just seems like a tall order to integrate.
reply