This post is about using using hashicorp vault and ansible.
Everyone that has used ansible knows you sometimes can’t get around storing secrets (passwords mostly) in an ansible playbook because for example an installer requires them. Or even simpler, because authentication must be done via a username and password.
The ansible embedded solution is to use ansible vault. To me, ansible vault is a solution to the problem of storing plain secrets in an ansible playbook by obfuscating them. However, these secrets are static, and still require the actual decryption key on runtime. In a lot of cases, it is delivered by putting the password in a file.
This is where hashicorp vault comes in. Vault is a standalone server for authentication and storing secrets. Using vault as a server, you can request information on runtime from the playbook, so that information is stored and maintained totally outside and independent from the ansible playbook.
In order to setup vault, I created a playbook to do that on Centos 7: https://gitlab.com/FritsHoogland/hashicorp_vault/blob/master/install_vault.yml
In order to use ansible with vault, a plugin (lookup plugin ‘hashi_vault’) can be used, however it has some python dependencies which must be resolved first, for which I created a playbook for Centos 7 too: https://gitlab.com/FritsHoogland/hashicorp_vault/blob/master/install_hashi_vault_plugin.yml
For the sake of testing, I assume this is installed on the same server. Of course in a true deployment situation, you don’t want to have anything else running on the vault server than vault, in order to keep potential attacks as far from the credentials away as possible.
After installation the vault server is “unsealed”, which means “usable”. However, it will be sealed after any stop and start, which means the server is not usable. You have to provide an “unseal token” in order for the server to be able to provide secrets. The default (production) installation provides 5 unseal tokens, and a minimum of 3 tokens necessary to unseal the vault. This installation is done using 1 unseal token and 1 that is needed to unseal vault.
At this point, the vault is empty (it contains no secrets) and there is a root token (which does not expire) to access the vault in root (superuser) mode.
Both the unseal token (unseal_key_1.txt) and the root token (root_token.txt) are left at the filesystem after the installation. Obviously, in a real deployment you don’t want these there. But for the sake of a proof-of-concept setup, I stored them on the filesystem. I also created a file that can be used to set some environment variables which are needed for the ‘vault’ commandline executable, and a script that can be used to set the root token:
$ . ./vault.env $ . ./set_root_token.sh
The next thing to do is enable an authentication method, username and password, to use, and set a username and password:
$ vault auth enable userpass $ vault write auth/userpass/users/test_read_user password=mypass
Next up, enable key-value store version 1 (‘kv’) and store dummy secrets:
$ vault secrets enable kv $ vault kv put kv/test/demo bar=foo pong=ping
What is needed additionally, is something that defines the rights which ‘test_read_user’ must have on it. This is done using a policy (file policy_test_read_kv.hcl):
path "kv/test/demo" {
capabilities = [ "list", "read" ]
}
This can be loaded as a policy in vault using:
$ vault policy write test_read_kv policy_test_read_kv.hcl
And then write this as a policy for test_read_user:
$ vault write auth/userpass/users/test_read_user policies="test_read_kv"
Now we can first test if this works on the CLI:
$ unset VAULT_TOKEN $ vault login -method=userpass username=test_read_user Password (will be hidden): Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token. Key Value --- ----- token s.OHNC9AFjnMC824pvjNPZ5aZ6 token_accessor 5AG7c00IPmqLofpwocp9yhHc token_duration 768h token_renewable true token_policies ["default" "test_read_kv"] identity_policies [] policies ["default" "test_read_kv"] token_meta_username test_read_user $ export VAULT_TOKEN=s.OHNC9AFjnMC824pvjNPZ5aZ6 $ vault vault kv get kv/test/demo ==== Data ==== Key Value --- ----- bar foo pong ping
Okay, now let’s do this in an ansible playbook (https://gitlab.com/FritsHoogland/hashicorp_vault/blob/master/1_kv_with_obtained_token.yml):
$ ansible-playbook 1_kv_with_obtained_token.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
PLAY [localhost] *********************************************************************************************************************************************************
TASK [Gathering Facts] ***************************************************************************************************************************************************
ok: [localhost]
TASK [show foo] **********************************************************************************************************************************************************
/home/vagrant/.local/lib/python2.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
ok: [localhost] => {}
MSG:
{u'pong': u'ping', u'bar': u'foo'}
PLAY RECAP ***************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
This shows all the key-values/secrets as a dict. You can do several things here, like specify the key explicitly:
lookup('hashi_vault', 'secret=kv/test/demo:bar token=s.OHNC9AFjnMC824pvjNPZ5aZ6 url=https://localhost:8200 validate_certs=false')
foo
Or specify it when you use the variable:
lookup('hashi_vault', 'secret=kv/test/demo token=s.OHNC9AFjnMC824pvjNPZ5aZ6 url=https://localhost:8200 validate_certs=false')
msg: "{{ demo.bar }}"
foo
I like the idea of handing out a token, so we don’t even have to think about username and passwords that need to be changed, a playbook gets to use a token, which holds all the access it needs, and expires automatically. If you watched closely, you saw that the token expiry is rather long (768 hours; 32 days), but you can specify the token duration in the policy. 24 hours look like a reasonable duration.
However, you could use the vault username and password in the lookup:
lookup('hashi_vault', 'secret=kv/test/demo auth_method=userpass username=test_read_user password=mypass url=https://localhost:8200 validate_certs=false')
Now there a second version of the key-value store, dubbed kv-v2. This version, as the name suggests, is a bit more advanced. It keeps more data about the key-value combinations, like versions and dates of versions. However, how to use this is not clearly documented, especially the ansible part.
This is how to setup kv-v2, insert some dummy secrets, create a policy and then retrieve them:
$ . ./vault.env $ . ./set_root_token.sh $ vault secrets enable kv-v2 $ vault kv put kv-v2/test/demo foo=bar ping=pong $ vault policy write test_read_kv-v2 policy_test_read_kv-v2.hcl $ vault write auth/userpass/users/test_read_user password="mypass" policies="test_read_kv,test_read_kv-v2"
So far it looks rather straightforward. However, if you look at the policy, you’ll see what is less obvious:
$ cat policy_test_read_kv-v2.hcl
path "kv-v2/data/test/demo" {
capabilities = [ "list", "read" ]
}
The data and metadata have been split, and explicit access to the DATA part of the secret must be written to.
This also causes the dict that is returned to be a bit different (https://gitlab.com/FritsHoogland/hashicorp_vault/blob/master/1_kv-v2_with_obtained_token.yml):
$ ansible-playbook 1_kv-v2_with_obtained_token.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
PLAY [localhost] *********************************************************************************************************************************************************
TASK [Gathering Facts] ***************************************************************************************************************************************************
ok: [localhost]
TASK [show demo] *********************************************************************************************************************************************************
/home/vagrant/.local/lib/python2.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
ok: [localhost] => {}
MSG:
{u'data': {u'foo': u'bar', u'ping': u'pong'}, u'metadata': {u'created_time': u'2019-10-06T13:48:04.378215987Z', u'destroyed': False, u'version': 1, u'deletion_time': u''}}
PLAY RECAP ***************************************************************************************************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
As you can see, some extra data is provided in the dict that is returned. In order to just list the value for the key ‘foo’, use:
msg: "{{ demo.data.foo }}"
Yes, this is another ‘data’ that is added. So the request in the lookup filter needs an added ‘data’, and when you want the value of a specific key, you need to add another ‘data’.
Frits Hoogland Weblog 