Archive

Tag Archives: vault

In first steps with with hashicorp vault and ansible I explained how to setup Hashicorp vault for use with Ansible.

The authentication of the playbook with Hashicorp vault in the playbooks was done in two ways:
– using a username and password in the playbook itself (which I discourage; then the authentication is readable).
– using a “authentication token” in the playbook.

The “authentication token” is obtained from vault using a username and password, and expires, so specifying that in a playbook does only spill the token. Please mind an authentication token and expires after a specified time, so it needs to created and provided just before execution, and should expire thus not being usable anymore.

There is an additional method for the playbook to authenticate itself to vault that can be used, which I spoiled in the title of this blogpost: using a certificate. You could argue that a certificate is just a longer version of a token.

I my opinion the principal difference is that a certificate is in a file, which can be produced upfront, made available before running the playbook(s), and removed afterwards. The authentication token is provided in the playbook itself.

This is how certificate based authentication is enabled on vault:

Set environment variables on the vault server to talk to it.

$ . ./vault.env
$ . ./set_root_token.sh

Create a certificate.

$ openssl req -newkey rsa:2048 -new -nodes -x509 -days 365 -keyout demo_key.pem -out demo_cert.pem

Enable certificate based authentication and upload the certificate, and bind it to policies.

$ vault auth enable cert
$ vault write auth/cert/certs/demo display_name=demo policies="test_read_kv-v2,test_read_kv" certificate=@demo_cert.pem

(please mind this assumes you created the demo setup as described in First steps with Hashicorp Vault and Ansible, otherwise you won’t have the policies test_read_kv and test_read_kv-v2 available to assign it to)

Now with the certificate available and loaded in vault, we can point to it in a playbook, which then uses the certificate to authenticate itself with vault for obtaining the secrets.

This is the playbook: demo playbook with certificate based authentication for vault

It’s important to understand that the lookup is done on the ansible master node, not on the managed node(s), so you only need to place the certificate (temporarily) there, and only need to open up the vault port for the ansible master node. This is also true for other authentication mechanisms using the hashi_vault lookup filter.

This post is about using using hashicorp vault and ansible.

Everyone that has used ansible knows you sometimes can’t get around storing secrets (passwords mostly) in an ansible playbook because for example an installer requires them. Or even simpler, because authentication must be done via a username and password.

The ansible embedded solution is to use ansible vault. To me, ansible vault is a solution to the problem of storing plain secrets in an ansible playbook by obfuscating them. However, these secrets are static, and still require the actual decryption key on runtime. In a lot of cases, it is delivered by putting the password in a file.

This is where hashicorp vault comes in. Vault is a standalone server for authentication and storing secrets. Using vault as a server, you can request information on runtime from the playbook, so that information is stored and maintained totally outside and independent from the ansible playbook.

In order to setup vault, I created a playbook to do that on Centos 7: https://gitlab.com/FritsHoogland/hashicorp_vault/blob/master/install_vault.yml

In order to use ansible with vault, a plugin (lookup plugin ‘hashi_vault’) can be used, however it has some python dependencies which must be resolved first, for which I created a playbook for Centos 7 too: https://gitlab.com/FritsHoogland/hashicorp_vault/blob/master/install_hashi_vault_plugin.yml

For the sake of testing, I assume this is installed on the same server. Of course in a true deployment situation, you don’t want to have anything else running on the vault server than vault, in order to keep potential attacks as far from the credentials away as possible.

After installation the vault server is “unsealed”, which means “usable”. However, it will be sealed after any stop and start, which means the server is not usable. You have to provide an “unseal token” in order for the server to be able to provide secrets. The default (production) installation provides 5 unseal tokens, and a minimum of 3 tokens necessary to unseal the vault. This installation is done using 1 unseal token and 1 that is needed to unseal vault.

At this point, the vault is empty (it contains no secrets) and there is a root token (which does not expire) to access the vault in root (superuser) mode.

Both the unseal token (unseal_key_1.txt) and the root token (root_token.txt) are left at the filesystem after the installation. Obviously, in a real deployment you don’t want these there. But for the sake of a proof-of-concept setup, I stored them on the filesystem. I also created a file that can be used to set some environment variables which are needed for the ‘vault’ commandline executable, and a script that can be used to set the root token:

$ . ./vault.env
$ . ./set_root_token.sh

The next thing to do is enable an authentication method, username and password, to use, and set a username and password:

$ vault auth enable userpass
$ vault write auth/userpass/users/test_read_user password=mypass

Next up, enable key-value store version 1 (‘kv’) and store dummy secrets:

$ vault secrets enable kv
$ vault kv put kv/test/demo bar=foo pong=ping

What is needed additionally, is something that defines the rights which ‘test_read_user’ must have on it. This is done using a policy (file policy_test_read_kv.hcl):

path "kv/test/demo" {
   capabilities = [ "list", "read" ]
}

This can be loaded as a policy in vault using:

$ vault policy write test_read_kv policy_test_read_kv.hcl

And then write this as a policy for test_read_user:

$ vault write auth/userpass/users/test_read_user policies="test_read_kv"

Now we can first test if this works on the CLI:

$ unset VAULT_TOKEN
$ vault login -method=userpass username=test_read_user
Password (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.OHNC9AFjnMC824pvjNPZ5aZ6
token_accessor         5AG7c00IPmqLofpwocp9yhHc
token_duration         768h
token_renewable        true
token_policies         ["default" "test_read_kv"]
identity_policies      []
policies               ["default" "test_read_kv"]
token_meta_username    test_read_user
$ export VAULT_TOKEN=s.OHNC9AFjnMC824pvjNPZ5aZ6
$ vault vault kv get kv/test/demo
==== Data ====
Key     Value
---     -----
bar     foo
pong    ping

Okay, now let’s do this in an ansible playbook (https://gitlab.com/FritsHoogland/hashicorp_vault/blob/master/1_kv_with_obtained_token.yml):

$ ansible-playbook 1_kv_with_obtained_token.yml
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'


PLAY [localhost] *********************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************
ok: [localhost]

TASK [show foo] **********************************************************************************************************************************************************
/home/vagrant/.local/lib/python2.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
ok: [localhost] => {}

MSG:

{u'pong': u'ping', u'bar': u'foo'}


PLAY RECAP ***************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

This shows all the key-values/secrets as a dict. You can do several things here, like specify the key explicitly:

lookup('hashi_vault', 'secret=kv/test/demo:bar token=s.OHNC9AFjnMC824pvjNPZ5aZ6 url=https://localhost:8200 validate_certs=false')
foo

Or specify it when you use the variable:

lookup('hashi_vault', 'secret=kv/test/demo token=s.OHNC9AFjnMC824pvjNPZ5aZ6 url=https://localhost:8200 validate_certs=false')
msg: "{{ demo.bar }}"
foo

I like the idea of handing out a token, so we don’t even have to think about username and passwords that need to be changed, a playbook gets to use a token, which holds all the access it needs, and expires automatically. If you watched closely, you saw that the token expiry is rather long (768 hours; 32 days), but you can specify the token duration in the policy. 24 hours look like a reasonable duration.

However, you could use the vault username and password in the lookup:

lookup('hashi_vault', 'secret=kv/test/demo auth_method=userpass username=test_read_user password=mypass url=https://localhost:8200 validate_certs=false')

Now there a second version of the key-value store, dubbed kv-v2. This version, as the name suggests, is a bit more advanced. It keeps more data about the key-value combinations, like versions and dates of versions. However, how to use this is not clearly documented, especially the ansible part.

This is how to setup kv-v2, insert some dummy secrets, create a policy and then retrieve them:

$ . ./vault.env
$ . ./set_root_token.sh
$ vault secrets enable kv-v2
$ vault kv put kv-v2/test/demo foo=bar ping=pong
$ vault policy write test_read_kv-v2 policy_test_read_kv-v2.hcl
$ vault write auth/userpass/users/test_read_user password="mypass" policies="test_read_kv,test_read_kv-v2"

So far it looks rather straightforward. However, if you look at the policy, you’ll see what is less obvious:

$ cat policy_test_read_kv-v2.hcl
path "kv-v2/data/test/demo" {
   capabilities = [ "list", "read" ]
}

The data and metadata have been split, and explicit access to the DATA part of the secret must be written to.

This also causes the dict that is returned to be a bit different (https://gitlab.com/FritsHoogland/hashicorp_vault/blob/master/1_kv-v2_with_obtained_token.yml):

$ ansible-playbook 1_kv-v2_with_obtained_token.yml
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'


PLAY [localhost] *********************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************
ok: [localhost]

TASK [show demo] *********************************************************************************************************************************************************
/home/vagrant/.local/lib/python2.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
ok: [localhost] => {}

MSG:

{u'data': {u'foo': u'bar', u'ping': u'pong'}, u'metadata': {u'created_time': u'2019-10-06T13:48:04.378215987Z', u'destroyed': False, u'version': 1, u'deletion_time': u''}}


PLAY RECAP ***************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

As you can see, some extra data is provided in the dict that is returned. In order to just list the value for the key ‘foo’, use:

msg: "{{ demo.data.foo }}"

Yes, this is another ‘data’ that is added. So the request in the lookup filter needs an added ‘data’, and when you want the value of a specific key, you need to add another ‘data’.

This blogpost is about using ansible vault. Vault is a way to encrypt sensitive information in ansible scripts by encrypting it. The motivation for this blogpost is the lack of a description that makes sense to me of what the possibilities are for using vault, and how to use the vault options in playbooks.

The basic way ansible vault works, is that when ansible-playbook reads a yaml file, it encounters $ANSIBLE_VAULT;1.1;AES256 indicating ansible vault is used to encrypt the directly following lines, it will use a password to decrypt it, and then uses the decrypted version in memory only. This way secrets can be hidden from being visible. Obviously, the password will allow decrypting it, and the password must be used in order for ansible-playbook to decrypt it.

The original use of vault is to encrypt an entire yaml file. As of Ansible version 2.3, ansible allows the encryption of single values in a yaml file.

1. File encryption
A very silly example of ansible-vault, which shouldn’t be used (!) is this:

$ echo "---
- hosts: localhost
  vars:
    a: aaa
    b: bbb
  tasks:

  - name: execute a shell command
    shell: >
      ls" > test1.yml

When this runs, it will execute on localhost, set to ansible variables (a and b) and then use the shell module to execute “ls”. Again, this is just an example, and is silly on itself. This is how it’s run:

$ ansible-playbook test1.yml
PLAY [localhost] ***********************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************
ok: [localhost]

TASK [execute a shell command] *********************************************************************************************************
changed: [localhost]

PLAY RECAP *****************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0

This can be encrypted using ansible vault in the following way:

$ ansible-vault encrypt test1.yml
New Vault password:
Confirm new Vault password:
Encryption successful

Now the yaml file is entirely encrypted:

$ cat test1.yml
$ANSIBLE_VAULT;1.1;AES256
32663536343532643864353135643431373830333236636230613264383531336562323530386632
3933353561316530333538363032633731336135396434630a626631303839646161363432396230
32336130633264343766393562656138363132393835646137633065393739336234336463336138
6564613730333436650a336537326233343566633761383436356136666533326561346530613566
64316336393036346538656338303734623161386435373138656532663737666662633765656438
37303666393939356161393134666130303435333463636630386434626335383164663761383035
35613437356361366665633931346664386535393936396637363561386164613337373562313264
63333530353263363437386536393765393930653234353733313433643637333932353638353339
31623366376566613766366236356163303837666664646465363439313265383562656637623266
6463663065383030626232643365623437666332313631376262

It still can be run, but you need to specify the password. In this case it asks to enter a password:

$ ansible-playbook test1.yml --ask-vault-pass
Vault password:

PLAY [localhost] ***********************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************
ok: [localhost]

TASK [execute a shell command] *********************************************************************************************************
changed: [localhost]

PLAY RECAP *****************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0

Of course this is not practical, the most common and probable way you want to use this is to have variables with secret information, like passwords, tokens, private keys, personal data, etc. encrypted, but the playbook still be readable. You should use a version control tool like git, and use functionality like diff!

The first thing is to move the variables out of the playbook into a dedicated section. In this case I chosen to assign them based on host, in a directory called ‘host_vars’, and then for localhost ‘host_vars/localhost’:

$ cat host_vars/localhost/vars.yml
---
a: aaa
b: bbb

And then of course remove them from the playbook itself:

$ cat test2.yml
---
- hosts: localhost
  tasks:

  - name: execute a shell command
    shell: >
      ls
  - debug: msg="{{ a }} / {{ b }}"

If this is run, it will pick up the variables (for which I put in a debug task):

$ ansible-playbook test2.yml
PLAY [localhost] ***********************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************
ok: [localhost]

TASK [execute a shell command] *********************************************************************************************************
changed: [localhost]

TASK [debug] ***************************************************************************************************************************
ok: [localhost] => {
    "msg": "aaa / bbb"
}

PLAY RECAP *****************************************************************************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0

Okay, now let’s say I need to encrypt the variable ‘a’. I can bluntly encrypt vars.yml, and that will work, however, then you don’t know where the variable comes from. The solution to that is ‘indirect referencing’. This is done by adding a second file to host_vars/localhost called ‘vault.yml’, which will be the encrypted file. In the case of encrypting variable a, do the following:
1. Change the vars.yml file to assign {{ vault_a }} to a:

$ cat host_vars/localhost/vars.yml
---
a: "{{ vault_a }}"
b: bbb

2. Define the value of a to vault_a in vault.yml:

$ cat host_vars/localhost/vault.yml
---
vault_a: aaa

3. Encrypt vault.yml:

$ ansible-vault encrypt host_vars/localhost/vault.yml
New Vault password:
Confirm New Vault password:
Encryption successful

Now it can be used by specifying the password again:

$ ansible-playbook test2.yml --ask-vault-pass
Vault password:
PLAY [localhost] ***********************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************
ok: [localhost]

TASK [execute a shell command] *********************************************************************************************************
changed: [localhost]

TASK [debug] ***************************************************************************************************************************
ok: [localhost] => {
    "msg": "aaa / bbb"
}

PLAY RECAP *****************************************************************************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0

This should not be a surprise, as I said, ansible will recognise an encrypted file, and decrypt it automatically. However, if you looked carefully you saw something else that is important to realise: I encrypted the variable a containing ‘aaa’, and it was revealed to me using the debug statement. I think this is reasonable, because it’s been explicitly programmed to do so. You should write your playbooks not to reveal any information that you encrypted previously.

2. Variable encryption
If you have larger sets of variables to be encrypted, the above method makes sense. However, if you want to encrypt only a few variables it’s probably simpler to encrypt only the variable you want to be encrypted. The ansible-vault executable provides a facility for that called ‘encrypt_string’. The way this works is that you simply specify the variable and the variable value with ansible-vault, which then outputs the encrypted value, which you need to put in a playbook yourself. For example your ‘a’ variable with the value ‘aaa’:

$ ansible-vault encrypt_string --name 'a' 'aaa'
New Vault password:
Confirm New Vault password:
a: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          37363334323763633562313934656631396431636139653036333036356636326163623238626132
          6138343534396232643638323830326462383831366165630a393335303939663138613437633835
          65396461383565316135316339383035626262343664643563373631366338393361623230666634
          3637363232313935310a323564663863336565623366633838616337393366313831396637386432
          3035
Encryption successful

Now use this to input to a playbook:

---
- hosts: localhost
  vars:
    a: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          37363334323763633562313934656631396431636139653036333036356636326163623238626132
          6138343534396232643638323830326462383831366165630a393335303939663138613437633835
          65396461383565316135316339383035626262343664643563373631366338393361623230666634
          3637363232313935310a323564663863336565623366633838616337393366313831396637386432
          3035
    b: bbb
  tasks:

  - name: execute a shell command
    shell: >
      ls
  - debug: msg="{{ a }} / {{ b }}"

3. Vault passwords
The next thing to look at is how to specify the decryption password. One way or another, you need a password specified to decrypt. The first option is to specify the password at the CLI, which is done in the previous examples by specifying ‘–ask-vault-pass’ as an argument to ansible-playbook. This is a good option for running playbooks ad-hoc.

If you want playbooks to run from a facility like ansible tower, rundeck or semaphore, you need to specify the password in non-interactive, in a file. This is often implemented as a hidden file, like ‘.vault_pass.txt’. Of course this then means the password is readable if you got command line access. The main idea here is this gives you an option to encrypt sensitive data in a playbook and store it in a version control system, and do NOT store the password (file) in version control.

$ echo "mypassword" > .vault_pass.txt
$ ansible-playbook --vault-password-file .vault_pass.txt test2.yml
PLAY [localhost] **************************************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************************
ok: [localhost]

TASK [execute a shell command] ************************************************************************************************************************************************
changed: [localhost]

TASK [debug] ******************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "aaa / bbb"
}

PLAY RECAP ********************************************************************************************************************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0

There is a caveat here too, into which I ran because my removable HDD enforces 777 on my files, which means the execute bit is set:

$ ansible-playbook --vault-password-file .ttt test2.yml
 [WARNING]: Error in vault password file loading (default): Problem running vault password script /Volumes/Samsung_T3/ansible/.ttt ([Errno 8] Exec format error). If this
is not a script, remove the executable bit from the file.

ERROR! Problem running vault password script /Volumes/Samsung_T3/ansible/.ttt ([Errno 8] Exec format error). If this is not a script, remove the executable bit from the file.

The message is self-explanatory: if ansible finds the passwordfile specified to be executable, it will execute it. In my case, I created a little shell script to do that:

#!/bin/bash
echo "mypassword"

3.a. Vault-id for passwords
Since ansible version 2.4 the official way to use encryption is by using vault-id. Vault-id is more or less the same as the above usage, however, it adds a new feature that could be important: the option to use multiple vault passwords in the same playbook. The way this works is quite simple: ansible tries all the passwords it has been given until it finds one that works. Please mind vault-id only works with encrypted files, not with encrypted variables! Below is an example of the use of vault-id with multiple passwords:

$ echo "---
- hosts: localhost
  tasks:

  - name: execute a shell command
    shell: >
      ls
  - debug: msg="{{ a }} / {{ b }}" > test3.yml
$ echo '---
a: "{{ vault_a }}"
b: "{{ vault_b }}"' > host_vars/localhost/vars.yml
$ echo "password_a" > .a.vault_pass.txt
$ echo "password_b" > .b.vault_pass.txt
$ echo "---
vault_a: aaa" > host_vars/localhost/vault_a.yml
$ echo "---
vault_b: bbb" > host_vars/localhost/vault_b.yml
$ ansible-vault --vault-id a@.a.vault_pass.txt encrypt host_vars/localhost/vault_a.yml
Encryption successful
$ ansible-vault --vault-id b@.b.vault_pass.txt encrypt host_vars/localhost/vault_b.yml
Encryption successful
$ ansible-playbook --vault-id a@.a.vault_pass.txt --vault-id b@.b.vault_pass.txt test3.yml
PLAY [localhost] **************************************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************************
ok: [localhost]

TASK [execute a shell command] ************************************************************************************************************************************************
changed: [localhost]

TASK [debug] ******************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "aaa / bbb"
}

PLAY RECAP ********************************************************************************************************************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0