Wed 15 January 2014


Virtually secure with openvpn, pillars, and salt


I use logstash to process all of my logs. I use icinga + check_mk to monitor all my systems. I still use munin as a crutch.

All the servers use beaver to ship logs. Beaver uses stunnel for encrypting transport. stunnel uses supervisord to stay running. At the other end is redis. Redis refuses to do encryption, so it is behind another stunnel. logstash outputs to elasticsearch in the clear. check_mk uses ssh for encrypting transport. ssh needs key management and host side command sanitization. munin is wide open but naively trusted to reject all but one ip adress set in a munin.conf.

elasticsearch is designed to run behind a firewall broadcasting its traffic via multicast. Three more stunnels.

And mongo. And mysql. No more!

OpenVPN offers an easy, NSA unfriendly way to tunnel IP traffic over authenticated and encrypted TLS connections. To my surprise, it even allows clients to create on demand encrypted connections to each other.

The Question

How do I use salt to create and install the openvpn and client specific config files for each minion --on demand?

It turned into a but of an oddesy. The last bit was the toughest nut, so lets start there.

On Demand Pillar Data

Before an openvpn client can be initiate a connection it needs two things:

  1. Certificates signed by the openvpn server. sorry NSA
  2. A configuration file that agrees with the server configuration.

openvpn 2.1+ allows you to embed the keys in the client config file. Great! One less thing to worry about.

I first thought of using the output of salt-run manage.up in a bash script to pre-generate configuration files for each existing minion. Then I would have had a problem adding those per-minion conf files in a pillar tree. That can get ugly. There are tricks that can be done in jinja to include files by minion name if they exist in a directory, but I didn't want to go that route either.

I didn't want the certificates stored in pillar. I wanted them stored on the client in /etc/openvpn. I wanted the file to be created on demand on the server, signed on the server, but saved on the minion.

Be sure to change vpn.example.com to your vpn server!


#!/bin/bash -e
die() { echo "$@"; exit 1; }

[ "x$1" == "x" ] && die "$0 <hostname>"


cd /etc/openvpn/easy-rsa
if [ ! -f keys/$minion.key ]; then
  . vars > /dev/null 2>&1
  ./pkitool $1 > /dev/null 2>&1

cat <<EOF
dev tun
proto udp
remote vpn.example.com 1194
resolv-retry infinite
ns-cert-type server
verb 3

$(cat keys/ca.crt)

$(cat keys/$minion.crt | awk 'BEGIN { flag=0 } /^-----BEGIN CERTIFICATE-----/ {flag=1} flag==1 {print}')

$(cat keys/$minion.key)

I wondered if I should somehow not use such a simple bash script and instead somehow embed this in salt.

...but that worry passed in about two seconds. This works.

Now I just needed to pass the minion id and capture the output. I always wanted to learn salt's python renderer.


import subprocess

def run():
    cmd = ["./get-conf.sh", __grains__['id']]
    cwd = "/etc/openvpn/easy-rsa"
    output = subprocess.Popen(cmd, cwd=cwd, stdout=subprocess.PIPE).communicate()[0]
    return {'openvpn': output}

if __name__ == "__main__":
    global __grains__
    __grains__ = {'id':'zinc'}
    print run()

# vim: ft=python ts=4 sts=4 sw=4 et

This works, and it is fast, and the configuration files and certificates aren't stored in pillar. Win.

Notice the \__name\__ == "\__main\__"? Allows you to test the pillar sls file from the command line.

Now the glue to associate this pillar data with minions.


    - openvpn

# vim: ft=yaml

Now we can test the whole pillar system:

salt minion1 saltutil.refresh_pillar
salt minion1 pillar.get openvpn | wc -l

Configuration state

Once we have the pillar data we can tackle the state file.

I already knew about file.managed's contents argument. But that plays havoc with embedded newlines. In old versions of salt I would have used template to pass a single string though jinja {{piller['openvpn']}}. But the almighty issue queue has prodded the saltstack wizards and they have since obviated this hack. Now we can use contents_pillar.


    - user: root
    - group: adm
    - mode: 0750

    - contents_pillar: openvpn
    - user: root
    - group: adm
    - mode: 0640
    - require:
      - file: /etc/openvpn

    - pkgs:
      - openvpn

    - enable: True
    - watch:
      - pkg: openvpn
      - file: /etc/openvpn/client.conf

  # vim: ft=yaml

The above works. You don't want to run it on the vpn server itself, so a created an init.sls file to do some dispatching:

  {% if "vpn" == grains['id'] %}
  - openvpn.server
  {% else %}
  - openvpn.client
  {% endif %}

# vim: ft=yaml

After some testing I felt confident enough to push this out to all minions.

salt \* state.sls openvpn


Now to rewrite beaver.sls, logstash.sls, elasticsearch.sls, mongo.sls, redis.sls, stunnel.sls, and mysql.sls. All formerly using their own ad-hoc security solutions, now simply need to either bind to tun0 or have eth0 blocked. Nice.

Go Top
comments powered by Disqus