Skip to content

Category: Technology

Deploying Docker Containers without Leaking Secrets

I maintain a Django application at the company at which I am employed. Initially, changes weren’t very frequent, so logging into the host and running a few commands when the application needed to be updated wasn’t a big deal. However, feature creep has set in and changes to the code base are not only more frequent, but are done by more people. Something had to be done.

Whatever solution I was going to implement had a few requirements:

  1. No secrets should be visible in source control
  2. No secrets should be built into the Docker image (anyone who can inspect it can view said secrets)
  3. All code for the process should be stored in source control
  4. The deployment process should not be limited to being run from my laptop or the application server.

Numbers one and three are problematic together, as it’s very easy to just leave a few files in your .gitignore that contain all of your secrets, but that means that #4 is impossible then. Clearly, many people have done this before, right? I mean, isn’t this what CI/CD and DevOps is all about? So, I took to Google.

The third principal of the Twelve Factor App is that the “app stores config in environment variables.” I was already doing that: setting the environment variables for things like my database password and secret key in the supervisor config. A lot of blog posts recommend setting those environment variables with the ENV command in the Dockerfile, but that violates requirement #2 and possibly #1, depending on the method. So that was right out.

The docker run command supports passing environment variables to the container with the -e flag. This seemed like the place to do it, but the question was how.

Another project I had worked on was doing a proof-of-concept of Ansible Tower. For a multitude of reasons (including very high licensing costs), we went with the open-source upstream, AWX, instead. As I’d been implementing that, I’d become aware of Ansible Vault, the a feature of Ansible that allows keeping sensitive data such as passwords or keys in encrypted form instead of plaintext. As Ansible has modules for Docker, AWX was looking like a great tool to use for deployment.

I started writing a simple playbook:

- hosts: appservers
become: yes
become: appuser
- "vars/Production.yml"
- name: Clone git repository
repo: ssh://
dest: /opt/myproject/src
accept_hostkey: yes
version: master
key_file: /opt/myproject/.ssh/id_rsa
force: yes
register: gitpull
- name: Build Docker Image
name: myproject-prod
path: /opt/myproject/src
state: present
force: yes
settingsfile: "{{ settings_file }}"
when: gitpull.changed
register: build
- name: Run the container
name: myproject-prod
image: myproject-prod:latest
published_ports: '8000:8000'
state: started
restart_policy: always
recreate: '{{ "yes" if (build.changed) else "no" }}'
SECRET_KEY: "{{ secretkey }}"
SQL_HOST: "{{ sqlhost }}"
SQL_USERNAME: "{{ sqluser }}"
SQL_PASSWORD: "{{ sqlpassword }}"
BIND_PASSWORD: "{{ bindpassword }}"

I’m doing a number of things here. Most of them should be fairly simple, but I’ll talk about a few of the more interesting things I did. First, I’m only building the container when cloning the git repo reports a change. There’s no need to rebuild the image if it’s going to build the same thing. Second, I’m setting recreate to yes if I did rebuild the image, as I want to run the new container, not keep the old one chugging along. I could have left it statically as yes and set the task to run based on “when” conditions, but there are situations in which I may need to start the container via deployment (like if my host goes down).

You’ll also note that I brought in a vars_file. This file contains definitions of all of the variables you see in the “env” section of the docker_container task and looks something like this:

This was created using the ansible-vault command, like this:

ansible-vault encrypt_string 'super_secret_password' --name 'sqlpassword'

This tool then asks you for the vault password, then spits out the encrypted version of the string.

From there, we can run the playbook like this:

ansible-playbook --ask-vault-pass -K -i prod deploy.yml

After entering our sudo and vault passwords, the application is deployed.

Next time, I’ll talk about turning this into a one-click operation with AWX.

Building a Ruby RPM for armv7hl

We’ve recently decided to supplement the temperature monitoring provided by the UPS units in our IDFs with Raspberry Pis and DHT22s. There’s a nice article on setting them up to pipe data into Zabbix, the monitoring system we use at work, so this is incredibly useful to us.

We use CentOS here, so converting from Raspbian to CentOS wasn’t too tricky, although the Zabbix agent is only available in the un-official, un-QAed armv7hl EPEL repository. Again, nothing too tough here. Not something I’d want on a critical system, perhaps, but if the agent stops working, at least we’ll get alerts. ūüėČ

The problem arose when I went to install Puppet, which we use for long-term configuration maintenance. It turns out that the CentOS 7 OS repository for armv7hl doesn’t include Ruby, which Puppet requires. Now, I’m currently trying to rebuild a Ruby 2.3 RPM on the Raspberry Pi so that I can get Puppet installed. Will post updates here (including a link to the RPM) once I get it figured out.

NuPass Released

NuPass, my user-readable password generator, is now released (currently 0.2.1) and is available on PyPI.

This means that I should get to updating NuPassWeb and its live version, I’m totally open to pull requests, if you feel like helping out.

Setting up oVirt 4

At work, we’re currently evaluating oVirt as a replacement for our VMware infrastructure. We had evaluated it around 3.5 and found it wanting, but the 4.0 release led us to reevaluate. I found the setup process a little lacking, but once setup correctly, it works delightfully well.

Setting up the First Host

Storage Configuration

Shared storage is required to setup oVirt. This can be an NFS share from your first host, but separate storage is highly recommended. In my setup, I used an NFS share from an illumos box we have laying around. Ensure that the NFS shares for oVirt are owned by 36:36 (vdsm).

For the rest of the post, I will assume that your hosts have at least two NICs: one for management and VM traffic and another for storage traffic.

Installation Process

Grab a copy of the oVirt Node ISO and begin installing it on the first host.

The installation process is very similar to CentOS 7. Configure the hostname and assign an IP address to the first network interface. Assign a root password and don’t bother creating a user.

Once the installation completes, reboot the host.

Hosted Engine Setup

In oVirt parlance, the “Engine” is the component that manages the hosts, much like vCenter or XenCenter. A Hosted Engine is when you run the Engine as a VM.

SSH into the first host and install the latest ovirt-engine-appliance package, which can be found here.

# yum install -y

Edit /etc/hosts and add entries for this host, as well as the oVirt Engine we will be setting up. This step is crucial, as name resolution is required to complete the setup and oVirt will drop your current DNS servers when it creates the bridge interface during Hosted Engine setup.   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 	ovengine	ov01

Open a web browser and login to the Cockpit page on the oVirt host (https://<<IP_ADDRESS>>:9090/). Login as root and click on the Networking link, then the add VLAN button. Set the Parent Interface as the second NIC, likely enp2s0f1, and the VLAN ID as your SAN. Assign the new VLAN interface an IP address on that VLAN, ignore IPv6 traffic, connect automatically, and enable the interface.

Enable the parent interface (enp2s0f1), as well, but disable IPv4, ignore IPv6, and connect automatically.

Click on the Virtualization tab, then on Hosted Engine, then click the Start button.

Answer the following questions:

  1. Are you sure you want to continue? Yes
  2. Please specify the storage you would like to use: nfs3
  3. Please specify the full shared storage connection path to use:
  4. Please indicate a nic to set ovirtmgmt bridge on: enp2s0f0
  5. iptables was detected on your computer, do you wish setup to configure it? Yes
  6. Please indicate a pingable gateway IP address:
  7. Please specify the device to boot the VM from: disk
  8. Please specify the console type you would like to use to connect to the VM: vnc
  9. Please choose which OVF archive you would like to use: 1
  10. Would you like to use cloud-init to customize the appliance on the first boot? Yes
  11. Would you like to generate on-fly a cloud-init ISO image? Generate
  12. Please provide the FQDN you would like to use for the engine appliance:
  13. Automatically execute engine-setup on the engine appliance on first boot? Yes
  14. Automatically restart the engine VM as a monitored service after engine-setup? Yes
  15. Please provide the domain name you would like to use for the engine appliance:
  16. Enter root password that will be used for the engine appliance: <Same as host>
  17. Confirm root password: <Same as host>
  18. Please specify the CPU type to be used by the VM: <Use default>
  19. Please specify the number of virtual CPUs for the VM: 4
  20. You may specifya unicast MAC address for the VM or accept a randomly generated default: <Use default>
  21. Specify the memory size of the VM in VM: 12722
  22. How should the engine VM network be configured: Static
  23. Please enter the IP address to be used for the engine VM: <The IP address selected previously and added the /etc/hosts>
  24. Please provide a comma-separated list (max 3) of IP addresses of domain name servers for the engine VM:,
  25. Add lines for the appliance itself and for this host to /etc/hosts on the engine VM? <If you’ve added DNS records, you can safely enter No. Else, Yes>
  26. Enter engine admin password: <password>
  27. Confirm engine admin password: <password again>
  28. Enter the name which will be used to identify this host inside the Administrator Portal: ov01
  29. Provide the name of the SMTP server through which we will send notifications:
  30. Please provide the TCP port number of the SMTP server: 25
  31. Please provide the email address from which notifications will be sent: ‚Äč
  32. Please provide a comma-separated list of email addresses which will get notifications:
  33. Please confirm installation settings: Yes

Let the process do it’s thing and eagerly await the message “Hosted Engine Setup successfully completed!”.

Next time, we’ll discuss configuring the default datacenter.

Connecting Network Gear to Avocent/Cyclades ACS Console Server

There’s a lot of differing information about what the proper cabling for connecting Cisco/HP/whatever network gear to an ACS console server. I’ve seen everything from straight-through to rollover to a modified version of straight-through. However, Avocent provided the correct version and I’ve verified that it works.

From the Avocent/Emerson Power site:

RJ-45 (Cyclades) to RJ-45 (Sun/Cisco), crossoverRJ-45 RJ-45
Cyclades Sun/Cisco
——– ———
3 TxD ———— RxD 6
6 RxD ———— TxD 3
4 Gnd ——–|— Gnd 4
|— Gnd 5
2 DTR ———— DSR 7
7 DCD ———— DTR 2
1 RTS ———— CTS 8
5 CTS ———— RTS 1

To build this, take a standard TIA-568B straight-through cable and cut of one end. Remake the cut end as follows:

Pin 1 2 3 4 5 6 7 8
Color Brown White/Brown Green Blue White/Orange White/Green Orange White/Blue (optional)

The factory end would go into the console port of the network gear, while your custom end would go into the ACS.

Updating the Firmware on Quanta LB4M Switches

There seems to be a lack of information (that isn’t misinformation) on how to upgrade the firmware on Quanta LB4M switches. Posts abound containing warnings about bricking your switch. Likely, these people attempted some XMODEM transfer that was not needed. All you need is a TFTP server, which can even be running on your local computer.

Grab a copy of the firmware from Jared’s site¬†(or my mirror: / I used lb4m. as it’s the latest and seems to be the most featureful, although I’m still trying to locate a copy of the release notes for each version.

Make that file available via your TFTP server and ensure that your switch can see your TFTP server (ping should do nicely). I’ll assume that your TFTP server is on, so if it isn’t replace that with the IP you’re using. Once your switch can see the TFTP server, it’s time to begin.

The LB4M has two firmware slots: image1 and image2. You’ll need to determine which is currently active, as you cannot download firmware to the active slot.

(Switching) #show bootvar

Image Descriptions

image1 : default image
image2 :

Images currently available on Flash

unit      image1     image2      current-active         next-active

  1              image1              image1

This shows that we need to download the firmware into image2, as image1 is the currently loaded firmware. We do that with the following command:

(Switching) #copy tftp:// image2

Select ‘y’ at the prompt of “Are you sure you want to start?” and the transfer will begin. It will take a few minutes, but should complete without issue, assuming your TFTP server is reachable and working properly. If all goes well, you’ll see the following:

Verifying CRC of file in Flash File System

File transfer operation completed successfully.

All that’s left is activating the image and rebooting.

(Switching) #boot system image2
Activating image image2...

(Switching) #reload

re2c on EPEL7

I’ve just pushed a build of re2c to epel-testing. This has been asked for in a bugzilla ticket and is required for building openvpn-auth-ldap, which I’ll be working on¬†next.

You can look at the specific build here and, if you test it, please make your comments on its bodhi page.

openvpn-auth-ldap on EL7 (CentOS 7/RHEL 7)

Following-up on my last post, I was able to successfully build and test openvpn-auth-ldap on EL 7. You can get it from my copr by adding the repository to your system and running the following:

# yum install openvpn-auth-ldap

The plugin will install to /usr/lib64/openvpn/plugin/lib/ and you can use that in your OpenVPN configuration file.


I built the RPM from the openvpn-auth-ldap-2.0.3-14.fc21 SRPM, but modified it to build against the 2.3.10 (current) version of OpenVPN, which implements plugin version 3.

I’ve reached out to see about getting both this and my re2c package included in EPEL7, which will probably require me to become the maintainer. We shall see how that plays out.

Never Gonna Catch Me, Copr

The openvpn-auth-ldap package doesn’t exist for EL7. Since the new OpenVPN servers I’m trying to setup run CentOS 7 and use LDAP (Active Directory) for user authentication, this is problematic.

So, I’m trying to build it. I’ve built the single dependency that didn’t already exist for EL7 (re2c), but am still having issues. Although the chroot environment installs gcc-objc, configure ¬†complains that there’s no Objective C environment available.

Will update with progress, but you can check out my Copr here.