Skip to content

Category: Software

Deploying Docker Containers without Leaking Secrets

I maintain a Django application at the company at which I am employed. Initially, changes weren’t very frequent, so logging into the host and running a few commands when the application needed to be updated wasn’t a big deal. However, feature creep has set in and changes to the code base are not only more frequent, but are done by more people. Something had to be done.

Whatever solution I was going to implement had a few requirements:

  1. No secrets should be visible in source control
  2. No secrets should be built into the Docker image (anyone who can inspect it can view said secrets)
  3. All code for the process should be stored in source control
  4. The deployment process should not be limited to being run from my laptop or the application server.

Numbers one and three are problematic together, as it’s very easy to just leave a few files in your .gitignore that contain all of your secrets, but that means that #4 is impossible then. Clearly, many people have done this before, right? I mean, isn’t this what CI/CD and DevOps is all about? So, I took to Google.

The third principal of the Twelve Factor App is that the “app stores config in environment variables.” I was already doing that: setting the environment variables for things like my database password and secret key in the supervisor config. A lot of blog posts recommend setting those environment variables with the ENV command in the Dockerfile, but that violates requirement #2 and possibly #1, depending on the method. So that was right out.

The docker run command supports passing environment variables to the container with the -e flag. This seemed like the place to do it, but the question was how.

Another project I had worked on was doing a proof-of-concept of Ansible Tower. For a multitude of reasons (including very high licensing costs), we went with the open-source upstream, AWX, instead. As I’d been implementing that, I’d become aware of Ansible Vault, the a feature of Ansible that allows keeping sensitive data such as passwords or keys in encrypted form instead of plaintext. As Ansible has modules for Docker, AWX was looking like a great tool to use for deployment.

I started writing a simple playbook:

---
- hosts: appservers
become: yes
become: appuser
vars_files:
- "vars/Production.yml"
tasks:
- name: Clone git repository
git:
repo: ssh://git@git.mycompany.com/myproject.git
dest: /opt/myproject/src
accept_hostkey: yes
version: master
key_file: /opt/myproject/.ssh/id_rsa
force: yes
register: gitpull
- name: Build Docker Image
docker_image:
name: myproject-prod
path: /opt/myproject/src
state: present
force: yes
buildargs:
settingsfile: "{{ settings_file }}"
when: gitpull.changed
register: build
- name: Run the container
docker_container:
name: myproject-prod
image: myproject-prod:latest
published_ports: '8000:8000'
state: started
restart_policy: always
recreate: '{{ "yes" if (build.changed) else "no" }}'
env:
SECRET_KEY: "{{ secretkey }}"
SQL_HOST: "{{ sqlhost }}"
SQL_USERNAME: "{{ sqluser }}"
SQL_PASSWORD: "{{ sqlpassword }}"
BIND_PASSWORD: "{{ bindpassword }}"

I’m doing a number of things here. Most of them should be fairly simple, but I’ll talk about a few of the more interesting things I did. First, I’m only building the container when cloning the git repo reports a change. There’s no need to rebuild the image if it’s going to build the same thing. Second, I’m setting recreate to yes if I did rebuild the image, as I want to run the new container, not keep the old one chugging along. I could have left it statically as yes and set the task to run based on “when” conditions, but there are situations in which I may need to start the container via deployment (like if my host goes down).

You’ll also note that I brought in a vars_file. This file contains definitions of all of the variables you see in the “env” section of the docker_container task and looks something like this:

This was created using the ansible-vault command, like this:

ansible-vault encrypt_string 'super_secret_password' --name 'sqlpassword'

This tool then asks you for the vault password, then spits out the encrypted version of the string.

From there, we can run the playbook like this:

ansible-playbook --ask-vault-pass -K -i prod deploy.yml

After entering our sudo and vault passwords, the application is deployed.

Next time, I’ll talk about turning this into a one-click operation with AWX.

NuPass Released

NuPass, my user-readable password generator, is now released (currently 0.2.1) and is available on PyPI.

This means that I should get to updating NuPassWeb and its live version, nupass.pw. I’m totally open to pull requests, if you feel like helping out.

re2c on EPEL7

I’ve just pushed a build of re2c to epel-testing. This has been asked for in a bugzilla ticket and is required for building openvpn-auth-ldap, which I’ll be working on next.

You can look at the specific build here and, if you test it, please make your comments on its bodhi page.

openvpn-auth-ldap on EL7 (CentOS 7/RHEL 7)

Following-up on my last post, I was able to successfully build and test openvpn-auth-ldap on EL 7. You can get it from my copr by adding the repository to your system and running the following:

# yum install openvpn-auth-ldap

The plugin will install to /usr/lib64/openvpn/plugin/lib/openvpn-auth-ldap.so and you can use that in your OpenVPN configuration file.


 

I built the RPM from the openvpn-auth-ldap-2.0.3-14.fc21 SRPM, but modified it to build against the 2.3.10 (current) version of OpenVPN, which implements plugin version 3.

I’ve reached out to see about getting both this and my re2c package included in EPEL7, which will probably require me to become the maintainer. We shall see how that plays out.

Never Gonna Catch Me, Copr

The openvpn-auth-ldap package doesn’t exist for EL7. Since the new OpenVPN servers I’m trying to setup run CentOS 7 and use LDAP (Active Directory) for user authentication, this is problematic.

So, I’m trying to build it. I’ve built the single dependency that didn’t already exist for EL7 (re2c), but am still having issues. Although the chroot environment installs gcc-objc, configure  complains that there’s no Objective C environment available.

Will update with progress, but you can check out my Copr here.

DODSCP: A Web Control Panel for Linux Game Server Managers

I’ve been working on a Flask-based web application for controlling my latest game server. This time around, I used LGSM to run the server. It’s a painless way to get it up and running and does all of the steam_cmd dirty work for me. The only issue is that I don’t want to hand out shell access to the other people who may need to restart or update the server from time-to-time. So, DODSCP was born.

I am currently working on determining how to best distribute it, but hope to have version 0.1 available by this weekend.

 

SSH Login without Password

Have you copied your public key to your remote ~/.ssh/authorized_keys file and are still being prompted for your password? There’s a good change that the permissions are wrong on that file. If you look at the ssh logs, you can see entries like this:

$ sudo tail /var/log/secure
Jun  5 17:27:16 server sshd[12001]: Authentication refused: bad ownership or modes for file /home/sean/.ssh/authorized_keys

Change the permissions mode to 600 and you should be able to login as intended.

Installing Cisco ASDM on Linux

Cisco’s Adaptive Security Device Manager is a GUI tool for managing and configuring Cisco security appliances. It runs perfectly well under Linux, but can be a little tricky to get running. Today, I’ll show you how.

I am currently running the following:

  • Fedora 22 Workstation w/ Gnome 3.16
  • Oracle Java 8 (1.8.0_45)

Adding a Security Exception

The first thing we need to do is add a security exception for the ASA. Open up the Java Control Panel with the following command:

$ /usr/java/latest/bin/ControlPanel &

Click on the Security tab and then on the Edit Site List… button.

Once the Exception Site List window opens, click on Add and type in “https://” followed by the IP of your ASA and a trailing forward-slash. If you’ve configured ASDM to be available on a different port, you’ll need to specify that. For example, if your ASA has the IP address of 192.168.10.1 and you’ve configured ASDM to be on port 4430, you’d enter the following:

https://192.168.10.1:4430/

Click OK to close the Exception Site List window, then OK again to close the Java Control Panel.

Installing ASDM

Go back to your terminal window and enter the following command, replacing <SITE_ADDRESS> with the IP and port number, if changed, of your ASA:

$ javaws https://<SITE_ADDRESS>/admin/public/asdm.jnlp

Accept the security warnings and login to your ASA. ASDM will install itself and, if you have the Applications Menu extension turned on, you’ll find it under Java WebStart.

Domain Controller Woes

Over the course of my work at my current place of employment, I have run into all sorts of issues either promoting new Active Directory domain controllers or demoting them.

Most recently, in a project to remove Windows Server 2003 boxes from an environment, I have four domain controllers running: two Windows Server 2003 and two Windows Server 2008 R2.

Issues Demoting Windows Server 2003 DCs

The RPC server is unavailable.

After transferring all of the FSMO roles off to a 2k8r2 server, I prepared to demote the first of the domain controllers. What happened?

The operation failed because:

Active Directory could not transfer the remaining data in directory partition
CN=Schema,CN=Configuration,DC=xxx,DC=NET to domain controller yyy.xxxx.NET.

"The RPC server is unavailable."

Retrying failed again, even though the RPC service was running on the referred to DC. Damn.

Well, the fix isn’t as hard as I thought, even though I had to piece the solution together from a number of sources.

  1. Change the to-be-demoted DC’s DNS servers to your new DCs.
  2. Run ipconfig /flushdns
  3. Run dcpromo again.

That’s it. It works. Horray.

I’ll expand this post as I discover more solutions.