jmtd → log → Ansible

I've just recently built the large bulk of VMs that we use for first semester teaching. This year that was 112. We use the same general approach for these as our others: get a generic base image up and running, with just enough configuration complete so a puppet client starts up; get it talking to our master; let puppet take it from there.

There are pragmatic balances between how much we do in the kickstart versus how much we do in puppet, but also when we build a new VM from scratch versus when we clone an existing image, and how specialisation we do in the clone image.

Unfortunately this year we ended up in a situation where our clone image wouldn't talk to our puppet master out of the box, due to some changes we'd made to our master set up since the clone image was prepared. We didn't really have enough time to re-clone the entire set of VMs from a fixed base image, and instead needed to fix them whilst up. However we couldn't rely on puppet to do that, since they wouldn't talk to the puppet master.

We needed to manually reset the puppet client state per VM and then re-establish a trust relationship with the correct master (which is not the default master hostname in our environment anymore). Luckily, we deploy a local account with a known passphrase via the kickstart, which also has sudo access, as an interim measure before puppet strips it back out again and sets up proper LDAP and Kerberos authentication. So we can at least get into the boxes. However logging into 112 VMs by hand is not a particularly pleasant task.

In the past I might have tried to achieve this using something like clusterssh but this year I decided to give ansible a try instead.

Ansible started life, I believe, as a tool that would let you run arbitrary commands on remote hosts, including navigating ssh and sudo as required, without needing any agent software on the remote end. It has since seemed to grow into an enterprise product in its own right, seemingly in competition with puppet, chef, cfengine et al.

Looking at the Ansible website now I'd be rather put off by just how "enterprisey" it has become - much as I am by the puppet website, if I'm honest - but if you persevere past the webinars, testimonials, etc. etc., you can find yourself to the documentation, and running an arbitrary command is as simple as

• defining a list of hosts
• running an ansible command line referencing some or all of those hosts

The hosts file format is simple

[somehosts]
host1
host2
...
[otherhosts]
host3

The command line can be a little bit more complex, especially if you need to use one username for ssh, another for sudo, and you don't want to use ssh key auth:

ansible -i ./hostsfile somehosts -k -u someuser \
    --sudo -K -a 'puppet agent --onetime --no-daemonize --verbose’

"all" would work where I've used somehosts in the example above.

So there you go: using one configuration management system to bootstrap another. I'm sure I've reserved myself a special place in hell for this.