Below are the five most recent posts in my weblog. You can also see a chronological list of all posts, dating back to 2003.

Wed 08 Oct 2014 09:12:46 PM BST

I've just recently built the large bulk of VMs that we use for first semester teaching. This year that was 112. We use the same general approach for these as our others: get a generic base image up and running, with just enough configuration complete so a puppet client starts up; get it talking to our master; let puppet take it from there.

There are pragmatic balances between how much we do in the kickstart versus how much we do in puppet, but also when we build a new VM from scratch versus when we clone an existing image, and how specialisation we do in the clone image.

Unfortunately this year we ended up in a situation where our clone image wouldn't talk to our puppet master out of the box, due to some changes we'd made to our master set up since the clone image was prepared. We didn't really have enough time to re-clone the entire set of VMs from a fixed base image, and instead needed to fix them whilst up. However we couldn't rely on puppet to do that, since they wouldn't talk to the puppet master.

We needed to manually reset the puppet client state per VM and then re-establish a trust relationship with the correct master (which is not the default master hostname in our environment anymore). Luckily, we deploy a local account with a known passphrase via the kickstart, which also has sudo access, as an interim measure before puppet strips it back out again and sets up proper LDAP and Kerberos authentication. So we can at least get into the boxes. However logging into 112 VMs by hand is not a particularly pleasant task.

In the past I might have tried to achieve this using something like clusterssh but this year I decided to give ansible a try instead.

Ansible started life, I believe, as a tool that would let you run arbitrary commands on remote hosts, including navigating ssh and sudo as required, without needing any agent software on the remote end. It has since seemed to grow into an enterprise product in its own right, seemingly in competition with puppet, chef, cfengine et al.

Looking at the Ansible website now I'd be rather put off by just how "enterprisey" it has become - much as I am by the puppet website, if I'm honest - but if you persevere past the webinars, testimonials, etc. etc., you can find yourself to the documentation, and running an arbitrary command is as simple as

  • defining a list of hosts
  • running an ansible command line referencing some or all of those hosts

The hosts file format is simple

[somehosts]
host1
host2
...
[otherhosts]
host3

The command line can be a little bit more complex, especially if you need to use one username for ssh, another for sudo, and you don't want to use ssh key auth:

ansible -i ./hostsfile somehosts -k -u someuser \
    --sudo -K -a 'puppet agent --onetime --no-daemonize --verbose’

"all" would work where I've used somehosts in the example above.

So there you go: using one configuration management system to bootstrap another. I'm sure I've reserved myself a special place in hell for this.

Tags:

Thu 02 Oct 2014 07:02:43 PM BST

For my 31st birthday I decided to build myself a computer, specifically a NAS and backup server which could do some other bits and pieces. I ended up buying a system based on the Gigabyte J1900N-D3V SoC from Mini-ITX (who's after sales support is great, by the way).

I hope to write up a more comprehensive overview of what I've ended up with (probably in my rather dusty hardware section), but in the meantime I have a question for anyone else with this board:

If you've upgraded the BIOS, do the more recent BIOS versions insist on there being a display connected in order to boot?

Sadly the V1 BIOS version does, which seriously limits the utility of this board for my purposes. I did manage to flash the board up to V3, once, but it later decided to downgrade itself (believing the flashed BIOS to be corrupt). I haven't managed a second time. The EFI implementation in this board is... interesting. Convincing it to boot anything legacy is a tricky task.

As an aside, I recently stumbled across this suggestion on reddit to use an old-ish, Core-era Thinkpad T-series with a dock for this exact purpose: the spare ultrabay gives you two SATA drive slots; the laptop battery serves as a crude UPS and there's a built in keyboard and mouse, avoiding the issue I'm having with the J1900N-D3V. A Core i5 is more than fast enough for what I want to do and it will have vt. Hindsight is a wonderful thing...

Tags:

Mon 29 Sep 2014 05:46:17 PM BST

I recently read a few issues of Starburst magazine which is good fun, but a brief mention of the Man Booker prize in issue 404 stoked the fires of the age old SF-versus-mainstream argument, so I wrote the following:

Dear Starburst,

I found it perplexing that, in "Brave New Words", issue 404, whilst covering the Man-Booker shortlist, Ed Fortune tried to simultaneously argue that genre readers "read broadly" yet only Howard Jacobson's novel would be of passable interest. Asides from the obvious logical contradiction he is sadly overlooking David Mitchell's critically lauded and undisputably SF&F novel "The Bone Clocks", which it turned out was also overlooked by the short-listers. Still, Jacobson's novel made it, meaning SF&F represents 16% of the shortlist. Not too bad I'd say.

All the best & keep up the good work!

As it happens I'm currently struggling through "J". I'm at around the half-way mark.

Tags:

Sun 28 Sep 2014 07:30:50 PM BST

Well, not long after writing my last post I've found some time to write up some of my puppet adventures, sooner than I imagined...

Outside work, I sys-admin a VPS instance that is shared by a few friends. We recently embarked in a project to migrate to a different VPS instance and I took the opportunity to revisit how we managed home directories.

I've got all the disk space allocated to the VM set up as LVM physical volumes. This has proven very useful for later expansion: we can do it all live. Each user on the VM may have one or more UNIX accounts that they use. Therefore, in the old scheme, for the jon user, we mounted an allocation of disk space at /home/jons, put the account home directories under it at e.g. /home/jons/jon, symlinked /home/jon -> /home/jons/jon for brevity, and set that as the home field in the passwd entry. This worked surprisingly well, but I was always uncomfortable with having a symlink in the home path (and some software was, too.)

For the new machine, I decided to try bind mounts. Short story: they just work. However, the mtab (and df output) can look a little cluttered, and mount order becomes quite important. To manage the set-up, I wrote a few puppet snippets. First, a convenience definition to make the actual bind-mounts a little less verbose.

define bindmount($device) {
  mount { $name:
    device  => $device,
    ensure  => mounted,
    fstype  => 'none',
    options => 'bind',
    dump    => 0,
    pass    => 2,
    require => File[$device],
  }
}

Once that was in place, we then needed to ensure that the directories to which the LV were to be mounted, and to where the user's home would be bind-mounted, actually exist; we also need to mount the underlying LV and set up the bind mount. The dependency chain is actually a graph, but with the majority of dependencies quite linear:

define bindmounthome() {
  file { ["/home/${name}s", "/home/${name}"]:
    ensure  => directory,
  } -> # depended upon by
  mount { "/home/${name}s":
    device  => "LABEL=${name}",
    ensure  => mounted,
    fstype  => 'ext4',
    options => 'defaults',
    dump    => 0,
    pass    => 2,
  } -> # depended upon by
  bindmount { "/home/${name}":
    device  => "/home/${name}s/${name}",
  }
  file { "/home/${name}s/${name}":
    ensure  => directory,
    owner   => $name,
    group   => $name,
    mode    => 0701, # 0701/drwx-----x
    require => [User[$name], Group[$name], Mount["/home/${name}s"]],
  }
}

That covers the underlying mounts and the "primary" accounts. However, the point of this exercise was to support the secondary accounts for each user. There's a bit of repetition here, and with some refactoring both this and the preceding bindmounthome definition could be a bit shorter, but I'm not sure whether that would be at the expense of legibility:

define seconduser($parent) {
  file { "/home/${name}":
    ensure => directory,
  } -> # depended upon by
  bindmount { "/home/${name}":
    device => "/home/${parent}s/${name}",
  }
  file { "/home/${parent}s/${name}":
    ensure  => directory,
    owner   => $name,
    group   => $name,
    mode    => 0701, # 0701/drwx-----x
    require => [User[$name], Group[$name], Mount["/home/${parent}s"]],
  }
}

I had to re-read the above a couple of times just now to convince myself that I hadn't missed the dependencies between the mount invocations towards the bottom, but they're there: so, puppet will always run the mount for /home/jons before /home/jons/jon. Since puppet is writing to the fstab, this means that the ordering is correct and a sequential start-up will work.

If you want anything cleverer than serialised, one-at-a-time mounting at boot, I think one would have to use something other than trusty-old fstab for the job. I'm planning to look at Systemd's mount unit type, but there's no rush as this particular host is still running sysvinit for the time being.

Tags:

Sun 28 Sep 2014 06:17:59 PM BST

It's been a little while since I've written about what I've been up to. The truth is I've been busy with moving house - and I'll write a bit more about that at another time. But asides from that there have been some bits and bobs.

I use a little tool called archivemail to tidy up old listmail (my policy is to retain 30 days of listmail for most lists). If I unsubscribe to a list, then eventually I end up with an empty mail folder corresponding to that list. I decided it would be nice to extend archivemail to delete mailboxes if, after the archiving has taken place, the mailbox is empty. Doing this properly means adding delete routines to Python's "mailbox" library, which is part of the Python standard library. I've therefore started work on a patch for Python.

Since this is an enhancement, Python would only accept a patch for Python 3. Therefore, eventually, I would also have to port archivemail from Python 2 to 3. "archivemail" is basically abandonware at the moment, and the principal Debian maintainer is MIA. There was a release critical bug filed against it, so I joined the Debian Python team to co-maintain archivemail in Debian. I've worked around the RC bug but a proper fix is still to come.

In other Debian news, I've been mostly quiet. A small patch for squishyball to get it to build on Hurd, and a temporary fix patch for lhasa to get it to build on the build daemons for all architectures (problems with the test suite). All three of lhasa, squishyball and archivemail need a little bit of love to get them into shape before the jessie freeze.

I've had plans to write up some of the more interesting technical things I've been up to at work, but with the huge successes of the School we've been so busy I haven't had time. Hopefully you can soon look forward to some of our further adventures with puppet, including evaluating Shibboleth modules, some stuff about handling user directories, bind mounts and LVM volumes and actually publishing some of our more useful internal modules; I hope we will also (soon) have some useful data to go with our experiments with Linux LXC containers versus KVM-powered virtual machines in some of our use-cases. I've also got a few bits and pieces on Systemd to write up.

Tags:

Older posts are available on the all posts page.