VMs in my home lab

TL;DR: Read the scripts here.  There is a simple “if; this; then; that” at the bottom.

It’s often I see some new bit of software I’d like to deploy in my home lab to test against my infrastructure. I’m not super keen on loading it on a machine I use for some other function, especially if I end up tossing the software.

With this in mind, I started using Linux’s Kernel Virtual Machines (KVM) and Quick Emulator (Qmeu) to test things out.  This worked quite well, with one exception: Spin up time was too long. WAY too long waiting 20 mins for a new VM wasn’t acceptable for me.  So I started digging around.

Caveat emptor: I am running dhcpd, kvm/qemu, and all the rest on one massive box.  This stuff CAN be split out into multiple hosts, but will require a bit more work to pull it together. I know you can do it. You are a smart cookie, after all…

I found that cloning VMs was possible and actually, pretty straight forward. I was just a matter of copying the image file, mounting it on a loopback device, editing a few OS specific files and importing a new libvirt host definition from a modified XML file.

Obviously, doing this over and over meant putting in a Bash script new-vm.sh, which can be found: https://github.com/sawasy/build_vm Of course, there is also a teardown script for VMs called destroy-vm.sh.

There are a few extra bits or gotchas in this script, which you’ll want to pay attention to.  It checks dhcpd for an existing MAC definition and uses that for the hostname provided, if available.  If dhcpd is going be on a different host, you’ll probably want to pull that over or check it remotely.

For a long while this worked pretty well.  Test hosts could now be rebuilt very quickly, sub 5 mins.

$ time new-vm.sh testhost
Domain golden-host is being shutdown

error: failed to get domain 'testhost'
error: Domain not found: no domain with matching name 'testhost'

error: failed to get domain 'testhost'
error: Domain not found: no domain with matching name 'testhost'

Copying image...
sending incremental file list
3,130,195,968 100% 43.68MB/s 0:01:08 (xfr#1, to-chk=0/1)
Dumping XML...
Domain golden-host started

Checking for dhcp definition
Clean up stale mounted....
umount: /mnt/loopback: not mounted
Fixing hostname
/dev/nbd0 disconnected
Importing testhost's definition
Domain testhost defined from /tmp/testhost.xml

Starting testhost
Domain testhost started

new-vm.sh testhost 15.24s user 4.98s system 21% cpu 1:33.50 total

As you can see in the example above, there is a golden host that runs.  This golden host is just that, the base host you want to load all your custom bits and bobs on to and then clone from with each new VM.

This worked pretty well for about a year. The disk on the golden-host became corrupted and I had to rebuild the golden host from scratch.  This worked well for a long while again, until the host wouldn’t boot. At this point, I decided to spend some time automating the golden-image build process.  Having worked with Packer for AWS AMIs, I used it for this.  I cooked up some JSON to do the Packer things I needed it to in the above github repo ubuntu-14.04-server-amd64.json. If you open the JSON file, you’ll see I have the ubuntu ISO on a internal host. This is the same host I run packer from.  If you look in the build.sh script, you’ll see it reaches out and checks that we have the freshest ISO available. If not, it downloads it and puts it where the webserver can access it.

If we plan to use a separate host for serving the various pieces for this, you can just dump it in a cron job. I digress…

The build.sh script sets up the ISO and preseed.txt file, cleans up old build, opens up the UFW firewall (because you are using firewalls, right? RIGHT?) and then kicks off the build.

Side note: Those with a keen eye will notice some scary passwords, sshd settings and such. Those are all fixed up by my ansible playbooks post install. They are there for bootstrapping.

Once that is done, Jenkins will run the post-build.sh script.  Sorry? What do I mean Jenkins? Shit. Right. I have a Jenkins install managing the automated Recurring build of all this goodness.  I’ve set up Jenkins to pull down a fresh copy of my Ansible repo (soon Saltstack because it is so bad ass), and handle the build and post build stuff. Also send me a little mail when it’s done. The post install script, really, just sets up a bunch of the environment variables for Jenkins.

And, that’s about it.

In a nutshell: Every morning at 3AM, Jenkins kicks off. -> Jenkins runs build.sh. -> Build.sh sets up the ISO and preseed.txt in the webserver -> Jenkins runs Packer. -> Packer creates a VM and preseeds it. -> Jenkins runs the post-install.sh. -> post-install.sh cleans up, if needed, and kicks off Ansible and sends and email out when done!


Cloud all the things!

Recently, I found that Amazon is offering their Cloud Drive service, with unlimited storage, for $60/year.  As someone who has had more than one catastrophic disk crash on my personal data stores, I was very interested in this service.  They are offering a 3 month trial so, I jumped to sign-up.

And was immediately very confused to find I was only offered 5 GB of photo space. After puzzling for some time, I realized I was not allowed this service because my Canadian Amazon site was linked to my US Amazon site. Lame.

Fast forward a week, I was relaying my story to a friend who said “Just sign up again”. I said I need a new email address like I need a new hole in the head.  He mentioned that Gmail has a neat feature. After the local part, if you can add a plus “+” symbol with anything you like, it gets dropped by Google and delivered to you. An example will make this simpler.

ie – matt@gmail.com would become matt+syslo.gd@gmail.com would get delivered to matt@gmail.com

Sold. I signed up for a new Amazon account and started mucking with their service. The GUI sucks. I mean, I really don’t want to upload things by clicking on them. They have some app for Windows, but that isn’t super useful for me.

I found a few projects for Linux which interface with ACD before landing on rclone.  The big sale point for me was the bandwidth throttling, which I’ll get to in minute.

I started shovelling things into the service to test it out. It is consuming like a champ. I found my home broadband provider, or their upstream, is doing some poor QOS so, my Upload/Download to AWS is shit.  I found one of my VPS had WAY better speeds, so I futzed around with that as a downloading station. Immediately, I tripped over saturating the virtual network driver caused hard-locking of the machine. It required a reboot in the console to recover. I found throttling total transfers below 10MB/s seemed to do the trick, hence the rclone feature.

A couple of package installs and some trivial shell scripts and I  had a tidy little pipe line to pull down data and shovel it into the Cloud Drive. Side-note: If there is interest, I can share the scripts.  Perhaps I will clean them up and publish them here.

So far, I’ve crammed about 420 GB into the storage and it’s behaving nicely.  There are a couple gotchas:

  1. you can’t play any video longer than 20 minutes. I think this is so people don’t upload their movie collection and play it from there.
  2. There is a 40 GB file limit. I believe this is so people don’t do DB dumps and stuff them in here, or something like that.

It’s nice piece of mind knowing some old television shows, movies and what not have a permanent place to live without the fear of hard drive crash.  It’s also nice to know, that when I leave this mortal coil, my credit cards will get terminated and this account a year later.