Which command do I use the most?

After waking to a ringing phone, I was attempting to fall back asleep. As I drifted, I wondered which command I used the most? I thought it a neat exercise, so I found out.

As I type this, I suspect it is “ls”. Which is second and third should be interesting.

I gathered up .zsh_history and .bash_history files from 5 different machines I work on a fair bit and put them in a pile, erm, directory.

 $ cat bash_history? > big_bash_history
 $ wc -l big_bash_history              
 2897 big_bash_history
 $ sed -e 's/^.*;//g' zsh_history? >big_zsh_history
 $ wc -l big_zsh_history
 31188 big_zsh_history

Wow! 35 thousand commands. Wait… There’s a snafu in there. SUDO. Hmm….

$ awk '{print $1}' big_zsh_history|grep sudo |wc -l
$ awk '{print $1}' big_bash_history|grep sudo |wc -l

Okay. I’ve got some numbers around that…Let’s see about stripping those and get the underlying command.

$ sed -r -e '/sudo -i/d' -e 's/sudo //g' big_bash_history big_zsh_history|awk '{print $1}'|while read CMD;do basename "$CMD"; done |sort |uniq -c |sort -rn |head -20
   4571 cd
   3845 ls
   2884 vi
   2436 git
   1719 ssh
   1195 cat
   1032 grep
    928 rm
    797 ansible-playbook
    659 mv
    619 pass
    619 cp
    443 python
    438 scp
    394 apt-get
    373 salt
    372 mkdir
    339 more
    297 find
    291 done

Okay. No surprises in the top five. ‘ansible-playbook’ surprised me. Didn’t realize I entered that so much. There were more points of interest after the first twenty, but I won’t post those.

Interesting ‘done’ showed up in the top twenty. Thinking about this, it makes sense, as the first command is usually input to while, ie:

 ls | while read FILE; do echo $FILE; done

No ‘if’ or ‘fi’ in the top twenty. How about that.

Actually, thinking about this more, there are probably many more missing commands that were piped into another or, say ‘rm’ used in an ‘exec’ for ‘find’.

VMs in my home lab

TL;DR: Read the scripts here.  There is a simple “if; this; then; that” at the bottom.

It’s often I see some new bit of software I’d like to deploy in my home lab to test against my infrastructure. I’m not super keen on loading it on a machine I use for some other function, especially if I end up tossing the software.

With this in mind, I started using Linux’s Kernel Virtual Machines (KVM) and Quick Emulator (Qmeu) to test things out.  This worked quite well, with one exception: Spin up time was too long. WAY too long waiting 20 mins for a new VM wasn’t acceptable for me.  So I started digging around.

Caveat emptor: I am running dhcpd, kvm/qemu, and all the rest on one massive box.  This stuff CAN be split out into multiple hosts, but will require a bit more work to pull it together. I know you can do it. You are a smart cookie, after all…

I found that cloning VMs was possible and actually, pretty straight forward. I was just a matter of copying the image file, mounting it on a loopback device, editing a few OS specific files and importing a new libvirt host definition from a modified XML file.

Obviously, doing this over and over meant putting in a Bash script new-vm.sh, which can be found: https://github.com/sawasy/build_vm Of course, there is also a teardown script for VMs called destroy-vm.sh.

There are a few extra bits or gotchas in this script, which you’ll want to pay attention to.  It checks dhcpd for an existing MAC definition and uses that for the hostname provided, if available.  If dhcpd is going be on a different host, you’ll probably want to pull that over or check it remotely.

For a long while this worked pretty well.  Test hosts could now be rebuilt very quickly, sub 5 mins.

$ time new-vm.sh testhost
Domain golden-host is being shutdown

error: failed to get domain 'testhost'
error: Domain not found: no domain with matching name 'testhost'

error: failed to get domain 'testhost'
error: Domain not found: no domain with matching name 'testhost'

Copying image...
sending incremental file list
3,130,195,968 100% 43.68MB/s 0:01:08 (xfr#1, to-chk=0/1)
Dumping XML...
Domain golden-host started

Checking for dhcp definition
Clean up stale mounted....
umount: /mnt/loopback: not mounted
Fixing hostname
/dev/nbd0 disconnected
Importing testhost's definition
Domain testhost defined from /tmp/testhost.xml

Starting testhost
Domain testhost started

new-vm.sh testhost 15.24s user 4.98s system 21% cpu 1:33.50 total

As you can see in the example above, there is a golden host that runs.  This golden host is just that, the base host you want to load all your custom bits and bobs on to and then clone from with each new VM.

This worked pretty well for about a year. The disk on the golden-host became corrupted and I had to rebuild the golden host from scratch.  This worked well for a long while again, until the host wouldn’t boot. At this point, I decided to spend some time automating the golden-image build process.  Having worked with Packer for AWS AMIs, I used it for this.  I cooked up some JSON to do the Packer things I needed it to in the above github repo ubuntu-14.04-server-amd64.json. If you open the JSON file, you’ll see I have the ubuntu ISO on a internal host. This is the same host I run packer from.  If you look in the build.sh script, you’ll see it reaches out and checks that we have the freshest ISO available. If not, it downloads it and puts it where the webserver can access it.

If we plan to use a separate host for serving the various pieces for this, you can just dump it in a cron job. I digress…

The build.sh script sets up the ISO and preseed.txt file, cleans up old build, opens up the UFW firewall (because you are using firewalls, right? RIGHT?) and then kicks off the build.

Side note: Those with a keen eye will notice some scary passwords, sshd settings and such. Those are all fixed up by my ansible playbooks post install. They are there for bootstrapping.

Once that is done, Jenkins will run the post-build.sh script.  Sorry? What do I mean Jenkins? Shit. Right. I have a Jenkins install managing the automated Recurring build of all this goodness.  I’ve set up Jenkins to pull down a fresh copy of my Ansible repo (soon Saltstack because it is so bad ass), and handle the build and post build stuff. Also send me a little mail when it’s done. The post install script, really, just sets up a bunch of the environment variables for Jenkins.

And, that’s about it.

In a nutshell: Every morning at 3AM, Jenkins kicks off. -> Jenkins runs build.sh. -> Build.sh sets up the ISO and preseed.txt in the webserver -> Jenkins runs Packer. -> Packer creates a VM and preseeds it. -> Jenkins runs the post-install.sh. -> post-install.sh cleans up, if needed, and kicks off Ansible and sends and email out when done!