Reducing vagrant box size

Here are some tricks I use to make my vagrant boxes as small as possible:

Tips:

Booting in single user mode:

I boot in single user mode since it will prevent running services that could output logs. I do this because I zero out all my logs before packaging the box.

Updating:

After updating any package, run yum clean (or the apt equivalent).

When booted in single user mode, don’t forget to start-up your network before updating.

When updating kernels, install the kernel packages, reboot and remove the old kernel packages that are no longer in use. Remember to re-install the VirtualBox add-ons too after a kernel update.

Cleanup:

After doing whatever you need to do with the box, I do some rather nasty stuff to make sure the box uses as little as possible place. If you are using a RAW hard-disks, these might be a bad idea (stuff gets BIG).

  • Zero out all remaining unused disk space
  • Zero out the swap
  • Clear out all log files (I just make them empty, I do NOT delete them)
Script:

(You can find this script – or an older version in /root/tools/cleanup_diskspace.sh on my newer vagrant boxes.)

cat - << EOWARNING 
WARNING: This script will fill up your left over disk space.
 
DO NOT RUN THIS WHEN YOUR VIRTUAL HD IS RAW!!!!!
 
You should NOT do this on a running system. 
This is purely for making vagrant boxes damn small. 
 
Press Ctrl+C within the next 10 seconds if you want to abort!! 
 
EOWARNING 
sleep 10; 
 
echo 'Cleanup log files';
find /var/log -type f | while read f; do echo -ne '' &gt; $f; done;
 
echo 'Whiteout root';
count=`df --sync -kP / | tail -n1  | awk -F ' ' '{print $4}'`;
dd if=/dev/zero of=/tmp/whitespace bs=1024 count=$count;
rm /tmp/whitespace;
 
echo 'Whiteout /boot'
count=`df --sync -kP /boot | tail -n1 | awk -F ' ' '{print $4}'`;
dd if=/dev/zero of=/boot/whitespace bs=1024 count=$count;
rm /boot/whitespace;
 
### Repeat the above for other partitions you have.
 
swappart=`cat /proc/swaps | tail -n1 | awk -F ' ' '{print $1}'`
swapoff $swappart;
dd if=/dev/zero of=$swappart
mkswap $swappart;
swapon $swappart;

Furthermore – about this script – USE IT AT YOUR OWN RISK

Puppet modules in Jenkins.

Code style checking

Prerequisites:

  • You will need a recent enough version of puppet-lint that supports the --log-format flag. Install the gem so that the Jenkins can use it.
  • On Jenkins, you will need the Warnings Plugin and the HTML Publisher Plugin.
  • Make sure that when checking the module from your VCS, it ends up in WORKSPACE/modules/module_name.

Configuration:

Jenkins

Go to the Configure System page and find the Compiler Warnings settings. Add a new console log parser and call it puppet-lint. I use following configuration for parsing puppet-lint warnings and errors.

Name:

puppet-lint

Regular Expression:

^\s*([^:]+):([0-9]+):([^:]+):([^:]+):\s*(.*)$

Mapping Script:

import hudson.plugins.warnings.parser.Warning
// map regular expression to strings
String fileName = matcher.group(1);
String lineNumber = matcher.group(2);
String kind = matcher.group(3);
String check = matcher.group(4);
String message = matcher.group(5);
// return a Warning.
return new Warning(fileName, Integer.parseInt(lineNumber), check, kind, message);

Example Log Message:

./manifests/params.pp:25:autoloader_layout:error:apache::params not in autoload module layout

Jenkins job configuration

We will add several build steps that will run certain actions on our puppet modules.

  1. Check syntax
  2. Check style
  3. Generate documentation

1. For the syntax check, I use following shell script (add a build step):

for file in $(find . -iname '*.pp'); do
  puppet parser validate --color false --render-as s --modulepath=modules $file || exit 1;
done;

2. For the style check, we use puppet-lint (add another build step):

find . -iname *.pp -exec puppet-lint --log-format "%{path}:%{linenumber}:%{check}:%{KIND}:%{message}" {} \;

3. And for generating documentation:

## Cleanup old docs.
[ -d doc/ ] &amp;&amp; rm -rf doc/
## Dummy manifests folder.
! [ -d manifests/ ] &amp;&amp; mkdir manifests/
## Generate docs
puppet doc --mode rdoc --manifestdir manifests/ --modulepath ./modules/ --outputdir doc
 
## Fix docs to how I want them, I don't like that the complete workspace is included in all file paths.
if [ -d ${WORKSPACE}/doc/files/${WORKSPACE}/modules ]; then
  mv -v "${WORKSPACE}/doc/files/${WORKSPACE}/modules" "${WORKSPACE}/doc/files/modules"
fi;
grep -l -R ${WORKSPACE} * | while read fname; do sed -i "s@${WORKSPACE}/@/@g" $fname; done;

In your post build section:

  • Enable Scan for compiler warnings and select puppet-lint.
  • Enable publish HTML reports (use ‘doc‘, ‘index.html‘ and ‘Puppet Docs‘ as values). This will add a link to the Job page linking your generated puppet docs.

That’s about it! Any suggestions / improvements on this are always welcome!

Notes:

  • I have some examples/tests setup on my Jenkins instance for testing at http://jenkins.vstone.eu. Since I use this for testing, it might be offline / broken / buggy at times.
  • The scripts I use may also require some changes if you are using an older version of puppet. I’m currently using 2.7.x for testing my modules.

Puppet modules and using dot graphs (both are unrelated but related to each other)

Puppet modules…  How I feel about them in a dot file:

digraph PuppetModules {
  node [
    fontname = "Bitstream Vera Sans"
    fontsize = 10
    shape    = "record"
  ]
  edge [
    fontname = "Bitstream Vera Sans"
    fontsize = 10
  ]
  question [label="Do I need to edit a file in your module for changing settings?", shape="oval"]
  ok  [label="Great.", shape="oval"]
  bah [label="You are doing it WRONG!", shape="oval"]

  question -> ok [label="No"]
  question -> bah [label="Yes"]
}

Read on if you want a rendered version.
(more…)

Vagrant quickstart for Puppet dev(op)s

A quick introduction on how I use vagrant for developing my puppet manifests/modules/. You can almost certain also use this for other purposes. In general, this will get you up to speed fast!
We will quickly go over installation and/or updating and maybe even removing an old version using ruby gems.
Furthermore: adding a vagrant box and preparing a project to develop a puppet module (or something of the likes).
(more…)

Simulating hash tables in bash

On my previous blog (which I did not migrate), I posted this snippet to simulate hash tables in bash. Is it useful? Maybe. In any case, this is an slightly improved version which allows ‘values’ to have spaces in them. Apparently (I’ve never tested it before today), this was kind of a problem.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#!/bin/bash
hash_table=(
    key1='value1'
    key2='value2'
)
 
hash_get() {
    local _table="$1";
    local _key="$2";
    local _value="${_table}[@]";
    local "${!_value}";
    eval echo "\$${_key}";
}
 
hash_get "hash_table" "key2";

Hello world!

My new wordpress installation! I have yet to figure out wether or not to import my old blog entries. I don’t feel much like it for now, but I might change my mind later. For now, it’s going to be empty! (unless I feel like blogging).

Some info on the setup? I’m using wordpress svn with some git-svn magic to keep my code in version control. Using git-svn also solved my biggest hastle I used to have that I couldn’t easily mix a wordpress installation in my document root with existing files and/or folders (error pages, some stats folder, various other thingies).

The wordpress plugins I use are also managed in git, but not with git-svn. I’m unsure if its possible to use git-svn with git-submodules. And even if it is, it may be too much of a command mess to update them all. Instead, I’ll just write a little script I can run whenever wordpress complains about updating plugins that will loop over them and run a svn update (with a git commit with a nicely predefined commit message.

Code will follow!