Chris Kimball of Cook’s Illustrated at the American Museum of Natural History. (CC-Att-NC)

How do you test your Chef cookbooks without firing up a real machine and uploading the recipes to a Chef Server?

I get asked this question all the time, especially after I’ve taught the basics of Chef at Opscode’s public training classes. For unit testing, there is ChefSpec — RSpec plus Chef primitives to allow you to make assertions about your recipes. However, that only goes so far without actually converging a real node & running external tests on the services that were configured.

Last year, Opscode released Test Kitchen, allowing you to use workstation-based virtualization (in the form of VirtualBox) to fire up test nodes, converge them, and run Minitests and Cucumber behavioural-driven development (BDD) tests on them after the converge. In this article, I’ll show you how to set up Test Kitchen 1.0 with Vagrant 1.1 to write and run integration tests.

Before we begin, I have to warn you that Test Kitchen 1.0.0 is a major rewrite and still under heavy development — it is currently in alpha, so there may will still be bugs. In particular, the directions in the README on the Test Kitchen GitHub refer to the use of Bundler, and this is no longer required or recommended.

One of the major new features in Test Kitchen is to abstract out the drivers for different virtualization providers from the actual Test Kitchen framework. This allows you to run your Kitchen tests on any virtualization provider: you could use Kitchen to converge on a VirtualBox machine when you’re on an airplane, but when you’re in the office, you could converge on an Amazon EC2 instance. For this demonstration, though, we’re going to continue to use VirtualBox for simplicity.

I’m going to assume you have both Chef and VirtualBox installed on your workstation. If you’ve used Opscode’s recommended Omnibus installer, Chef will be installed in /opt/chef (or C:\chef if on Windows) so those are the paths I’m going to refer to.

Vagrant Installation

Vagrant is a framework for virtualization providers that allows you to rebuild environments from scratch, using a base image and applying the same provisioning logic you would normally run in your production environment to manage your infrastructure. For Puppet, that might mean your Puppet manifests; for Chef, it means your Chef cookbooks. There’s more information about Vagrant at vagrantup.com, but suffice it to say that we’ll be using it as the abstraction layer to VirtualBox, so go ahead and download & install that.

Berkshelf Installation, Part 1

Berkshelf is a cookbook artifact manager, which obviates you from having to store “upstream” or community cookbooks on your local machine, or even in your own source control management system. Test Kitchen uses it to satisfy the dependencies for the cookbook under test, so we’re going to install that as well.

There are two parts to the installation. The first part is to install the Berkshelf plugin for Vagrant, so that it knows to call out to Berkshelf when running the provisioning phase. So go ahead and do that:

$ vagrant plugin install berkshelf-vagrant --plugin-version 1.0.6
Installing the 'berkshelf-vagrant' plugin. This can take a few minutes...
Installed the plugin 'berkshelf-vagrant (1.0.6)'!

Note: berkshelf-vagrant 1.1.0 won’t work until this patch gets merged to the Kitchen Vagrant driver. (Remember what I said about alpha software?)

We’ll install the other part of Berkshelf after we’ve installed Test Kitchen.

Test Kitchen Installation

Let’s install the pre-release version of Test Kitchen into Chef’s embedded Ruby:

$ sudo /opt/chef/embedded/bin/gem install test-kitchen --pre
Successfully installed test-kitchen-1.0.0.alpha.4
1 gem installed
Installing ri documentation for test-kitchen-1.0.0.alpha.4...
Installing RDoc documentation for test-kitchen-1.0.0.alpha.4...

We also need a virtualization driver for Test Kitchen; by default, no drivers are installed. Let’s install the Vagrant driver, since that’s what we’re going to use:

$ sudo /opt/chef/embedded/bin/gem install kitchen-vagrant
Succesfully installed kitchen-vagrant-0.7.4
1 gem installed
Installing ri documentation for kitchen-vagrant-0.7.4...
Installing RDoc documentation for kitchen-vagrant-0.7.4...

Berkshelf Installation, Part 2

We also need to make Test Kitchen aware that we’re using Berkshelf for cookbook artifact management, so let’s do that:

$ sudo /opt/chef/embedded/bin/gem install berkshelf
Successfully installed berkshelf-1.3.1
1 gem installed
Installing ri documentation for berkshelf-1.3.1...
Installing RDoc documentation for berkshelf-1.3.1...

That’s it! Now we’re ready to roll.

Running Test Kitchen Tests

Let’s not go into how to write tests for Test Kitchen just yet: let’s just get cooking using some pre-written Chef minitests in the Opscode bluepill cookbook. Let’s clone that from GitHub and do some testing.

$ git clone https://github.com/opscode-cookbooks/bluepill.git

The .kitchen.yml file describes all the platforms we’re interested in testing, how to set them up to serve as a sensible test fixture for our tests, and finally, what the test suite consists of. In this situation, the test suite just sets up a couple of required attributes, and adds a bluepill_test cookbook to the test harness. You’ll see the code for that cookbook under test/cookbooks/bluepill_test. The suite also adds the minitest-handler cookbook to the run list, which is how the minitests (located in that test cookbook under files/default/tests/minitest) will actually get run.

Ok, so let’s start up our kitchen:

$ /opt/chef/embedded/bin/kitchen test
-----> Starting Kitchen
-----> Cleaning up any prior instances of 
-----> Destroying 
       Finished destroying  (0m0.00s).
-----> Testing 
-----> Creating 
       [kitchen::driver::vagrant command] BEGIN (vagrant up --no-provision)
       Bringing machine 'default' up with 'virtualbox' provider...
       [default] Importing base box 'canonical-ubuntu-12.04'...

I’ve elided the rest of the output as it’s very long, but basically what Kitchen will do is to start up and try to converge a VM for every platform you’ve defined in the .kitchen.yml, and throw an exception if it fails on any one. Neat, huh?

How Do I Write My Test Cases?

The Chef Minitest framework provides you a lot of assertions out of the box; look at the documentation on GitHub for some examples of how to write regular Minitest-style testcases, or spec-style testcases. There are also custom assertions bundled with Minitest.

As with many open-source projects, the README can be a bit lacking with respect to the features currently in Minitest Chef. The default_test.rb in the Minitest Chef Handler project shows a very comprehensive list of tests that can be made.

Wrap-Up

I’ve shown you how to integration test your cookbooks using Opscode Test Kitchen 1.0 Alpha, Berkshelf and Vagrant with VirtualBox as the underlying virtualization technology. Obviously, this is a field under heavy development, and some features could change between now and the actual release of Test Kitchen 1.0. But hopefully it’s enough to get you started.

Additional Resources

Joshua Timberman has a great two-part series on his blog, delving into more detail about Test Kitchen 1.0.

Vagrant 1.1 represents a major rewrite. While the vagrant command-line tool remains the same, the package format changes significantly. Vagrant 1.1.x is only being distributed as an “omnibus” package, which bundles Ruby and all the Rubygems that Vagrant needs. (In another post, we can argue about whether the trend towards omnibus Ruby installers is a good one or not.) Vagrant 1.1 also introduces a new plugin architecture, and developers hoping to add functionality to Vagrant must rewrite their plugins to be conformant with the new API. Some of the developers who have already rewritten their plugins include Jamie Winsor, the author of Berkshelf, and Cassiano Leal, the author of vagrant-butcher (to clean up test node and client objects from Chef when using a Chef Server provisioner).

Veewee, however, presents a special challenge. Veewee depends on Vagrant being available as a Rubygem, a format that Mitchell is no longer distributing officially. To solve this, we need to use Bundler to create a Gem directly from Git, and manage some of Veewee’s other bleeding-edge dependencies.

If you haven’t guessed already, this is a set of technologies that is in enormous flux, and my instructions here merely provide for a workaround until a more elegant solution comes about (if it does). That said, let’s write ourselves a Gemfile that’ll let us successfully install and run Veewee, based on Patrick’s own Gemfile:

source "https://rubygems.org"

gem "veewee", :git => 'https://github.com/jedi4ever/veewee.git', :ref => '333ffcaba291a3cca9a54ebb9bf48ef534d60cfd'

group :windows do
  gem "em-winrm", :git => 'https://github.com/hh/em-winrm.git', :ref => '31745601d3'
  gem "log4r"
end

group :test do
  gem "rake"
  gem "vagrant", :git => 'https://github.com/mitchellh/vagrant.git', :tag => 'v1.1.5'
  #gem "chef"
  #gem "knife-windows"
end

I’m not sure if all the groups are necessary, but I’m not familiar enough with Bundler to know what they’re used for.)

You’ll notice a couple key things here. First, I’m tying veewee itself to a particular commit hash – this points to HEAD at the time of writing, but since the project is under heavy development, there will be more commits after this. The reason I’m pointing to GitHub for veewee itself is because the version on Rubygems.org, 0.3.7, is quite old and a new one hasn’t been released for a while.

Second, in group :test, the Vagrant gem points to Mitchell’s GitHub repository with a particular tag attached to it — this represents the source code of the currently-released version, 1.1.5.

Now that we have a Gemfile set up, we can run bundle install to install all these Gems on our system, including the bleeding-edge ones (which won’t show up when you type gem list, but are nonetheless on your system).

To test, let’s try to build a CentOS 6.4 box. Veewee’s convention is that box definitions live in a definitions directory, so let’s make a directory called definitions/CentOS-6.4-x86_64-minimal and copy the files out of Patrick’s master repo into there.

Now, we can try to build the box with the command

bundle exec veewee vbox build CentOS-6.4-x86_64-minimal

(assuming we’re going to use VirtualBox to build it). If this works, you should get a lot of output as veewee brings up a new VirtualBox machine, automates the operating system installation, and installs Chef, Puppet, and makes a few other system customizations to make it conform to the Vagrant box interface. Once done, you can type

bundle exec veewee vbox validate CentOS-6.4-x86_64-minimal

to run some basic tests against that box to ensure it meets that interface.

Finally, once we’re satisfied, we can then export this box using

bundle exec veewee vbox export CentOS-6.4-x86_64-minimal

Voilà, we have a box file we can use with VirtualBox and Vagrant in the usual fashion.

To summarize: the current interactions between Vagrant 1.1 and some older plugins are still shaky, but developers are working hard to adapt to the new plugin architecture. (Another example where heavy development is occurring is on the vagrant-windows plugin, which is still not compatible with Vagrant 1.1.) Until these problems are worked out, workaround procedures like the above, for veewee, are necessary.

Later in the week, I’ll talk about how to get a local Chef cookbook development environment set up using Vagrant 1.1 and Berkshelf. Stay tuned.

The Jenkins Amazon EC2 Plugin

We decided to implement the Jenkins Amazon EC2 plugin on the master to spin up new slaves when there are jobs waiting. After a certain configurable period of inactivity, the EC2 plugin will suspend or terminate the instances. The main problem we had to solve is that SecondMarket’s build process & test suite require the presence of many packages, representing the key parts of our stack: a JDK, PostgreSQL, MongoDB, etc. We’ve spent a lot of time automating the provisioning process for these services using Chef. But we don’t want to bootstrap all these transient nodes off our Chef server and have lots of dead client and node objects lying around. How can we leverage the work we’ve done with Chef in a world where a “golden image” AMI is required?

Instance Metadata to the Rescue

Amazon EC2 allows you to pass in “instance metadata” (or just “instance data”) on bootup, so does the Jenkins plugin. By convention, instance data is a shell script run once by the target system after the machine is up. (Of course, the target system’s AMI must include some way to run it: most people have cloud-init in EC2 AMIs for this reason.) Initially, I’d hoped to call Chef Solo directly in the instance data script, so that I could just use a bare, generic CentOS 6 AMI in Jenkins. This proved to be infeasible due to the provisioning time. In addition to the startup time for the instance (about five minutes), the first run of Chef Solo took another five to seven minutes, so it would be almost fifteen minutes before a new slave could come online.

Instead, I agreed on a compromise with our build master: we’d use Chef Solo to build the golden image and automate that process. If changes are required to the AMI, we could commit them to our Chef repository, create a new Chef Solo bundle, and rebuild a new AMI. (Down the road SecondMarket might even consider having Jenkins perform the whole process, which would be a strange Malkovichian development to this whole scheme.)

The Code

First we need a script to generate the Chef Solo tarball. I wrote a very simple one to check out the master branch of every relevant cookbook and create a tarball of it, which could be put on our internal repository server, accessible by HTTP.

Then, we pass an instance data script to the EC2 bootstrap that installs Chef, sets up a dummy role called jenkins-node-solo-role, and runs Chef Solo to provision the machine. This is done using the Amazon EC2 tools:

% ec2-run-instances -f instance-data.sh -g whatever -k some-key -t m1.small ami-XXXXXXXX

Once satisfied, we can snapshot the instance and create an AMI from it:

% ec2-create-image -n "jenkins-node-image-20130226" i-XXXXXXXX

Now just configure Jenkins to use the new AMI ID as the slave image and we’re done. (Don’t forget to kill off the temporary instance.)

Conclusions and Future Work

I learned a couple of lessons from doing this, chief amongst them being that if your role’s JSON doesn’t have "json_class": "Chef::Role" you get some truly bizarre error messages. That took me a half day to track down.

Ideally I’d love to have some BDD tests that would describe the target state of the slave machine (e.g. “must have MongoDB installed”) that could be run right after the Chef Solo run. I spent perhaps ten iterations of building AMIs using this process only for the build master to complain that something was missing on the image. It took a lot of tweaking to get the run list right.

I’m extremely excited to be working for a company whose product has been revolutionizing the job responsibilities of the traditional system administrator, and even those of the software engineer. It’s easier to break down the walls between operations and development when all your infrastructure is code, and Chef makes that a no-brainer. Frankly, it’s also more fun for everyone — yes, it’s possible for web operations to be fun again, just like it was back in 1996 when I got into this sort of thing.

I’m looking forward to working with all of the really smart people at Opscode, and, if you’re part of the Chef community, with you as well. See you around, maybe at a conference, training session, or just in IRC!

I recently started to adopt Bryan Berry’s application & library cookbook model as outlined in his excellent and funny blog post, "How to Write Reusable Chef Cookbooks, Gangnam Style". But I quickly ran into a blocker, because people are trying to solve problems using the compile phase and not the execute phase of Chef. Perhaps this calls into question the entire viability of compile-phase providers like chef_gem.

Let’s recap Bryan’s post quickly. He argues for vastly simplifying the number and complexity of bespoke cookbooks that sysadmins have to maintain by leveraging existing, high-quality community cookbooks as “libraries”. (The most common source of these library cookbooks is, of course, the Opscode Cookbooks GitHub repo.) Then, by creating a small bespoke cookbook (the “application” cookbook) that overrides certain attributes & behaviors, you can achieve the desired customizations for your own environment. As a trivial example, suppose you had a popular web application called “instachef” that just got bought by MyFace, and you need to handle more traffic in the Apache VirtualHost that’s fronting it. Well, you might create an instachef::server recipe that does nothing more than

node.set['apache']['prefork']['maxclients'] = 31337
include_recipe "apache2"

thereby overriding the default attributes in the apache2 cookbook but re-using all the logic that Opscode and other community members have put in there. Seems logical, right?

I wanted to do the same thing with the PostgreSQL cookbook: that is, I wanted to wrap the community’s with one that sets up Yum repos from the PostgreSQL Global Development Group (PGDG) and then installs PostgreSQL 9.2 instead of PostgreSQL 8.x on CentOS servers. The PGDG recipe looks something like this:

and then the run_list in my Berkshelf-managed Vagrant VM is just “recipe[smpostgresql::pgdb], recipe[postgresql::server]“. This works great: I get PostgreSQL 9.2 installed on the VM. So far, so good.

Things break down, however, when I want to use the database and database_user LWRPs to manage a set of databases and users. In order to use the LWRPs which run in the execute phase, I need to have the “pg” Rubygem installed in the compile phase. But the “pg” Gem has native extensions which must be compiled against the headers of the PostgreSQL I want, and I can’t retrieve those for PostgreSQL 9.2 until the execute phase, when the pgdg recipe sets up a Yum repo for me to retrieve them from! Argh, chicken and egg problem.

I think the root cause of this problem is that people are abusing the compile-phase of the Chef run to do things that normally would be done in the execute phase. Just look at the source code of postgresql::ruby: it’s almost like an entire recipe, forcibly run in the compile phase. Whenever I see code that breaks the boundaries between execute and compile, I think something’s seriously wrong.

I don’t know the internals of Chef and I’m no Ruby expert, so I don’t know how viable my solution is. But conceptually, this would all be solved if chef_gem was an execute-time resource only. Doing so would mean that other recipes that actually belong in the execute phase — like downloading postgresql92-devel and a C compiler — could be done ahead of installing the gem. It’s almost like we need lazy evaluation of the require "pg" call, until the execute phase, at which point the LoadError could be rescued, the Gem installation could proceed, and the require retried.

I’m interested to know what other Chef practitioners think. In the meantime, I’m working around the issue by simply avoiding using the LWRPs.

I understand that sometimes you want to comment on someone’s opinions by simply showing their own words. In that situation, why not add your voice to the conversation by prepending a couple words to the retweet, or condensing/summarizing the original tweet using a MT (Modified Tweet)? People follow you on Twitter because they want to know your perspective on things. You shouldn’t just be a mouthpiece for other people.

In short: If you don’t completely agree with or endorse an opinion on Twitter, don’t just blindly retweet it and think that “RT ≠ endorsement” will cover you. Add some color to the retweet to clarify where you stand. Not only will you show that you’re not afraid to have an opinion, but your followers will thank you for continuing the conversation. Isn’t that what Twitter is about, after all?

Image from holeymoon on Flickr. CC-licensed.

I’m pleased to announce that the slides from the presentation are now available, and the video (eek!) will be shortly.

Hope everyone’s having a great holiday and I promise there will be more blog posts in the New Year.

The products themselves, though, can be a nightmare to install, despite the fact that they are mostly just Java web applications living in a WAR file. The products have improved immensely from the days when setting them up involved hacking up a multitude of XML files in WEB-INF (though there still is some of that), and it’s still annoying that Atlassian doesn’t support running the applications as unexploded WARs within Tomcat or another servlet container, probably for the aforementioned reasons. All that aside, though, it’s satisfying when everything is working together and users can single-sign-onto the entire Atlassian suite because of the magic of Crowd, Atlassian’s SSO directory server.

Last week, I released a set of Chef cookbooks I wrote at SecondMarket to ease the installation of the Atlassian tools on a server. I’m still looking to automate more parts of this, including the ability to edit the aforementioned XML files in-place in an idempotent way, so pull requests against our GitHub repo would be welcome.

Special Note on Using Atlassian Products in the Amazon Cloud

I should also mention that my first attempt to set up Atlassian’s products using Amazon Relational Database Service (RDS) as a backing store was a failure. To spare you the pain of finding this out yourself, I’ll just mention the reason: Crowd, JIRA and Confluence expect MySQL to be configured with READ-COMMITTED transaction isolation level, which means you need to configure MySQL to have row-based binary-logging. Unfortunately, binlog_format is not a parameter you can configure in RDS’s DB Parameter Groups, for obvious reasons; it would affect all other clients on that MySQL instance. This has been confirmed with Amazon support, so JIRA/Crowd/Confluence with RDS is a no-go.