#339 Chef Solo Basics pro

In episode 335 we showed how to deploy a Rails application to a Virtual Private Server. Doing this required a large number of commands to be run on the server to get it all set up and if an application runs on multiple servers this is something that we don’t want to have to do manually every single time. In episode 337 we automated the server setup with Capistrano. This works quite well but it stretches the limits of what Capistrano is designed to do as it isn’t really a server provisioning tool. This brings us to Chef. Chef is a complicated framework and we won’t be able to do much more than scratch the surface of it in this episode but hopefully we’ll show you enough of the basics so that you can explore Chef on your own afterwards if it seems to fit your needs. Chef helps us to automate the setup and management of our servers and when we use it one of the first decisions we need to make is whether to go with Chef Server or Chef Solo. Chef Server runs on its own server and other servers can communicate with it to determine how they should be set up. By contrast Chef Solo runs on the server that it’s setting up and as we’re working with a single VPS we’ll be using that here.

Installing Chef on a Server

We’ll be using Chef on a Linode VPS and we’ll start by rebuilding the server so that we’ve got clean install to work with. Once the server has finished rebuilding we’ll boot it up and SSH in to it.

$ ssh root@178.xxx.xxx.xxx

Chef uses Ruby which isn’t installed on our server by default. We’ll install it from source using the commands listed here. Note that the last command shown installs two gems to get Chef installed on our system.

#!/usr/bin/env bash
apt-get -y update
apt-get -y install build-essential zlib1g-dev libssl-dev libreadline5-dev libyaml-dev
cd /tmp
wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p125.tar.gz
tar -xvzf ruby-1.9.3-p125.tar.gz
cd ruby-1.9.3-p125/
./configure --prefix=/usr/local
make
make install
gem install chef ruby-shadow --no-ri --no-rdoc

If you prefer to install Ruby through rvm or rbenv you can do so with Chef afterwards through a recipe; here we’re installing an initial version of Ruby to get Chef up and running. We can use the Gist shown above to run these commands in one go, like this:

root@li349-144:~# curl -L https://gist.github.com/raw/2307959/ff2d251c9f4f149c5ca73c873ad8990711b3ca74/chef_solo_bootstrap.sh | bash

This script will take a while to run but once it’s finished we should have both Ruby and Chef Solo installed.

root@li349-144:~# ruby -v
ruby 1.9.3p125 (2012-02-16 revision 34643) [i686-linux]
root@li349-144:~# chef-solo -v
Chef: 0.10.8

Creating Our First Chef Recipe

Now we can set Chef up. Chef Solo expects all its configuration files and cookbooks to be on the same system that it’s running on. We’ll create a chef directory under /var where we can put all our Chef-related files then move into that directory.

root@li349-144:~# mkdir /var/chef
root@li349-144:~# cd /var/chef

In this directory we’ll make our first Chef cookbook under a new cookbooks directory. Normally we’d make a separate cookbook for each thing we want to install and set up on the server, such as Nginx, MySql or Git, but as we’re working with Chef Solo on a single VPS we’ll create a cookbook that we’ll call main which will set up everything we need. Each cookbook has many recipes and these go in a recipes subdirectory.

root@li349-144:/var/chef# mkdir -p cookbooks/main/recipes

Chef will look inside the cookbooks directory for a recipe called default.rb and this will be the primary recipe. Other recipes will usually be variations of this. Inside this recipe we have access to methods that will help us set up our system. For example if we want to install Git on our server we could use package like this:

package "git-core"

This method will use the server’s package manager, in this case apt-get, to install git-core. We also need to set up a configuration file for Chef Solo so that we can tell it where are cookbooks are stored. We’ll do this in a file called solo.rb.

cookbook_path File.expand_path("../cookbooks", __FILE__)

We can now run the chef-solo command and use this file as its config file.

root@li349-144:/var/chef# chef-solo -c solo.rb 
[Thu, 12 Apr 2012 17:40:55 -0400] INFO: *** Chef 0.10.8 ***
[Thu, 12 Apr 2012 17:40:56 -0400] INFO: Run List is []
[Thu, 12 Apr 2012 17:40:56 -0400] INFO: Run List expands to []
[Thu, 12 Apr 2012 17:40:56 -0400] INFO: Starting Chef Run for li349-144.members.linode.com
[Thu, 12 Apr 2012 17:40:56 -0400] INFO: Running start handlers
[Thu, 12 Apr 2012 17:40:56 -0400] INFO: Start handlers complete.
[Thu, 12 Apr 2012 17:40:56 -0400] INFO: Chef Run complete in 0.00146873 seconds
[Thu, 12 Apr 2012 17:40:56 -0400] INFO: Running report handlers
[Thu, 12 Apr 2012 17:40:56 -0400] INFO: Report handlers complete

Even though Chef ran successfully here it didn’t really do anything. This is because its Run List is empty and we can see this in the output. Even though we’ve created a recipe we haven’t told Chef to run it. We can create as many recipes as we like but Chef will only run the ones we tell it to. We need to make a Run List configuration and with Chef Solo this is normally done in a JSON file in a file called node.json. Don’t confuse this with Node.js, this is completely unrelated. “Node” is a term that Chef uses to represent a server that it needs to set up. Inside this file we need to specify a run_list option and tell it to run our main recipe.

{
  "run_list":["recipe[main]"]
}

We can run chef-solo again now and use its -j option to tell it to use our new node.js file. This time we’ll can see our recipe in the Run List and also that the git-core package is installed.

root@li349-144:/var/chef# chef-solo -c solo.rb -j node.json 
[Thu, 12 Apr 2012 17:54:40 -0400] INFO: *** Chef 0.10.8 ***
[Thu, 12 Apr 2012 17:54:41 -0400] INFO: Setting the run_list to ["recipe[main]"] from JSON
[Thu, 12 Apr 2012 17:54:41 -0400] INFO: Run List is [recipe[main]]
[Thu, 12 Apr 2012 17:54:41 -0400] INFO: Run List expands to [main]
[Thu, 12 Apr 2012 17:54:41 -0400] INFO: Starting Chef Run for li349-144.members.linode.com
[Thu, 12 Apr 2012 17:54:41 -0400] INFO: Running start handlers
[Thu, 12 Apr 2012 17:54:41 -0400] INFO: Start handlers complete.
[Thu, 12 Apr 2012 17:54:41 -0400] INFO: Processing package[git-core] action install (main::default line 1)
[Thu, 12 Apr 2012 17:54:45 -0400] INFO: package[git-core] installed version 1:1.7.0.4-1ubuntu0.2
[Thu, 12 Apr 2012 17:54:45 -0400] INFO: Chef Run complete in 3.912841062 seconds
[Thu, 12 Apr 2012 17:54:45 -0400] INFO: Running report handlers
[Thu, 12 Apr 2012 17:54:45 -0400] INFO: Report handlers complete

If we don’t want to specify the -j option each time we run chef-solo we can use the json_attribs option in our solo.rb file to specify the path of our JSON configuration file.

cookbook_path File.expand_path("../cookbooks", __FILE__)
json_attribs File.expand_path("../node.json", __FILE__)

Now we can call chef-solo without the -j option and when we do the output is a little different.

root@li349-144:/var/chef# chef-solo -c solo.rb
[Thu, 12 Apr 2012 18:09:19 -0400] INFO: *** Chef 0.10.8 ***
[Thu, 12 Apr 2012 18:09:20 -0400] INFO: Setting the run_list to ["recipe[main]"] from JSON
[Thu, 12 Apr 2012 18:09:20 -0400] INFO: Run List is [recipe[main]]
[Thu, 12 Apr 2012 18:09:20 -0400] INFO: Run List expands to [main]
[Thu, 12 Apr 2012 18:09:20 -0400] INFO: Starting Chef Run for li349-144.members.linode.com
[Thu, 12 Apr 2012 18:09:20 -0400] INFO: Running start handlers
[Thu, 12 Apr 2012 18:09:20 -0400] INFO: Start handlers complete.
[Thu, 12 Apr 2012 18:09:20 -0400] INFO: Processing package[git-core] action install (main::default line 1)
[Thu, 12 Apr 2012 18:09:20 -0400] INFO: Chef Run complete in 0.135240883 seconds
[Thu, 12 Apr 2012 18:09:20 -0400] INFO: Running report handlers
[Thu, 12 Apr 2012 18:09:20 -0400] INFO: Report handlers complete

The output still says that Chef ran the install action for git-core but it doesn’t say that it installed it. The idea here is that we can run chef-solo as many times as we want and it will only install and perform the actions necessary on the system and not rerun actions that have already been run. This means that we can run Chef quite frequently, such as every time we deploy our Rails application.

Adding More Functionality To Our Recipe

There’s more that we want to do to our recipe but editing it on the server isn’t the easiest way to do it so we’ll move everything over to our local machine. We can use scp to copy everything over.

$ scp -r root@178.xxx.xxx.xxx:/var/chef .

In our default.rb file we currently just install Git but there are other things we can do here. If we look at the Resources page of Chef’s documentation we’ll find a list of the many methods that we can call in our recipe. For example we can make new directories and files and and even create new users on the server. Each of these methods has its own documentation page explaining how its used. We want to create a deployer user on the server so we’ll add a call to the user method to our recipe. We need to pass this a name and a block and in the block we specify the user’s other attributes. The password attribute is interesting: we don’t use a plain text password here but a shadow hash and the documentation for this page shows us how to create one. We’ll use the openssl command to do this. We pass this the password we want to use and it will return a hash that we can use in our recipe.

$ openssl passwd -1 "theplaintextpassword"
$1$5mk003zO$GDCEKOIAsqvm4RDrJMX5Z.

Now we can write the code to create our new user using this hash for the password.

package "git-core"

user "deployer" do
  password "$1$5mk003zO$GDCEKOIAsqvm4RDrJMX5Z."
  gid "admin"
  home "/home/deployer"
  supports manage_home: true
end

We also set gid to set the users group to admin so that the user has sudo privileges and set the user’s home directory. This directory may not be set by default and it can be necessary to use the supports option to ensure that it’s created as we have here. To run this recipe we’re going to have to copy these changes back to the server and we can use rsync to do this.

$ rsync -r . root@178.xxx.xxx.xxx:/var/chef

We can then run the recipe on the server by SSHing in and running chef-solo with our solo.rb file.

$ ssh root@178.xxx.xxx.xxx "chef-solo -c /var/chef/solo.rb"
root@178.xxx.xxx.xxx's password: 
[Fri, 13 Apr 2012 13:54:09 -0400] INFO: *** Chef 0.10.8 ***
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: Setting the run_list to ["recipe[main]"] from JSON
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: Run List is [recipe[main]]
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: Run List expands to [main]
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: Starting Chef Run for li349-144.members.linode.com
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: Running start handlers
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: Start handlers complete.
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: Processing package[git-core] action install (main::default line 1)
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: Processing user[deployer] action create (main::default line 3)
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: user[deployer] created
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: Chef Run complete in 0.15908905 seconds
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: Running report handlers
[Fri, 13 Apr 2012 13:54:10 -0400] INFO: Report handlers complete

We can see from the output that the deployer user has been created so the changes to our recipe have worked.

Each time we run these commands we need to enter the server’s root user’s password, but there’s a useful utility called ssh-copy-id that we can use so that we don’t have to. This isn’t included by default in OS X but if we have Homebrew we can easily install it by running

$ brew install ssh-copy-id

We can now use this, passing in a username and the address of our server. We’ll need to enter the password one last time but after we do we won’t need to type it in again when we log in to the server.

Making Our Recipe More Configurable

Back to our recipe now. The way we create out user works fine but we have a lot of static strings in our recipe. The username and password are settings that we might want to change depending on the server we’re deploying to so it would be a good idea to move these out into an external configuration file. Chef supports this by giving us access to the content in node.json file. We’ll add a new option to this file and define the username and password there.

{
  "user": { 
    "name": "deployer",
    "password": "$1$5mk003zO$GDCEKOIAsqvm4RDrJMX5Z."
  },
  "run_list":["recipe[main]"]
}

In our recipe we can now access these values through a node hash.

package "git-core"

user node[:user][:name] do
  password node[:user][:password]
  gid "admin"
  home "/home/#{node[:user][:name]}"
  supports manage_home: true
end

Templates

Next we’ll look at templates. These allow us to generate files dynamically using erb and to use one all we have to do is call template and pass it the name of the file we want to write and a block. In the block we use source, passing in the name of the template. We want deployer to have ZSH as the default shell and so we’ll use the shell option to set this. We want to create a zshrc file for deployer in their home directory and we’ll do this by using a template. Note that we’re adding the zsh package here so that it’s installed on the server.

package "git-core"
package "zsh"

user node[:user][:name] do
  password node[:user][:password]
  gid "admin"
  home "/home/#{node[:user][:name]}"
  supports manage_home: true
  shell "/bin/zsh"
end

template "/home/#{node[:user][:name]}/.zshrc" do
  source "zshrc.erb"
  owner node[:user][:name]
end

As well as setting the source file, which we’ll create next, we use owner to set the file’s owner to that user. Templates belong in a templates directory under cookbooks/main and the templates for our default recipe should go in a default directory here. We’ll create zshrc.rb here and paste some simple code into it that customizes the prompt, sets the default Rails environment to production and adds some colour to the ls command.

export PS1='%m:%3~%# '
export RAILS_ENV=production

alias ls='ls --color=auto'

The great thing about using a template to do this is that we can use erb code to make it dynamic. We have access to the node object like we did in our recipe so we could customize our template so that the command to set the ls colouring is only included if an ls_color option is present.

export PS1='%m:%3~%# '
export RAILS_ENV=production

<% if node[:user][:ls_color] %>
alias ls='ls --color=auto'
<% end %>

Now we can change this file based on the contents of node.json and we’ll add the ls_color option there now.

{
  "user": { 
    "name": "deployer",
    "password": "$1$5mk003zO$GDCEKOIAsqvm4RDrJMX5Z.",
    "ls_color": true
    },
  "run_list":["recipe[main]"]
}

Even though we’ve already created deployer on the server running this recipe again will update that user and set their default shell. This is one of the great things about Chef: if something is already set up on the server and we alter a recipe Chef will update it to match the changes.

We can try this out now. We’ll run rsync to copy the changes to the server and then run chef-solo again.

$ rsync -r . root@178.xxx.xxx.xxx:/var/chef
$ ssh root@178.xxx.xxx.xxx "chef-solo -c /var/chef/solo.rb"
[Fri, 13 Apr 2012 16:25:23 -0400] INFO: *** Chef 0.10.8 ***
[Fri, 13 Apr 2012 16:25:24 -0400] INFO: Setting the run_list to ["recipe[main]"] from JSON
[Fri, 13 Apr 2012 16:25:24 -0400] INFO: Run List is [recipe[main]]
[Fri, 13 Apr 2012 16:25:24 -0400] INFO: Run List expands to [main]
[Fri, 13 Apr 2012 16:25:24 -0400] INFO: Starting Chef Run for li349-144.members.linode.com
[Fri, 13 Apr 2012 16:25:24 -0400] INFO: Running start handlers
[Fri, 13 Apr 2012 16:25:24 -0400] INFO: Start handlers complete.
[Fri, 13 Apr 2012 16:25:24 -0400] INFO: Processing package[git-core] action install (main::default line 1)
[Fri, 13 Apr 2012 16:25:24 -0400] INFO: Processing package[zsh] action install (main::default line 2)
[Fri, 13 Apr 2012 16:25:26 -0400] INFO: package[zsh] installed version 4.3.10-5ubuntu3
[Fri, 13 Apr 2012 16:25:26 -0400] INFO: Processing user[deployer] action create (main::default line 4)
[Fri, 13 Apr 2012 16:25:26 -0400] INFO: user[deployer] altered
[Fri, 13 Apr 2012 16:25:26 -0400] INFO: Processing template[/home/deployer/.zshrc] action create (main::default line 12)
[Fri, 13 Apr 2012 16:25:26 -0400] INFO: template[/home/deployer/.zshrc] owner changed to 1000
[Fri, 13 Apr 2012 16:25:26 -0400] INFO: template[/home/deployer/.zshrc] updated content
[Fri, 13 Apr 2012 16:25:26 -0400] INFO: Chef Run complete in 1.876843613 seconds
[Fri, 13 Apr 2012 16:25:26 -0400] INFO: Running report handlers
[Fri, 13 Apr 2012 16:25:26 -0400] INFO: Report handlers complete

We can see from the output that ZSH has been installed, that the deployer user has been altered and that the .zshrc template has been processed and copied into the right directory. If we log into the server as deployer now we should have ZSH as our shell and if we cat the .zshrc file we’ll see the code from the template.

li349-144:~% cat .zshrc
export PS1='%m:%3~%# '
export RAILS_ENV=production

alias ls='ls --color=auto'

Using Other Cookbooks To Install Software

We’ll need to install more software on the server before we can host a Rails application on it. For example we want to install Nginx and while we could do this by installing its package doing this will install an older version. If we want to install a more recent version or compile it from source we can do so. There’s a rich community around Chef and plenty of cookbooks available on the Internet. A good place to start is the Opscode Community site and there’s an Nginx cookbook available there. We can download this and move it into our Chef directory. This is a better example of a cookbook than our simple example and it comes complete with a metadata.rb file that contains a lot of useful information. When using a third-party cookbook it’s a good idea to check its calls to depends. Cookbooks often have dependencies on other cookbooks and the Nginx cookbook is no exception, depending on build-essential, runit, bluepill and ohai.

%w{ build-essential runit bluepill }.each do |cb|
  depends cb
end

depends 'ohai', '~> 1.0.2'

Depending on our configuration not all of these will be used but it’s still a good idea to find these on the Opscode Community site and download them. We’ll do that now and copy each one into our cookbooks directory.

The metadata file also tells us what recipes the cookbook makes available. The Nginx cookbook has two recipes, a default nginx one, which uses the normal package install, and an nginx::source one which will install Nginx from source.

recipe "nginx", "Installs nginx package and sets up configuration with Debian apache style with sites-enabled/sites-available"
recipe "nginx::source", "Installs nginx from source and sets up configuration with Debian apache style with sites-enabled/sites-available"

We want to install Nginx from source so we’ll use this one here. We can view the source code for this recipe by looking in its recipes directory and if we look in the source.rb file there we’ll find the code that downloads, un-tars and compiles the Nginx source code.

There are a number of configuration options in this recipe that we can override in our node.json configuration file. One option, for example, sets the version of Nginx that will be downloaded.

src_filepath  = "#{Chef::Config[:file_cache_path]}/nginx-#{node[:nginx][:version]}.tar.gz"

The default value for this and all the other settings is defined in a file in the recipe’s attributes directory. The version number appears in a default.rb file. We’ll leave all the settings at their defaults for now.

default[:nginx][:version] = "1.0.14"

Exploring cookbooks like this is great way to get an understanding of how Chef works and of how to structure our own cookbooks. We’ll go back to our own recipe now and use the nginx::source recipe from the cookbook to install Nginx on our server.

package "git-core"
package "zsh"

include_recipe "nginx::source"

user node[:user][:name] do
  password node[:user][:password]
  gid "admin"
  home "/home/#{node[:user][:name]}"
  supports manage_home: true
  shell "/bin/zsh"
end

template "/home/#{node[:user][:name]}/.zshrc" do
  source "zshrc.erb"
  owner node[:user][:name]
end

When we rsync our changes to the server then run our chef-solo there again we’ll see a lot more output now as it goes through the process of downloading and compiling Nginx. We now have Nginx running on our server but it doesn’t have any enabled sites. We’ll add the following code to our recipe to make a simple Nginx site.

directory "/home/#{node[:user][:name]}/example" do
  owner node[:user][:name]
end

file "/home/#{node[:user][:name]}/example/index.html" do
  owner node[:user][:name]
  content "<h1>Hello World!</h1>"
end

file "#{node[:nginx][:dir]}/sites-available/example" do
  content "server { root /home/#{node[:user][:name]}/example; }"
end

nginx_site "example"

Here we make a new example directory under the deployer user’s home directory and add a simple HTML file to it. We then create another file under Nginx’s sites-available directory with a very simple Nginx configuration file in it. Making a site available doesn’t make it enabled but the Nginx cookbook provides an nginx_site method to do this and we’ve used this in our recipe. This will enable our site and also reload Nginx. When we rsync the changes to the server and run chef-solo again we should be able to view our simple site by visiting the server’s IP address in a browser.

That’s as far as we’re going to take this recipe. It’s still quite simple but it should give you some idea as to how to go about setting up a server for a Rails application. We’ve only scratched the surface of Chef in this episode but there’s much more documentation available at the Chef Wiki and it’s well worth taking a look through this if you need more information. If you need to install a specific piece of software on your server the chances are that someone has already written a cookbook to do this so remember to check the community site before writing your own recipes. There are even more cookbooks available on Github. 37 Signals have a large cookbooks repository which is worth taking a look at.

If you decide that Chef isn’t for you there are a number of alternatives available such as Puppet, Sprinkle, Rubber and Sunzi. If your Rails application is fairly simple don’t forget about the Capistrano recipe solution that we showed last week.