#339 Chef Solo Basics pro

Apr 06, 2012 | 17 minutes | Tools, Deployment, Production

Chef is a provisioning tool which allows you to spin up production servers automatically. Here I show how to get started with Chef Solo and create a custom cookbook.

Click to Play Video ▶

Download:
source codeProject Files in Zip (83.5 KB)
mp4Full Size H.264 Video (46.4 MB)
m4vSmaller H.264 Video (21.9 MB)
webmFull Size VP8 Video (22.1 MB)
ogvFull Size Theora Video (49.3 MB)

Kenn Ejima over 12 years ago

Ryan,

chef-solo is cool, but it could be quite involving when your need is pretty simple.

With Sunzi, installing git-core on Linode is about 5 commands:

          
        
$ sunzi create
$ cd sunzi
$ sunzi setup linode
$ echo "apt-get -y install git-core" >> install.sh
$ sunzi deploy yourhost.com

Could you try that out? :)

Ryan Bates over 12 years ago

Thanks for sharing, I wasn't familiar with Sunzi. I'll add it to the list of alternatives.

Radha over 12 years ago

What a great episode. Thnx Ryan.

I am a bit disappointed that most cookbooks ignore Mac OS X server. :(

joonas over 12 years ago

They are open source, so you can just patch them and submit a pull request :)

Jamie Winsor over 12 years ago

Radha,

There are actually a number of cookbooks which have open pull requests to add OSX support to them. It's quite easy to add support yourself and contributing back to the project!

Aaron Renner over 12 years ago

Nice episode. There is definitely a steep learning curve with chef. Here are a couple of libraries that make it simpler:

Knife-Solo - This allows you to remotely install chef on a blank machine using knife prepare and also allows you to push your recipes to your remote server using knife cook.
Librarian - This works like bundler for chef recipes. You define a Cheffile and list out what cookbooks you need, and it will download them and their dependencies into your cookbooks directory. If you have custom recipes, it is good to move them into a site-cookbooks folder so they don't get overwritten by librarian when you import your cookbooks. When you're ready, you can go back to knife-solo to push them out to your server.

Michael van Rooijen over 12 years ago

^ pro-tip.

Dane O'Connor over 12 years ago

Comment of the week!

bertomartin over 12 years ago

Hey, thanks for the comment. I've been playing around with chef using the hosted version. I realized that some of the opscode cookbooks are a bit outdated, so I've turned to github to find cookbooks. My problem now is how to mix these cookbooks with the ones I've already downloaded using knife cookbook install site install... I think Librarian should help, but I'm lost after you mentioned bundler. What does bundler have to do with chef?

Akshay Rawat about 12 years ago

Should be awesome if Knife-Solo and/or Librarian are covered.

Vincent Bray over 12 years ago

So having covered Capistrano and Chef, can Chef be used without Capistrano for regularly deploying Rails apps? Does moving to Chef obviate Cap? I know I could find out myself but I'm really looking for confirmation that zsh history search for a long command is giving me less than these tools.

Artellectual over 12 years ago

Hey,

I think chef is more for provisioning the server, I would still stick with capistrano for deployment.

Michael van Rooijen over 12 years ago

You'll want to provision the server(s) with Chef (or similar tools like Puppet) to prepare it for deployment. Then use Capistrano to do the actual deployment(s).

Jamie Winsor over 12 years ago

Chef has a deployment resource to deploy out of SVN or Git - it acts similar to what you're familiar with in Capistrano.

Because Chef's job is to control state of a node, it's actually an anti-pattern to use Capistrano for service control or deployments.

Braden Wright almost 12 years ago

I use chef server (open source) and got rid of Capistrano for my deploys. Chef has a "deploy resource" http://wiki.opscode.com/display/chef/Deploy+Resource

Few things I really like:
1) Its a pull architecture, so you can cron machines to check back in and pull updates for production. But from the cli (knife) you can use the search feature and run whatever command you want. So for testing env I use it like cap (and essentially have a push deploy)

2) You can rollout upgrades to your application stack as well as your code. This is really nice for testing b/c it has environments so you can roll out new recipes to testing envs and then to production. Recipes etc are also usually maintained in a git repo.

3) I really like that it can be used in so many places. I use it to setup my laptop which runs ubuntu, virtual machines for testing environments, and a physical hosted server. I have also started to play with it in the cloud a little. For the physical machines I run a bootstrap command to setup the machine, with vms & cloud there are plugins that will also provision machines. If you write your recipes to scale and add monitoring... you can actually have you app scale on its own!

I have been playing with it for the last 6 months or so (on and off). I will say that chef like rails has a learning curve, but it gets easier... And its very easy to see the power. Infrastructure as Code gotta love it

Roland Moriz over 12 years ago

Nice episode!

After you read the wiki at http://wiki.opscode.com/ to understand the basic commands and architecture of chef and maybe some of the available opensource chef cookbooks you probably want to check footcritic:

http://acrmp.github.com/foodcritic/

It's a quite new project that analyzes cookbook code regarding the best/common practice of writing chef cookbooks.

E.g. I see Ryan uses the (old) Ruby-symbol syntax to access node data. As far as I remember this was deprecated in favour of the typical 'string'-keys syntax: http://acrmp.github.com/foodcritic/#FC001

I hope, some day, the opscode guys will also provide an "official" tool that makes life with chef-solo easier without having to use 3rd party tools...

Until that, you may want to check:

https://github.com/matschaffer/knife-solo
or:
https://github.com/tobami/littlechef

chadoh over 12 years ago

Daaamn Chef is serious. It doesn't even make sense to use it for the tiny one-server example you gave; it's power only really starts to make sense once you see how Chef Server works to let you implement an EC2-type setup on your own infrastructure.

I've seen the value of it for a long time, but two different attempts to ease into it were thwarted by the bad introductory materials. This is a great introduction. I look forward to using this as a jumping off point to actually get into Chef someday.

Thanks!

Roland Moriz over 12 years ago

Even large chef users (e.g. Engine Yard) tend to use chef-solo and their own services to replicate/push cookbooks.

Chef-Server adds a runtime dependency and needs it's own complex replication/failover strategy. If you rely on (a hosted) chef-server and it goes down, you can't reconfigure existing or launch new servers. A big no-go imho.

Sure, for a couple of unimportant VMs it's awesome to play and get a feeling with. But if your setup grows, you really don't want to have implicit configuration (like every 30mins with chef-client) and go with an explicit/push-style configuration.

Alex Howells over 12 years ago

A decent number of large users are not using chef-solo and there are definitely approaches that you can take to resiliency with the chef-server infrastructure.

Obviously Engine Yard is one of the poster children of chef-solo but last time I spoke to anyone from over there, they are also still on 0.6.x whereas the rest of the world has largely moved onto much newer releases. They're probably not the best example of a "large scale" user unless your needs marry closely with what they wanted.

Speaking personally, the benefits provided by Search are more than worth any hassle involved in making sure the API calls always get answered!

Roland Moriz over 12 years ago

Srsly?

What do you do if your chef-server vm crashes and you need to provision new app servers to keep your main business running? ... exactly.

So either you have a redundant chef-server setup OR you'll get into trouble. It's just a matter of time.

Trading in a SPOF-less environment for convenience is a very, very bad DevOps guide.

Brandon over 12 years ago

If you really provision new servers that often, that you don't have time to reboot your chef-server, then have as many chef-servers as you need. I don't see why the preferred solution would be to exclusively use chef-solo. If you have some reason to prefer that approach, please do tell.

chef-server != SPOF

Alex Howells over 12 years ago

It is not hard to make chef-server run redundantly.

Evrone over 12 years ago

Dear Ryan,
Use use rsync -r . root@72.14.183.209:/var/chef & ssh root@72.14.183.209 "chef-solo -c /var/chef/solo.rb" each time when you want to roll up new recipes. Is there any way to do it with one command?

Jack Kinsella over 12 years ago

Create a file deploy.sh containing that command then all you have to do is run ./deploy.sh in future.

Christian Joudrey over 12 years ago

Another idea, you could add a Capistrano task for this.

Jonathan Spooner over 12 years ago

On my Ubumtu server on AWS I'm not able to use rsync with the root user. Any ideas why?

Potiguar Faga over 12 years ago

In my AWS server I need to use "ubuntu" server instead as the root user is disabled.

Jonathan Spooner over 12 years ago

Yep your correct. I probably didn't word my question very well.

Since the root user doesn't have access to ssh you cant rsync the files to /var/chef so I have to figure out a work around for that. I don't think enabling the root user is a good option.

Potiguar Faga over 12 years ago

True, I needed also to chown ubuntu /var/chef in order for it to work

emrass over 12 years ago

You need to rsync using ssh and your keyfile like that:
rsync -r <dir> -e "ssh -i <your pem file>" --rsync-path='sudo rsync' ubuntu@<your ip>:/var/chef

Jesse Dearing over 12 years ago

You can use Chef to install Ruby on the server instead of having to log in and set it up yourself.

You can use knife bootstrap host instead of logging in and installing everything chef needs manually. Chef even allows you to build your own custom bootstrap script based on what type of Linux machine your using.

Check out knife help bootstrap

Peter de Croos over 12 years ago

https://gist.github.com/2307959

Shota Fukumori over 12 years ago

sudo apt-get build-dep ruby1.9.1

you can use apt-get build-dep to get dependency for building ruby!

yocha over 12 years ago

did anyone have to restart the server before running nginx?

Mike Ni over 12 years ago

How does chef compare to puppet?

Deepak Kannan over 12 years ago

Joshua Sierles is from 37signals.
His cookbook at https://github.com/jsierles/chef_cookbooks
is better maintained than the 37signals cookbook

The 37signals cookbook was last updated 2 years ago, by jsierles
jsierles cookbooks was updated 14 days ago

bertomartin over 12 years ago

Thanks for that! I was wondering why they weren't update more often.

Tilo about 12 years ago

Thanks for posting the link to Joshua's cookbooks. It looks like 37Signals cookbook has been removed from their Git account, or made private.

Christian Joudrey over 12 years ago

A good practice is to place site specific cookbooks in a folder called site-cookbooks instead of putting them all in cookbooks.

http://wiki.opscode.com/display/chef/Cookbooks#Cookbooks-SiteSpecificCookbooks

dei79 about 12 years ago

Thanks for this screencast, during my migration to chef-solo I started working on a solution which makes it super easy to install a more or less standard rails stack so that you can start deploying via capistrano:

https://github.com/dei79/solify

udit99 about 12 years ago

For some reason the chef_solo_bootstrap.sh didnt work for me. I had to end up installing the following packages:

apt-get install libreadline-gplv2-dev lib64readline-gplv2-dev make zlib1g-dev

udit99 about 12 years ago

Even after installing the above packages and finishing the ruby/chef installation process on the Ubuntu 12.04 VPS, I would get an error on launching 'chef -v':

          Terminal
        
"/usr/local/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- openssl (LoadError)"

I ended up doing this to install ruby and chef:

          Terminal
        
apt-get update
apt-get install make ruby1.9.3
gem install chef ruby-shadow --no-ri --no-rdoc

Scott Shea over 11 years ago

I was installing from source and had to do this to get it to work

          
        
cd ruby-1.9(source directory used to install ruby)/ext/openssl
sudo ruby extconf.rb
sudo make
sudo make install

udit99 about 12 years ago

Also, the admin group didnt exist on the Ubuntu 12 VPS that I used. So the useradd failed. I ended up doing:

          default.rb
        
group "admin" do
  gid 420
end

user "deployer" do
  password "$%^&shadowhash&$%&#*(."
  gid 420
  home "/home/deployer"
  supports manage_home: true
end

This worked for me. But anyone know if this is because of a difference in distros from the one used on the Railscast? Is admin usually a pre-created default group?

Jefferson Queiroz Venerando almost 12 years ago

Just in case someone is interested to know, in Ubuntu versions 11.10 (Oneiric Ocelot) and onwards the "admin" group is not going to be provided by default. The power that be are favouring the "sudo" group instead of "admin" group. If you're upgrading from a lower version, your "admin" group will persist but on a fresh install "admin" group will be missing.

So if you get an error that "admin" group does not exist, simply replace it with "sudo".

adduser deployer --ingroup sudo

Credits to Aditya Sanghi, here: http://railscasts.com/episodes/337-capistrano-recipes?view=comments#comment_157369

Denis almost 12 years ago

Installing Ruby on CentOS:
https://gist.github.com/denispeplin/5011950

Benjamin Reed over 11 years ago

Any reason somebody would recommend Chef over Puppet?

Tair Assimov over 11 years ago

I got inspired by your screencast, and wrote a comprehensive guide to Chef Solo and Vagrant: https://deveo.com/blog/2013/03/11/underneath-your-ruby-app-deployment-strategies-extracted/

Thanks!

lamont-opscode over 11 years ago

If it isn't obvious from my screen-name, disclaimer is that I work at opscode...

The latest community cookbooks maintained by opscode are here:

https://github.com/opscode-cookbooks/

Since there are 137+ repos in there, you can sometimes find specific cookbooks on github which may be more up to date or may better fit your needs, but that's a good place to start looking. Since those cookbooks are designed to be multi-operating system (CentOS, Ubuntu, Solaris, etc) they also aren't necessarily the simplest designs.

Installing chef has also gotten considerably easier, and this:

curl -L https://www.opscode.com/chef/install.sh | sudo bash

Will install chef-client and chef-solo into /opt/chef and setup symlinks into /usr/bin/chef-client and works across most major operating systems. You can still gem install chef-client, but the omnibus installation into /opt/chef gets us a consistent (and usually substantially more up-to-date) version of ruby and rubygems across platforms including Solaris 9 and CentOS 5 which we support.

The site-cookbooks method of separating your own cookbooks has also been deprecated in favor of using berkshelf:

http://berkshelf.com/

There's good integration between berkshelf and vagrant for easily testing chef-solo recipes:

http://www.slideshare.net/bluepojo/introduction-to-berkshelfvagrant

The chef community has changed fairly fast in the past 12 months...

Ashley Connor over 11 years ago

Great post!

Andrew Havens about 11 years ago

Thanks for the update lamont. I agree, things have changed and it's hard to find good documentation. I'm interested in learning how to use berkshelf and chef-solo together but can't find a good tutorial. Can you recommend one?

Tom Marshall about 11 years ago

I'm looking for the same. I've found these 2 posts helpful, but part 3 is yet to be published:

http://blog.atwam.com/blog/2013/04/27/preparing-a-rails-app-for-deployment-using-chef-solo/
http://blog.atwam.com/blog/2013/06/10/configuring-chef-solo-for-our-servers/

I've also found this post helpful for filling in some of the gaps:

http://teohm.github.io/blog/2013/04/17/chef-cookbooks-for-busy-ruby-developers/

Have you found anything elsewhere?

Brandon Weaver over 11 years ago

I'd suggest fairly strongly getting a nice deep familiarity with SSH and how it works.

There's a lot you can do between that and some sage knowledge on the sudo command to write a fairly well locked down server that's ready to rock the world.

Combine this with knowledge on OpenBSD and how it works and you have a tank of a server that's going to be near impossible to break into.