#339 Chef Solo Basics pro
Apr 06, 2012 | 17 minutes | Tools, Deployment, Production
Chef is a provisioning tool which allows you to spin up production servers automatically. Here I show how to get started with Chef Solo and create a custom cookbook.
- source codeProject Files in Zip (83.5 KB)
- mp4Full Size H.264 Video (46.4 MB)
- m4vSmaller H.264 Video (21.9 MB)
- webmFull Size VP8 Video (22.1 MB)
- ogvFull Size Theora Video (49.3 MB)
chef-solo is cool, but it could be quite involving when your need is pretty simple.
With Sunzi, installing
git-coreon Linode is about 5 commands:
Could you try that out? :)
Thanks for sharing, I wasn't familiar with Sunzi. I'll add it to the list of alternatives.
What a great episode. Thnx Ryan.
I am a bit disappointed that most cookbooks ignore Mac OS X server. :(
They are open source, so you can just patch them and submit a pull request :)
There are actually a number of cookbooks which have open pull requests to add OSX support to them. It's quite easy to add support yourself and contributing back to the project!
Nice episode. There is definitely a steep learning curve with chef. Here are a couple of libraries that make it simpler:
Knife-Solo - This allows you to remotely install chef on a blank machine using
knife prepareand also allows you to push your recipes to your remote server using
Librarian - This works like bundler for chef recipes. You define a Cheffile and list out what cookbooks you need, and it will download them and their dependencies into your cookbooks directory. If you have custom recipes, it is good to move them into a site-cookbooks folder so they don't get overwritten by librarian when you import your cookbooks. When you're ready, you can go back to knife-solo to push them out to your server.
Comment of the week!
Hey, thanks for the comment. I've been playing around with chef using the hosted version. I realized that some of the opscode cookbooks are a bit outdated, so I've turned to github to find cookbooks. My problem now is how to mix these cookbooks with the ones I've already downloaded using knife cookbook install site install... I think Librarian should help, but I'm lost after you mentioned bundler. What does bundler have to do with chef?
Should be awesome if Knife-Solo and/or Librarian are covered.
So having covered Capistrano and Chef, can Chef be used without Capistrano for regularly deploying Rails apps? Does moving to Chef obviate Cap? I know I could find out myself but I'm really looking for confirmation that zsh history search for a long command is giving me less than these tools.
I think chef is more for provisioning the server, I would still stick with capistrano for deployment.
You'll want to provision the server(s) with Chef (or similar tools like Puppet) to prepare it for deployment. Then use Capistrano to do the actual deployment(s).
Chef has a deployment resource to deploy out of SVN or Git - it acts similar to what you're familiar with in Capistrano.
Because Chef's job is to control state of a node, it's actually an anti-pattern to use Capistrano for service control or deployments.
I use chef server (open source) and got rid of Capistrano for my deploys. Chef has a "deploy resource" http://wiki.opscode.com/display/chef/Deploy+Resource
Few things I really like:
1) Its a pull architecture, so you can cron machines to check back in and pull updates for production. But from the cli (knife) you can use the search feature and run whatever command you want. So for testing env I use it like cap (and essentially have a push deploy)
2) You can rollout upgrades to your application stack as well as your code. This is really nice for testing b/c it has environments so you can roll out new recipes to testing envs and then to production. Recipes etc are also usually maintained in a git repo.
3) I really like that it can be used in so many places. I use it to setup my laptop which runs ubuntu, virtual machines for testing environments, and a physical hosted server. I have also started to play with it in the cloud a little. For the physical machines I run a bootstrap command to setup the machine, with vms & cloud there are plugins that will also provision machines. If you write your recipes to scale and add monitoring... you can actually have you app scale on its own!
I have been playing with it for the last 6 months or so (on and off). I will say that chef like rails has a learning curve, but it gets easier... And its very easy to see the power. Infrastructure as Code gotta love it
After you read the wiki at http://wiki.opscode.com/ to understand the basic commands and architecture of chef and maybe some of the available opensource chef cookbooks you probably want to check footcritic:
It's a quite new project that analyzes cookbook code regarding the best/common practice of writing chef cookbooks.
E.g. I see Ryan uses the (old) Ruby-symbol syntax to access node data. As far as I remember this was deprecated in favour of the typical 'string'-keys syntax: http://acrmp.github.com/foodcritic/#FC001
I hope, some day, the opscode guys will also provide an "official" tool that makes life with chef-solo easier without having to use 3rd party tools...
Until that, you may want to check:
Daaamn Chef is serious. It doesn't even make sense to use it for the tiny one-server example you gave; it's power only really starts to make sense once you see how Chef Server works to let you implement an EC2-type setup on your own infrastructure.
I've seen the value of it for a long time, but two different attempts to ease into it were thwarted by the bad introductory materials. This is a great introduction. I look forward to using this as a jumping off point to actually get into Chef someday.
Even large chef users (e.g. Engine Yard) tend to use chef-solo and their own services to replicate/push cookbooks.
Chef-Server adds a runtime dependency and needs it's own complex replication/failover strategy. If you rely on (a hosted) chef-server and it goes down, you can't reconfigure existing or launch new servers. A big no-go imho.
Sure, for a couple of unimportant VMs it's awesome to play and get a feeling with. But if your setup grows, you really don't want to have implicit configuration (like every 30mins with chef-client) and go with an explicit/push-style configuration.
A decent number of large users are not using chef-solo and there are definitely approaches that you can take to resiliency with the chef-server infrastructure.
Obviously Engine Yard is one of the poster children of chef-solo but last time I spoke to anyone from over there, they are also still on 0.6.x whereas the rest of the world has largely moved onto much newer releases. They're probably not the best example of a "large scale" user unless your needs marry closely with what they wanted.
Speaking personally, the benefits provided by Search are more than worth any hassle involved in making sure the API calls always get answered!
What do you do if your chef-server vm crashes and you need to provision new app servers to keep your main business running? ... exactly.
So either you have a redundant chef-server setup OR you'll get into trouble. It's just a matter of time.
Trading in a SPOF-less environment for convenience is a very, very bad DevOps guide.
If you really provision new servers that often, that you don't have time to reboot your chef-server, then have as many chef-servers as you need. I don't see why the preferred solution would be to exclusively use chef-solo. If you have some reason to prefer that approach, please do tell.
chef-server != SPOF
It is not hard to make chef-server run redundantly.
rsync -r . firstname.lastname@example.org:/var/chef & ssh email@example.com "chef-solo -c /var/chef/solo.rb"each time when you want to roll up new recipes. Is there any way to do it with one command?
Create a file deploy.sh containing that command then all you have to do is run ./deploy.sh in future.
Another idea, you could add a Capistrano task for this.
On my Ubumtu server on AWS I'm not able to use rsync with the root user. Any ideas why?
In my AWS server I need to use "ubuntu" server instead as the root user is disabled.
Yep your correct. I probably didn't word my question very well.
Since the root user doesn't have access to ssh you cant rsync the files to /var/chef so I have to figure out a work around for that. I don't think enabling the root user is a good option.
True, I needed also to chown ubuntu /var/chef in order for it to work
You need to rsync using ssh and your keyfile like that:
rsync -r <dir> -e "ssh -i <your pem file>" --rsync-path='sudo rsync' ubuntu@<your ip>:/var/chef
You can use Chef to install Ruby on the server instead of having to log in and set it up yourself.
You can use
knife bootstrap hostinstead of logging in and installing everything chef needs manually. Chef even allows you to build your own custom bootstrap script based on what type of Linux machine your using.
knife help bootstrap
you can use apt-get build-dep to get dependency for building ruby!
did anyone have to restart the server before running nginx?
How does chef compare to puppet?
Joshua Sierles is from 37signals.
His cookbook at https://github.com/jsierles/chef_cookbooks
is better maintained than the 37signals cookbook
The 37signals cookbook was last updated 2 years ago, by jsierles
jsierles cookbooks was updated 14 days ago
Thanks for that! I was wondering why they weren't update more often.
Thanks for posting the link to Joshua's cookbooks. It looks like 37Signals cookbook has been removed from their Git account, or made private.
A good practice is to place site specific cookbooks in a folder called
site-cookbooksinstead of putting them all in
Thanks for this screencast, during my migration to chef-solo I started working on a solution which makes it super easy to install a more or less standard rails stack so that you can start deploying via capistrano:
For some reason the chef_solo_bootstrap.sh didnt work for me. I had to end up installing the following packages:
apt-get install libreadline-gplv2-dev lib64readline-gplv2-dev make zlib1g-dev
Even after installing the above packages and finishing the ruby/chef installation process on the Ubuntu 12.04 VPS, I would get an error on launching 'chef -v':
I ended up doing this to install ruby and chef:
I was installing from source and had to do this to get it to work
Also, the admin group didnt exist on the Ubuntu 12 VPS that I used. So the useradd failed. I ended up doing:
This worked for me. But anyone know if this is because of a difference in distros from the one used on the Railscast? Is admin usually a pre-created default group?
Just in case someone is interested to know, in Ubuntu versions 11.10 (Oneiric Ocelot) and onwards the "admin" group is not going to be provided by default. The power that be are favouring the "sudo" group instead of "admin" group. If you're upgrading from a lower version, your "admin" group will persist but on a fresh install "admin" group will be missing.
So if you get an error that "admin" group does not exist, simply replace it with "sudo".
adduser deployer --ingroup sudo
Credits to Aditya Sanghi, here: http://railscasts.com/episodes/337-capistrano-recipes?view=comments#comment_157369
Installing Ruby on CentOS:
Any reason somebody would recommend Chef over Puppet?
I got inspired by your screencast, and wrote a comprehensive guide to Chef Solo and Vagrant: https://deveo.com/blog/2013/03/11/underneath-your-ruby-app-deployment-strategies-extracted/
If it isn't obvious from my screen-name, disclaimer is that I work at opscode...
The latest community cookbooks maintained by opscode are here:
Since there are 137+ repos in there, you can sometimes find specific cookbooks on github which may be more up to date or may better fit your needs, but that's a good place to start looking. Since those cookbooks are designed to be multi-operating system (CentOS, Ubuntu, Solaris, etc) they also aren't necessarily the simplest designs.
Installing chef has also gotten considerably easier, and this:
curl -L https://www.opscode.com/chef/install.sh | sudo bash
Will install chef-client and chef-solo into /opt/chef and setup symlinks into /usr/bin/chef-client and works across most major operating systems. You can still gem install chef-client, but the omnibus installation into /opt/chef gets us a consistent (and usually substantially more up-to-date) version of ruby and rubygems across platforms including Solaris 9 and CentOS 5 which we support.
The site-cookbooks method of separating your own cookbooks has also been deprecated in favor of using berkshelf:
There's good integration between berkshelf and vagrant for easily testing chef-solo recipes:
The chef community has changed fairly fast in the past 12 months...
Thanks for the update lamont. I agree, things have changed and it's hard to find good documentation. I'm interested in learning how to use berkshelf and chef-solo together but can't find a good tutorial. Can you recommend one?
I'm looking for the same. I've found these 2 posts helpful, but part 3 is yet to be published:
I've also found this post helpful for filling in some of the gaps:
Have you found anything elsewhere?
I'd suggest fairly strongly getting a nice deep familiarity with SSH and how it works.
There's a lot you can do between that and some sage knowledge on the sudo command to write a fairly well locked down server that's ready to rock the world.
Combine this with knowledge on OpenBSD and how it works and you have a tank of a server that's going to be near impossible to break into.
SSH Mastery: http://www.amazon.com/SSH-Mastery-OpenSSH-Tunnels-ebook/dp/B006ZO9ULK
Absolute OpenBSD: http://www.amazon.com/Absolute-OpenBSD-Practical-Paranoid-ebook/dp/B00CH96VB4/ref=sr_1_1?s=digital-text&ie=UTF8&qid=1368591922&sr=1-1&keywords=absolute+openbsd
Another great screencast!
You have mentioned that for minimal proposes (installing chef) you compile ruby from source and later on RVM can be installed with recipe.
Is there a common method to install RVM through chef? I didn't find something popular among the community cookbooks.
Finally, after installing RVM, how do you make sure that everything runs through the ruby version installed through RVM?
Here's alternative chef solo wrapper as a gem:
If you're getting
NoMethodError: undefined method
' for nil:NilClasserrors in the nginx compile:
Looks like as of Chef 11, you need to add
depends 'nginx'to a metadata.rb file if you do an 'include_recipe'. (In this example, the medata.rb would be in the 'main' recipe directory.) See http://tickets.opscode.com/browse/COOK-2342
This is not necessary if you just add the recipe to your runlist in the node.json file.
I have written an updated Gist for installing Ruby and Chef on a Ubuntu 12.04 32 Bit Server. You can check out the gist here: https://gist.github.com/codemis/6041247.