#335 Deploying to a VPS pro

Mar 21, 2012 | 18 minutes | Tools, Deployment, Production

Deploying a Rails application can be overwhelming because there are so many different options. Here I present a pattern for deploying a Rails app to a VPS using nginx, Unicorn, PostgreSQL, rbenv and more.

Click to Play Video ▶

Download:
source codeProject Files in Zip (98 KB)
mp4Full Size H.264 Video (48.2 MB)
m4vSmaller H.264 Video (22.3 MB)
webmFull Size VP8 Video (21.7 MB)
ogvFull Size Theora Video (51.3 MB)

Levi Figueira over 13 years ago

Great screencast, Ryan! I'm comfortable setting servers and, over time, I have created my own system to it but I'm always learning and some of your techniques might be added to my workflow… :)

Question: I always add an nginx conf file for the domain I'm hosting. You simply deleted the default. How does deleting a default automatically make it load the application from the deployer's home directory? Magic sauce there? :)

fschwahn over 13 years ago

The capistrano recipe symlinks it to /etc/nginx/sites-enabled/

Levi Figueira over 13 years ago

AWESOME! Gotta get that going on my systems… :o <3

Thanks man :)

lolski over 11 years ago

Great question. I was just about to ask the same

John Rees over 13 years ago

Thanks Ryan, this is superb.

Reassuring to see that it's pretty similar to what I bodged together after watching your Vagrant/Unicorn screencasts https://gist.github.com/1985879

Panggi Libersa Jasri Akadol over 13 years ago

in Linux,instead of using:
cat ~/.ssh/id_rsa.pub | ssh user@host 'cat >> ~/.ssh/authorized_keys'

i prefer to use:
ssh-copy-id user@host

Mikko Kokkonen over 13 years ago

That's easy command, and can be found on OS X by homebrew as well
brew install ssh-copy-id

Alexander Zubkov over 13 years ago

Thanks! Really useful. Didn't know about it.

John Rees over 13 years ago

sanjaynair about 13 years ago

sha@cluncle.com
+1

Ratnadeep Deshmane about 13 years ago

Jason Wong over 13 years ago

I wonder if this could apply to services like EC2.

John Rees over 13 years ago

Yep, you just need to do things a little differently when initially ssh'ing/setting up your keys

Ammar over 13 years ago

Okay, so I'm developing a web app that's going to have around 600 users - creating about 1500 records a day.

After watching this I'm tempted to check out a VPS, but I've only got experience deploying with Heroku ( <- total n00b ).

For my use case, would you recommend a VPS (such as Linode) or Heroku?

Mike Goggin over 13 years ago

I've used three different VPS solutions and each are pretty good. They all provide different features for their price points as well.

ChicagoVPS - They offer a few solutions which are reasonably priced. OS options are somewhat limited. I'm pretty sure you're locked into CentOS 5.
ChunkHost - ChunkHost is slightly more expensive, but offers more options. OS options available are: Ubuntu (8.04 LTS, 10.04 LTS, 10.10), Debian (lenny, squeeze), and CentOS (5, 6). They're offering a free VPS for a limited time as a "beta test." Good for testing.
Linode - Linode is my ol' standard. I've used them for years and have never been disappointed. They offer several solutions and have quite a few features that the others don't have. They also offer flexibility with your Linode (add more storage or an extra IP instantly). OS options available are pretty much anything you can think of. Each option available also comes in a 64-bit flavor if that's your preference (or requirement).

Ammar over 13 years ago

So for my use case would you recommend a VPS over something like Heroku or EngineYard?

Mike Goggin over 13 years ago

It depends on how much control you want over your environment. If you're not expecting your app to have to scale much, you might go with a VPS. However, if you're expecting the app to grow rapidly and for your userbase and daily record commit to increase by a large margin, you might consider a hosted solution like Heroku or EngineYard. Both of those solutions scale very well up and down.

debprado over 13 years ago

I've been using Slicehost for a few years

this articles helped me a lot: articles.slicehost.com

Richard Schneeman over 13 years ago

I know you've gotten some responses but I'll throw in my 2 cents. If you're doing this for fun only then go with whatever you enjoy or challenges you. If you're doing it for money or for recognition, determine what your competitive advantage is/could-be and if either a VPS or Heroku can help you provide your customers with an advantage choose them. If neither of those are true pick one that you trust and can use effectively. I learned a ton by building systems in VPS and on various cloud providers, but at the end of the day rails is my passion (p.s. I work for Heroku). Everyone should know something about how servers are set up, only you can choose which provider is right for you.

fschwahn over 13 years ago

Most rails applications have more dependencies, like a sphinx server for example. How would you go about launching these (e.g. in case of a server restart)?
Can something like Foreman be used for this task?

Torey Heinz over 13 years ago

The key is to use foreman to export the proper config for your server http://ddollar.github.com/foreman/#UPSTART-EXPORT

bertomartin over 13 years ago

weblee over 13 years ago

If Linode is your thing another quick way to setup a VPS is to use a StackScript.

http://www.linode.com/stackscripts/view/?StackScriptID=1291

Includes:
*Rails 3
*Ruby 1.9.2
*Nginx with Passenger
*MySQL
*git
*Updates rubygems
*Install rails 3
*Install mysql gem
*Add deploy user

Nice work again Ryan but would love a full cast on deployment.

einstyne over 13 years ago

I've never commented here before, but felt like this episode was worth the comment.

Fantastic. I wish this had been done weeks ago prior to my deployment headaches.

Lastly, I've been a fan of the railsready script found here: https://github.com/joshfng/railsready

Kevin Bende over 13 years ago

What are some security concerns for VPS (linode) vs managed solution (heroku)?

Tim Lovett over 13 years ago

Linode is less secure by default until you do some configuration changes when you get the box -- it comes very unsecure -- Ryan didn't go into it in the screencast (probably for the purposes of time) but you should lock down your VPS root access and only really sudo up to it from another user. Changing your SSH port and turning off password authentication can help too. Also setting up a firewall (even just ufw) will help lock it further down. You may also want to add some blacklisting software to handle multiple login attempts.

http://library.linode.com/security/basics

I'd follow those steps when setting up.

As for Heroku it's a great solution that makes scaling very very easy. The only problem I've had with it is you tend to have to bend off backwards at times to get things working (asset pipeline issues among other things) and you will bump into issues with some gems needing to be configured differently. It's also much pricier relative to using a solution such as Linode.

It does have free plans, however, so you should really try it out and see if you like the workflow. You pretty much make a local git repository for it (or just add another remote host to an existing) and push to it. It handles bundling, getting the server up and running, etc. You can even set environmental variables directly from the command prompt for the server using: heroku config:add KEY_NAME=value

Short answer: Heroku handles all that configuration for you since you don't get root access.

Kevin Bende over 13 years ago

Thanks for the information, will definitely look into the linode security tips!

Kasper 'kapöw' Grubbe about 13 years ago

Sorry, but calling that setup 'Very unsecure' is a big exaggeration. It is not optimal, but definitely not insecure.

Jaimie van Santen over 13 years ago

For some reason I'm getting this while doing a cold deploy, deploy:setup succeeds however, the repo is on the same server. Tried referencing it through the full ssh url or by giving a local absolute url, no avail.

          shell
        
$ cap deploy:cold 
  * executing `deploy:cold'
  * executing `deploy:update'
 ** transaction: start
  * executing `deploy:update_code'
    updating the cached checkout on all servers
    executing locally: "git ls-remote /home/deployer/projects/blog master"
fatal: '/home/deployer/projects/blog' does not appear to be a git repository
fatal: The remote end hung up unexpectedly
*** [deploy:update_code] rolling back
  * executing "rm -rf /home/deployer/apps/blog/releases/20120322194319; true"
    servers: ["192.168.1.49"]
    [192.168.1.49] executing command
    [192.168.1.49] sh -c 'rm -rf /home/deployer/apps/blog/releases/20120322194319; true'
    command finished in 186ms
Command git ls-remote /home/deployer/projects/blog master returned status code pid 9997 exit 128

Torey Heinz over 13 years ago

I would double check that /home/deployer/projects/blog is actually a git repo like the error says.

When creating the repo on the server I usually name the directory #{app_name}.git for clarity, then you need git init --bare to make sure the there is no working directory just the repo.

Paul over 13 years ago

Im getting this error with deploy:cold..any ideas?

failed: "sh -c 'cd /home/deployer/apps/burden_farming/releases/20120322200251 && bundle install --gemfile /home/deployer/apps/burden_farming/releases/20120322200251/Gemfile --path /home/deployer/apps/burden_farming/shared/bundle --deployment --quiet --without development test'" on 74.207.240.214

Paul over 13 years ago

Hey Guys,

Ryan helped me figure this out...I was having issues installing mysql, so I had to manually install the gem:

apt-get install libmysqlclient-dev

then I changed the mysql socket in database.yml to:

/var/run/mysqld/mysqld.sock

Hope that helps anybody who runs into this issue.

HaNdTriX about 13 years ago

thanks a lot (+1)

That was the last piece of my puzzle!
Now everything is running properly!

I am so proud!
Thanks Ryan & Paul

Liev about 13 years ago

Hi Paul,

I'm getting the same error, but after running the command you mentioned and retrying, i'm still getting it... the command ran properly as i see when trying to rerun it tells me it already has the latest version.

          ruby
        
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libmysqlclient-dev is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 42 not upgraded.

in my Gemfile i have:

gem 'mysql2'

and then the error when deploying:

failed: "sh -c 'cd /home/deployer/apps/sample_linode/releases/20120502195419 && bundle install --gemfile /home/deployer/apps/sample_linode/releases/20120502195419/Gemfile --path /home/deployer/apps/sample_linode/shared/bundle --deployment --quiet --without development test'" on 176.58.101.41

what do you think I'm doing wrong?

Liev about 13 years ago

OK, I got it to get past this point, but now i'm getting the following error when trying to run the migration.

rake aborted!
database configuration does not specify adapter

David Douglas almost 13 years ago

I got this error because my VPS server had installed Rails 3.2.8 when my app was running Rails 3.2.6.

Definitely check your Gemfile and your database.yml file (of course). The problem here is clearly stated---Rails is not communicating with your database specifically due to an adapter (aka gem)

Emmanuel BOUGEROLLE over 12 years ago

how have you passed this probleme ?

Ikenna Okpala over 13 years ago

Excellent!! as usual...

Would like contribute a little tweek to this with installing gems on production.

create a gemrc (touch ~/.gemrc)

and then copy paste this to avoid fetching the gem docs on install, update and sources, plus some other defualts i use.

          bash
        
---
:verbose: true
:bulk_threshold: 1000
install: --no-ri --no-rdoc --env-shebang
:sources:
- http://gemcutter.org
- http://gems.rubyforge.org/
- http://gems.github.com
:benchmark: false
:backtrace: false
update: --no-ri --no-rdoc --env-shebang
:update_sources: true

bertomartin over 13 years ago

Fayimora Femi-Balogun about 13 years ago

Paul over 13 years ago

Anybody know how to fire up the ruby console in the linode?

Mike Goggin over 13 years ago

Once you're ssh'd in you should just be able run irb at the command prompt.

JP Slavinsky over 13 years ago

In the rails app directory should be able to run

bundle exec rails c production

To access a ruby console on the production db

Paul over 13 years ago

Thanks JP - I was using this:

RAILS_ENV=production bundle exec rails c

but your is much easier to type :)

Nico Ritsche over 13 years ago

I prefer Heroku, so I can focus on development not setting up and maintaining servers. ;-)

Stan BIENAIVES over 13 years ago

Hello,
I am interrested in Heroku as well, it seems so easy and restful. My only concern is about the price, for a small commercial web-site, let's say 200 visit a day, some mailing processing and a bit of image processing, how much will I pay.

I would be greatful to have an estimation... could you help me?

thx a lot.

Nico Ritsche over 13 years ago

You can do a whole lot with a single dyno - and that is for free.

One dyno can serve one request at a time, that is for example 10 100ms requests/sec.
See here for more info: http://devcenter.heroku.com/articles/dynos.

You can also send mails from that one dyno, unless you send bulk mails. I recommend sending those in a background process to not block your web dyno(s). Also note that dynos drop requests that are longer than 30 secs.

A permanently running background process needs at least 1 extra dyno which is 35$/month. But if you don't need a permanent process you can programatically start a dyno when a job is added to the job queue and stop it once it's finished. That way you keep cost low, as dynos are paid per hour.

For image processing I recommend storing the images on Amazon S3 which is rather cheap, check their pricing info for details. Heroku is not suitable for storing files, as dynos are not persisted so you'd loose all uploads after dyno restarts or deploys.

Hope this helps.

Richard Schneeman over 13 years ago

Thanks Nico!

Dan Neumann over 13 years ago

I think the most immediate downside of Heroku is that the free tier comes with one dyno that must be booted back up after a short period of inactivity. In other words, the first user to visit your website in a while has to endure the wait until the dyno is spun back up.

Once you cough up the $35/month minimum for an extra dyno, then both dynos stay on. Just something to keep in mind when you're comparing practical price points.

Nico Ritsche over 13 years ago

There is a simple fix for this: Just use a pinging service that pings the site every minute or so. I do this with new relic. And voila, your app doesn't fall asleep anymore.

John Rees over 13 years ago

Shhh... that's a dev secret :P

Nico Ritsche over 13 years ago

I doubt heroku doesn't know about this possibility... ;-)

James Jelinek over 10 years ago

I'm not sure if heroku allows cron jobs or not but I had a similar issue with Passenger going to sleep after inactivity on a low traffic app. So in my crontab I added the following:

*/5 * * * * wget -O /dev/null -o /dev/null host.domain.com

This will issue a GET request every 5 minutes to your host and keep the app spun up. Note we output the results of wget to /dev/null to avoid any files being created on the server. Works like a charm.

James Jelinek about 13 years ago

Only thing I dislike about Heroku is the lack of Ruby 1.9.3 support

Michael van Rooijen about 13 years ago

Heroku is good in a lot of situations. However, what disappoints me a little is the lack of JRuby and Rubinius support. This is where EngineYard shines a little more. A lot of Ruby developers probably aren't aware how much more concurrency you can gain using a VPS running JRuby/Rubinius with Puma/Trinidad/TorqueBox rather than running Thin or Unicorn.

For example, TorqueBox (a JRuby-specific App Server) is a "real" application server that runs independently (it's practically a whole infrastructure, not something that just binds to a port), and you deploy your application to this app server. This one single instance uses threads (JVM, not MRI) for everything you would normally have to spin up processes for with MRI (which translates to Dynos on Heroku).

There have been people that took their worker farm (120 worker processes) from around 60gb in memory usage (yes, 60 gigabyte) spread across like 10 dedicated servers, down to about 1gb memory usage on a single server by switching from process-based scaling to thread-based with JRuby or Rubinius. That is huge. Not only does it scale well, you reduce a lot of costs, the infrastructure becomes more simple and imagine running 120 workers on Heroku, the bill goes through the roof. It really makes me wonder why there aren't more people supporting JRuby and Rubinius. (Note that this also applies to concurrent web requests of course, not just workers).

However, I definitely agree you when it comes to being more at ease when deploying to Heroku, simply because you cannot do any administrative tasks. When something breaks, needs updating, and what not, Heroku is the one taking care of it all while you sleep. It mainly depends on your application's requirements. If you have a basic app that doesn't really need a lot of concurrency (be it in the web process, or worker process) then it's all good. However, once you have use cases where you run large worker farms or tremendous amounts of traffic, it might change your game completely.

These are all interesting discussions, I guess basically it comes down to picking the right tools for the job and dealing with certain tradeoffs on each side. But it'd be good to let everyone know it's not just MRI, Thin and Unicorn out there.

What'd I'd love to see from Heroku is (real) support for JRuby (now) and Rubinius (once RBX hits 2.0), perhaps 1GB of ram per Dyno (at a higher price) and the ability to use TorqueBox flawlessly. I think this would be a total game changer for deployments and power per Dyno.

tl;dr:

Check out Ruby Implementations:
- JRuby
- Rubinius
Check out thread-enabled app servers:
- Puma
- Trinidad
- TorqueBox (this is pretty much one-of-a-kind, a beast, not comparable to any other Ruby app server to date)
@Heroku: Please add real support for JRuby and TorqueBox.

Diego Plentz about 13 years ago

Barry Jones over 11 years ago

For what it's worth, if you want to use Torquebox (I heart Torquebox) but want the hands off style of infrastructure that you can get with Heroku it might be worth it to check out OpenShift from RedHat.

It's very similar to Heroku but a little more flexible for your configuration needs. It's worth noting that it's also in the same Amazon data center that Heroku resides in, so any 3rd party services or databases should be able to connect just as easily within the network.

https://github.com/openshift-quickstart/torquebox-quickstart

Bharat Ruparel over 13 years ago

This has got to be one of the most thoroughly researched and comprehensive Screencast produced by Ryan and he shines.

Nico Ritsche has a great point though. As one lone developer working on a big project (relatively speaking of course), I can simply rely on Heorku to do all my sys admin work and focus on the development. The Developer's docs are also of a high quality and very helpful.

Perhaps Ryan can do one screencast on Heroku?

Stan BIENAIVES over 13 years ago

I would be greatful to have an estimation... could you help me?

thx a lot.

Bharat Ruparel over 13 years ago

I don't think you will pay much more than $40.00.
Though, as you see here, if you do this on Linode, the cost will easily be halved. This is a rough estimate based on my personal experience however and I may easily be off.

Stan BIENAIVES over 13 years ago

thx a lot

weblee over 13 years ago

I would love to use Heroku but there servers are US based and for SEO purposes my clients are all UK and UK based services.

Would be nice if Heroku could supply UK based service.

Nico Ritsche over 13 years ago

What has SEO to do with server location?

weblee over 13 years ago

It has everything. see link

Although this is a little old it still stands true.

Nico Ritsche over 13 years ago

I doubt that the server IP has much influence on search engine rankings these days. But I'm curious: does anyone have experience with this? Done some tests?

weblee over 13 years ago

We have tested yes! A resent client with heavy SEO development was ranking around bottom of page one for a number of years. With just the movement to UK server we have seen a 4 place growth with no extra SEO work done. Its a clear indication that it has an effect.

SEOMOZ still stands true with this as well if you search out there forums. But I do think It has come the day for it not to with cloud being so big now.

Nico Ritsche over 13 years ago

Okay, good to know. I hope that server location will lose SEO significance. No idea why it keeps having influence these days. Wonder what's the reasoning behind it.

will over 12 years ago

One issue could be speed. The latency for sites to get to our users is quite significant when hosting in the US and users are in Australia.

bertomartin over 13 years ago

Great job Ryan. Question: what if you're deploying to a test server first, then up to production? like so: Dev --> Test <--> Prod. The arrow goes both ways between test and prod because the prod database is sent to testing db and the testing code sent to prod server for releases.

Pushing from dev -> test is fine. But how about copying the code from test to prod? I don't want to ever push code from dev --> prod because multiple developers are working on the project and they're all at different points in the code.

I could simply scp the code over, but is there a more efficient way? Thanks all.

bertomartin over 13 years ago

Figured out a way from your recent post on Capistrano using Multistage-extension. thnx for that one.

Werner Laude over 13 years ago

In Europe we smile about the pricing of VPS and RootServers Hosting's in USA ...

Linode: 512mb, 20Gb, 200Gb Traffice => $19.95
Hetzner: 512mb, 20Gb, 1Tb Traffic => €7.90
Heroku..unpayable..

When you work with clients you def. also need an Adminplatform for them to install their own email accounts etc..

Anyway thanks for this screencast..and all the others. So helpful.
Werner

Aaron Cruz over 13 years ago

I pay €10.85 for my hetzner servers with 1024 MB RAM. They come highly recommended. They're in Germany.

Thorpe Obazee over 13 years ago

I might be asking too much but where should seeds be run?

Bharat Ruparel over 13 years ago

you can run rake db:seeds as a remote task under Capistrano's control. See the revised screencast.

Paul over 13 years ago

Im using carrier wave to store uploaded files, but when I do a deploy, those files get overwritten and are not longer accessible, cause they are in previous releases...I know I could use something like a3 for file storage, but I don't really require that right now...

Any ideas on what I might be able to do copy the previous deploys "upload" dir back into the "current" app?

Thanks!

Joe Jackson over 13 years ago

I just had this problem, you need to set capistrano to symlink youre public/whatever directory to a shared directory of your choosing.

See the answer to this stack overflow question
http://stackoverflow.com/questions/4648180/keep-unversioned-files-when-deploying-with-capistrano

Jonny Dalgleish over 13 years ago

anyone got a clue why this would happen to me?

sh -c 'cd /home/deployer/apps/appname/releases/20120324031224 && bundle exec rake RAILS_ENV=production RAILS_GROUPS=assets assets:precompile'
** [out :: 106.187.88.49] rake aborted!
** [out :: 106.187.88.49] FATAL: password authentication failed for user "deployer"

Can't figure it out to save my life. :)

Thorpe Obazee over 13 years ago

I'd check if you aren't typing the correct deployer password.

Jonny Dalgleish over 13 years ago

strangely its in a script and doesn't ask for a password... perhaps thats why?

Bharat Ruparel over 13 years ago

I would check the following:
1. Make sure that your target server, e.g., Linode is known to your source, e.g., GitHub server.
2. Make sure that you can follow your script manually, i.e., type in each command on the terminal and see where you get asked a password related question and try to use public/private key authentication to automate the authentication process.
That should help you isolate the problem.

Will Cosgrove about 13 years ago

Did you ever figure out what was going on there? I'm running into the same issue

Jere Salonen almost 13 years ago

All of a sudden I'm facing the same error as well, and it's happening on more then one projects, so shouldn't be in the asset files...any ideas anyone?

Patrick Glandien almost 13 years ago

The issue will be that Rails tries to connect to the database even just for assets precompiling, and the login for it fails. Check your database.yml so that it works when run on the web server etc.

Eugene Korneev about 13 years ago

check your database.yml file. Rake tries to connect to DB.

Chandresh Pant almost 13 years ago

I solved this by adding bundle exec like:

bundle exec cap deploy:setup

Cathy Zoller over 12 years ago

11 months later...the answer to this question is capistrano (as well as some other applications) expects to find a database owned by a user of the same name 'deployer' (read that in the postgres documentation). I'm wondering if you happen to be developing in windows so won't be using agent-forwarding, and I suspect this is the root cause of permissions errors.

A perhaps related problem...I'm getting a permission-denied error when the capistrano task deploy:start/restart attempts to run. Anyone know why? Anyone still read these 11 months later?

steve fox over 12 years ago

I still read these. :)

Did you resolve your permissions issue?

Mohamad El-Husseini about 12 years ago

Were you able to solve this issue? I'm running into the same problem.

Mohamad El-Husseini about 12 years ago

Did you solve this issue? I've having the same problem.

Cathy Zoller about 12 years ago

Yes I did. May have commented below somewhere but this is a windows specific problem because it doesn't support agent-forwarding. If you need more help I wrote everything down, but in short:

Make sure the postgresql database owner and password match the deployer username & password.
Then, I also needed to add
run "chmod +x #{release_path}/config/unicorn_init.sh"
to the symlink_config, so it would remember it has permission.
Set up the git repository on your vps, since again it won't forward from git.

cqcn1991 almost 12 years ago

How is the 3rd step? I'm getting the same problem on windows
Is it something like git clone?

Michael Dausmann over 11 years ago

I had the same issue. I found that I needed to do the # ssh setup steps before I attempted cap deploy:setup. In the video, the ssh is done after and appears to be convenience only but in the show notes its before.

Frederic ZINGG over 13 years ago

Problem with assets:precompile
Thanks to ryan I can deploy my app with capistrano now but evertime come the assets:precompile command it takes longer and longer.

In fact assets subdirectories are added everytime :

app/public/assets/assets/assets/assets/assets etc .....

It looks like it compiles all assets recursively.
I thought it because I have : config.assets.precompile += ['*.js', '*.css'] but even if without the problem shows up.

Any idea ?

Frederic ZINGG over 13 years ago

I just tried on local. No link with capistrano.
But the problem still remains.

From scratch, first compilation, it generates 3 levels of assets directory. /public/assets/assets/assets

Frederic ZINGG over 13 years ago

I found out, it's because in application.rb :
config.assets.paths << "#{Rails.root}/public"

OwnLocalDevelopers over 12 years ago

Thank you!! I don't understand why this is causing this problem though. I've commented this line, and it seems to work, but doesn't this line tell Rails where to find other non-compiled assets that should be served statically? What is the right way to configure this? I've posted on StackOverflow as well: