#271 Resque

Jun 20, 2011 | 12 minutes | Plugins, Background Jobs

Resque creates background jobs using Redis. It supports multiple queue and comes with an administration interface for monitoring and managing the queues.

Click to Play Video ▶

Download:
source codeProject Files in Zip (206 KB)
mp4Full Size H.264 Video (18.8 MB)
m4vSmaller H.264 Video (12.1 MB)
webmFull Size VP8 Video (13.8 MB)
ogvFull Size Theora Video (28 MB)

Anatoly Mikhailov over 13 years ago

I'd like to use that Redis installation script:

          bash
        
  $ cd /tmp
  $ git clone --depth=1 git://github.com/defunkt/resque.git
  $ cd resque
  $ rake redis:install dtach:install
  $ vim /etc/redis.conf
        "daemonize yes
         bind 127.0.0.1
         loglevel notice
         logfile /var/log/redis.log
         dir /home/app/public_html/application/shared/db_backup/" 
  $ cd /tmp
  $ wget https://gist.github.com/raw/892578/bf55748800e3ca812c5ad8233b933bd6283d3aff/redis.sh
  $ adduser --system --no-create-home --disabled-login --disabled-password --group redis
  $ mv /tmp/redis.sh /etc/init.d/redis
  $ chmod +x /etc/init.d/redis
  $ touch /var/log/redis.log
  $ chown redis:redis /var/log/redis.log
  $ update-rc.d -f redis defaults

  $ /etc/init.d/redis start

nessur over 13 years ago

This last part starts redis in the foreground?

Maybe my Ubuntu config is disagreeable somewhere.

nessur over 13 years ago

For a production grade environment, on an Ubuntu server, I installed redis on its own:

sudo apt-get install redis-server

which then also gave me the 'redis' script in /etc/init.d, let things run in the background, etc. Though, note that the redis.conf get's put in /etc/redis/redis.conf.

fschwahn over 13 years ago

I guess it makes sense to take precautions if the webservice you are using is down.
Is it possible to automatically re-queue the job if it fails because it cannot reach the webservice?

jrn over 13 years ago

fschwahn, see Resque, how to requeue failed jobs

Ryan, thanks, another clear and well structured episode.

Jean-Paul Bernadina over 13 years ago

Great screencast as always Ryan!

You might want to consider writing tutorials for SmashingMagazine.com or net.tutsplus.com on How to create a rails app that does X. They pay really well when it someone who knows as much about their niche like you do :)

Rajib Ahmed over 13 years ago

@Ryan,

Does eventmachine serve the same purpose? And when will you use eventmachine?

Great information :)

Mathieu Arnold over 13 years ago

Must be me, but lib/tasks/resque.rake has the content of config/initializers/resque_auth.rb.

Ryan Bates over 13 years ago

Oops! Fixed. Thanks for reporting.

Gus Gollings over 13 years ago

This fork of resque adds some nice features:

Vary the interval between polling queues
Skip queues under certain conditions
Expand queue name patterns into the real queue names
Specify the target queue when scheduling a job

Jaroslaw Zabiello over 13 years ago

Hi Ryan, have you ever considered doing a cast about Torquebox? Torquebox supports all Rack-based (J)Ruby frameworks and it comes with job scheduling and asynchronous task scheduling out of the box (no extra installs necessary) Torquebox is the most enterprise-oriented platform I've seen to date.

Ken Hart over 13 years ago

Does anyone else think the app directory is getting too big to fast? I'm worried things are getting thrown in there. If you're using this technique, plus carrierwave, maybe some caching, presenters, and emails, your app directory would look like this:

assets
controllers
helpers
mailers
views
uploaders
models
sweepers
workers
presenters

Jack over 13 years ago

Another great screencast from you, Ryan. Went through Resque's github Readme, saw the comparison between DJ and it, and got a couple of questions:

For few background jobs each costing a few hours to run, which is preferable, DJ or Resque?
Resque doesn't rely on Activerecord. It should work fine with MongoDB, right?

Tilo about 12 years ago

there are two GitHub projects which fork Resque and replace Redis with MongoDB:

https://github.com/streamio/mongo-resque

https://github.com/nfo/resque-mongo

This looks pretty awesome!

aselder over 13 years ago

Love Resque and your screencasts as always.

I know this screencast is focused on Resque and not necessarily Rails best practices, but wouldn't it be more appropriate to have the enqueuing logic to an after_create callback on the Snippet class?

Arcath over 13 years ago

Love the ep

As I don't use devise and HTTP Basic seems a bit weak for my uses I got the route to be conditional based on CanCan permissions:

http://www.arcath.net/post/Controlling_access_to_Resque_with_CanCan

Joe Johnston over 13 years ago

Big fan of Resque (and RailsCasts!).

Here's a slight modification to Arcath's CanCan support to check if a user can manage Resque instead of relying on route checking.

http://www.simple10.com/resque-admin-in-rails-3-routes-with-cancan/

Michael de Silva about 13 years ago

Hi Joe, I'm getting

undefined method `user' for nil:NilClass

This seems to be an issue with request.env['warden'].user - any ideas?

Peter Sankauskas almost 12 years ago

Same here... I can't work it out.

Ajasja Ljubeti? over 13 years ago

A good missing piece of this screencast would be an Ajax hook, so that the page would get automatically updated when the background worker completes it's job. (So one would not need to reload the page manually to see the highlighted snippet).

Best regards,
Ajasja

Florent Monbillard over 13 years ago

Hi Ryan, great screencast as always.
Another solution would be to use an AMQP server (like RabbitMQ), it's way faster to fetch the queue item as it's not polling but pushing to the subscriber (i.e. worker). It's maybe less light than Redis though, but it does the job as well.
And you are not tied to load the entire Rails environment too.
Thanks a lot and keep up the good work !

citizencompass over 13 years ago

Maybe the answer to this question is too obvious.In the video, when you submit the snippet, the page does not show you the colored code immediately but instead requires a refresh. For actual production site, how do you avoid having to refresh? Is there an easy way to implement a progress bar?

Frédéric over 13 years ago

Based on the source code provided with this screencast (for the <%= yield(:head) %> in the app/views/layouts/application.html.erb ), update the app/views/snippets/show.html.erb to add:

          ruby
        
<% content_for :head do %>
  <meta http-equiv="refresh" content="5"><!-- refresh every 5s -->
<% end %>

into the else branch (the one with <pre><%= @snippet.plain_code %></pre>) that displays the plain code when the highlighted version is not available.

If you wan't a progress bar, you certainly would have to use some javascript, and have the worker in charge of the background processing to update some attribute with its progress.

Matthijs Langenberg over 13 years ago

What should you do when you have jobs that really should not be done by a web-app server (like resizing images and S3 uploading), but users are waiting for?

So while the backend benefits to be asynchronous, the user is actually waiting for the end-result.

Nicolai Seerup over 13 years ago

How would you recommend installing Redis on Ubuntu server?

mistergiri over 10 years ago

sudo apt-get install redis-server

Nicolai Seerup over 13 years ago

When I go to the resque interface, i don

jack dempsey over 13 years ago

I get the layout, but intermittently. Some refreshes don't request the style.css file for some reason. Not sure what's going on--has that happened to you at all Ryan?

nessur over 13 years ago

Anyone figure out what this problem is?

I'm running rails 3.0.9, ruby 1.8.7, with passenger.

nessur over 13 years ago

There is a /public directory inside resque's gem directory, and you need to symlink it to your application's /public directory.

That, or change this line:

config.serve_static_assets = true

in your config/production.rb file.

Having no desire to make/let rails serve assets, I went with the former option.

Chuck Vose over 13 years ago

Fantastic intro, we just started using resque and I was going to have to learn it the hard way. But then you show up with a great screencast and now I get to take a long lunch. :)

Thanks Ryan!

Cayce Balara over 13 years ago

Killer preso, as always. Beyond the obvious stellar info on Resque, the pointer to the pygments syntaxer was a great take-away for me. Thanks, Ryan.

nessur over 13 years ago

This comment is slightly implementation specific, but if anyone is trying to use Postgres with Passenger for a production environment, Resque workers seems to lose their DB connections when the forking happens.
see
https://github.com/rails/rails/issues/1339

Basically, everything can still work properly, but your resque.rake file needs to be setup like this:

require 'resque/tasks'
require 'resque_scheduler/tasks'

task "resque:setup" => :environment do  
  Resque.before_fork = Proc.new { ActiveRecord::Base.establish_connection }
end

Josh Sharpe over 13 years ago

Hi Ryan,

In staging/production passenger doesn't serve up the static assets associated with the resque server after mounting it in routes.rb. Otherwise, it works just fine, just no styles/javascript. Is there a workaround for this?

Thanks!

Jesse Wolgamott over 13 years ago

There's a public folder in ..../gems/resque-1.17.1/lib/resque/server/public -- if you copy those files to RAILS_ROOT/public/resque it'll show in production.

Joseph Pawletko over 13 years ago

Thank you for this, Ryan.
Really, really useful.

David Gerber over 13 years ago

I am using resque-scheduler for scheduling. Is anyone using this? I am getting this error message when I submit to a queue.

(Job{news_scraper_queue} | NewsScraper | []) failed: #

has anyone tried to use scheduling with the workers being in the app/workers directory?

coin3d over 13 years ago

Wouldn't it be possible to use ruby threads for the use case shown in the cast?

Michael de Silva about 13 years ago

Here's how I hardened the login via resque_auth.rb. You'll need the bcrypt-ruby gem installed.

Fire up IRB or PRY
require 'bcrypt'
Generate a salt: salt = BCrypt::Engine.generate_salt
puts salt // make sure you copy this somewhere.
Now, salted_hash = BCrypt::Engine.hash_secret("YOUR PASSWORD GOES HERE", salt)
make sure you save the longer output!

Update resque_auth.rb as follows:

          ruby
        
Resque::Server.use(Rack::Auth::Basic) do |user, password|
  
  [SALTED_HASH] == BCrypt::Engine.hash_secret(password, [SALT])
  
end

Make sure you replace the above respectively. Voila!

Michael de Silva about 13 years ago

A slight improvement:

          ruby
        
Resque::Server.use(Rack::Auth::Basic) do |user, password|
  if ['admin'].include? user do
    [SALTED_HASH] == BCrypt::Engine.hash_secret(password, [SALT])
  end
end

Since it's a mounted sinatra app, I guess inheriting from AdminController won't be possible?

Onno Faber about 13 years ago

Anyone knows how to keep the redis process alive? I'm using Deamon Controller for a sphinx server, but have to say that I'm running into a lot of small problems (eg. using Capistrano to deploy the code).

I'm not deploying to Heroku right now (we're in Europe), so still need to figure this out myself. :)

Jamie Cobbett about 13 years ago

I wanted only Users with the admin attribute set to true to access resque-web.

Since we're using Devise, the current user is accessible like this:

          ruby
        
request.env['warden'].user

So I made the below off the back of the example here:
http://www.scottw.com/securing-resque-server-rails-3

          ruby
        
require 'resque/server'

class SecureResqueServer < Resque::Server
  before do
    redirect '/' unless request.env['warden'].user.admin
  end
end

If we were using Devise's "roles" (where each role has it's own table) we could simply do this in the routes file:

          ruby
        
authenticate(:admin) do
  mount Resque::Server, :at => "/resque"
end

C R about 13 years ago

Thanks. If you use .try(:admin) instead, the redirect works correctly even if you aren't logged in:

          ruby
        
require 'resque/server'

class Resque::SecureResqueServer < Resque::Server
  before do
    redirect '/' unless request.env['warden'].user.try(:admin)
  end
end

That also uses the Resque:: namespace and is loaded by Rails automatically if you use it in e.g. app/initializers/resque_auth.rb.

In your routes.rb, you would include:

          ruby
        
mount Resque::SecureResqueServer, :at => "/resque"

John Dyer about 13 years ago

I am not sure I am clear on how you would update the view w/ the progress of a job, or the results of a job? Having to refresh is obviously not an ideal implementation, bu it seems like you did that in this screencast as well as the BeadStalkd + Stalker one... Am I missing something ? Do you have any suggestions on how I could update the view automagically when the job is complete?

Thanks!

-John

RussellSnow almost 13 years ago

I added a task just like shown and it works great. So I needed another and it refuses to recognize the second one. The first call to Rescue works but the second barf because it thinks it a member of the model. It will not add a second queue, nor will it work if I put it in the first one.

          ruby
        
    if params[:commit] == "Upload File"
      Resque.enqueue(Import360,params[:file].tempfile.path,current_user.id, cookies['Neighborhood'],cookies['Storm'],cookies['Company'])
    else
      Resque.enqueue(FusionTableSync)
    end 

gives the error:
uninitialized constant LocationsController::FusionTableSync
Any ideas?

RussellSnow almost 13 years ago

The trouble was I had not named the worker file correctly. Sometimes convention over configuration can be a pain to debug.

Leszek Zalewski almost 13 years ago

A note to myself, also maybe someone else will find it helpful.

If you wondered how to render resque-web inside your own layout, then use an iFrame with src to resque_server_url. Works like a harm. :)

Akshay Rawat over 12 years ago

More applications of Redis: Obie at RailsConf

Tilo about 12 years ago

there are two GitHub projects which fork Resque and replace Redis with MongoDB:

https://github.com/streamio/mongo-resque

https://github.com/nfo/resque-mongo

This looks pretty awesome!

Not sure which one is more stable and/or feature-complete..

Ευάγγελος Αυγερινός about 12 years ago

Hi.all :)

Does anyone know how to deploy this project on Heroku?

Thanks in advance!

Sadiksha about 12 years ago

Thank you for the railscast. I am having some problem redirecting the job to the resque-worker. I followed the similar steps as shown in the railscast. When I do

          ruby
        
Resqueue.enqueue(SnippetHighlighter, @snippet.id)

I get the error that SnippetHighlighter is undefined in the rails server, not in the resque-web. I also have problem opening resque-web. It gives me strange error says:
/usr/lib/ruby/1.8/net/http.rb:560: in 'initialize': getaddrinfo: Name or service not known(SocketError).

Does anyone here know what is wrong.? I am using Ubuntu 12.04.

miraclecoder almost 12 years ago

here's a nice slideshare presentation that covers rspec among other details: http://www.slideshare.net/homanj/background-jobs-with-resque

Eric Peterson over 11 years ago

I'm curious: How can the tasks be configured to be more lightweight? (i.e. not loading the entire rails environment)

willmoss about 11 years ago

This tutorial needs information about how to configure redis to work with resque, otherwise you're going to get errors.

From: http://hafizbadrie.wordpress.com/2011/03/23/implementing-resqueredis-on-rails-3/

Create redis.yml in config folder :

defaults: &defaults
host: localhost
port: 6379

development:
<<: *defaults

staging:
<<: *defaults

production:
<<: *defaults

Create resque.rb in config/initializers :

Dir[File.join(Rails.root, 'app', 'jobs', '*.rb')].each { |file| require file }
config = YAML::load(File.open("#{Rails.root}/config/redis.yml"))[Rails.env]
Resque.redis = Redis.new(:host => config['host'], :port => config['port'])

Sandeep Kumar almost 11 years ago

How to start resque in verbose mode for production.

Thanks in advance!!

Gustavo Caso over 10 years ago

Great screencast Thanks Ryan !!!

Chris P almost 10 years ago

Hi - I cannot run the rake task and I cannot find much on the internet that can assist me with resolving this. it fails loading this file which I dont have.. any ideas would be appreciated

admin$ QUEUE=* rake resque:work --trace
** Invoke resque:work (first_time)
** Invoke resque:preload (first_time)
** Invoke resque:setup (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute resque:setup
** Execute resque:preload
rake aborted!
LoadError: No such file to load -- helpers
/Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:247:in require' /Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:247:inblock in require'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:230:in block in load_dependency' /Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:633:innew_constants_in'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:230:in load_dependency' /Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:247:inrequire'
/Users/admin/sites/extendity/app/controllers/account_controller.rb:1:in <top (required)>' /Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:443:inload'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:443:in block in load_file' /Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:633:innew_constants_in'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:442:in load_file' /Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:342:inrequire_or_load'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:307:in depend_on' /Users/admin/.rvm/gems/ruby-2.1.2/gems/activesupport-4.1.4/lib/active_support/dependencies.rb:225:inrequire_dependency'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/railties-4.1.4/lib/rails/engine.rb:468:in block (2 levels) in eager_load!' /Users/admin/.rvm/gems/ruby-2.1.2/gems/railties-4.1.4/lib/rails/engine.rb:467:ineach'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/railties-4.1.4/lib/rails/engine.rb:467:in block in eager_load!' /Users/admin/.rvm/gems/ruby-2.1.2/gems/railties-4.1.4/lib/rails/engine.rb:465:ineach'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/railties-4.1.4/lib/rails/engine.rb:465:in eager_load!' /Users/admin/.rvm/gems/ruby-2.1.2/gems/resque-1.25.2/lib/resque/tasks.rb:61:inblock (2 levels) in '
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:240:in call' /Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:240:inblock in execute'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:235:in each' /Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:235:inexecute'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:179:in block in invoke_with_call_chain' /Users/admin/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/monitor.rb:211:inmon_synchronize'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:172:in invoke_with_call_chain' /Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:201:inblock in invoke_prerequisites'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:199:in each' /Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:199:ininvoke_prerequisites'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:178:in block in invoke_with_call_chain' /Users/admin/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/monitor.rb:211:inmon_synchronize'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:172:in invoke_with_call_chain' /Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/task.rb:165:ininvoke'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/application.rb:150:in invoke_task' /Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/application.rb:106:inblock (2 levels) in top_level'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/application.rb:106:in each' /Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/application.rb:106:inblock in top_level'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/application.rb:115:in run_with_threads' /Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/application.rb:100:intop_level'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/application.rb:78:in block in run' /Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/application.rb:176:instandard_exception_handling'
/Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/lib/rake/application.rb:75:in run' /Users/admin/.rvm/gems/ruby-2.1.2/gems/rake-10.4.2/bin/rake:33:in'
/Users/admin/.rvm/gems/ruby-2.1.2/bin/rake:23:in load' /Users/admin/.rvm/gems/ruby-2.1.2/bin/rake:23:in'
Tasks: TOP => resque:work => resque:preload

Chris P almost 10 years ago

never mind on the previous error. i just had a reference to include a file which didnt exist. I removed the include.

I have another issue though

I am using Resque.enqueue_to(queue_name, ClassName, params) and I get this error:

wrong argument type Class (expected Module)

my class file has this:
class MyClass
def self.perform(parameters, item_id)

etc

alexmoruz almost 10 years ago

How deploy that?
What specify in deploy.rb or how call lib/taks/resque.rake

dheena over 9 years ago

How to use multiple resque with single application.

My aim is:
I am using multiple database postgres in single application.
Is that possible to have multiple resque (each redis for each postgres database)

Sajjad Murtaza over 9 years ago

You should have to pass @snippet instead of snippet_id.

By this you don't need to again

snippet = Snippet.find(snippet_id)