#271 Resque
- Download:
- source codeProject Files in Zip (206 KB)
- mp4Full Size H.264 Video (18.8 MB)
- m4vSmaller H.264 Video (12.1 MB)
- webmFull Size VP8 Video (13.8 MB)
- ogvFull Size Theora Video (28 MB)
In this episode we’ll take a break from our series on the new features of Rails 3.1 and take a look at Resque which is a great way to handle background jobs in Rails applications. We’ve covered several ways of handling background processing in previous episodes; each one serves a slightly different need and Resque is no exception. At the end of this episode we’ll give you some tips on choosing the right one to fit your needs but for now, let’s dive into Resque and add it to a Rails application.
The application we’ll be using is a simple code-sharing snippet site a little like Pastie. With this site we can enter a code sample and give it a name and language.
When we submit a snippet it is shown with the appropriate syntax highlighting for the selected language applied.
The syntax highlighting is handled by an external web service and it’s this part of the code that we want to be handled by a background job. It is currently executed inline as part of the SnippetController
’s create
action.
def create @snippet = Snippet.new(params[:snippet]) if @snippet.save uri = URI.parse('http://pygments.appspot.com/') request = Net::HTTP.post_form(uri, {'lang' => @snippet.language, 'code' => @snippet.plain_code}) @snippet.update_attribute(:highlighted_code, request.body) redirect_to @snippet, :notice => "Successfully created snippet." else render 'new' end end
The syntax highlighting happens when the snippet is saved. It uses the service available at http://pygments.appspot.com/, which was set up by Trevor Turk, to provide highlighting without using a local dependency. The code makes a POST request to the service, sending the plain code and the language, and populates the snippet’s highlighted_code
attribute with the response from that request.
Communicating with external services through a Rails request is generally a bad idea as they might be slow to reply and so tie up your entire Rails process and any other requests that trying to connect to it. It’s much better to move external requests out into an external process. We’ll set up Resque so that we can move the request into a Resque worker.
Getting Resque Running
Resque depends on Redis which is a persistent key-value store. Redis is fairly awesome in itself and well worth an episode on its own, but here we’re just going to use it with Resque.
As we’re running OS X the easiest way to get Redis installed is through Homebrew which we can do by running this command.
$ brew install redis
Once it has installed we can start up the server by running
$ redis-server /usr/local/etc/redis.conf
With Redis running we can add Resque to our application’s Gemfile
and then install it by running bundle.
source 'http://rubygems.org' gem 'rails', '3.0.9' gem 'sqlite3' gem 'nifty-generators' gem 'resque'
Next we have to add the Resque Rake tasks. We’ll do this by adding a resque.rake
file in our application’s /lib/tasks
directory. In this file we need to require "resque/tasks"
so that the gem’s custom tasks are loaded. We’ll also load the Rails environment when the workers start.
require "resque/tasks" task "resque:setup" => :environment
This gives us access to our application’s models in the workers. If we want to keep our workers light, however, it might be worth implementing a customized solution so that the entire Rails environment isn’t loaded.
We now have a Rake task that we can use to start up the Resque workers. To run it we need to pass in a QUEUE argument. We can either pass in the name of a specific queue that we want to work on or '*' to work on any queue.
$ rake resque:work QUEUE='*'
This script won’t show any output but it is working.
Moving The Web Service Code
Now that we have Resque set up we can focus on moving the code that calls the web service into a background process and handling it through a worker. We need to add a job to the queue so that the work is handled by a Resque worker. Here’s the create
action again.
def create @snippet = Snippet.new(params[:snippet]) if @snippet.save uri = URI.parse('http://pygments.appspot.com/') request = Net::HTTP.post_form(uri, {'lang' => @snippet.language, 'code' => @snippet.plain_code}) @snippet.update_attribute(:highlighted_code, request.body) redirect_to @snippet, :notice => "Successfully created snippet." else render 'new' end end
We add a job to the queue by calling Resque.enqueue
. This method takes a number of arguments, the first of which is the class of the worker that we want to use to handle the job. We haven’t created any workers but we’ll create one shortly and call it SnippetHighlighter
. We also need to pass any additional arguments that we want to pass to the worker. In this case we could pass in the snippet but whatever we pass in to enqueue
is converted to JSON so that it can be stored in the Redis database. This means that we shouldn’t pass complex objects like ActiveRecord models into here so instead of passing the whole snippet we’ll pass it’s id
and refetch the snippet in the worker.
def create @snippet = Snippet.new(params[:snippet]) if @snippet.save Resque.enqueue(SnippetHighlighter, @snippet.id) redirect_to @snippet, :notice => "Successfully created snippet." else render 'new' end end
Next we’ll create the worker and move the code that we’ve taken from the controller into it. We’ll put the worker in a new workers directory under /app
. We could put this class under the /lib
directory but by having it in /app
it’s automatically loaded and already in Rails’ loadpath.
A worker is just a class with two features. Firstly it needs an instance variable called @queue
that holds the name of the queue. This limits the queues that the worker handles. Secondly it needs a class method called perform
that takes the arguments that we passed to enqueue
, in this case the snippet’s id
. In this method we can put the code we extracted from the create
action that calls the remote server and returns the highlighted code, replacing the calls to the @snippet_id
instance variable with the local variable with the same name.
class SnippetHighlighter @queue = :snippets_queue def self.perform(snippet_id) snippet = Snippet.find(snippet_id) uri = URI.parse('http://pygments.appspot.com/') request = Net::HTTP.post_form(uri, {'lang' => snippet.language, 'code' => snippet.plain_code}) snippet.update_attribute(:highlighted_code, request.body) end end
Let’s try this out and see if it works. We’ll create a new snippet and submit it. When we do and we’ll see the snippet without any highlighting.
We won’t see the highlighting straightaway as it’s now done in the background. If we wait a few seconds and reload the page it’s still not applied so let’s try debugging the code to see what’s not working. Resque comes with a web interface, written in Sinatra. This makes is easy to monitor and manage its jobs. We can start it up by running
$ resque-web
When we run this the admin interface opens and we’ll see that we have a failed job.
If we click on the failed job we’ll see the details of the error: uninitialized constant SnippetHighlighter
.
The SnippetHighlighter
class isn’t being found and the reason for this is that we started the Rake task before we wrote it. Let’s restart it and see if it fixes the issue.
Once we’ve restarted the Rake task we can click the ‘Retry’ link to run the job again. When we do and we go back to the ‘Overview’ page there is still only one failed task listed so it appears that the job has run successfully this time. We can confirm this by reloading the last snippet we uploaded. This time the code is highlighted which means that the background job has run successfully.
If we upload a new snippet now it will be syntax-highlighted, though there may be a few seconds’ delay before the highlighting appears.
Embedding The Resque Web Interface
We now have Resque all set up and handling the jobs we give it. It would be useful though if we could embed Resque’s web interface into our Rails application so that we don’t have to start it up and manage another separate process.
Rails 3 works well with Rack applications and Sinatra is just another Rack application so it’s easy to do this by just mounting the application in the routes file.
Coderbits::Application.routes.draw do resources :snippets root :to => "snippets#new" mount Resque::Server, :at => "/resque" end
The Resque web interface will now be mounted in our Rails application at http://localhost:3000/resque
. We need to make sure that the Resque server is loaded for this to work and so inside the Gemfile
we’ll add a require option to the resque
gem.
source 'http://rubygems.org' gem 'rails', '3.0.9' gem 'sqlite3' gem 'nifty-generators' gem 'resque', :require => 'resque/server'
If we restart our application’s server now and visit http://localhost:3000/resque
we’ll see Resque’s web interface.
We don’t want this page to be publicly visible. How can we add some authorization to it to keep prying eyes out? If we were using something like Devise in our application it would be easy to secure this by wrapping the route in a call to authenticate
.
Coderbits::Application.routes.draw do resources :snippets root :to => "snippets#new" authenticate :admin do mount Resque::Server, :at => "/resque" end end
We’re not using Devise or any other authentication system in our application so instead we’ll use HTTP Basic Authentication. To do this we’ll create a new initializer in the config/initializers
directory called resque_auth.rb
.
Resque::Server.use(Rack::Auth::Basic) do |user, password| password == "secret" end
In this file we call use
on Resque::Server
, which is a Rack app, and add the Basic Authentication. In the block we check that the password matches. Obviously we can customize this code to set any username or password and add any other logic we want in here. When we restart the server and reload the page we’ll see the login prompt and we’ll need to enter the password we specified in the initializer to view the page.
Resque and its Alternatives
That’s it for our coverage of Resque. How do we choose between this and the other background processing tools available to us? One of the benefits of Resque is the admin interface that lets us monitor the queue, retry failed jobs and so on. Another reason to consider Resque is that Github uses it. Github handles a heavy load and so it should be able to handle anything we can throw at it.
If the Redis dependency is a problem then Delayed Job is a great alternative. This was covered in episode 171 [watch, read] and is an easier solution as it uses our application’s database to manage the job queue. It does, however, load the application’s Rails environment for all the workers. If we want to keep our workers light then this may not be the best solution.
Both Resque and Delayed Job use polling on the queue so there might be a slight delay between a job being added to the queue and when it’s processed. If the queue is often empty and we want jobs to be processed straightway then Beanstalkd, covered in episode 243 [watch, read], is the better option. Beanstalkd handles jobs through a push event so it can begin processing a job that’s added to the queue immediately.