#344 Queue Classic
Apr 24, 2012 | 8 minutes | Plugins, Background Jobs
PostgreSQL can act as a worker queue which can replace the need for a separate process to manage the background jobs. Here you will learn how to do this with the queue_classic gem.
- Download:
- source code
- mp4
- m4v
- webm
- ogv
Nice introduction to QC - thanks heaps. What's with that video poster though? ;-)
I'm definitely considering the jump to Postgres, it would be really nice if there was a PG engine for Faye.
So, the only advantage of QC over DJ is that in uses some internal postgres hooks and doesn't hit database every n seconds like DJ, right?
In that case DJ is still far more advanced and feature-rich and i see no reasons considering QC over it.
While I admit that QC does not have all of the features that delayed jobs boasts, I would also admit that it does not have all of the shortcomings as well. QC takes job locking quite serious and has proven to be more robust in high-volume scenarios. For instance, take a look at the PL/pgSQL functions that queue_classic uses to lock a job. All of the code in that function is tuned to ensure quick access to jobs while guaranteeing that no two workers can lock the same job. While the chance that delayed job will allow multiple workers to lock the same chance is low, it will happen eventually --especially in production environments that churn through tens of millions of jobs per day
Also, as Ryan Bates mentioned, queue_classic is not dependent on active_record. This means a whole lot when you want to write simple ruby programs that utilize a message queue.
Finally, we use queue_classic at Heroku because we have production systems that have daily message throughput in the tens of millions. We needed a queuing system that was reliable and easy to reason about; queue_classic satisfied both of these conditions.
Feel free to reach out if you have any more questions on how queue_classic might be able to help you out. @ryandotsmith or ryan@heroku.com
Awesome ! but, anyone can tell if queue_classic load the whole application in the workers ? I'm running on a small VPS and it's hard to have 80mb of RAM used by background job workers... just for send registration email (like DJ)
Answer to your question is in the screencast (between 6:30-6:50) and QC documentation; simply write your own worker "binary" (weird to use that term for a script file...) that only loads what's needed.
I'm experiencing issue with
require "your_app"what it should be required? the name of my application does not work
It must run with rake or "./bin/worker" ? (both does not work)
Ok so here we need to require mailers classes & action_mailer only, work great
You mentioned that the main advantage of queue_classic over something like resque would be that you don't need a daemonized process running all the time. I'm curious what you would use to poll the queue then? Wouldn't you daemonize the rake task you had running throughout the screencast? Or would you run that or the lightweight worker script from cron?
resque needs redis running all the time. that's the daemon in question. queue_classic only needs the database.
Really nice and handy episode (as usual !), keep up the good work.
Another queue system worth trying is sidekiq, it's quite close of resque but doesn't spawn a process per worker. It uses celluloid under the hood that would worth an episode too !
I wonder how easy it would be to use this with Apartment. Apartment currently supports DJ. Also, it would be cool to see an episode on Apartment. It'd fit right into the current Postgres theme.
I don't see any PostgreSQL specific features here.
MySQL and any another RDBMSes can be used to achieve the same thing AFAICT.
PostgreSQL specific features used include:
SELECT FOR UPDATE NOWAITandLISTEN/NOTIFY. Take a look at the PL/pgSQL function that is called to lock a job. https://github.com/ryandotsmith/queue_classic/blob/master/sql/ddl.sqlCool. Oracle also supports
SELECT FOR UPDATE NOWAIT. It also has a feature named Database Change Notification that is similar toLISTEN/NOTIFY.Ah yes. Oracle.... Oracle certainly has these features and more. However, I typically exclude Oracle from comparisons when talking about Ruby related programming. Oracle has so many features but they are out of reach for simple projects that exist in a non-enterprise environment.
Does loading the QC functions once with a migration work for dev environments? We have it set up so we drop and reload them around db:schema:load and load them after db:test:load.
We are working to add support to allow symbols in job params (e.g. hash keys or values) when enqueuing jobs in QC (by turning them into strings before encoding with OkJSON).
We have a queue for sending emails, so we'll have to investigate using workers that aren't running the full environment for those to save memory. Great idea.
One thing to watch out for when using QC (or any background job manager) with Rails is jobs enqueued from after_create / after_save callbacks that use the new record. We've found that you must use after_commit instead to ensure that the record has been committed to the DB before the worker tries to find it by ID.
You are right about your input/comment - One thing to watch out for when using QC (or any background job manager) with Rails is jobs enqueued from after_create / after_save callbacks that use the new record. We've found that you must use after_commit instead to ensure that the record has been committed to the DB before the worker tries to find it by ID.
I am interested to see Ryan Smiths response or comments on this too.
How does one use queue_classic with rubber/capistrano? It seems like
rake qc:workneeds to run, just not sure where to put that command (in deploy.rb?) Thanks!I needed a:
ENV["QC_DATABASE_URL"] = "postgres://localhost/mailer_development" if Rails.env.development?
instead of:
ENV["DATABASE_URL"] = "postgres://localhost/mailer_development"
in config/initializers/queue_classic.rb
The two differences:
1) Only set the url in development so when I deploy to Heroku the production database url gets picked up.
2) Use QC_DATABASE_URL instead of DATABASE_URL. Apparently Rails actually uses DATABASE_URL instead of database.yml if DATABASE_URL is present. This will cause your tests to start failing, despite the Rails.env.development? guard, due to the initializer running prior to Rails.env getting set to "test". This is confusing, but put a puts Rails.env into the initializer and watch the output when you run rake test and you'll see "development" and then "test" as Rails runs your tests.
Hope that helps someone. Great Railscast as always Ryan!
thanks! ENV["DATABASE_URL"] = "postgres://localhost/mailer_development" doesn't work for me as well.
but i get "undefined method `load_functions' for QC::Queries:Module" when trying to migrate, why is this?
take a look at :
https://github.com/ryandotsmith/queue_classic/blob/master/lib/queue_classic/setup.rb
I think the names have changed to :
QC::Setup.create_functions
Thanks Yoda! I have this problem and also my postgres installation is kind of weird so was all messed. but thanks to you i can narrow down the prob and its now fixed!
How can I use queue_classic to send emails? I can pretty much make queue_classic call any method and Queue it successfully, but when it comes the time to call an ActionMailer method or a class method that wraps an ActionMailer call, it simply Doesn't work!
is there an additional configuration needed?
PS. I can see the method being queued in the queue_classic_jobs table and then being removed, but it basically does nothing(does not send the e-mail) :(
Sure I have used it to send email, and it works great.
It was pretty simple. The send_confirmation looks up the user, and then gets their email address and whatever other information needed.
QC.enqueue("UserMailer.send_confirmation", @user.id)
You may want to enable delivery errors in your development.rb config file. Maybe there's a failure you are not seeing.
config.action_mailer.raise_delivery_errors = true
What is the best way of running a worker process on my vps? Is a detached screen session a good idea?
Should i add cron entry to lunch it on restart in case of a restart? Maybe there is a simpler way of doing it?
Try Foreman (https://github.com/ddollar/foreman)
I found a admin interface for queue classic.
https://github.com/rainforestapp/queue_classic_admin
Don't miss a better alternative: Que.