#366 Sidekiq

Jul 18, 2012 | 10 minutes | Plugins, Background Jobs

Sidekiq allows you to move jobs into the background for asynchronous processing. It uses threads instead of forks so it is much more efficient with memory compared to Resque.

Click to Play Video ▶

Download:
source codeProject Files in Zip (59.6 KB)
mp4Full Size H.264 Video (30.6 MB)
m4vSmaller H.264 Video (13.4 MB)
webmFull Size VP8 Video (14.5 MB)
ogvFull Size Theora Video (29.4 MB)

Dom 10 months ago

Would be nice if Sidekiq could use PG instead of Redis.

Kasper 'kapöw' Grubbe 10 months ago

Wouldn't that require that PG got push/pull support for its clients?

Mike Perham 10 months ago

People have asked me to use ${FAVORITE_DATA_STORE} instead of Redis since day one of the project.

Sidekiq's mantra is simple and efficient. Having a storage layer with pg, redis, etc adapters and inherent performance trade-offs does not fit with those ideals.

Dom 10 months ago

Yeah I understand, and I'm not knocking Sidekiq. I've been using sidekiq since it's early days, and in a very big project. It performs flawlessly. I was just expressing a sentiment that's all... I would rather just have one database to install/maintain/monitor. Kudos to you for creating and sharing sidekiq.

hani elabed 10 months ago

Mike, thanks for Sidekiq and all your hard work.
It is my understanding that Redis is a memory only database, so if it crashes or the Rails app crashes, then you loose your queues, correct ? I could be wrong here or my info is outdated, please correct me as to how to deal with this kind of situation.
And let me say again, from watching the Railscast, and hearing about it on Ruby5, Sidekiq is awesome.

Mike Perham 10 months ago

I chose Redis for Sidekiq's storage because it's been extremely reliable for me in production. That said, I can't comment on how well or poorly it works when full because I've never hit that edge case and I'd caution against trusting the word of 2-3 year old blog posts. If running out of memory is a worry, you should load test that scenario and see how Redis actually performs.

Houen 10 months ago

You are both right and wrong. Redis is a memory-only key-value store, but you are free to specify dump intervals in your redis config where it saves its contents to disk. Check the docs for more.

Ohad Levy 10 months ago

Worth noting that celluloid only works on ruby 1.9

fawzyj 10 months ago

How this compared to GirlFriday? it uses the same concepts mainly

Michael van Rooijen 10 months ago

Sidekiq runs as a separate process like Resque, Delayed Job, Qu, QueueClassic, etc. GirlFriday runs in the same process as your app server.

andrewhavens 10 months ago

It's also worth mentioning that Girl Friday (along with a slew of other worker/queueing gems) were also written by Mike Perham. Sidekiq is his latest and greatest attempt at efficient message queueing in Ruby.

Damon Cali 10 months ago

Very cool, but check out the license before using commercially.

Derrick Johnson 10 months ago

Could you poll Sidekiq to determine when the job finished and update the syntax highlighting with ajax?

tspike 9 months ago

Potapov Sergey 10 months ago

We've ported resque-status gem to sidekiq. We're gonna open source it soon.

Trevor Hiley 10 months ago

Is anyone working on a rubber template for sidekiq? This would make deployment to ec2 much easier. I would do it myself if I knew what I as doing with Capistrano (still learning).

Thijs de Vries 10 months ago

Any reason why you could not just call Thread.new instead of using a job queue?

Diego Plentz 7 months ago

Calling Thread.new every time you need to do something async could create more threads then you probably want. Besides that, using sidekiq you can have a dedicated machine just to process the queues.

Joe Meirow 10 months ago

Has anyone faced this issue:

# example code
def member_logged_in(*args)
binding.pry
@member = Member.find(args[0])
@club = Club.find(@member.club_id])
mail(:to =>.....)
end

Under 'rails console', I can 'find' a member from the console prompt => @member.club_id returns a value.

When the same code is invoked via worker in Sidekiq, from the pry prompt, @member.club_id returns nil.

Thoughts?

Thomas Klemm 10 months ago

Joe asked this question on the sidekiq mailing list. Here are the responses:

By Darren Boyd:
It's possible this is a race condition with your ActiveRecord transactions.

If you are creating a job in an after_save (or after_create/update) callback, it's possible that sidekiq is starting the job before the database transaction is committed. Since the sidekiq doesn't see the DB change inside the job creator's transaction, the record isn't there (yet).

There's a longer explanation and a solution here: http://rails-bestpractices.com/posts/695-use-after_commit

Hope that helps.

By Mike Perham (Creator of Sidekiq):
Yep, we see this all the time at The Clymb. The answer is to use
either after_commit or delay_for(5.seconds).

terryjray 9 months ago

Looks very promising. Unfortunately, for Postgres users there is an open issue with pg documented here:

https://github.com/mperham/sidekiq/issues/164

and

https://bitbucket.org/ged/ruby-pg/issue/135/segmentation-fault-on-postgresql_adapterrb

Ran into this myself trying to implement sidekiq. Hopefully it will be resolved soon.

Diego Plentz 8 months ago

Already solved :)

Daniel Lorenz 8 months ago

Hi guys.
I tried sidekiq out for the first time. It is really cool. But I have a question. Which is the best way to starting sidekiq "bundle exec sidekiq" on server restart.

iamjoshua 8 months ago

Looks like the new version of the slim gem broke the monitoring sinatra app:
https://github.com/mperham/sidekiq/issues/411

Adding this to your gem file will fix it:
gem 'slim', '<= 1.3.0'

Just wanted to help others avoid the frustration until the docs are updated!

Wayne 8 months ago

This is fixed with slim 1.3.2 https://github.com/stonean/slim/issues/301

Anderson Dias 7 months ago

Hey, I've released sidekiq_mailer. A gem that handles asynchronous mail sending using sidekiq.

Robin Chou 7 months ago

Hi, interesting gem, but can you explain what advantages there are over using the delay extensions available by default?

https://github.com/mperham/sidekiq/wiki/Delayed-Extensions

Ellin Tolstov 7 months ago

I'm trying to set up Sidekiq to work with Monit, but it seems to be needing to cd to current_path before executing bundle exec. Does anyone know how to tell Monit to cd to my app's directory before executing the start command? The commands in Monit also fail if I try to start them without /usr/bin/env

          
        
check process sidekiq
  with pidfile <%= current_path %>/tmp/pids/sidekiq.pid
  start program = "/usr/bin/env bundle exec sidekiq -C <%= current_path %>/config/sidekiq.yml -P <%= current_path %>/tmp/pids/sidekiq.pid" as uid deployer and gid admin with timeout 90 seconds
  stop program = "/usr/bin/env kill -s INT `cat <%= current_path %>/tmp/pids/sidekiq.pid`" as uid deployer and gid admin with timeout 90 seconds
  if totalmem is greater than 800 MB for 2 cycles then restart # eating up memory?
  group sidekiq

Danial Pearce 6 months ago

You probably need an /etc/init.d wrapper for sidekiq. Whenever I want to monitor anything with monit, i write a simple wrapper for it. You can find them around pretty easily, plenty available for unicorn and other gems that require bundle exec functionality.

Akash Soti 3 months ago

I am unable to start sidekiq, it throws an error, "Timed out connecting to Redis on localhost:6379". My Redis server is running on port 6379. I am using mac os x 10.8.2.

First sign in through GitHub to post a comment.