#366 Sidekiq
Jul 18, 2012 | 10 minutes | Plugins, Background Jobs
Sidekiq allows you to move jobs into the background for asynchronous processing. It uses threads instead of forks so it is much more efficient with memory compared to Resque.
- Download:
- source code
- mp4
- m4v
- webm
- ogv
Would be nice if Sidekiq could use PG instead of Redis.
Wouldn't that require that PG got push/pull support for its clients?
People have asked me to use ${FAVORITE_DATA_STORE} instead of Redis since day one of the project.
Sidekiq's mantra is simple and efficient. Having a storage layer with pg, redis, etc adapters and inherent performance trade-offs does not fit with those ideals.
Yeah I understand, and I'm not knocking Sidekiq. I've been using sidekiq since it's early days, and in a very big project. It performs flawlessly. I was just expressing a sentiment that's all... I would rather just have one database to install/maintain/monitor. Kudos to you for creating and sharing sidekiq.
Mike, thanks for Sidekiq and all your hard work.
It is my understanding that Redis is a memory only database, so if it crashes or the Rails app crashes, then you loose your queues, correct ? I could be wrong here or my info is outdated, please correct me as to how to deal with this kind of situation.
And let me say again, from watching the Railscast, and hearing about it on Ruby5, Sidekiq is awesome.
I chose Redis for Sidekiq's storage because it's been extremely reliable for me in production. That said, I can't comment on how well or poorly it works when full because I've never hit that edge case and I'd caution against trusting the word of 2-3 year old blog posts. If running out of memory is a worry, you should load test that scenario and see how Redis actually performs.
You are both right and wrong. Redis is a memory-only key-value store, but you are free to specify dump intervals in your redis config where it saves its contents to disk. Check the docs for more.
Worth noting that celluloid only works on ruby 1.9
How this compared to GirlFriday? it uses the same concepts mainly
Sidekiq runs as a separate process like Resque, Delayed Job, Qu, QueueClassic, etc. GirlFriday runs in the same process as your app server.
It's also worth mentioning that Girl Friday (along with a slew of other worker/queueing gems) were also written by Mike Perham. Sidekiq is his latest and greatest attempt at efficient message queueing in Ruby.
Quick note for anyone scouring these comments for updates:
Check out Sucker Punch https://github.com/brandonhilkert/sucker_punch for simple background jobs that can run within the same process as your app.
Very cool, but check out the license before using commercially.
Could you poll Sidekiq to determine when the job finished and update the syntax highlighting with ajax?
+1
We've ported resque-status gem to sidekiq. We're gonna open source it soon.
Hello Sergey!
Look at Sidekiq Status. I tried to make very clean and use at little hooks as possible. It works well with scheduled jobs (perform_at/perform_in). It has nice sidekiq status web UI, allows to kill long running status jobs.
Is anyone working on a rubber template for sidekiq? This would make deployment to ec2 much easier. I would do it myself if I knew what I as doing with Capistrano (still learning).
Any reason why you could not just call Thread.new instead of using a job queue?
Calling Thread.new every time you need to do something async could create more threads then you probably want. Besides that, using sidekiq you can have a dedicated machine just to process the queues.
Has anyone faced this issue:
# example code
def member_logged_in(*args)
binding.pry
@member = Member.find(args[0])
@club = Club.find(@member.club_id])
mail(:to =>.....)
end
Under 'rails console', I can 'find' a member from the console prompt => @member.club_id returns a value.
When the same code is invoked via worker in Sidekiq, from the pry prompt, @member.club_id returns nil.
Thoughts?
Joe asked this question on the sidekiq mailing list. Here are the responses:
By Darren Boyd:
It's possible this is a race condition with your ActiveRecord transactions.
If you are creating a job in an after_save (or after_create/update) callback, it's possible that sidekiq is starting the job before the database transaction is committed. Since the sidekiq doesn't see the DB change inside the job creator's transaction, the record isn't there (yet).
There's a longer explanation and a solution here: http://rails-bestpractices.com/posts/695-use-after_commit
Hope that helps.
By Mike Perham (Creator of Sidekiq):
Yep, we see this all the time at The Clymb. The answer is to use
either after_commit or delay_for(5.seconds).
Looks very promising. Unfortunately, for Postgres users there is an open issue with pg documented here:
https://github.com/mperham/sidekiq/issues/164
and
https://bitbucket.org/ged/ruby-pg/issue/135/segmentation-fault-on-postgresql_adapterrb
Ran into this myself trying to implement sidekiq. Hopefully it will be resolved soon.
Already solved :)
Hi guys.
I tried sidekiq out for the first time. It is really cool. But I have a question. Which is the best way to starting sidekiq "bundle exec sidekiq" on server restart.
Looks like the new version of the slim gem broke the monitoring sinatra app:
https://github.com/mperham/sidekiq/issues/411
Adding this to your gem file will fix it:
gem 'slim', '<= 1.3.0'Just wanted to help others avoid the frustration until the docs are updated!
This is fixed with slim 1.3.2 https://github.com/stonean/slim/issues/301
Hey, I've released sidekiq_mailer. A gem that handles asynchronous mail sending using sidekiq.
Hi, interesting gem, but can you explain what advantages there are over using the delay extensions available by default?
https://github.com/mperham/sidekiq/wiki/Delayed-Extensions
I'm trying to set up Sidekiq to work with Monit, but it seems to be needing to
cdtocurrent_pathbefore executingbundle exec. Does anyone know how to tell Monit tocdto my app's directory before executing the start command? The commands in Monit also fail if I try to start them without/usr/bin/envYou probably need an /etc/init.d wrapper for sidekiq. Whenever I want to monitor anything with monit, i write a simple wrapper for it. You can find them around pretty easily, plenty available for unicorn and other gems that require bundle exec functionality.
I am unable to start sidekiq, it throws an error, "Timed out connecting to Redis on localhost:6379". My Redis server is running on port 6379. I am using mac os x 10.8.2.
Has anyone know how to move create method into worker?
Photo.create( params)
If you want to use sidekiq in https://teatro.io/ you need to create
Procfilelike this:Hi I am working with rails4,mongoid4,sidekiq . I am unable to start sidekiq ,when I run the bundle exec sidekiq I am getting the following error I am not sure what happend and what to do , can anyone help me
I fixed it by downgrading the sidekiq version to 3.1.x , I am getting issues with 3.2.x
Even I tried to define autoload paths , still getting the same error . I am calling worker from my controller
Can any one have an idea how to fix it
I have been getting the same problem
http://stackoverflow.com/questions/29056608/sidekiq-worker-class-not-working-with-carrierwave-upload
Two updates for this Railscast:
bundle exec sidekiq -q high,5 -q defaultYes, I had to read this:
https://github.com/mperham/sidekiq/wiki/Advanced-Options
It is useful to take a look of that documentation too.
Regards,
Daniel.
So I am trying to get this set up for my AWS EC2 instance, I currently have 2 tiers (staging / prod) and deploying with capistrano. Followed the tutorial video and some other tutorials elsewhere to supplement
Looking for some expert input on my approach and pointing out things that I might be missing. I manage to get Redis / sidekiq installed on the staging instance ( as I am confirming this works before moving onto production ).
I have one worker EmailSender
my deploy.rb
Then my sidekiq.yml
When I deployed, it correctly brings up the sidekiq instance - which I can confirm via ps aux | grep sidekiq...
But when I try to process the emailsender job...it seems like it isn't doing anything, wondering if anything above or if I am missing causing the job to not be picked up?
I tried to the perform job manually and it works.
Thanks!
A quick reminder about older versions of Redis' default security settings... be sure to secure your Redis server from outside access. (bind to localhost and set up AUTH as default installs were accessible from the outside).
See: http://redis.io/topics/security and http://antirez.com/news/96
Thanks Ryan.
Seems to be uri changed by Pygments.
here is the new link
http://pygments.simplabs.com/
I tried with the above link and it works.
Can anyone explain about below line of code-
:concurrency: 25
:verbose: false
:queues:
- [often, 7]
- [default, 5]
- [seldom, 3]
I'm facing an issue where data is interchanging when it processed through Sidekiq in backgroud but if I run it manually then no data issue.