#129 Custom Daemon

Ryan,
As part of your series on background processing, have you considered covering daemon_controller (http://github.com/FooBarWidget/daemon_controller/tree/master) by the Phusion guys and how it might tie in with some of these other approaches?

Thanks and keep the great screencasts coming! :)

@Kevin, I'm planning to cover Monit or God to help manage daemon processes in the near future. I may cover daemon controller there too. Thanks for bringing this up.

thanks Ryan, I was using the OpenWFEru scheduler, but I preffer this method, this is exactly what I was looking for.

Ryan, nicely done. I'm very pleased to hear you are planning a future episode on monit and god. Thanks

Great episode.

Just a quick question - does this method load the rails env when the job is picked up?

Thanks.

Fred C - DG does load the rails env when the job is picked up. By default this is production, but you could change it to whatever you wished.

@fred, the Rails environment will be loaded when the daemon starts. At that point it will stay loaded for each job.

Hi Ryan. Thanks for great screencasts! I will need one of this solution for background processing, but I'm not sure witch one i should go with. My application will receive xml file via http upload. It's quit time critical data and it has to processed as soon as possible. I can have a daemon that process the files in a directory or go with the solution like workling/starling. /Jon

Ryan, in case you're planning to add a followup:

1) using this method is tricky if you have more than one schedule, e.g. one every 15sec, one daily and one weekly schedule. Unless you can live with the memory load of having 3 separate daemons (which you usually don't want).

2) add a auto startup/shutdown to the deploy scripts, I guess stopping via :before_update_code might be appropriate.

@Jon, if you need a job to be processed on demand then it sounds like Starling + Workling will be a good solution for you.

Another question to ask yourself is how frequently this task will be called. If you expect it to be only once an hour or less then you may want to try Backgroundjob as that will start/stop the Rails process each time. No need to have something constantly running in the background if you're not using it constantly.

@aleco, are you referring to recurring scheduled tasks? This solution was primarily designed for dynamically scheduled tasks where the frequency is not known up front. What you're referring to is definitely possible with one daemon, but the logic can get a little bit tricky in determining what task needs to be run at what given time.

You may want to look into BackgrounDRb for this. It has some cron-like scheduling which will work much better for this kind of thing.

Regarding the deployment, I honestly haven't tried doing this yet. Ideally you should just add a "lib/daemons/mailer_ctl restart" call to your deploy.rb restart task. I'm not certain if this works, if not then I think the fault is with the generator and should be fixed.

Using "restart" will do the trick, but if you use the daemon for longer DB tasks, it might be a good idea to stop the daemon before deploying and restarting it afterwards. Because otherwise you might end up migrating a DB you're updating from the daemon at the same moment. This is why I suggested to use :before_update_code and :after_update_code.

Oh, another hint: most people will want to add

RAILS_DEFAULT_LOGGER.auto_flushing = 1

otherwise they will spend hours trying to find out why their production log isn't updated, assuming their daemon code is buggy. The explanation is that production log uses MAX_BUFFER_SIZE = 1000 and no auto flushing, meaning that it will only write to disk once it has 1000 bytes in buffer.

Hi Ryan, thanks for the screencast.
Just an info. What about the simplest solution? I mean, what do you think to create a method in a given class and then class that method every X minutes by a cron job?
Is there any pros to go with a daemon or stuff like that instead of a cron job? (obviously for methods that can be called every X minutes)
What's the pros and cons of each approach?
(now i'm using a cron job :) )

This came at a perfect time for me. I'm now successfully using this method but wanted to comment that I had a horrible time debugging my daemon. It seems the log file spits out a lot of confusing errors. Sample:

*** below you find the most recent exception thrown, this will be likely (but not certainly) the exception that made the application exit abnormally ***
#<MissingSourceFile: no such file to load -- action_mailer/ar_mailer_helper.rb>
*** below you find all exception objects found in memory, some of them may have been thrown in your application, others may just be in memory because they are standard exceptions ***
#<NoMemoryError: failed to allocate memory>
#<SystemStackError: stack level too deep>
#<fatal: exception reentered>
#<LoadError: no such file to load -- active_support>

The confusing part is that, at least for my env, all of those can be ignored. The more confusing part is that I couldn't figure out how to get a backtrace out of my daemon :( So I spent a good chunk of time on the errors shown in the log only to later realize they were red herrings. I ended up having to do a lot of inspecting of my daemon and finally got it working.

Anyway, it was frustrating. Hopefully the debug situation will get better? All that said, this is a great solution for the problem I had :-D So was worth the trouble.

As you said in the "Advanced search" screencast it would be nice to clean up the search table from time to time. Is this a good way doing that? I mean shall I have a sleep time for about 24 hours?

:O

Thanks, Ryan! I've been using this for a couple of months now to do some pretty hairy background processing and it's been working well. I'm looking forward to seeing your screencast on monitoring these types of processes with God and Monit. Keep it up!

Hi Ryan,

I get this error message wheen i try to start the Daemon on windows machine.

http://gist.github.com/14555

Ryan,
I was able to start the daemon fine, changed code while($running) loop and saw the changes. But now I can't stop or restart the daemon using lib/daemons/whatever_ctl stop command. I had to force kill the process from terminal to stop the daemon.

@Rajesh,

As far as I know Daemons won't work on Windows, so you'd probably have to create a service and work through that (a much bigger problem than doing it the *nix way). I found this to give you a bit of help getting started, but really it should just tell you why it is better to use Linux or OS X for some things:

http://rubyforge.org/forum/forum.php?thread_id=14841&forum_id=320

My lame ISP (dreamhost) won't let me install gems.

#20

Thanks Carl.

I'm wondering how much memory usage this approach uses. BackgroundRB is the method I've had experience with but it took like 140mb of memory to run. Any ideas on how this approach is on memory?

I followed the instructions but when I start the daemon, I get this error:

http://pastie.org/292348

Any ideas on how I might get it working?

@Melvin: i have the same issue...

I also have the same issue as Melvin and lolcatz. Running on linux.

All I put in the daemon code were puts statements and it doesn't run. I have installed all the gems it complains about.

@melvin, @nick:
i completely removed daemons.yml and lib/daemons/gc_ctl.

here's my script/daemons and lib/daemons/gc.rb http://pastie.textmate.org/293636

removing options/daemons.yml solved my problem.

i know it's cheesy but works like a charm.

I'm unable to stop or restart the daemon. Anyone figure out how to fix this?

The log file also has this:
#<NoMemoryError: failed to allocate memory>
#<SystemStackError: stack level too deep>
#<fatal: exception reentered>
#<Errno::ENOENT: No such file or directory - /home/sthapit/workspace/rezz/log/mailer.rb.pid>

lolcatz, that didn't work for me

Thanks for posting the topic, it's great idea! I'm getting this in the log file smiliar to comments above, any ideas?
# Logfile created on Sun Nov 02 11:16:11 -0800 2008 by /
*** below you find the most recent exception thrown, this will be likely (but not certainly) the exception that made the application exit abnormally ***
#<Gem::Exception: can't activate activesupport (= 2.0.2, runtime), already activated activesupport-2.1.0>
*** below you find all exception objects found in memory, some of them may have been thrown in your application, others may just be in memory because they are standard exceptions ***
#<NoMemoryError: failed to allocate memory>
#<SystemStackError: stack level too deep>
#<fatal: exception reentered>
#<LoadError: no such file to load -- active_support>

This is fits my scenario perfectly. I did run into a problem in using this with Capistrano however. I get daemon buildup over time. Every time I do a "cap deploy" I end up with a few more daemons running around.

Any idea why this might be happening?

http://pastie.org/316048

Hi Brian, thank you very much for your great and helpful screencasts!

I want to add one hint for a problem I run into:

If your daemon requires Rails and it's not the latest version you actually have on your machine, the daemon startup fails, because the

require 'activesupport'

from the _ctl script first loads the higher version and afterwards the real daemon tries to load a lower one. Replacing this "require" by

gem 'activesupport', '=2.1.2'
require 'active_support'

did the job. Maybe something one could add to the templates / script_ctl somehow?

Your's Frank

Hi Ryan,

Thanks for the wonderful web cast. I am using daemon, instead of cron job, to periodically rebuild sphinx delta index. The reason is that my data update very frequently and I want to udpate the delta index every 10 seconds or so whereas cron job can not be scheduled by seconds.

I tried to run rake command in the daemon to rebuild index for sphinx. But the rake command is never successfully executed. The source code is as following. I just don't know what is going on.

#!/usr/bin/env ruby

ENV["RAILS_ENV"] ||= "development"

require File.dirname(__FILE__) + "/../../config/environment"

$running = true
Signal.trap("TERM") do
  $running = false
end

while($running) do
  time_start = Time.now
  system "/usr/local/bin/rake thinking_sphinx:index:delta &"
  time_finish = Time.now
  time_span = time_finish - time_start
  ActiveRecord::Base.logger.info "/usr/local/bin/rake thinking_sphinx:index --- #{time_span} seconds. \n"
  
  sleep 10
end

You mentioned in http://railscasts.com/episodes/127-rake-in-background that rake task is quite resource-consuming for frequent using. So I am wondering whether I am even in the right track or not.

Thank you very much. Any suggestion is appreciated.

cheers
--
Canvas

Hi Ryan,

I tried command
" system 'rake thinking_sphinx:index' " in IRB and controller action, it worked in both cases. It seems that it dose not work only in the daemon. Any ideas? Thanks.

Cheers
--
Canvas

I built another rake task and tried it in the daemon. But it dose not
work neither. It seems that the daemon just dose not like any rake
tasks in it for some reason. It really drives me crazy.

This is exactly what I need, but not every 5 seconds, how can I change the interval?

I've just changed to Ubuntu 8.10 from Win XP. I now need background.
For some reason this episode (only) will not download/play in Ubuntu ?
Is it corrupt ? or possibly encoded differently ? Ajacent episodes, 128,127, 130 play ok.... But 129 ! No go :(

In XP , it plays no problem.

Denis

Update _ just to clarify....
I *can* save it to my Ubuntu desktop and play it.

Denis
ps
These railcasts are brilliant. Take a bow Ryan.

I have a question. I need to set up recurring job which can be send any minute of the day. I need to load rails environment every minute as cron job is running every minute. Which is the best way to do this ? Cron/Daemon or Starling ?

Hey, just wondering if a queuing system could be used for uploading files? not sure how this would work and if so which one would be recommended?

If like me you want to debug the daemon and want full backtrace logs instead of the truncated logs then do:

log_output: true

in your daemons.yml file (from the daemon generator)

then you should have in your log directory a file named daemon-name.rb.output

which has full output including backtrace and line numbers for errors etc.

If like me you want to debug the daemon and want full backtrace logs instead of the truncated logs then do

Here is the error i am getting

 /Library/Ruby/Site/1.8/rubygems/requirement.rb:72:in `parse': Illformed requirement [""] (ArgumentError)
from /Library/Ruby/Site/1.8/rubygems/requirement.rb:96:in `initialize'
from /Library/Ruby/Site/1.8/rubygems/requirement.rb:96:in `map!'
from /Library/Ruby/Site/1.8/rubygems/requirement.rb:96:in `initialize'
from /Library/Ruby/Site/1.8/rubygems/requirement.rb:35:in `new'
from /Library/Ruby/Site/1.8/rubygems/requirement.rb:35:in `create'
from /Library/Ruby/Site/1.8/rubygems/dependency.rb:61:in `initialize'
from /Library/Ruby/Site/1.8/rubygems.rb:207:in `new'
from /Library/Ruby/Site/1.8/rubygems.rb:207:in `activate'
from /Library/Ruby/Site/1.8/rubygems.rb:1056:in `gem'
from lib/daemons/mailer_ctl:14

I get a:
/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- rails/generators (MissingSourceFile)
When i try to run the generator. (rails 2.3.8)

Installing from http://github.com/kozy/daemon_generator.git fixes the bug.

@Alfrenovsky - thanks for sharing the http://github.com/kozy/daemon_generator.git bugfix.

Hey guys,

if you find that you have rails 3 gems conflicting with your script/generate daemon mailer...
error like: http://pastie.org/1736483

Then this version of the plugin will work for you: https://github.com/jmazzi/daemon_generator
You can also solve the problem by implementing bundler into your rails 2.3.X application:
http://ryanbigg.com/2010/08/three-dot-oh/

I got it working using the first technique and didn't bother to try the second. happy coding!

Hey Ryan,

if you have time, it would be great if you could revise this episode!

I'm struggling with how to pass parameters through to the .rb script. Been battling away for weeks now, with no success. :-(

Ryan, this screencast is very clear! I need to know if work with custom daemons: https://github.com/mirasrael/daemons-rails is the "best" solution for synchronize a db in an app that read/write external hardware memory (Programmable Logic Controller). It is about a long running task (maybe years) and it is desirable not to break the execution at all along - ..... Or can I consider other solution that fit in this types of problems?
Thank You...