#127
Sep 15, 2008
43 comments

Rake in Background

In need of a background process? You may be able to accomplish this with a simple Rake task. See how in this episode.
Tags: tools controllers performance
Download (15.5 MB, 10:38)
alternative download for iPod & Apple TV (12.4 MB, 10:38)

Resources

Note: Ending the rake command with “&” will not work in Windows. Instead You’ll need to prefix it with “start” to run the command in the background. See the comments below for details.

# controllers/application.rb
def call_rake(task, options = {})
  options[:rails_env] ||= Rails.env
  args = options.map { |n, v| "#{n.to_s.upcase}='#{v}'" }
  system "/usr/bin/rake #{task} #{args.join(' ')} --trace 2>&1 >> #{Rails.root}/log/rake.log &"
end

# mailings_controller.rb
def deliver
  call_rake :send_mailing, :mailing_id => params[:id].to_i
  flash[:notice] = "Delivering mailing"
  redirect_to mailings_url
end

# lib/tasks/mailer.rake
desc "Send mailing"
task :send_mailing => :environment do
  mailing = Mailing.find(ENV["MAILING_ID"])
  mailing.deliver
end

# models/mailing.rb
def deliver
  sleep 10 # placeholder for sending email
  update_attribute(:delivered_at, Time.now)
end

RSS Feed for Episode Comments 43 comments

1. Towbie Sep 15, 2008 at 05:10

Finally the series I've been waiting for so long! Thanks a lot for your brilliant screencasts! They're one of the best Rails resources I know and extraordinarily well prepared.

And, by the way, nice new intro!

2. Cassiano Sep 15, 2008 at 07:25

Ryan,

Very nice article!

I believe the forking (&) won't work under Windows. Any ideas on how to accomplish it in this case?

3. Nathaniel Bibler Sep 15, 2008 at 07:31

I was just about to mention, this won't work in Windows. The ampersand forking is a *nix only thing.

I don't believe there is a simple way to do this in windows, except to use the Ruby Process module with #fork. Not as easy, but similar functionality, just using the Ruby library.

http://www.ruby-doc.org/core-1.8.7/classes/Process.html#M000968

4. Martin S Sep 15, 2008 at 07:50

Thanks for these wonderful screencasts!

5. Jaime Iniesta Sep 15, 2008 at 07:56

Interesting alternative to backgroundrb or starling, seems easier to set up.

I use rake tasks, launched with a cron job, for my background processing, but this is suitable only for periodical tasks. Your solution is fine for user-launched tasks.

Thanks!

6. tony petruzzi Sep 15, 2008 at 07:59

for those asking about background processes in windows.

the best way to accomplish this from the command line is to use the "start" command.

http://www.ss64.com/nt/start.html

has a pretty good reference.

7. Jean-Marc Lagacé Sep 15, 2008 at 08:17

@Cassiano

if you want to "spawn" a process under windows you can do so by typing "start rake ..." then again you're better off having the whole path to the rake command.

Start will trigger a new command shell.

Jean-Marc

8. Cassiano Sep 15, 2008 at 08:38

@Jean-Marc (and others)

Thanks for the Windows tip.

It does work, except that the command shell window remains open after execution of the task, making subsequent executions be silently ignored (the window needs to be manually closed before clicking 'Deliver' again).

9. Henrik N Sep 15, 2008 at 08:40

Since it wasn't made fully explicit: the code in the screencast doesn't escape the env values (or keys), only upcases the key and puts some quotes around the value.

If regular users can pass in the parameters used, they can exploit it to run arbitrary code.

It was hinted at, but could be misunderstood to mean that the current code does proper escaping.

10. Ryan Bates Sep 15, 2008 at 08:52

@tony and Jean-Marc, thanks for pointing out how to accomplish this in Windows. I don't have access to one at home so I can't fully test these. I'll add this to the show notes.

@Henrik, thanks for clarifying this. Along that line, does anyone know of a good way to escape a string to be entered into a shell command?

11. Reuben Sep 15, 2008 at 09:55

This is great, but what about firing off processes whose status the user is interested in later? For example, whether it finished successfully, unsuccessfully, what percentage complete it is, etc.?

Perhaps you'll cover this in the upcoming background processing episodes, but it would be nice to have some sort of rails-model driven status mechanism that could be tied into the rake technique shown here.

12. Jean-Marc Lagacé Sep 15, 2008 at 10:44

@Reuben

Taking the Mailing as an example, you could use an attributed called status where the background task updates it and your web application displays the current value (you could do it with a periodical refresh).

13. Neil Sep 15, 2008 at 11:44

Would 'rake thinking_sphinx:index' or delta indices make for good uses of the approach outlined in this screencast? Or, are we better off with backgroundrb and the convenience of its :trigger_args intervals?

14. Amir Sep 15, 2008 at 11:55

I've got an excel file as a view that takes ages to render. Can I generate it in the background and direct it to disk so I can have a link to it appear later?

15. AC Sep 15, 2008 at 13:32

Unrealiable, suboptimal, fast and easy. Just the way the rails community likes it ;]

16. AC Sep 15, 2008 at 13:47

I really don't mean to be an asshole, but how hard is it to set up backgroundrb and do this properly ?

17. Ryan Bates Sep 15, 2008 at 14:07

@Reuben, this can be accomplished by regularly updating the database record associated to the task. In this case it could be the given Mailing record. True this could be extracted out into a generic jobs table, but at that point I think you're better off going with another solution (covering in future episodes).

@Neil, it depends on how often the indexing will be done. This approach is not optimal for frequent tasks, so if you're calling it on each record update then I'd say no.

@Amir, yep, that's definitely possible. The hardest part is notifying the user after it's done. One way to do that is just have the browser reload periodically (or ask them to come back later). You can update a database record at the end of the task so your app knows when it's done building and can display the link.

@AC, I plan to cover more "proper" ways on doing this kind of thing in the future. But as far as reliability is concerned, I haven't had any problems with this approach. Admittedly it is a bit of a hack, but it works okay (in my testing) and is much simpler than most alternatives.

18. Cassiano Sep 15, 2008 at 14:33

@Amir,

How about sending an e-mail to the user (with an embedded link) when it's done?

19. AC Sep 15, 2008 at 14:53

@Ryan: Sure - it works, when it works. But if a task is killed for whatever reason (say, power outage), you've lost it along with all the data it carried. That is bad enough. And unless you check your logs religously, you won't even notice. Hell, it might not even be reflected in the log, that the task wasn't completed successfuly. Apart from that, there's the memory consumption and CPU time wasted. Seems ugly to me. I wouldn't dare rely on something like this in a serious production environment..

20. Matthew Sep 15, 2008 at 16:48

You probably want to add 2>&1 into that command somewhere so that you capture STDERR into your log file as well as STDOUT.

I've noticed that you tend to use Rake for all command line tasks in place of just a straight call to a Ruby script. Is there any reason for this other than the ability to easily load the Rails environment ? Is there any overhead to running Rake as opposed to a straight Ruby script ?

21. Jeff Pritchard Sep 15, 2008 at 16:55

Hi Ryan,
Thanks for another great screencast. In your future ones about other ways to do background tasks, could you spend some time covering how/where to put the code that starts the process so that each time you redeploy your app it will automatically start up the background processes too? (and not wind up with a bunch of extra ones running either)

I've been using "spawn" for background stuff, but it requires a fair amount of manual starting and killing the way I'm doing it now. There must be a (much) better way!

thanks,
jp

22. Carl Sep 15, 2008 at 18:04

@Jeff,

Deamons is pretty easy to use and will prevent the "more than one copy running at a time" problem ( http://daemons.rubyforge.org/ ).

The Rails Way book goes into this a little bit (enough to get started anyways) and you can avoid most of the problems listed above by making updates to the database as tasks complete. If you have 1,000 emails you need to send and after you send each one you update the row to show it was sent you'd know where it stopped, and with only a little more work you could have it pick up there again latter, if you had to.

I suppose this wouldn't work as well if the tasks take a very long time to execute, but I generally find that if that's the case there are probably ways to make it faster. Some code that is inside a loop that should be outside, or a way to break it down into smaller tasks rather than one giant one. Ask for help if you get stuck, sometimes you just can't see the answer yourself because you are too close to the problem and need a fresh persective.

That's my 2 cents anyways.

23. Matthew Higgins Sep 15, 2008 at 20:14

Ruby 1.8.7 introduced Shellwords.escape and String#shellescape.

24. Michael Graff Sep 15, 2008 at 22:08

I believe the best way to protect yourself is to not escape at all, but instead use the multiple-argument syntax to system:

Rather than system("ls -l"), use:

system("ls", "-l")

I believe this will bypass the shell and therefore any risk of injected characters.

> system("ls", "`ps ax`")
ls: `ps ax`: No such file or directory

Clearly those back-ticks we so worry about were considered arguments, not parsed by the shell.

Warning though, if you pass those to poorly written shell scripts, you are still risking a lot. Be careful, perhaps converting known-integer arguments using params[:id].to_i

Also, should you write an 'escape' program, you might just consider not doing that at all, and instead using a whitelist (not blacklist) to pass only specific character types through, such as A-Z a-z 0-9 and so on. Whitelisting is preferred because if you miss one, the app won't work, but if you blacklist you could leave a hole open.

25. Trevor Turk Sep 15, 2008 at 22:52

Totally awesome new intro!

26. Carl Sep 17, 2008 at 12:29

Since we were just talking about optimizing Rails code I thought I'd post this:

http://lsrc2008.confreaks.com/02-james-edward-gray-ii-hidden-gems.html

The guys at Railsenvy.com just posted a video about the talks that were done at Lone Star Ruby Conf and Confreaks have their videos of the individual talks online now.

27. Ryan Bates Sep 17, 2008 at 12:39

@Matthew, good point. I'll add that to the code in the notes. As for using Rake, I generally prefer it for two reasons: 1) it's an easy way to load the Rails environment. 2) it is easy to organize and have built in descriptions.

There are definitely times I prefer not using Rake, especially when needing to pass a lot of arguments/options.

@Michael, I originally planned to pass the arguments as separate parameters to system, but then realized this doesn't allow you to end it with "&" or direct the output to something else. Anyone know of a workaround?

Also, good points about security. It's a good idea to be extremely careful whenever passing web parameters to a shell command.

28. Michael Graff Sep 17, 2008 at 13:54

@Ryan (re system and background tasks)

You will probably need to end up writing your own background thing, using something like:

Kernel.fork { system()... }

This is more or less what system() does internally. It will more than likely do a few more things (like closing specific file descriptors in the child, and making certain things are correct there.)

Another option is to have a simple shell script (forker.sh?) which you run instead of directly running rake. This shell script could even have commands in it to run "rake ... &" based on input.

Of course, it need not be a shell script, it could be a Ruby script too. All this is more complicated, but in all actuality, I dislike calling programs directly from the Rails framework unless they are intended to display something to the user, or to prod other processes into action.

I have done background tasks before (not in Rails, but in roll-your-own PHP coding) for a large DNS management system. Sometimes actions could take seconds and sometimes hours. We queued up the requests in a database table, and used PostgreSQL's NOTIFY command to tell listeners there was something to do. Doing a periodic (even once a second) SQL query was also added when we had to support MySQL as well. It wasn't a major delay.

29. Sam Millar Sep 20, 2008 at 15:51

Loving the new intro, wonderfully useful tutorial too!

30. Nathan Sep 25, 2008 at 18:37

@8 Regarding the start command:

You can make sure your cmd windows close after the emails are sent like this:

start cmd /c "title Email@%TIME% & cd c:/www & rake -T > emails.log 2>&1"

/c closes cmd window
2>&1 redirects STDERR

31. Asís García Oct 01, 2008 at 10:19

Hey Ryan, nice one! However it's not working under passenger here... Maybe its the way passenger processes reacts to the system call.

Any advice?

Thanks and keep up the good work!

32. Dennis Krupenik Oct 05, 2008 at 20:11

glad to note that new intro is way better ;)

thanks a lot for your work, keep it up

33. cheap wow gold Oct 29, 2008 at 02:57

good detailed post which show us the process.

34. warhammer gold Oct 31, 2008 at 02:55

good site

36. Justin Nov 04, 2008 at 11:32

@Asis
I had a similar problem using passenger and system() calls. I was on a CentOS box, and the root user owned my rails app. In passenger whoever owns the environment.rb file is the user passenger tries to run the rails app as, unless it is the root user. Then passenger runs as the PassengerDefaultUser, which is set to nobody. I changed that setting in my httpd.conf file to root and it worked fine. Obviously you don't want root running around doing things on a system exposed to the public. Try using chown and setting your rails app to someone who can run whatever command you want to run.

37. cheap wow gold Nov 05, 2008 at 01:09

well done.

38. Uzytkownik Nov 06, 2008 at 15:25

Is there any benefit of rake over the creation of new Thread?
- It is fast (it do not load separate env)
- It is cheap
- It is much less code to write and the code is much simpler
- You don't have to worry about injections
- You can create mutex and do not worry about 2 tasks in pararell

PS. Actually rake is better command then /usr/bin/rake. You don't know if the rake is in /usr/bin/rake (not in /usr/local/bin/rake) but there is great chance it is in $PATH.

39. warhammer power leveling Nov 16, 2008 at 19:17

<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy WAR gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Cheap Warhammer Gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy warhammer online gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> cheap warhammer online gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy warhammer gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> buy warhammer online gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> buy cheap warhammer gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> War Gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Buy Warhammer Gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Warhammer Online Gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Warhammer Gold for sale </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Cheap Warhammer Gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Account/Cheap-Warhammer-Account.html
"> Warhammer Accounts </a></p>
<p><a

40. warhammer time card Nov 17, 2008 at 21:56

<p><a href="http://www.bysneaker.com/"> Air Jordan Shoes</a> </p>
<p><a href="http://www.gmbar.com/">WoW Gold</a> </p>
<p><a href="http://www.gmbar.com/">WoW Power Leveling</a> </p>
<p><a href="http://www.gamevive.com/">WoW Gold</a> </p>
<p><a href="http://www.gamevive.com/">WoW Power Leveling</a> </p>
<p><a href="http://www.gmbar.com/"> Warhammer Time Card </a> </p>
<p><a href="http://www.gmbar.com/"> Warhammer CD Key </a> </p>
<p><a href="http://www.gmbar.com/cheap-war-gold.php"> Warhammer Gold </a> </p>
<p><a href="http://www.gmbar.com/cheap-war-gold.php"> Warhammer Online Gold </a> </p>
<p><a href="http://www.gmbar.com/cheap-war-gold.php"> Warhammer Accounts </a> </p>
<p><a href="http://www.gmbar.com/cheap-war-gold.php">Warhammer Power Leveling</a> </p>
<p><a href="http://www.gmbar.com/cheap-war-gold.php"> Warhammer Online Key </a> </p>
<p><a href="http://www.gmbar.com/cheap_war_gold.php"> WAR gold </a> </p>
<p><a href="http://www.gmbar.com/cheap_war_gold.php"> warhammer gold </a> </p>
<p><a href="http://www.gmbar.com/cheap_war_gold.php"> warhammer online gold </a> </p>
<p><a href="http://www.gmbar.com/cheap_war_gold.php"> Buy WAR gold </a> </p>
<p><a href="http://www.gmbar.com/cheap_war_gold.php"> Buy warhammer gold </a> </p>
<p><a href="http://www.gmbar.com/cheap_war_gold.php"> Buy warhammer online gold </a> </p>
<p><a href="http://www.gmbar.com/war-accounts.php"> WAR Accounts </a> </p>
<p><a href="http://www.gmbar.com/war-accounts.php"> warhammer Accounts </a> </p>
<p><a href="http://www.gmbar.com/war-accounts.php"> warhammer online Accounts </a> </p>
<p><a href="http://www.gmbar.com/war-accounts.php"> Buy WAR Accounts </a> </p>
<p><a href="http://www.gmbar.com/war-accounts.php"> warhammer Accounts for sale </a> </p>
<p><a href="http://www.gmbar.com/war-power-leveling.php "> Warhammer Power Leveling </a> </p>
<p><a href="http://www.gmbar.com/war-power-leveling.php "> Warhammer Online Power Leveling </a> </p>
<p><a href="http://www.gmbar.com/war-power-leveling.php "> War Power Leveling </a> </p>
<p><a href="http://www.gmbar.com/war-power-leveling.php "> Buy Warhammer Power Leveling </a> </p>
<p><a href="http://www.gmbar.com/war-power-leveling.php "> Warhammer PowerLeveling </a> </p>
<p><a href="http://www.gmbar.com/war-cdkey.php"> Warhammer CD Key </a> </p>
<p><a href="http://www.gmbar.com/war-cdkey.php"> Warhammer online CD Key </a> </p>
<p><a href="http://www.gmbar.com/war-cdkey.php"> Warhammer Timecard </a> </p>
<p><a href="http://www.gmbar.com/war-cdkey.php"> Buy Warhammer Time Card </a> </p>
<p><a href="http://www.gmbar.com/war-cdkey.php"> Warhammer 60 days Time Card </a> </p>
<p><a href="http://www.gmbar.com/cheap-war-gold-eu.php"> Warhammer EU Gold </a> </p>
<p><a href="http://www.gmbar.com/cheap-war-gold-eu.php"> Warhammer EU Power Leveling </a> </p>
<p><a href="http://www.gmbar.com/cheap-war-gold-eu.php"> Warhammer EU CD Key </a> </p>
<p><a href="http://www.gmbar.com/cheap-war-gold-eu.php"> Warhammer EU Accounts </a> </p>
<p><a href="http://www.gmbar.com/cheap_war_gold_eu.php"> Cheap Warhammer Gold </a> </p>
<p><a href="http://www.gmbar.com/cheap_war_gold_eu.php"> Cheap Warhammer online gold </a> </p>
<p><a href="http://www.gmbar.com/cheap_war_gold_eu.php"> Buy Cheap Warhammer Gold </a> </p>
<p><a href="http://www.gmbar.com/cheap_war_gold_eu.php"> Buy WAR Gold </a> </p>
<p><a href="http://www.gmbar.com/war-accounts-eu.php"> Cheap WAR Accounts </a> </p>
<p><a href="http://www.gmbar.com/war-accounts-eu.php"> Cheap warhammer Accounts </a> </p>
<p><a href="http://www.gmbar.com/war-accounts-eu.php"> Cheap warhammer online Accounts </a> </p>
<p><a href="http://www.gmbar.com/war-accounts-eu.php"> Buy Cheap WAR Accounts </a> </p>
<p><a href="http://www.gmbar.com/war-power-leveling-eu.php"> Cheap Warhammer Power Leveling </a> </p>
<p><a href="http://www.gmbar.com/war-power-leveling-eu.php"> Cheap Warhammer Online Power Leveling </a> </p>
<p><a href="http://www.gmbar.com/war-power-leveling-eu.php"> Buy War Power Leveling </a> </p>
<p><a href="http://www.gmbar.com/war-power-leveling-eu.php"> Warhammer EU Power Leveling </a> </p>
<p><a href="http://www.gmbar.com/war-cdkey-eu.php"> buy Warhammer CD Key </a> </p>
<p><a href="http://www.gmbar.com/war-cdkey-eu.php"> buy Warhammer online CD Key </a> </p>
<p><a href="http://www.gmbar.com/war-cdkey-eu.php"> cheap Warhammer CD key </a> </p>
<p><a href="http://www.gmbar.com/war-cdkey-eu.php"> warhammer time card </a> </p>
<p><a href="http://www.gmbar.com/war-cdkey-eu.php"> Warhammer prepaid time card </a> </p>

41. wow gold Nov 17, 2008 at 22:12

ow gold
http://www.gamevive.com/

warhammer gold
http://www.gmbar.com/

Air Jordan Shoes
http://www.bysneaker.com

warhammer power leveling
http://www.gmbar.com/war-power-leveling.php

Add your comment:

(SKIP THIS ONE)

(required)

(not shown)


(use pastie or gist for code)

navigation:
sponsored by:
tags:
if you want to help:
required:
Get Quicktime Player