Railscasts - Starling and Workling

1. Tor Sep 22, 2008 at 06:18

Thank's
Starling is so easy

do you have MemCache screencast?

2. Andreas Sep 22, 2008 at 06:18

Great. This sounds like a lightweight easy to use replacement for backgroundrb. Can't wait to try it myself. Thanks :)

3. Ryan Bates Sep 22, 2008 at 07:06

@Tor, I have an episode on caching in Rails 2.1, but it's not specific to MemCache.

http://railscasts.com/episodes/115-caching-in-rails-2-1

4. Momo Sep 22, 2008 at 07:31

Hi another great episode!
I was wondering if you plan on doing an episode on google maps in rails? I think a lot of people would like that. Thanks

5. Tracy LOISEL Sep 22, 2008 at 07:32

Hi Ryan,

Thanks you for your screencasts that are so awesome and useful.

Do you know backgroundrb ?
(http://backgroundrb.rubyforge.org/)

I would like to know if you choose starling and workling over backgroundrb. Then, can you give me your reasons ?

I find your solution nice, but i like backgroundrb too. And at that time, i could not see some interesting differences between these two solutions.

Thank you for your light ;-).

Tracy, from Paris :o

6. Neil Sep 22, 2008 at 07:49

Cool. Would it take much work in Starling/Workling to turn the Mailings in to a persistent queue?

And let's say you were to run the MailingWorker in both Starling and BackgroundRB; would you expect any performance difference between the two?

7. Ryan Bates Sep 22, 2008 at 08:18

@Tracy, I haven't tried Backgroundrb yet, so I can't really comment on it. From what I've heard many people prefer Starling + Workling due to better memory usage. I also really like Starling's simplicity and how you can interact with it in many different ways. That said, I may do an episode on backgroundrb in the future as it has some additional features.

@Neil, I'm not sure what you mean by persistent queue? This is persistent in the fact that it's always running. It's not starting/stopping a Rails process each time it needs to send a mailing.

8. Jim Sep 22, 2008 at 08:37

I have jobs that I need to run every 2 mins and some that I need to run every 5 mins. Currently I have just set up various cron jobs to call script/runner and run the jobs. This has the downside of the Rails environment being loaded each time a job is run.

Is there a good way to set up these recurring jobs using Starling?

9. Ryan Bates Sep 22, 2008 at 10:00

@Jim, you can do that with an endless looping ruby script or rake task which sleeps for two minutes at the end of each iteration. There's no need to use Starling or other background job managers because you don't need to start up jobs arbitrarily. You may want to check out the "daemons" gem for managing this.

On a related note, I do plan to address dynamically scheduled jobs in the next episode.

10. Neil Sep 22, 2008 at 13:06

Ryan, by persistent queue I meant; if something goes wrong and the mailings aren't sent by Starling (maybe someone crashes a truck in to a power station), can we store the jobs in the database so Starling can carry on where it left off?

11. aleco Sep 22, 2008 at 15:33

Ryan, thanks for the new series on background jobs. As you will start preparing the episode on scheduled jobs in the next few days, may I ask you to also provide a suggestion on how to deal with both scheduled and user triggered jobs, without the need of two background instances (one starling workling, one daemon), aka both scheduled and triggered background jobs within a single background instance?

The reason why I'm asking for this is that memory on the average 256MB VPS host is limited, and still many apps require both scheduled and user triggered jobs. Here's a few examples:

- sending user registration/password emails
- daily calculation of e.g. a kind of "top 10"
- sending daily email newsletter
- background import of data via various APIs

PS: I'm currently using daemons for all this, but my background process keeps growing steadily, which basically forces me to terminate and restart it every few days. I can only assume that GC is optimized for the typical web request, and the infrequent, memory intense methods never really cause a cleanup.

12. aleco Sep 22, 2008 at 15:38

Oh, another question: will ruby 1.9 fibers the new way of spawning background jobs, as they seem to be much nicer to used memory? Ruby libraries such as Neverblock sound interesting, but I'm still somewhat clueless on how to use them (as I'm waiting for a Ruby 1.9 compatible Rails+Mongrel before installing it).

13. Ryan Bates Sep 22, 2008 at 18:06

@Neil, I believe starling has some persistence built in. If it goes down and there are messages in the queue, it will grab them from the queue log when it starts back up. However, once workling pulls the message from the queue it's up to you to provide the persistence in the Worker class. Usually you can do this by keeping a log of some kind or updating the database during the process. That's rather application specific.

@aleco, the scheduling system I plan to tackle is user driven schedules. For example, the user will be able to choose what time a given mailing goes out. That would be able to handle both user-triggered and scheduled jobs.

Regarding the garbage collection, I'm not certain what the problem is there, but it may be application specific. It sounds like some object never gets freed in the loop.

As for Ruby 1.9 and fibers, I haven't looked in this yet so I can't say much about it. I have a feeling this will get a lot more attention in the coming months as Rails 2.2 goes thread safe.

14. Rich Sep 23, 2008 at 10:57

How would you set up your controller tests? Any chance you could update the mailit code to reflect this? Much appreciated.

15. Jim Jones Sep 23, 2008 at 12:35

Great screencast. I've just begun looking into options for running background jobs (primarily user-triggered - ie. user activation emails, admin alert emails, etc)

Other than the ones mentioned in this thread (Starling, backgroundrb, daemons) I'm also looking at BackgroundJob (BJ) -
http://agilewebdevelopment.com/plugins/bj

Ryan, did you plan on covering this plugin? Anyone have any thoughts or opinions on it?

Thanks.
Jim-

16. Ho-Sheng Hsiao Sep 23, 2008 at 15:36

@Jim in reference to recurring tasks,

I have had great success with this:

http://openwferu.rubyforge.org/rquickstart.html

They renamed it as "Rufus", but it is easier to find up docs for the old name.

17. andy Sep 24, 2008 at 10:37

is the workling svn the same as the github one?

http://github.com/purzelrakete/workling/tree/master

18. Wilson Sep 24, 2008 at 11:07

I am getting a memcache error.

Workling::WorklingError: MemCache::MemCacheError - No connection to server

Any ideas?

19. andy Sep 24, 2008 at 11:52

@Wilson - I'm getting the same memcache error

I tried instructions here & from the github plugin site. Neither work. So I'm pulling the plug on this thing, looked nice in the video but my mileage has been zero.

Hope that next weeks idea pans out...

20. andy Sep 24, 2008 at 12:23

OK, cancel last statement

followed Ryan's directions precisely this time and now it works - just be sure you ignore the stuff in Workling's README

21. Wilson Sep 24, 2008 at 12:55

@andy, I can get Ryan's mailit app to worked as detailed -- but, like @Rich -- I'm trying to test the "deliver" method in the controller and get the MemCacheErr. I've gone ahead and added the "Workling::Remote.dispatcher" reference in the config/test.rb as we did in the development.rb but i can't seem to pass go on this one.

My test looks like this http://gist.github.com/12646

I'm getting ready to eject on this one.

22. purzelrakete Sep 24, 2008 at 15:48

glad that people are finding workling useful :)

i've tried to keep the svn repo synched with github, this hasnt really worked out so well ;). i think i'lll just kick it to the curb.

@andy: the README might be a bit out of date - i'll update this asap.

@wilson re: the memcache error: i'll look into this now. a few people have had this, probably just a case of old README instructions.

23. purzelrakete Sep 24, 2008 at 16:39

okay, the documentation needs a good update - agreed ;).

i'll fix it up within the next day and write a post up on http://playtype.net. i'll open a group on google groups, too.

and i'll keep the svn repo synched, after all, so that the instructions here keep working.

if anybody needs help setting this up, drop me a line and i'll do what i can!

24. Kurian Sep 25, 2008 at 00:39

I am facing while using workling it gives some error like
FAILED to process queue mailings_workers__send_mailing. #<MailingsWorker:0xb7234db4> could not handle invocation of send_mailing with {:uid=>"mailings_workers:send_mailing:af30186f8c4acff1449353f2a65fbb81", :email=>"kurian_1024@yahoo.co.in"}: wrong number of arguments (1 for 0).

25. purzelrakete Sep 25, 2008 at 02:00

@kurian your worker methods need to have an 'options' argument.

26. Pimpmaster Sep 26, 2008 at 00:41

Excellent 'cast, Ryan :)

Would love to see a Part 3 on how to provide more feedback to user when jobs re actually completed. Even better, if multiple jobs are being processed, to somehow update the view so user sees whats going on in queue

27. Shy Sep 27, 2008 at 08:24

Hi Ryan,

Do you have any opinion about ar_mailer for this purpose?

http://blog.segment7.net/articles/2006/08/15/ar_mailer

Thanks!

28. Ryan Bates Sep 28, 2008 at 16:46

For those getting the MemCacheError, make sure the Starling server is started up and running on the proper port. You may also want to try connecting using IRB as I show at the end of this episode to see if it's a problem with Workling or Starling.

@Jim, I'm investigating BJ as well. It's a bit of a cross between this episode (Workling) and the last one (Rake) because it queues command line calls. This means the Rails app will need to start up each time a job is requested. I'm not too fond of this for frequently recurring jobs, so I may not do an episode on it unless there's some demand.

@purzelrakete, I had trouble getting the github repository working which is why I went with the SVN one here. It looks like it requires fiveruns-memcache-client which may have been my problem. Once you update the README I'll change the instructions here.

@Pimpmaster, I find reporting progress to the user is very application specific. For example, in the case of sending emails, we could update the database every 50 emails and use a bit of math to determine what percentage is done so far. At that point it's just a matter of reporting this to the user, maybe with a polling AJAX request. On second thought, the AJAX part might be a bit interesting and more generic, so I may end up doing one on that.

@Shy, I haven't looked into ar_mailer. I'm mainly using sending email as an example background process and only looking into solutions which will work for other situations. But if you're just sending email it's worth investigating further.

29. Jim Jones Sep 29, 2008 at 08:07

@Ryan - thanks for the response. That is correct BJ will load the rails env for each job - however, BJ allows only one copy of itself to run (for a given hostname/rails_env combination) at a time.

So while the loading the rails env will take some cycles, at least processes will not continue to spawn. BJ just handle each job in the queue accordingly. The docs specifically state this was a design decision so as to converse memory.

A pretty reasonable trade-off I'd say. Here are the docs:
http://codeforpeople.rubyforge.org/svn/bj/trunk/README

30. pimpmaster Sep 29, 2008 at 11:59

Cool, Ryan!

Definitely looking forward to seeing some AJAX love on this one.

Cheers :)

31. Ilia Oct 01, 2008 at 12:50

I am getting the following error intalling Starling - is something else required to install on Windows?:

"gem install starling
Building native extensions. This could take a while...
ERROR: Error installing starling:
ERROR: Failed to build gem native extension.

C:/InstantRails/ruby/bin/ruby.exe extconf.rb install starling"

32. purzelrakete Oct 01, 2008 at 14:58

i've updated the README for workling. i also made the error messages much friendlier - if your starling isn't running on the correct port, workling will tell you where it's expected.

workling will now also function without the fiveruns memcache client, if you're runnign an older version of starling.

so go and grab the latest workling version and look at the README :)

33. Morten Oct 01, 2008 at 15:09

Ryan "...I do plan to address dynamically scheduled jobs in the next episode" - check out Rufus, superb :-)

http://rufus.rubyforge.org/rufus-scheduler/files/README_txt.html

34. Mitja Oct 02, 2008 at 04:07

Hello there and thanks for this great tutorial!

I have some question regarding automated background jobs.

I am building an application which is getting some data from the remote RSS feed every 5 minutes. Then it checks some values and makes comparison of those values with user values. If condition is true e-mail is sent to the user.

Now, i am doing this every 5 minutes via cron job and rake.

Because of environment being loaded every time i am considering using starling+workling.

Someone already asked similar question here, about how to do starling jobs in a scheduled manner, and the answer was to "do" a sleep after the job is finished for let's say 5 minutes.

I am not a very good programmer nor specialist, but i doubt sleeping for 5 minutes or triggering action every 5 minutes is the same.

When i trigger the an action via cron it will allways be triggered at same time (0m, 5m, 10m etc...), however the job done with sleep will be allways run at different times, depending on how long it took it to run (0m, 5m20s, etc...), which is completely out of option for me :)

Please, correct if i got this wrong.

So, another solution i am thinking about is to create starling job which would be triggered with a cron job which would call a script with simulated http trigger of action (like when you click on deliver e-mail in your screencast).

As i said before i am not a specialist so my question is wether this is a stupid solution or what could be best.

Thanks

35. Ilia Oct 03, 2008 at 00:17

Re: Starling install on windows. The problem was that there is no Windows binary yet for the latest version of eventmachine gem which is a dependency for starling. I have VC++ installed so it should compile, but anyway...I installed the previous version (which appears to be all Starling requires) and then installed Starling using --ignore-dependencies.

Now I am getting a syslog error when trying to start Starling. This is documented as being fixed in version 0.9.9 in Starling Readme, but I can't figure out how to install 0.9.9...I always get 0.9.8.

Any ideas on how to get Starling 0.9.9 installed?

36. purzelrakete Oct 03, 2008 at 01:55

@Ilia you have to get it off github. grab the lastest copy of workling (0.3), the README explains all :)

37. Ilia Oct 03, 2008 at 13:41

@purzelrakete - I actually tried that, but I get version 0.9.7.10 instead of 0.9.9.

I am doing:

gem sources -a http://gems.github.com/
gem install starling-starling

38. manos Oct 28, 2008 at 23:05

why not check out <a href="http://www.3zoom.com">Warhammer gold</a> at 3zoom? i think it might be better.
<a href="http://www.3zoom.com">Warhammer online gold</a>,
<a href="http://www.3zoom.com/war-all/cheap-powerleveling-5-all.html">warhammer online powerleveling</a>,
<a href="http://www.3zoom.com/war-all/cheap-powerleveling-5-all.html">Warhammer Powerleveling</a> as well
<a href="http://www.3zoom.com/war-all/cheap-account-5-all.html">Warhammer online account</a> 
 <a href="http://www.3zoom.com/war-all/cheap-account-5-all.html">Warhammer online accounts</a> 
 <a href="http://www.3zoom.com/uswarhammer-all/warhammer-account-5-all.html">Warhammer account</a> 
 <a href="http://www.3zoom.com/uswarhammer-all/warhammer-account-5-all.html">Warhammer accounts</a>

39. Matt Brown Oct 29, 2008 at 07:44

So I started using this. I needed a persistence service, so I stuck an object on the queue and had the worker save it. Nothing was happening though. It turns out that I needed to add to my load path the model I was trying to use (require 'event'). Does the workling_starling_client not include models from rails??? It does not look like the rails cast does this? Is there some config I am missing?

41. Jules Nov 05, 2008 at 06:16

@Ilia same for me. I'm trying to get version 0.9.9 but only get 0.9.8.2008....
Did you manage to get 0.9.9 ?

42. Matías Nov 11, 2008 at 14:39

Great episode.
Greetings from Paraguay.

43. kalle Nov 14, 2008 at 05:41

hey all,
i try to get this working but it looks like i can't call the Worker from my controller. if i put a logger.info("TEST") in the worker i don't get a output. any ideas ?

regards + thx for your good work !

44. warhammer gold Nov 16, 2008 at 19:16

<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy WAR gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Cheap Warhammer Gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy warhammer online gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> cheap warhammer online gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy warhammer gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> buy warhammer online gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> buy cheap warhammer gold </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> War Gold </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Buy Warhammer Gold </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Warhammer Online Gold </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Warhammer Gold for sale </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Cheap Warhammer Gold </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Account/Cheap-Warhammer-Account.html
"> Warhammer Accounts </a>
<a

45. wow gold Nov 16, 2008 at 19:53

Thank's
Starling is so easy

Starling and Workling

47 comments

Add your comment: