#128 Starling and Workling

Thank's
Starling is so easy

do you have MemCache screencast?

Great. This sounds like a lightweight easy to use replacement for backgroundrb. Can't wait to try it myself. Thanks :)

@Tor, I have an episode on caching in Rails 2.1, but it's not specific to MemCache.

http://railscasts.com/episodes/115-caching-in-rails-2-1

Hi another great episode!
I was wondering if you plan on doing an episode on google maps in rails? I think a lot of people would like that. Thanks

Hi Ryan,

Thanks you for your screencasts that are so awesome and useful.

Do you know backgroundrb ?
(http://backgroundrb.rubyforge.org/)

I would like to know if you choose starling and workling over backgroundrb. Then, can you give me your reasons ?

I find your solution nice, but i like backgroundrb too. And at that time, i could not see some interesting differences between these two solutions.

Thank you for your light ;-).

Tracy, from Paris :o

Cool. Would it take much work in Starling/Workling to turn the Mailings in to a persistent queue?

And let's say you were to run the MailingWorker in both Starling and BackgroundRB; would you expect any performance difference between the two?

@Tracy, I haven't tried Backgroundrb yet, so I can't really comment on it. From what I've heard many people prefer Starling + Workling due to better memory usage. I also really like Starling's simplicity and how you can interact with it in many different ways. That said, I may do an episode on backgroundrb in the future as it has some additional features.

@Neil, I'm not sure what you mean by persistent queue? This is persistent in the fact that it's always running. It's not starting/stopping a Rails process each time it needs to send a mailing.

I have jobs that I need to run every 2 mins and some that I need to run every 5 mins. Currently I have just set up various cron jobs to call script/runner and run the jobs. This has the downside of the Rails environment being loaded each time a job is run.

Is there a good way to set up these recurring jobs using Starling?

@Jim, you can do that with an endless looping ruby script or rake task which sleeps for two minutes at the end of each iteration. There's no need to use Starling or other background job managers because you don't need to start up jobs arbitrarily. You may want to check out the "daemons" gem for managing this.

On a related note, I do plan to address dynamically scheduled jobs in the next episode.

Ryan, by persistent queue I meant; if something goes wrong and the mailings aren't sent by Starling (maybe someone crashes a truck in to a power station), can we store the jobs in the database so Starling can carry on where it left off?

Ryan, thanks for the new series on background jobs. As you will start preparing the episode on scheduled jobs in the next few days, may I ask you to also provide a suggestion on how to deal with both scheduled and user triggered jobs, without the need of two background instances (one starling workling, one daemon), aka both scheduled and triggered background jobs within a single background instance?

The reason why I'm asking for this is that memory on the average 256MB VPS host is limited, and still many apps require both scheduled and user triggered jobs. Here's a few examples:

- sending user registration/password emails
- daily calculation of e.g. a kind of "top 10"
- sending daily email newsletter
- background import of data via various APIs

PS: I'm currently using daemons for all this, but my background process keeps growing steadily, which basically forces me to terminate and restart it every few days. I can only assume that GC is optimized for the typical web request, and the infrequent, memory intense methods never really cause a cleanup.

Oh, another question: will ruby 1.9 fibers the new way of spawning background jobs, as they seem to be much nicer to used memory? Ruby libraries such as Neverblock sound interesting, but I'm still somewhat clueless on how to use them (as I'm waiting for a Ruby 1.9 compatible Rails+Mongrel before installing it).

@Neil, I believe starling has some persistence built in. If it goes down and there are messages in the queue, it will grab them from the queue log when it starts back up. However, once workling pulls the message from the queue it's up to you to provide the persistence in the Worker class. Usually you can do this by keeping a log of some kind or updating the database during the process. That's rather application specific.

@aleco, the scheduling system I plan to tackle is user driven schedules. For example, the user will be able to choose what time a given mailing goes out. That would be able to handle both user-triggered and scheduled jobs.

Regarding the garbage collection, I'm not certain what the problem is there, but it may be application specific. It sounds like some object never gets freed in the loop.

As for Ruby 1.9 and fibers, I haven't looked in this yet so I can't say much about it. I have a feeling this will get a lot more attention in the coming months as Rails 2.2 goes thread safe.

How would you set up your controller tests? Any chance you could update the mailit code to reflect this? Much appreciated.

Great screencast. I've just begun looking into options for running background jobs (primarily user-triggered - ie. user activation emails, admin alert emails, etc)

Other than the ones mentioned in this thread (Starling, backgroundrb, daemons) I'm also looking at BackgroundJob (BJ) -
http://agilewebdevelopment.com/plugins/bj

Ryan, did you plan on covering this plugin? Anyone have any thoughts or opinions on it?

Thanks.
Jim-

@Jim in reference to recurring tasks,

I have had great success with this:

http://openwferu.rubyforge.org/rquickstart.html

They renamed it as "Rufus", but it is easier to find up docs for the old name.

is the workling svn the same as the github one?

http://github.com/purzelrakete/workling/tree/master

I am getting a memcache error.

Workling::WorklingError: MemCache::MemCacheError - No connection to server

Any ideas?

@Wilson - I'm getting the same memcache error

I tried instructions here & from the github plugin site. Neither work. So I'm pulling the plug on this thing, looked nice in the video but my mileage has been zero.

Hope that next weeks idea pans out...

OK, cancel last statement

followed Ryan's directions precisely this time and now it works - just be sure you ignore the stuff in Workling's README

@andy, I can get Ryan's mailit app to worked as detailed -- but, like @Rich -- I'm trying to test the "deliver" method in the controller and get the MemCacheErr. I've gone ahead and added the "Workling::Remote.dispatcher" reference in the config/test.rb as we did in the development.rb but i can't seem to pass go on this one.

My test looks like this http://gist.github.com/12646

I'm getting ready to eject on this one.

glad that people are finding workling useful :)

i've tried to keep the svn repo synched with github, this hasnt really worked out so well ;). i think i'lll just kick it to the curb.

@andy: the README might be a bit out of date - i'll update this asap.

@wilson re: the memcache error: i'll look into this now. a few people have had this, probably just a case of old README instructions.

okay, the documentation needs a good update - agreed ;).

i'll fix it up within the next day and write a post up on http://playtype.net. i'll open a group on google groups, too.

and i'll keep the svn repo synched, after all, so that the instructions here keep working.

if anybody needs help setting this up, drop me a line and i'll do what i can!

I am facing while using workling it gives some error like
FAILED to process queue mailings_workers__send_mailing. #<MailingsWorker:0xb7234db4> could not handle invocation of send_mailing with {:uid=>"mailings_workers:send_mailing:af30186f8c4acff1449353f2a65fbb81", :email=>"kurian_1024@yahoo.co.in"}: wrong number of arguments (1 for 0).

@kurian your worker methods need to have an 'options' argument.

Excellent 'cast, Ryan :)

Would love to see a Part 3 on how to provide more feedback to user when jobs re actually completed. Even better, if multiple jobs are being processed, to somehow update the view so user sees whats going on in queue

Hi Ryan,

Do you have any opinion about ar_mailer for this purpose?

http://blog.segment7.net/articles/2006/08/15/ar_mailer

Thanks!

For those getting the MemCacheError, make sure the Starling server is started up and running on the proper port. You may also want to try connecting using IRB as I show at the end of this episode to see if it's a problem with Workling or Starling.

@Jim, I'm investigating BJ as well. It's a bit of a cross between this episode (Workling) and the last one (Rake) because it queues command line calls. This means the Rails app will need to start up each time a job is requested. I'm not too fond of this for frequently recurring jobs, so I may not do an episode on it unless there's some demand.

@purzelrakete, I had trouble getting the github repository working which is why I went with the SVN one here. It looks like it requires fiveruns-memcache-client which may have been my problem. Once you update the README I'll change the instructions here.

@Pimpmaster, I find reporting progress to the user is very application specific. For example, in the case of sending emails, we could update the database every 50 emails and use a bit of math to determine what percentage is done so far. At that point it's just a matter of reporting this to the user, maybe with a polling AJAX request. On second thought, the AJAX part might be a bit interesting and more generic, so I may end up doing one on that.

@Shy, I haven't looked into ar_mailer. I'm mainly using sending email as an example background process and only looking into solutions which will work for other situations. But if you're just sending email it's worth investigating further.

@Ryan - thanks for the response. That is correct BJ will load the rails env for each job - however, BJ allows only one copy of itself to run (for a given hostname/rails_env combination) at a time.

So while the loading the rails env will take some cycles, at least processes will not continue to spawn. BJ just handle each job in the queue accordingly. The docs specifically state this was a design decision so as to converse memory.

A pretty reasonable trade-off I'd say. Here are the docs:
http://codeforpeople.rubyforge.org/svn/bj/trunk/README

Cool, Ryan!

Definitely looking forward to seeing some AJAX love on this one.

Cheers :)

I am getting the following error intalling Starling - is something else required to install on Windows?:

"gem install starling
Building native extensions. This could take a while...
ERROR: Error installing starling:
ERROR: Failed to build gem native extension.

C:/InstantRails/ruby/bin/ruby.exe extconf.rb install starling"

i've updated the README for workling. i also made the error messages much friendlier - if your starling isn't running on the correct port, workling will tell you where it's expected.

workling will now also function without the fiveruns memcache client, if you're runnign an older version of starling.

so go and grab the latest workling version and look at the README :)

Ryan "...I do plan to address dynamically scheduled jobs in the next episode" - check out Rufus, superb :-)

http://rufus.rubyforge.org/rufus-scheduler/files/README_txt.html

Hello there and thanks for this great tutorial!

I have some question regarding automated background jobs.

I am building an application which is getting some data from the remote RSS feed every 5 minutes. Then it checks some values and makes comparison of those values with user values. If condition is true e-mail is sent to the user.

Now, i am doing this every 5 minutes via cron job and rake.

Because of environment being loaded every time i am considering using starling+workling.

Someone already asked similar question here, about how to do starling jobs in a scheduled manner, and the answer was to "do" a sleep after the job is finished for let's say 5 minutes.

I am not a very good programmer nor specialist, but i doubt sleeping for 5 minutes or triggering action every 5 minutes is the same.

When i trigger the an action via cron it will allways be triggered at same time (0m, 5m, 10m etc...), however the job done with sleep will be allways run at different times, depending on how long it took it to run (0m, 5m20s, etc...), which is completely out of option for me :)

Please, correct if i got this wrong.

So, another solution i am thinking about is to create starling job which would be triggered with a cron job which would call a script with simulated http trigger of action (like when you click on deliver e-mail in your screencast).

As i said before i am not a specialist so my question is wether this is a stupid solution or what could be best.

Thanks

Re: Starling install on windows. The problem was that there is no Windows binary yet for the latest version of eventmachine gem which is a dependency for starling. I have VC++ installed so it should compile, but anyway...I installed the previous version (which appears to be all Starling requires) and then installed Starling using --ignore-dependencies.

Now I am getting a syslog error when trying to start Starling. This is documented as being fixed in version 0.9.9 in Starling Readme, but I can't figure out how to install 0.9.9...I always get 0.9.8.

Any ideas on how to get Starling 0.9.9 installed?

@Ilia you have to get it off github. grab the lastest copy of workling (0.3), the README explains all :)

@purzelrakete - I actually tried that, but I get version 0.9.7.10 instead of 0.9.9.

I am doing:

gem sources -a http://gems.github.com/
gem install starling-starling

So I started using this. I needed a persistence service, so I stuck an object on the queue and had the worker save it. Nothing was happening though. It turns out that I needed to add to my load path the model I was trying to use (require 'event'). Does the workling_starling_client not include models from rails??? It does not look like the rails cast does this? Is there some config I am missing?

@Ilia same for me. I'm trying to get version 0.9.9 but only get 0.9.8.2008....
Did you manage to get 0.9.9 ?

Great episode.
Greetings from Paraguay.

hey all,
i try to get this working but it looks like i can't call the Worker from my controller. if i put a logger.info("TEST") in the worker i don't get a output. any ideas ?

regards + thx for your good work !

@Ryan, Great episode! I have always enjoy learning from you. Each episode is like equivalent of a chapter of book, only easier to grasp!

By following your episode, I was able to get my starling and workling working in less than 30 minutes.

@All, Here are just few gotchas based on my experience that I would to share with other people learning.

1. It seems that every time I change the code in the worker, I have to reload the working client; otherwise, change will not take effect.

2. When calling the async_xxx method, its important to spell correctly, otherwise, no warning or error will be given.

Hello,

I have to following problem when I start workling :

ruby script/workling_client start -t

/Library/Ruby/Site/1.8/rubygems/custom_require.rb:36:in `gem_original_require': no such file to load -- rubyeventmachine (MissingSourceFile)

But gem list --local gives :

eventmachine (0.12.2)

Is there any thing to do ?

Thanks,

Pierre

Thanks Ryan for your screencast.

Have you or anyone tried using starling (and workling) on Windows?

I have currently installed starling-starling gem (0.9.9) on my Windows XP machine.

I am able to start Starling. However, I get a Rubygems Load Error when I attempt a require 'starling'.

Have you or anyone else encountered this problem with starling-starling gem on Windows?

Thanks,

Sam

Was wondering if anyone else had seen or heard of this problem or anything similar.

I placed 1000 messages onto the queue. In a matter of seconds they all get taken off the queue.

I have a sleep 5 and a logger.info "message" in the worker class. I do not see "message" 1000 times though. I might see it a couple of times but that is it.

Anyone else know what could be going on? (this only happens on my redhat box. on my macbook, it works fine)

starling 0.9.9
workling 0.4.2.2
fiveruns-memcache-client
ruby 1.8.7
rails 2.1.2

I'd just like some suggestions on how to do functional or integration tests when starling/workling is present!

might be useful to note that (at least with Rails 2.3.2), you need to setup your Workling dispatcher in an after_initialize block if you're defining it in each specific environment file.

see: http://gist.github.com/84308

Hey, I am trying to send email on specific times selected by the user. How can I schedule job of sending email like that. I did not get that. Pls help.

@Phil thanks for this bit of information it was very helpful.

Thanks so much for the tutorials!
Really appreciate it!
You have no idea how helpful these tutorials are. They are one of a kind and very very well made.
Great Job!

Update for the instructions:

To get the latest version of starling you should use

gem install starling-starling

and not just "starling", or you'll end up with an old version.

I'm trying to use workling and starling to expire_fragment() using a regexp. I'm using FileStore as my cache and running the expire fragment in an async workling worker. I've checked that the RAILS_ROOT is the same in my worker as it is in my rails server. I've checked that the Rails.cache.cache_path is the same in both. I puts'd the finished regexp to the console and it looks correct. But the cache fragments are not being destroyed. What am I doing wrong?

Code sample at http://gist.github.com/162963

I'm having some problems with my Worker on my production machine and I can't see any of the error logs.

I've set my env config.log_level to :debug and I still can't find any logs from workling.

I've been trying to figure this out for a week now. I hope someone can help.

has anyone been able to get workling starling work on windows?
i could finally install starling on windows and get the process to start. but workling_client fails to start.

is there a workaround or is it a lost cause?

thanks

If you're having trouble getting this to work with Rails 2.3, you might need to wrap the line in the development.rb like this: http://gist.github.com/294492

Does anyone know if there is some kinda fork of the workling pluging that works with rails 3? I tried the plugin Ryan posted above, but it misses the method allow_concurrency= for ActiveRecord::Base...

is this gem dead?

Hey, can anyone please tell me how to start workling server. I tried script/workling_starling_client start command but it keeps saying
-bash: script/workling_starling_client: No such file or directory

I am a newbie to ROR and working on Ubuntu.