Railscasts - Monitoring with God

#130

Oct 05, 2008

Monitoring with God

Need to ensure your background processes stay up and running and don't use too many resources? Check out the god gem as shown in this episode.

Tags: tools

Download (39.9 MB, 14:40)

alternative download for iPod & Apple TV (22.7 MB, 14:40)

Resources

I forgot to mention that this will not work on Windows (sorry!). But it’s primarily used on the production server which is likely running some unix based OS.

sudo gem install god
god -c config/mailit.god
god status
god terminate
god log mailit-workling
kill `cat log/workling.pid`

# script/workling_starling_client
workling = File.join(File.dirname(__FILE__), '..', 'vendor', 'plugins', 'workling', 'script', 'listen.rb')
options = {
  :app_name   => "workling",
  :ARGV       => ARGV,
  :dir_mode   => :normal,
  :dir        => File.join(File.dirname(__FILE__), '..', 'log'),
  :log_output => true,
  :multiple   => false,
  :backtrace  => true,
  :monitor    => false
}

# config/mailit.god
RAILS_ROOT = File.dirname(File.dirname(__FILE__))

def generic_monitoring(w, options = {})
  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 10.seconds
      c.running = false
    end
  end
  
  w.restart_if do |restart|
    restart.condition(:memory_usage) do |c|
      c.above = options[:memory_limit]
      c.times = [3, 5] # 3 out of 5 intervals
    end
  
    restart.condition(:cpu_usage) do |c|
      c.above = options[:cpu_limit]
      c.times = 5
    end
  end
  
  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

God.watch do |w|
  script = "#{RAILS_ROOT}/script/workling_starling_client"
  w.name = "mailit-workling"
  w.group = "mailit"
  w.interval = 60.seconds
  w.start = "#{script} start"
  w.restart = "#{script} restart"
  w.stop = "#{script} stop"
  w.start_grace = 20.seconds
  w.restart_grace = 20.seconds
  w.pid_file = "#{RAILS_ROOT}/log/workling.pid"
  
  w.behavior(:clean_pid_file)
  
  generic_monitoring(w, :cpu_limit => 80.percent, :memory_limit => 100.megabytes)
end

God.watch do |w|
  w.name = "mailit-starling"
  w.group = "mailit"
  w.interval = 60.seconds
  w.start = "/usr/bin/starling -d -P #{RAILS_ROOT}/log/starling.pid -q #{RAILS_ROOT}/log/"
  w.stop = "kill `cat #{RAILS_ROOT}/log/starling.pid`"
  w.start_grace = 10.seconds
  w.restart_grace = 10.seconds
  w.pid_file = "#{RAILS_ROOT}/log/starling.pid"
  
  w.behavior(:clean_pid_file)
  
  generic_monitoring(w, :cpu_limit => 30.percent, :memory_limit => 20.megabytes)
end

< Previous Episode

Next Episode >

28 comments

1. Guillermo Oct 06, 2008 at 04:17

How much is the amount of ram used by god, and how much it is increased with the addition of more procesess.

I come from monit, and i am asking why could be god better (more than is ruby).

Thanks.

2. Mike Breen Oct 06, 2008 at 04:41

Another great screencast. Thank you Ryan.

3. Morten Oct 06, 2008 at 04:41

Great as always.. keep up the good work. I like how you do 2-4 relating screencasts in row yet still keep them independent.

4. Francois Oct 06, 2008 at 10:40

Thanks for another great screencast. Question: if you have multiple rails projects on a given server, do you suggest running multiple instances of god (one for each rails application)? Or would you then move the config file outside of the rails config folder and place it somewhere else on the server and run only one instance of god?

5. Carl Oct 06, 2008 at 10:43

Great screencast! God is one of those tools that I've looked at, but haven't really had a reason to use yet, but wanted to.

6. Ernie Oct 06, 2008 at 11:45

Am I too late to add the obligatory "Of *course* God isn't compatible with Windows!" comment?

7. Ben Johnson Oct 06, 2008 at 12:04

Is it just me or does this seem like you are just sweeping the dirt under the rug?

If your background processes are consuming too much memory, or constantly shutting down, wouldn't it be smarter to find out why, instead of just having a script restart it?

Something like this just seems to promote bad programming practices / laziness.

8. Jonathan Nelson Oct 06, 2008 at 13:11

way to step up ryan and create an excellent railscast on God process monitoring. more people need to become familiar with its awesomeness.

9. Jonathan Nelson Oct 06, 2008 at 13:16

might i also add...jesse newland has created an awesome rails plugin called san_juan. use it for capistrano god recipes.

get it here: http://github.com/jnewland/san_juan/tree/master

10. Ryan Bates Oct 06, 2008 at 17:55

@Guillermo, that's a good question. On my setup it uses about 8 MB of RAM, so if you're running on a VPS with very limited memory you may want to use Monit.

@Francois, I'd just use one God process for the entire server (multiple Rails apps). This is one reason I prefixed the watch names with "mailit" so we could easily add other apps without conflict. It's easy to load as many config files as you need.

# all.god
load '/path/to/mailit.god'
load '/path/to/foo.god'
load '/path/to/bar.god'

Then just run "god -c /path/to/all.god" to load them all.

@Ben, I agree in some sense. I encourage everyone who uses this kind of tool to closely monitor the logs and set up a notification to warn you when something goes down. If a process is having to restart constantly then that should be fixed.

That said, I think this still has a very important role. Think of it as the net under the trapezist. It's not part of the act, but it's there if the unthinkable happens.

11. Freddy Andersen Oct 07, 2008 at 10:53

Hi,

Yet another great cast.

Testing this on Centos 5 I had to do
script = "export RAILS_ENV=production && #{RAILS_ROOT}/script/workling_starling_client"

If I didn't the workling startup process just failed...

12. Zach Inglis Oct 07, 2008 at 17:02

Nice work Ryan but single-letter variables are evil. They lack clarity and promote laziness.

13. Muni Oct 08, 2008 at 16:55

Thanks for a great tutorial!!!
Is there a way to start god automatically with the rails application, without any need for for external commands?

14. Carl Oct 08, 2008 at 17:16

@Zach,

Normally I'd agree with you, but inside of a block I'd say single letter variables are fine, since they have a very limited context. I suppose in a complicated example I'd also agree with you, but with every line starting the same way in the block, this seems like the closest you can get to DRY in a block context (correct me if I'm wrong).

15. Ashley Williams Oct 09, 2008 at 01:58

Maybe I'm just paranoid — but whats stopping God from dying? (if your possibly expecting your daemons to go down, it would be kind of ignorant to expect God to stay up 100% of the time I would think?)

But very good episode Ryan, thanks!

16. EppO Oct 09, 2008 at 06:43

Good screencast but what a pity you didn't talk about mail notifications.
The code you show up is on god homepage, your explanation is a real bonus but I think it lacks a bit of "advanced use".
Never mind, I am a fan of your screencasts, it helps so much to improve my RoR skills that I just "want more" I guess ;)

17. chris Oct 09, 2008 at 17:44

Thanks for another great railscast.

What are the advantages/disadvantages between God and monit? Anyone?

Cheers.

18. Chris Anderton Oct 11, 2008 at 03:36

Another nice one Ryan.

I originally used an init script to start god and load all config files under a specific directory (as per my blog post a while ago).

I'm now thinking that you could start up a god process in inittab, use respawn to keep it alive, and use the init script to load/unload the monitors.

Decisions, decisions.

19. Hari Oct 12, 2008 at 17:31

When i try starting the god process i get the following error:
The server is not available (or you do not have permissions to access it)

Any ideas?

20. Andrew Oct 15, 2008 at 07:18

@Hari
you have been denied by god!
On a more serious note, check to make sure that it has +x access.

21. Chris Gaffney Oct 16, 2008 at 12:45

We've been using God (0.7.8) to monitor a couple applications all running on Thin. The awesome bit with God is that it brings process back almost instantly. For example if we request a restart (thin -C path/to/config restart) it will usually catch the fact that they went down and bring them back up before Thin can finish the restart. This is an order of magnitude faster than we were seeing with Monit.

The only issue we've had so far with God is that it appears to have a memory leak (using Ruby 1.8.6-p114). Every now and again we'll notice it's running with 3g of virtual memory and taking up 100% processor. We've been able to solve this though by restarting god nightly (hooray cron).

We have played with the idea of having Monit monitor God but haven't gotten around to actually trying it.

23. venta base de datos Oct 24, 2008 at 12:47

Hola, como estas, queria felicitarte por tu pagina, esta muy buena, si queres pasa por mi pagina y dejame un comentario, estan entrando mucho en mi sitio, si te interesa podemos hacer un intercambio de links, banners, cualquier cosa avisame, te dejo mi email tvinternet08@gmail.com, te mando un abrazo.

24. manos Oct 28, 2008 at 23:03

why not check out <a href="http://www.3zoom.com">Warhammer gold</a> at 3zoom? i think it might be better.
<a href="http://www.3zoom.com">Warhammer online gold</a>,
<a href="http://www.3zoom.com/war-all/cheap-powerleveling-5-all.html">warhammer online powerleveling</a>,
<a href="http://www.3zoom.com/war-all/cheap-powerleveling-5-all.html">Warhammer Powerleveling</a> as well
<a href="http://www.3zoom.com/war-all/cheap-account-5-all.html">Warhammer online account</a> 
 <a href="http://www.3zoom.com/war-all/cheap-account-5-all.html">Warhammer online accounts</a> 
 <a href="http://www.3zoom.com/uswarhammer-all/warhammer-account-5-all.html">Warhammer account</a> 
 <a href="http://www.3zoom.com/uswarhammer-all/warhammer-account-5-all.html">Warhammer accounts</a>

25. cheap wow gold Oct 29, 2008 at 02:54

Are you a WAR player?Do you need <a href="http://www.gmbar.com/"> Warhammer Gold </a> ? If your answer is yes,please come here. We are professional, loyal and reliable and have been a paypal confirmed seller of MMOG service for years. So it is securest and safest to buy from us. We also have <a href="http://www.gmbar.com/"> Warhammer Time Card </a>, <a href="http://www.gmbar.com/"> Warhammer CD Key </a>,<a href="http://www.gmbar.com/cheap-war-gold.php">Warhammer Power Leveling</a> and <a href="http://www.gmbar.com/cheap-war-gold.php"> Warhammer Accounts </a>,etc.Our world class customer service will make you satisfied.If you live in Europe, you can buy <a href="http://www.gmbar.com/cheap-war-gold-eu.php"> Warhammer EU Gold </a>from us ,too.Our price is competitive, We guarantee lowest price and professional services. Choose Us, Don’t be irresolute! We are ready to serve you and help you to have a wonderful WAR life.

26. Air Jordan Shoes Nov 16, 2008 at 19:15

<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy WAR gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Cheap Warhammer Gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy warhammer online gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> cheap warhammer online gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy warhammer gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> buy warhammer online gold </a>
<a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> buy cheap warhammer gold </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> War Gold </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Buy Warhammer Gold </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Warhammer Online Gold </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Warhammer Gold for sale </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Cheap Warhammer Gold </a>
<a href="http://www.gamevive.com/Warhammer/Warhammer-Account/Cheap-Warhammer-Account.html
"> Warhammer Accounts </a>
<a

27. wow gold Nov 16, 2008 at 19:49

Thanks for another great screencast.

Monitoring with God

28 comments

Add your comment: