#130
Oct 05, 2008
28 comments

Monitoring with God

Need to ensure your background processes stay up and running and don't use too many resources? Check out the god gem as shown in this episode.
Tags: tools
Download (39.9 MB, 14:40)
alternative download for iPod & Apple TV (22.7 MB, 14:40)

Resources

I forgot to mention that this will not work on Windows (sorry!). But it’s primarily used on the production server which is likely running some unix based OS.

sudo gem install god
god -c config/mailit.god
god status
god terminate
god log mailit-workling
kill `cat log/workling.pid`
# script/workling_starling_client
workling = File.join(File.dirname(__FILE__), '..', 'vendor', 'plugins', 'workling', 'script', 'listen.rb')
options = {
  :app_name   => "workling",
  :ARGV       => ARGV,
  :dir_mode   => :normal,
  :dir        => File.join(File.dirname(__FILE__), '..', 'log'),
  :log_output => true,
  :multiple   => false,
  :backtrace  => true,
  :monitor    => false
}

# config/mailit.god
RAILS_ROOT = File.dirname(File.dirname(__FILE__))

def generic_monitoring(w, options = {})
  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 10.seconds
      c.running = false
    end
  end
  
  w.restart_if do |restart|
    restart.condition(:memory_usage) do |c|
      c.above = options[:memory_limit]
      c.times = [3, 5] # 3 out of 5 intervals
    end
  
    restart.condition(:cpu_usage) do |c|
      c.above = options[:cpu_limit]
      c.times = 5
    end
  end
  
  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

God.watch do |w|
  script = "#{RAILS_ROOT}/script/workling_starling_client"
  w.name = "mailit-workling"
  w.group = "mailit"
  w.interval = 60.seconds
  w.start = "#{script} start"
  w.restart = "#{script} restart"
  w.stop = "#{script} stop"
  w.start_grace = 20.seconds
  w.restart_grace = 20.seconds
  w.pid_file = "#{RAILS_ROOT}/log/workling.pid"
  
  w.behavior(:clean_pid_file)
  
  generic_monitoring(w, :cpu_limit => 80.percent, :memory_limit => 100.megabytes)
end

God.watch do |w|
  w.name = "mailit-starling"
  w.group = "mailit"
  w.interval = 60.seconds
  w.start = "/usr/bin/starling -d -P #{RAILS_ROOT}/log/starling.pid -q #{RAILS_ROOT}/log/"
  w.stop = "kill `cat #{RAILS_ROOT}/log/starling.pid`"
  w.start_grace = 10.seconds
  w.restart_grace = 10.seconds
  w.pid_file = "#{RAILS_ROOT}/log/starling.pid"
  
  w.behavior(:clean_pid_file)
  
  generic_monitoring(w, :cpu_limit => 30.percent, :memory_limit => 20.megabytes)
end

RSS Feed for Episode Comments 28 comments

1. Guillermo Oct 06, 2008 at 04:17

How much is the amount of ram used by god, and how much it is increased with the addition of more procesess.

I come from monit, and i am asking why could be god better (more than is ruby).

Thanks.

Report as Spam
2. Mike Breen Oct 06, 2008 at 04:41

Another great screencast. Thank you Ryan.

Report as Spam
3. Morten Oct 06, 2008 at 04:41

Great as always.. keep up the good work. I like how you do 2-4 relating screencasts in row yet still keep them independent.

Report as Spam
4. Francois Oct 06, 2008 at 10:40

Thanks for another great screencast. Question: if you have multiple rails projects on a given server, do you suggest running multiple instances of god (one for each rails application)? Or would you then move the config file outside of the rails config folder and place it somewhere else on the server and run only one instance of god?

Report as Spam
5. Carl Oct 06, 2008 at 10:43

Great screencast! God is one of those tools that I've looked at, but haven't really had a reason to use yet, but wanted to.

Report as Spam
6. Ernie Oct 06, 2008 at 11:45

Am I too late to add the obligatory "Of *course* God isn't compatible with Windows!" comment?

Report as Spam
7. Ben Johnson Oct 06, 2008 at 12:04

Is it just me or does this seem like you are just sweeping the dirt under the rug?

If your background processes are consuming too much memory, or constantly shutting down, wouldn't it be smarter to find out why, instead of just having a script restart it?

Something like this just seems to promote bad programming practices / laziness.

Report as Spam
8. Jonathan Nelson Oct 06, 2008 at 13:11

way to step up ryan and create an excellent railscast on God process monitoring. more people need to become familiar with its awesomeness.

Report as Spam
9. Jonathan Nelson Oct 06, 2008 at 13:16

might i also add...jesse newland has created an awesome rails plugin called san_juan. use it for capistrano god recipes.

get it here: http://github.com/jnewland/san_juan/tree/master

Report as Spam
10. Ryan Bates Oct 06, 2008 at 17:55

@Guillermo, that's a good question. On my setup it uses about 8 MB of RAM, so if you're running on a VPS with very limited memory you may want to use Monit.

@Francois, I'd just use one God process for the entire server (multiple Rails apps). This is one reason I prefixed the watch names with "mailit" so we could easily add other apps without conflict. It's easy to load as many config files as you need.

# all.god
load '/path/to/mailit.god'
load '/path/to/foo.god'
load '/path/to/bar.god'

Then just run "god -c /path/to/all.god" to load them all.

@Ben, I agree in some sense. I encourage everyone who uses this kind of tool to closely monitor the logs and set up a notification to warn you when something goes down. If a process is having to restart constantly then that should be fixed.

That said, I think this still has a very important role. Think of it as the net under the trapezist. It's not part of the act, but it's there if the unthinkable happens.

Report as Spam
11. Freddy Andersen Oct 07, 2008 at 10:53

Hi,

Yet another great cast.

Testing this on Centos 5 I had to do
script = "export RAILS_ENV=production && #{RAILS_ROOT}/script/workling_starling_client"

If I didn't the workling startup process just failed...

Report as Spam
12. Zach Inglis Oct 07, 2008 at 17:02

Nice work Ryan but single-letter variables are evil. They lack clarity and promote laziness.

Report as Spam
13. Muni Oct 08, 2008 at 16:55

Thanks for a great tutorial!!!
Is there a way to start god automatically with the rails application, without any need for for external commands?

Report as Spam
14. Carl Oct 08, 2008 at 17:16

@Zach,

Normally I'd agree with you, but inside of a block I'd say single letter variables are fine, since they have a very limited context. I suppose in a complicated example I'd also agree with you, but with every line starting the same way in the block, this seems like the closest you can get to DRY in a block context (correct me if I'm wrong).

Report as Spam
15. Ashley Williams Oct 09, 2008 at 01:58

Maybe I'm just paranoid — but whats stopping God from dying? (if your possibly expecting your daemons to go down, it would be kind of ignorant to expect God to stay up 100% of the time I would think?)

But very good episode Ryan, thanks!

Report as Spam
16. EppO Oct 09, 2008 at 06:43

Good screencast but what a pity you didn't talk about mail notifications.
The code you show up is on god homepage, your explanation is a real bonus but I think it lacks a bit of "advanced use".
Never mind, I am a fan of your screencasts, it helps so much to improve my RoR skills that I just "want more" I guess ;)

Report as Spam
17. chris Oct 09, 2008 at 17:44

Thanks for another great railscast.

What are the advantages/disadvantages between God and monit? Anyone?

Cheers.

Report as Spam
18. Chris Anderton Oct 11, 2008 at 03:36

Another nice one Ryan.

I originally used an init script to start god and load all config files under a specific directory (as per my blog post a while ago).

I'm now thinking that you could start up a god process in inittab, use respawn to keep it alive, and use the init script to load/unload the monitors.

Decisions, decisions.

Report as Spam
19. Hari Oct 12, 2008 at 17:31

When i try starting the god process i get the following error:
The server is not available (or you do not have permissions to access it)

Any ideas?

Report as Spam
20. Andrew Oct 15, 2008 at 07:18

@Hari
you have been denied by god!
On a more serious note, check to make sure that it has +x access.

Report as Spam
21. Chris Gaffney Oct 16, 2008 at 12:45

We've been using God (0.7.8) to monitor a couple applications all running on Thin. The awesome bit with God is that it brings process back almost instantly. For example if we request a restart (thin -C path/to/config restart) it will usually catch the fact that they went down and bring them back up before Thin can finish the restart. This is an order of magnitude faster than we were seeing with Monit.

The only issue we've had so far with God is that it appears to have a memory leak (using Ruby 1.8.6-p114). Every now and again we'll notice it's running with 3g of virtual memory and taking up 100% processor. We've been able to solve this though by restarting god nightly (hooray cron).

We have played with the idea of having Monit monitor God but haven't gotten around to actually trying it.

Report as Spam
22. lardawge Mar 04, 2009 at 15:19

FYI for anyone setting up god to monitor workling in a none standard environment. You need to pass the RAILS_ENV which Ryan touched on briefly... However, you need to wrap it with parentheses as so.. RAILS_ENV='staging'... In the Podcast Ryan adds it in as RAILS_ENV=production without parentheses which won't work! PS. I have learned so much from these railscasts it's ridiculous... keep up the great work!

Report as Spam
23. Aaron Gibralter Aug 25, 2009 at 15:00

In my setup, god.rb is reporting that my worklings are going over the memory limit (200MB) about every day or so, and it restarts them. I can't seem to figure out why the memory is blowing up from 70MB => 230MB every once in a while... does anyone have any suggestions on how to go about investigating?

Report as Spam
24. wholesale nike shoes Jan 13, 2010 at 23:37

A very good article, I will always come in.

Report as Spam
25. fashion scarves Jan 13, 2010 at 23:37

Such a good article, caught my sympathy!
-

Report as Spam
26. mallio Jan 25, 2010 at 15:41

@Chris

I haven't tried it out, but theoretically you could have God monitor its own resource usage. As far as I can tell it just checks the memory and processor percentage using the pidfile, so you could set up a simple watch that transitions immediately to 'up' (since it's obviously running). Then have it monitor memory and cpu usage and notify you when they reach a certain level.

If you want it to restart automatically, you can set up god as a service in /etc/init.d and have god issue a restart command to itself.

Like I said, I haven't tested it, but I don't see why it wouldn't work.

Report as Spam

Add your comment:

(SKIP THIS ONE)

(required)

(not shown)


(use pastie or gist for code)

navigation:
sponsored by:
tags:
if you want to help:
required:
Get Quicktime Player
Give Back to Open Source