#130 Monitoring with God

How much is the amount of ram used by god, and how much it is increased with the addition of more procesess.

I come from monit, and i am asking why could god be better (more than is ruby).

Thanks.

Great as always.. keep up the good work. I like how you do 2-4 relating screencasts in row yet still keep them independent.

Thanks for another great screencast. Question: if you have multiple rails projects on a given server, do you suggest running multiple instances of god (one for each rails application)? Or would you then move the config file outside of the rails config folder and place it somewhere else on the server and run only one instance of god?

Great screencast! God is one of those tools that I've looked at, but haven't really had a reason to use yet, but wanted to.

Am I too late to add the obligatory "Of *course* God isn't compatible with Windows!" comment?

Is it just me or does this seem like you are just sweeping the dirt under the rug?

If your background processes are consuming too much memory, or constantly shutting down, wouldn't it be smarter to find out why, instead of just having a script restart it?

Something like this just seems to promote bad programming practices / laziness.

way to step up ryan and create an excellent railscast on God process monitoring. more people need to become familiar with its awesomeness.

might i also add...jesse newland has created an awesome rails plugin called san_juan. use it for capistrano god recipes.

get it here: http://github.com/jnewland/san_juan/tree/master

@Guillermo, that's a good question. On my setup it uses about 8 MB of RAM, so if you're running on a VPS with very limited memory you may want to use Monit.

@Francois, I'd just use one God process for the entire server (multiple Rails apps). This is one reason I prefixed the watch names with "mailit" so we could easily add other apps without conflict. It's easy to load as many config files as you need.

# all.god
load '/path/to/mailit.god'
load '/path/to/foo.god'
load '/path/to/bar.god'

Then just run "god -c /path/to/all.god" to load them all.

@Ben, I agree in some sense. I encourage everyone who uses this kind of tool to closely monitor the logs and set up a notification to warn you when something goes down. If a process is having to restart constantly then that should be fixed.

That said, I think this still has a very important role. Think of it as the net under the trapezist. It's not part of the act, but it's there if the unthinkable happens.

Hi,

Yet another great cast.

Testing this on Centos 5 I had to do
script = "export RAILS_ENV=production && #{RAILS_ROOT}/script/workling_starling_client"

If I didn't the workling startup process just failed...

Nice work Ryan but single-letter variables are evil. They lack clarity and promote laziness.

Thanks for a great tutorial!!!
Is there a way to start god automatically with the rails application, without any need for for external commands?

@Zach,

Normally I'd agree with you, but inside of a block I'd say single letter variables are fine, since they have a very limited context. I suppose in a complicated example I'd also agree with you, but with every line starting the same way in the block, this seems like the closest you can get to DRY in a block context (correct me if I'm wrong).

Maybe I'm just paranoid — but whats stopping God from dying? (if your possibly expecting your daemons to go down, it would be kind of ignorant to expect God to stay up 100% of the time I would think?)

But very good episode Ryan, thanks!

Good screencast but what a pity you didn't talk about mail notifications.
The code you show up is on god homepage, your explanation is a real bonus but I think it lacks a bit of "advanced use".
Never mind, I am a fan of your screencasts, it helps so much to improve my RoR skills that I just "want more" I guess ;)

Thanks for another great railscast.

What are the advantages/disadvantages between God and monit? Anyone?

Cheers.

Another nice one Ryan.

I originally used an init script to start god and load all config files under a specific directory (as per my blog post a while ago).

I'm now thinking that you could start up a god process in inittab, use respawn to keep it alive, and use the init script to load/unload the monitors.

Decisions, decisions.

When i try starting the god process i get the following error:
The server is not available (or you do not have permissions to access it)

Any ideas?

@Hari
you have been denied by god!
On a more serious note, check to make sure that it has +x access.

We've been using God (0.7.8) to monitor a couple applications all running on Thin. The awesome bit with God is that it brings process back almost instantly. For example if we request a restart (thin -C path/to/config restart) it will usually catch the fact that they went down and bring them back up before Thin can finish the restart. This is an order of magnitude faster than we were seeing with Monit.

The only issue we've had so far with God is that it appears to have a memory leak (using Ruby 1.8.6-p114). Every now and again we'll notice it's running with 3g of virtual memory and taking up 100% processor. We've been able to solve this though by restarting god nightly (hooray cron).

We have played with the idea of having Monit monitor God but haven't gotten around to actually trying it.

FYI for anyone setting up god to monitor workling in a none standard environment. You need to pass the RAILS_ENV which Ryan touched on briefly... However, you need to wrap it with parentheses as so.. RAILS_ENV='staging'... In the Podcast Ryan adds it in as RAILS_ENV=production without parentheses which won't work! PS. I have learned so much from these railscasts it's ridiculous... keep up the great work!

In my setup, god.rb is reporting that my worklings are going over the memory limit (200MB) about every day or so, and it restarts them. I can't seem to figure out why the memory is blowing up from 70MB => 230MB every once in a while... does anyone have any suggestions on how to go about investigating?

@Chris

I haven't tried it out, but theoretically you could have God monitor its own resource usage. As far as I can tell it just checks the memory and processor percentage using the pidfile, so you could set up a simple watch that transitions immediately to 'up' (since it's obviously running). Then have it monitor memory and cpu usage and notify you when they reach a certain level.

If you want it to restart automatically, you can set up god as a service in /etc/init.d and have god issue a restart command to itself.

Like I said, I haven't tested it, but I don't see why it wouldn't work.

I made a gem called Prowly to send messages to iPhone via Prowl and after that I forked the God project and added it support to send notification via Prowl, so here are the repos: http://github.com/rafmagana/prowly and http://github.com/rafmagana/god

Now God supports sending notifications to an iPhone via Prowl using the Prowly gem (http://github.com/rafmagana/prowly)

Your website's super-tiny fixed-width one-column body text kinda sucks, guys. If you have horizontal scroll bars, I'm pretty sure you've already lost.

Read up about monit vs god and i think i'll give god a try at http://dealsking.de - hate it when delayed_job crashes without notice ;)

Running "god log mydaemon" only gives me "Please wait..." and nothing shows up. What could that be?

I know this is an old episode but I wanted to clarify the path used for the pid. In the config/mailit.god file in the God.watch block you have this path for the pid:

  w.pid_file = "#{RAILS_ROOT}/log/workling.pid"

Is not the convention for pids to be put in the tmp folder? Like this:

  w.pid_file = "#{RAILS_ROOT}/tmp/workling.pid"

Hope you can reply...

Hey guys.
The server is not available (or you do not have permissions to access it)
this error always happens when I run any command related to god.
god status
god start
can someone help me?
Thanks