#85 YAML Configuration (revised)

Oct 14, 2012 | 8 minutes | Tools

Keeping passwords and secret tokens in source control is a security risk. Here I show how to move these settings out into a YAML configuration file which can be loaded in as a hash or environment variables.

Click to Play Video ▶

Download:
source codeProject Files in Zip (59.4 KB)
mp4Full Size H.264 Video (23.4 MB)
m4vSmaller H.264 Video (11.7 MB)
webmFull Size VP8 Video (12.6 MB)
ogvFull Size Theora Video (30.2 MB)

Milovan Zogovic about 12 years ago

If you are using foreman, you can put env vars in .env file. Also, there is a dotenv gem worth checking out

Alberto Harres Rocha over 11 years ago

best $9 I've ever spent

Pierre GD almost 7 years ago

Thanks Milovan !

Michael J. McDermott about 12 years ago

Since Ruby is an interpreted language, what is the drawback to storing configurations/settings in a simple ruby hash in an rb file/Ruby module and then include the file/module? What advantage does parsing a YAML file into a simple ruby data structure yield since Ruby is already pretty darn readable?

I know you could make arguments that you might include your real config file in version control, but that's true for YAML et al. as well.

Steve Richert about 12 years ago

That's a really simple and interesting idea. Why isn't this more popular? I like it!

Hélio Pereira about 12 years ago

planetmcd comment makes me question if using YAML is the right way to use configurations. The only advantage that I see in YAML is that it's portable. But what about in terms of performance what is better YAML or ruby?

Michael J. McDermott about 12 years ago

I think the only performance issue would be load time. Loading the class is probably quicker than parsing the YAML and then loading the relevant data structures. I've never done a compare, but I'd imagine the difference would not matter much.

I just wondered whether setting up a nested data structure is spartan enough in Ruby that the YAML abstraction is often unnecessary.

I like YAML, and when I use Rails, that's how it works, so I use it. But when I've been doing non-rails stuff lately, I've been experimenting with just using Ruby, e.g.:

          ruby
        
module Config

  @ldap_array =     {
        :host => 'server',
        :port => 636,
        :base => '[base]',
        :method => :ssl,
        :bind_dn => "bind_dn",
        :password_block => Proc.new { '[password]' },
        :allow_anonymous => false,
        :try_sasl => false
    }

  def self.ldap_settings
    @ldap_array
  end

end

and then I can use it like:

          ruby
        
class LdapThingee
  include Config

  def some_method()
    LdapPerson.setup_connection(Config.ldap_settings)
  end
end

As best as I can figure, YAML lets you not worry about quoting strings. But the portability of YAML is not really a big selling point for configs in Ruby. In Java, XML was a idea since it meant you could use a standard, portable format, that didn't get compiled into your code. YAML's readability is a big improvement from XML for configs. But since the Ruby doesn't get compiled, why not use it for configs I wondered?

Brent about 12 years ago

I find this configuration very useful as well. Further, I prefer to add a defaults section to the YAML config file, and allow let the other sections to inherit from that.

          yaml
        
defaults: &defaults
  email: info@example.com

development:
  <<: *defaults
  email: me@example.com

test:
  <<: *defaults
  email: test@example.com

staging:
  <<: *defaults

production:
  <<: *defaults

Gustavo Beathyate about 12 years ago

If you use Heroku, it makes sense to use foreman in development. If you do that, you can put all your config vars in a .env file.

https://github.com/ddollar/foreman/blob/master/lib/foreman/cli.rb#L134

Paul Morganthall about 12 years ago

You mention (@7:10) that if you're not using Heroku, you don't use environment variables.

Wouldn't the CONFIG hash solution work on Heroku also? If Heroku itself were looking for a setting in the environment variables, that would be a reason to use environment variables. In all of the screencast examples (auth & mail settings), we have control of both the definition and usage of the configuration method.

Steve Richert about 12 years ago

That's absolutely true for the examples given in the screencast. However, many of Heroku's plugins depend on ENV configuration; SendGrid for example. So configuring your app to use ENV in those cases allows you to use Heroku's configuration in production and your own locally.

Drew Dara-Abrams about 12 years ago

Two relevant gems:

Brian M. Clapper about 12 years ago

I'll second the rails_config comment. I've been using it (in projects for two separate clients), and it does pretty much exactly what I want. It allows for default settings, environment-specific overrides, and local (i.e., not checked-in) overrides. As with the final approach in Ryan's screencast, it makes the config settings available in a global.

There are certainly edge cases where it might not work well, but for my uses (passwords, email configuration, API keys, and the like), it's cleaned things up considerably.

PetriWessman about 12 years ago

Thanks for the tip on rails_config, it's just what I was looking for over here. The built-in support for local overrides is nice, and more elegant than my current home-baked solution.

benrs44 about 12 years ago

I have been using Yettings which seems to work well.

narath about 12 years ago

Thanks Ryan, good episode. And interesting comments with good links. One thing that I think would complete the railscast well would be the capistrano deploy script. Since you have the file in .gitignore, it will not post to the server with your cap deploy, so you just need to add a command to put the yaml file to the server when you deploy.

Steve Richert about 12 years ago

Exactly, although I put application.yml in Capistrano's shared directory and symlink it rather than uploading it every time. That way, a developer doesn't need access to the latest production server configuration in order to deploy.

xanview about 12 years ago

How did you do it?

I have this in my deploy recipe:

          ruby
        
    run "mkdir -p #{current_path}/config"
    run "mkdir -p #{release_path}/config"
    run "mkdir -p #{shared_path}/config"
    template "application.yml.erb", "#{current_path}/config/application.yml"
    template "application.yml.erb", "#{release_path}/config/application.yml"
    template "application.yml.erb", "#{shared_path}/config/application.yml"

When I do cap deploy:setup, it copies the file to:

          ruby
        
    /home/deployer/xanview-portals/demo1/current/config/application.yml
    /home/deployer/xanview-portals/demo1/releases/20121031020720/config/application.yml
    /home/deployer/xanview-portals/demo1/shared/config/application.yml

But when I do cap deploy:cold I see:

          ruby
        
    2012-10-31 02:08:20 executing `deploy:assets:precompile'
    executing "cd /home/deployer/xanview-portals/demo1/releases/20121031020819 && bundle exec rake RAILS_ENV=production RAILS_GROUPS=assets assets:precompile"
    servers: ["zanview.com"]
    [zanview.com] executing command
    [out :: zanview.com] rake aborted!
    [out :: zanview.com] No such file or directory - /home/deployer/xanview-portals/demo1/releases/20121031020819/config/application.yml

And it dies :( - Any ideas?

xanview about 12 years ago

I fixed the problem by adding this:

          ruby
        
  task :setup do
    run "mkdir -p #{shared_path}/config"
    template "application.yml.erb", "#{shared_path}/config/application.yml"
  end
  
  before 'deploy:assets:precompile' do
    run "ln -s #{shared_path}/config/application.yml #{release_path}/config/application.yml"
  end

woo :)

Paul Vudmaska about 12 years ago

I like that you can prefix the yaml key with a colon, like so

:symbol:

in your yaml file so you don't have to use symobolize_keys!. Also, if I remember correctly, symbolize_keys! only works one level deep.

Vitor Oliveira about 12 years ago

I would recommend to use CONFIG.fetch(:key) instead of CONFIG[:key]. If for some reason the :key does not exist an KeyError exception will be raised.

Ali Ukani about 12 years ago

Another good gem to check out is Settingslogic. While Figaro is easier to set up, Settingslogic allows nested configurations which feels for cleaner code.

Rainer Kuhn about 12 years ago

Thanks for the tip!

I was using this settings logic: http://speakmy.name/2011/05/29/simple-configuration-for-ruby-apps/

Bad thing is, they aren't always loaded, I had some cases in the workers where the Settings weren't actually available.

Chris de Bruin about 12 years ago

What about deploying your app when the application.yml isn't part of your code in git? When working with different developers it's a pain in the ass to keep the application.yml for the production env in sync. Any tips on this one?

Josh Smith about 12 years ago

I'm also curious about this. If we're not supposed to keep config data in SCM, where does it go?

Justin Kolberg about 12 years ago

I usually deploy projects using a configuration management system such as Chef or Puppet. I create templates for the settings files and the templates are populated using data pulled from encrypted data bags on convergence.

Chris Johnson about 12 years ago

I do something similar. You may find the constantinople gem useful.

Put all your non-secret config settings (eg - hostname), as well as "dummy" secrets into a config/app.yml.default file that you store in source control.
Gitignore config/app.yml.
In production, use chef templates to generate config/app.yml. Devs can also maintain their own local over-rides there as well.

If you prefer, instead of keeping config/app.yml.default and overriding with gitignore'd config/app.yml... you can keep config/app.yml in source control, and override with a gitignore'd config/app.yml.override

Alexander Berezovsky about 12 years ago

I'd better use

          zsh
        
cp config/application{,.example}.yml

instead of

          zsh
        
cp config/application.yml config/application.example.yml

Seth Vargo about 12 years ago

Hey guys, I just packaged this all into a gem called magiconf. It's super simple and easy to use. Based off of this + figaro. https://github.com/sethvargo/magiconf

Balazs Nagy almost 12 years ago

My latest revision of reading yaml'ed data in application.rb (1.9 notation):

          ruby
        
read_yaml = ->(fname) do
  begin
    YAML.load(File.read(File.expand_path("../#{fname}", __FILE__)))
  rescue Errno::ENOENT
    {}
  end
end

appconfig = read_yaml.('application.yml')
defaults = read_yaml.('application_defaults.yml')
CONFIG = defaults.deep_merge(appconfig)
CONFIG.merge! CONFIG.fetch(Rails.env, {})
CONFIG.symbolize_keys!

One less mutation, and it allows specifying environment-dependent settings in both yaml files. It does use mutation though, but as far as I know they (merge! and symbolize_keys!) are both thread-safe.

Mattan Griffel almost 12 years ago

Wait, if you're gitignoring your application.yml file, how do you get around this in your application.rb?

config = YAML.load(File.read(File.expand_path('../application.yml', __FILE__)))

I'm getting this when I try to run on heroku:

No such file or directory - /app/config/application.yml

Tom Harrison over 11 years ago

config/application.example.yml should be a file containing the skeleton of the variables you want to keep secret (don't put real values there); check this in, then in the deployed application rename the skeleton to config/application.yml and manually add the private data.

The purpose is to keep private data out of source control. The need here is clear when your SCM is public, but even in a small organization with multiple developer, one might want to set up an environment where not everyone has access to everything.

shazebq over 11 years ago

Thanks for the explanation. I'm wondering though, how would you go about doing this if you're deploying to Heroku where it seems that you can't add or edit anything in the deployed application. I read that it's basically read only because any changes you make to any file are temporary and will be undone whenever you push your source code to Heroku.

dlegr250 almost 10 years ago

@shazebq, this is a little old topic but it's still applicable (and debated!) today. Heroku operates on ENV variables expected in the ENV namespace. It is used by Heroku itself for deployment as well as the 3rd-party plugins that Heroku provides (email, additional DBs, etc...).

Storing env variables in config/application.yml and reading them via code in your application.rb is only useful if your production environment can do the same. As you pointed out, you can't (easily) dynamically add the production-ready application.yml into Heroku's deployment process. Thus, using application.yml is not a good idea for Heroku deployment.

If you use Heroku (as I do for several projects) I recommend using the dotenv gem. In your local development, you have a .env file that stores environment variables. So your app code expects and uses the ENV namespace.

But you have to manually set those ENV vars on Heroku. Currently you can login to your Heroku app and they provide a clean interface to alter your ENV variables. Put your production-ready secrets and keys into that interface and your app will use them when it's deployed.

This way each developer can have their own local .env file with particular configs, yet the production system will always be consistent. The only "gotcha" is when you need to add a new env variable that you have to remember to add it everywhere.

What I have started doing is keeping application environment configs in another repo named "REPO_NAME-environments" where I have folders for development/test/staging/production and applicable config files there. This keeps them versioned and is a single place for them to exist. Goes without saying, but make sure this is a private repo.

Hope you or anyone else reading through these great comments finds this helpful! I recently did a major overhaul of some app configs myself and wanted to throw in my 2 cents.

Alexander Toder almost 10 years ago

Check out how I load YAML file config for a basic http authentication in my tutorial
http://aflexsystem.com/how-to-create-a-simple-admin-control-panel-with-ruby-on-rails-in-less-than-9-minutes/

You can use

          ruby
        
if File.exists?(File.expand_path('../application.yml', __FILE__))

to check if the file exists first.. and set your heroku variables manually

Ben Fischer almost 12 years ago

This was really helpful, just one typo in the ASCIIcast that was a gotcha for me when I copy and pasted.

CONFIG.merge! config.fetch(Rails.env, {})

should be

CONFIG.merge! CONFIG.fetch(Rails.env, {})

Mark Kremer almost 12 years ago

Thanks for letting us know, I've corrected the ASCIIcast text :)

William Limratana almost 12 years ago

Is it certain that "CONFIG.merge! CONFIG.fetch(Rails.env, {})" will pull in heroku environment variables pulled in via heroku config:push from the .env file?

Leo Gallucci over 11 years ago

Foreman + dotenv + dotenv-rails + heroku-config

My preferred configuration management tools.
Using them always with your .env file either on development or production, in heorku or VPS, with rake or capistrano, on Rails or any app.

Rails 3 / 4 with dotenv-rails

          ruby
        
gem 'dotenv-rails', :groups => [:development, :test]

Rails + Heroku with heroku-config

          shell
        
heroku config:push

Rails + Capistrano with dotenv-capistrano

          ruby
        
require "dotenv/capistrano"

Plain old Ruby apps with dotenv

          ruby
        
require 'dotenv'
Dotenv.load

Rake tasks

          ruby
        
require 'dotenv/tasks'
task :mytask => :dotenv do ... end

Note: do not commit your `.env` file but commit the example

          shell
        
cp .env.example .env

Riccardo Tacconi about 11 years ago

I am using:

          ruby
        
CONFIG = YAML.load(File.read(File.join(File.dirname(__FILE__), 'application.yml')))[Rails.env]

It is one line not two and it does not use slashes for OS compatibility.

Agen Poker almost 7 years ago

That's a really simple and interesting idea

Agen Poker almost 7 years ago

Permainan TOGEL ONLINE , Live 24D,Live 48D Dan Dragon Tiger
Permainan Togel Ini Juga Tersedia Hadiah Utama, Hadiah Kedua, Dan Ada Juga Hadiah Ketiga Untuk 3D Dan 4D.
2D : Prize 1 X 70
3D : Prize 1 X 400, Prize 2 X 200, Prize 3 X 100
4D : Prize 1 X 3000, Prize 2 X 2000, Prize 3 X 1000

Ada Juga Pasaran nya :
- Wuhan
- Saga
- Sisilia
- Singapore
- Koln

Ayuk Daftar Diri Anda Dan Bergabung Lah Bersama kami
Bonus Refferal 1% Seumur Hidup ( + Bonus hingga 100% )

#85 YAML Configuration (revised)

Foreman + dotenv + dotenv-rails + heroku-config

Rails 3 / 4 with dotenv-rails

Rails + Heroku with heroku-config

Rails + Capistrano with dotenv-capistrano

Plain old Ruby apps with dotenv

Rake tasks

Note: do not commit your .env file but commit the example

Note: do not commit your `.env` file but commit the example