#126
Sep 08, 2008
56 comments

Populating a Database

Have you ever wanted to fill up a database with a lot of test data? See how to do that in this episode using the populator and faker gems.
Tags: active-record plugins
Download (13.3 MB, 8:38)
alternative download for iPod & Apple TV (9.9 MB, 8:38)

Resources

gem install populator
gem install faker
rake db:populate
# lib/tasks/populate.rake
namespace :db do
  desc "Erase and fill database"
  task :populate => :environment do
    require 'populator'
    require 'faker'
    
    [Category, Product, Person].each(&:delete_all)
    
    Category.populate 20 do |category|
      category.name = Populator.words(1..3).titleize
      Product.populate 10..100 do |product|
        product.category_id = category.id
        product.name = Populator.words(1..5).titleize
        product.description = Populator.sentences(2..10)
        product.price = [4.99, 19.95, 100]
        product.created_at = 2.years.ago..Time.now
      end
    end
    
    Person.populate 100 do |person|
      person.name    = Faker::Name.name
      person.company = Faker::Company.name
      person.email   = Faker::Internet.email
      person.phone   = Faker::PhoneNumber.phone_number
      person.street  = Faker::Address.street_address
      person.city    = Faker::Address.city
      person.state   = Faker::Address.us_state_abbr
      person.zip     = Faker::Address.zip_code
    end
  end
end

Full Source Code

RSS Feed for Episode Comments 56 comments

1. Gruszks Sep 08, 2008 at 04:50

Nice! Thanks!

2. Tobias Sep 08, 2008 at 05:02

Ryan, I just wanna thank your for this awesome Screencast! Keep up the great work! It helps me a lot focussing on what really matters.

3. sthapit Sep 08, 2008 at 06:13

Awesome. This is exactly what I was looking for - thanks!

4. sthapit Sep 08, 2008 at 06:15

off topic, but one of your ads says "Powered by Railsmachine". Isn't railscasts being run on slicehost?

5. Ethan Vizitei Sep 08, 2008 at 06:18

Good work, Ryan. I've done something similar to the "populator" gem on the last on the last few projects I've done, but your interface is much more pleasent, and I'll be using it in the future. Thanks.

6. Mark Richman Sep 08, 2008 at 07:04

This is great. For the longest time I relied on my spreadsheet data based approach to generating random names as test data.

http://www.markrichman.com/2007/09/26/generating-random-names-as-test-data/

7. Luke Francl Sep 08, 2008 at 09:23

Cool!

I wrote an article about this a while back (http://railspikes.com/2008/1/25/fuzzing-your-database) but it was a more random approach. I like the approach of using Faker to generate more realistic data.

8. Carl Sep 08, 2008 at 09:49

Wow, I was just thinking about creating some loops to do this very thing to do some sort of stress testing on my slice to see how long different approaches would take. Perfect timing, thanks!

9. tilo Sep 08, 2008 at 12:07

Very cool & useful gem!

Almost sent you a comment on the future dates ;-)

   T.

10. Ren Sep 08, 2008 at 14:09

I'm getting this error:

rake aborted!
Could not find RubyGem echoe (>= 0)

I'm pretty sure I installed the gem properly.

11. Neil Sep 08, 2008 at 15:12

Ren, I have the same problem, and I also believe I installed the gem correctly. Have you updated your rubygems system too?

12. Don Sep 08, 2008 at 15:41

Having the same problem here...

13. Don Sep 08, 2008 at 15:52

Try putting this in your environment.rb file:

Rails::Initializer.run do |config|
   config.gem 'populator'
end

14. James Sep 08, 2008 at 20:04

@Ren & @Neil, do a gem install echoe...should install and you'll be good to go. I had the same issue, maybe he doesn't have it requiring it as a dependency upon installation.

15. Liam Morley Sep 09, 2008 at 00:40

Looks nice. I usually do this with a fixture http://pastie.org/268729 , but you'd still need something for really random data... this works well. :)

16. Ren Sep 09, 2008 at 05:44

@James

Thanks! Didn't even consider "echoe" to be a gem. It works. Thanks Ryan for an awesome tutorial/gem. Mucho helpful.

17. Ryan Bates Sep 09, 2008 at 11:08

Echoe isn't supposed to be a gem dependency, but looks like something is hosed in my gem config. I think I fixed it, so the latest populator version (0.2.4) should work without echoe installed.

18. Greg Sep 09, 2008 at 12:53

Thanks for the great screencast, Ryan. I have a question, however, concerning how you would populate a habtm (has_many :through is possible though) association? In your example, Category-to-Products is a one-to-many, but what if you modeled products to be part of many categories? Nesting the two like you have in your example wouldn't work in that case. Maybe you already came across this problem and have a nice solution? :)

19. Ryan Bates Sep 09, 2008 at 16:51

@Greg, currently HABTM associations aren't supported by Populator, I'll work on adding that though. Thanks. :)

20. Fredd Sep 10, 2008 at 07:08

Great technique, just a question. Is there any support for counter caches?

21. Stephen Brown Sep 10, 2008 at 08:15

sthapit: ping it:
PING railscasts.com (75.127.77.214) 56(84) bytes of data.
64 bytes from rm-75-127-77-214.railsmachina.com (75.127.77.214): icmp_seq=1 ttl=47 time=36.2 ms

22. Pascal Laplante Sep 10, 2008 at 10:01

Hi

If I add to many records at the same time, MySQL give me this error through the task:

Mysql::Error: Got a packet bigger than 'max_allowed_packet' bytes

I know that max_allowed_packet is a parameter into MySQL, but do you know if there is a way to tell Populator to insert the records by steps instead of waiting at the end to do a big insert ?

Thanks !

23. Per Velschow Sep 10, 2008 at 11:08

@Pascal: Just had the same problem. There is an option to #populate called :per_query that limits how many records are inserted per query.

24. Ryan Bates Sep 10, 2008 at 13:16

@sthapit I used to use slicehost and had no problems, but Rails Machine was kind enough to host it for free. :)

@Fredd, you'll need to set the counter caches directly. Usually this is possible with a query at the end to count the associated models. I'll look into providing a more convenient way to do this.

@Pascal, the :per_query option will do this, it defaults to 1000 so I may need to lower this. It will vary depending on the size of each record.

25. Carl Sep 10, 2008 at 21:24

@Ryan,

I'm at slicehost now and I'm happy with what I get for my $$, but I'm also curious about other providers. Ignoring the price difference (or what it would be if you had to pay for it) how would you compare the two for a production rails app? What kind of slice (size and OS) were you running railscasts on before?

26. Ryan Bates Sep 10, 2008 at 22:32

@Carl, I'm just running on a 256 MB slice, both here and at slicehost. Both hosts fit my needs equally well since this is such a simple site. Rails Machine offers quite a bit more with their plan so you should look into it and see if it fits your needs.

27. Mark Sep 11, 2008 at 11:53

Ryan, this is really a great screencast! Thanx for it and keep up the good work!

28. Mark Richman Sep 13, 2008 at 08:33

How can I use this to load up a bunch of dummy users using restful_authentication?

I can't seem to set user.password = 'foo' here:

    Account.populate 200 do |account|
      account.name = Faker::Company.name
      account.email = Faker::Internet.email
      account.created_at = 2.years.ago..Time.now
      User.populate 10 do |user|
        user.login = Faker::Internet.user_name
        user.password = 'foo'
        user.first_name = Faker::Name.name
        user.last_name = Faker::Name.name
        user.email = Faker::Internet.email
        user.account_id = account.id
      end
    end

29. Carl Sep 13, 2008 at 09:38

@Mark,

I wonder if that's because populator is working directly with the database for speed and bypassing the virtual attribute? Does it give you an error or just not set anything for the password, or just not create any users?

30. Ryan Bates Sep 13, 2008 at 11:02

@Mark, Carl is correct, the "account" passed to the block is not a true instance of account. You can only set columns direction, not through virtual attributes.

I'm thinking of changing how this is done so it allows you to use virtual attributes as well. But it doesn't work yet.

31. MarkHamlin Sep 15, 2008 at 00:04

This was great. Thank you! I have almost everything in your railscasts in my apps!

32. Darryl Ring Sep 15, 2008 at 13:28

Hey, this is a good idea. I just cloned Faker and spent a little time in the wee hours of the morning adding a couple features. I'll likely be adding a bit more too. I haven't created a patch yet for the SVN version, but I'll get around to it quickly.

Also I need to set up a gemspec so that it can be used with GitHub's gem thingy, but until then:

http://github.com/darrylring/faker/tree/master

33. Dave Spurr Sep 15, 2008 at 14:25

It would be nice to have the ability to supply a max length to Populator.words, Populator.sentences etc. as otherwise I think you'd hit the limits on some columns.

I thought I was hitting that problem but it appears that I'm hitting a problem with duplicate keys although it doesn't make any sense as the queries that it's complaining about aren't duplicating keys from what I can see.

34. Dave Spurr Sep 16, 2008 at 12:25

I figured out the duplicate key issue, it would be nice to be able to tell populator about what should be unique within a database.

35. Darin Sep 20, 2008 at 09:40

Hi, and thanks for the great screencast! I am having the following error when I run the rake:

rake aborted!
no such file to load -- spec/rake/spectask

So I read all the comments and tried:
  sudo gem install echoe
and got:
ERROR: Error installing echoe:
echoe requires RubyGems version = 1.2

I am running this on a fresh Rails 2.1 app and have not had any proplems with RubyGems before... Any suggestions?

36. Corey Woodcox Sep 21, 2008 at 00:33

Here's how I'm trying to populate now... http://pastie.org/276597

I know the last couple of lines won't work, and I know why. I'm just demonstrating how slick it would be if I could. Do you have a github project for the gem, or anything? I'd be happy to help accomplish that if you'd like.

Also, it would be slick if there was something available along the lines of "Populator.bool" that would return an array of true and false, so I don't have to set that up manually in my task.

Thanks a ton for this plugin, it's way slick. Hit me on twitter or GT if there's any way I can contribute. :)

37. Corey Woodcox Sep 21, 2008 at 00:35

And of course, as soon as I submit it, I find the github link. :)

38. Jared Sep 24, 2008 at 09:42

Is there any support for min/max or high/low fields? I have a model that has a price range that is stored as 2 separate fields. I can set pkg.min_price = 1..100 and pkg.max_price = 101..200, but that's not ideal.

39. Kris Steigerwald Sep 24, 2008 at 19:41

restful_authentication X 2 I would like to know the best approach for this as well

40. Trevor Turk Sep 27, 2008 at 13:39

Here's a quick example of how you can use Populator and Faker to create valid records for use in demos, acceptance testing, etc:

http://almosteffortless.com/2008/09/27/creating-valid-records-with-populator-and-faker/

41. Andy Oct 10, 2008 at 08:20

hello, great as usual!

i have a small issue though with the following code

deal.name = Populator.words(4..9).titleize
deal.permalink = PermalinkFu.escape(deal.name)

this code dose correctly set the permalink but it also sets deal.name to be a permalink, any thoughts why??

42. warhammer time card Oct 16, 2008 at 03:00

Echoe isn't supposed to be a gem dependency, but looks like something is hosed in my gem config. I think I fixed it, so the latest populator version (0.2.4) should work without echoe installed.

43. warhammer gold Oct 31, 2008 at 02:53

you are right.but i think...

44. Air Jordan Shoes Nov 02, 2008 at 23:23

thank you very much, i will try.

45. Tony Nov 05, 2008 at 14:34

This is awesome - can't wait to try it!

I do have one issue. This is more of a general question, but...

In one of my applications, a number of my models have one uploaded file (ex. User has one uploaded_image). I'm using attachment_fu, which provides an "uploaded_data=" setter that works perfectly via the web interface. However, I can't seem to get this working in the rake task. I'm pretty sure that uploaded_data= expects a binary file, but none of my attempts with File, Tempfile, StringIO, or ActionController::UploadedTempfile have worked.

Is there a way to create a model with attachments via a rake task?

46. wotlk cd key Nov 12, 2008 at 22:20

Hey! don't you know Wrath of the Lich King release in about 20 days later? All of us is wating for Lich King release, you still worry about WOTLK cdkey? Surely no! we start preorder wrath of the lich king cd key, wrath cd key is not hart to get, we don't think it's happy that waiting for the wow wrath of the lich king cdkey, if you still don't have WOW WOTLK cd key,wlk cd key,wrath key, put an order for preorder WOTLK cd key , wlk cdkey on WOW cd key page, then see you in game when the Wrath of the Lich King release!

47. wrath of the lich king cd Nov 12, 2008 at 22:23

Preorder Wrath of the Lich King cd key on Gamevive
Hey! don't you know Wrath of the Lich King release in about 20 days later? All of us is wating for Lich King release, you still worry about WOTLK cdkey? Surely no! we start preorder wrath of the lich king cd key, wrath cd key is not hart to get, we don't think it's happy that waiting for the wow wrath of the lich king cdkey, if you still don't have WOW WOTLK cd key,wlk cd key,wrath key, put an order for preorder WOTLK cd key , wlk cdkey on WOW cd key page, then see you in game when the Wrath of the Lich King release!

48. wrath of the lich king cd key Nov 12, 2008 at 22:23

Preorder Wrath of the Lich King cd key on Gamevive
Hey! don't you know Wrath of the Lich King release in about 20 days later? All of us is wating for Lich King release, you still worry about WOTLK cdkey? Surely no! we start preorder wrath of the lich king cd key, wrath cd key is not hart to get, we don't think it's happy that waiting for the wow wrath of the lich king cdkey, if you still don't have WOW WOTLK cd key,wlk cd key,wrath key, put an order for preorder WOTLK cd key , wlk cdkey on WOW cd key page, then see you in game when the Wrath of the Lich King release!

49. wotlk cd key Nov 12, 2008 at 22:25

Hey! don't you know Wrath of the Lich King release in about 20 days later? All of us is wating for Lich King release, you still worry about WOTLK cdkey? Surely no! we start preorder wrath of the lich king cd key, wrath cd key is not hart to get, we don't think it's happy that waiting for the wow wrath of the lich king cdkey, if you still don't have WOW WOTLK cd key,wlk cd key,wrath key, put an order for preorder WOTLK cd key , wlk cdkey on WOW cd key page, then see you in game when the Wrath of the Lich King release!

50. wrath of the lich king cd key Nov 12, 2008 at 22:25

Preorder Wrath of the Lich King cd key on Gamevive
Hey! don't you know Wrath of the Lich King release in about 20 days later? All of us is wating for Lich King release, you still worry about WOTLK cdkey? Surely no! we start preorder wrath of the lich king cd key, wrath cd key is not hart to get, we don't think it's happy that waiting for the wow wrath of the lich king cdkey, if you still don't have WOW WOTLK cd key,wlk cd key,wrath key, put an order for preorder WOTLK cd key , wlk cdkey on WOW cd key page, then see you in game when the Wrath of the Lich King release!

51. Eric Nov 13, 2008 at 12:40

Is there a way to get populator to add records to a HABTM relationship?

52. Anthony Ettinger Nov 13, 2008 at 17:28

Two questions:

1) How do I enforce that this db:populate only gets ran on 'test' environment? Don't want to accidently do it on prod :)

2) What about test scripts that rely on data to be the same, is there a better way of writing tests that don't rely on the data itself in the db? (ie: selenium tests for UI)

53. foobar Nov 13, 2008 at 20:15

For whatever reason, I get a TON of failures during testing phase:

$ rake db:populate
Ftest
F
Finished in 0.872049 seconds.

  1) Failure:
test_should_create_directory(ThingControllerTest)
    [/usr/lib/ruby/gems/1.8/gems/activesupport-

... about 100 failures form this pint.

54. warhammer power leveling Nov 16, 2008 at 19:17

<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy WAR gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Cheap Warhammer Gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy warhammer online gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> cheap warhammer online gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> Buy warhammer gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> buy warhammer online gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/cheap-Warhammer-gold/Warhammer-Anlec(Oceanic)-24-2093-Order.html"> buy cheap warhammer gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> War Gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Buy Warhammer Gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Warhammer Online Gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Warhammer Gold for sale </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Gold/Cheap-Warhammer-Gold.html
"> Cheap Warhammer Gold </a></p>
<p><a href="http://www.gamevive.com/Warhammer/Warhammer-Account/Cheap-Warhammer-Account.html
"> Warhammer Accounts </a></p>
<p><a

Add your comment:

(SKIP THIS ONE)

(required)

(not shown)


(use pastie or gist for code)

navigation:
sponsored by:
tags:
if you want to help:
required:
Get Quicktime Player