Railscasts - Feed Parsing

#168

Jun 29, 2009

18 comments

Feed Parsing

Learn two different techniques for parsing an RSS feed using Feedzirra in this episode!

Tags: plugins active-record

Download (18.1 MB, 10:31)

alternative download for iPod & Apple TV (13.5 MB, 10:31)

Read it on ASCIIcasts

Resources

sudo gem install pauldix-feedzirra
script/generate model feed_entry name:string summary:text url:string published_at:datetime guid:string
rake db:migrate

# config/environment.rb
config.gem "pauldix-feedzirra", :lib => "feedzirra", :source => "http://gems.github.com"

# script/console
FeedEntry.update_from_feed("http://feeds.feedburner.com/railscasts")
FeedEntry.count

# models/feed_entry.rb
class FeedEntry < ActiveRecord::Base
  def self.update_from_feed(feed_url)
    feed = Feedzirra::Feed.fetch_and_parse(feed_url)
    add_entries(feed.entries)
  end
  
  def self.update_from_feed_continuously(feed_url, delay_interval = 15.minutes)
    feed = Feedzirra::Feed.fetch_and_parse(feed_url)
    add_entries(feed.entries)
    loop do
      sleep delay_interval
      feed = Feedzirra::Feed.update(feed)
      add_entries(feed.new_entries) if feed.updated?
    end
  end
  
  private
  
  def self.add_entries(entries)
    entries.each do |entry|
      unless exists? :guid => entry.id
        create!(
          :name         => entry.title,
          :summary      => entry.summary,
          :url          => entry.url,
          :published_at => entry.published,
          :guid         => entry.id
        )
      end
    end
  end
end

<!-- articles/index.html.erb -->
<div id="recent_episodes">
  <h3>Recent Railscasts Episodes</h3>
  <ul>
  <% for entry in FeedEntry.all(:limit => 10, :order => "published_at desc") %>
    <li><%= link_to h(entry.name), entry.url %></li>
  <% end %>
  </ul>
</div>

< Previous Episode

Next Episode >

18 comments

1. sthapit Jun 29, 2009 at 00:53

always enjoy your screencasts. thanks!

Report as Spam

2. Kieran P Jun 29, 2009 at 04:32

Been using FeedZirra for some time now. It's been really great. But long/complex urls on Ruby Enterprise Edition have caused problems recently.

Report as Spam

3. Michael Jun 29, 2009 at 04:59

Thanks again mate!

Report as Spam

4. Ricky Jun 29, 2009 at 06:26

Thanks Ryan

any idea how to combine multiple feeds into one master feed?

Report as Spam

5. Nils R Jun 29, 2009 at 06:52

It is starting to become kinda creepy … it's as if you were reading my mind. Always coming with stuff I need just in time!

Love your screencasts!

Report as Spam

6. RailsCasts Fan Jun 29, 2009 at 10:21

Thank you Ryan!
That's exactly what I needed!
So far, I used Nokogiri. The performance of XML processing is really better than Nokogiri and than Hpricot?

Report as Spam

7. Tim Matheson Jun 29, 2009 at 10:51

I second the notion that Ryan has ESP. His screencast's always seem to be right on target with what I am working on. We joke that he has bugged our office space :D.
Thanks Ryan another great episode.

Report as Spam

8. Gavin Jun 29, 2009 at 11:16

Great! Thanks for this Ryan

ps - I think feedzirra is supposed to rhyme with Godzilla... it's a joke

Report as Spam

9. Julien GENESTOUX Jun 29, 2009 at 13:06

Anybody dealing with RSS parsing for a lot of feeds and who don't want to have the hassle of polling should check out http://superfeedr.com

Let me know what you think! (You included Ryan ;))

Report as Spam

10. Orion Engleton Jun 30, 2009 at 00:48

This is awesome .. I had to do this a couple weeks ago.. I had the same moving parts as you but not as clear and concise!! Thanks for totally showing me the right way to do this!
I agree with Nils.. when I first saw the title I had deja vu!

Report as Spam

11. Nacho Jun 30, 2009 at 06:43

I'm getting errors installing it on a Windows box because of curl. Is there a way to install curl on windows?

Report as Spam

12. wrax Jun 30, 2009 at 22:54

@ryan what problems were you having with updating? I cant seem to get it to work either. It fetches and parses fine. Anyone else getting this?

http://pastie.org/530375

Report as Spam

13. bob d Jul 03, 2009 at 17:57

I found that this plugin works very well for any xml, but the documentation is not available. Just do the following:
- freeze the feedzirra gem
- in the parser directory clone the rss and rss_entry files, rename them and change the mappings to match you xml structure
- locate feedzirra.rb and add references to your new parser files
- in feed.rb update @feed_classes array with your new parser

In general you should be able to use the example files as reference. Also make sure to update the specs.

Report as Spam

14. Pooja Gupta Jul 07, 2009 at 04:30

Is there a way to install curl on windows? does feedzirra work with windows or not ?

Report as Spam

15. Jason Cohen Jul 07, 2009 at 07:47

Can you give guidelines for how you would unit-test this code?

For example, can you use a local file URL to ensure it properly parses and stuffs in the database various test data?

Thanks.

Report as Spam

16. Joris Jul 26, 2009 at 12:27

Feedzirra needs information from the stored feed entries to get the update working. You can read more on this here: http://groups.google.com/group/feedzirra/browse_thread/thread/6eb16d9a6d4d168e#

Report as Spam

17. Michael Aug 14, 2009 at 18:27

Does anyone have a code sample for a daemonized version of this? I'm not quite sure how to get the initial statement to run only once within daemon_generator's while clause.

I used the code as is, but it will never terminate since it will never complete the while($running) loop because of the update loop.

Report as Spam

18. anon_anon Dec 02, 2009 at 17:26

You might want to look at vtd-xml, the latest and most advanced xml processing model, far better than DOM or SAX

Report as Spam

19. Valentin Jan 06, 2010 at 03:26

Might be a bit late to post here, but I can't get this to work at all...

I followed step by step but get this error:

NoMethodError: undefined method `update_from_feed' for #<Class:0x103804fb8>

I don't really see where I am going wrong, as I have that method defined in the model ...

Thanks!

Report as Spam

20. Brad May 17, 2010 at 00:36

I'm a bit of a newbie to all this and after a few hours of experimenting today, I couldn't for the life of me work out how to the Feedzirra in Cron with Whenever. Any ideas?

Report as Spam

21. Nike pas cher Jul 13, 2010 at 00:40

thank you for your resources.

Report as Spam

22. iPhone Ringtone Maker for mac Jul 20, 2010 at 19:09

ah ha .so ^^^^

Report as Spam

23. Chirantan Jul 23, 2010 at 04:20

How to parse custom feed attributes? There is a way to do that for entries using

Feedzirra::Feed.add_common_feed_entry_element("wfw:commentRss", :as => :comment_rss)

But I want the same feature for the feed object, not just its entries. Is there a way to do that? Like say

Feedzirra::Feed.add_common_feed_element("geo:lat", :as => :latitudes)

Help!

Report as Spam

25. optical fiber Cable Aug 01, 2010 at 23:59

I am very appreciate the issue that it can benefit to the low income guys, thanking for your sharing, it is wonderful.

Report as Spam

25. solar collector solar water heater Aug 02, 2010 at 00:03

It's one of the most important facto to fix this time.

Report as Spam

26. timberlandbootsuk Aug 02, 2010 at 02:07

we provide our buyers with an efficient and manageable procurement process covering every phase of the international supply chain and

streamlining trade channels. Also welcome wholesaling, feedback now!

Report as Spam

27. authentic nike shoes Aug 17, 2010 at 20:01

I am very appreciate the issue that it can benefit to the low income guys, his is awesome .. I had to do this a couple weeks ago.. I had the same moving parts as you but not as clear and concise!! thanking for your sharing, it is wonderful.

Report as Spam

28. Wholesale hats Aug 20, 2010 at 20:30

Took me awhile to read all the comments, but I really love the article. It proved to be very useful to me and I am sure to all the commenters here! It's always nice when you can not only be informed, but also entertained! I'm sure you had fun writing this article. Comfortably, the article is really the sweetest on this precious topic.

Report as Spam

29. jimmy choos Aug 21, 2010 at 01:09

Thank you for sharing!

Report as Spam

30. Wholesale Electronics Aug 25, 2010 at 01:52

Discount Wholesale Electronics, Wholesale Cell Phones, Electronic Gadgets and More from the Best Dropship Wholesaler

Report as Spam

31. silly bandz Aug 25, 2010 at 03:30

<a href="http://www.toptoys2trade.com/power-balance-wholesale-2-c-40/ "> power balance</a>

Report as Spam

32. louis vuitton shoes Aug 26, 2010 at 21:02

Thanks for sharing your article. I really enjoyed it. I put a link to my site to here so other people can read it. My readers have about the same interets

Report as Spam

33. WiFi Cell Phones Aug 30, 2010 at 20:17

thanks for your sharing the post, i like it so much for the useful information.

Report as Spam

34. snow boots Aug 30, 2010 at 20:55

I'm getting errors installing it on a Windows box because of curl. Is there a way to install curl on windows?

Report as Spam

35. levis belts Sep 01, 2010 at 20:54

Good article! Thank you so much for sharing this post.Your views truly open my mind.

Report as Spam