Railscasts - Feed Parsing

#168

Jun 29, 2009

34 comments

Feed Parsing

Learn two different techniques for parsing an RSS feed using Feedzirra in this episode!

Tags: plugins active-record

Download (18.1 MB, 10:31)

alternative download for iPod & Apple TV (13.5 MB, 10:31)

Read it on ASCIIcasts

Resources

sudo gem install pauldix-feedzirra
script/generate model feed_entry name:string summary:text url:string published_at:datetime guid:string
rake db:migrate

# config/environment.rb
config.gem "pauldix-feedzirra", :lib => "feedzirra", :source => "http://gems.github.com"

# script/console
FeedEntry.update_from_feed("http://feeds.feedburner.com/railscasts")
FeedEntry.count

# models/feed_entry.rb
class FeedEntry < ActiveRecord::Base
  def self.update_from_feed(feed_url)
    feed = Feedzirra::Feed.fetch_and_parse(feed_url)
    add_entries(feed.entries)
  end
  
  def self.update_from_feed_continuously(feed_url, delay_interval = 15.minutes)
    feed = Feedzirra::Feed.fetch_and_parse(feed_url)
    add_entries(feed.entries)
    loop do
      sleep delay_interval
      feed = Feedzirra::Feed.update(feed)
      add_entries(feed.new_entries) if feed.updated?
    end
  end
  
  private
  
  def self.add_entries(entries)
    entries.each do |entry|
      unless exists? :guid => entry.id
        create!(
          :name         => entry.title,
          :summary      => entry.summary,
          :url          => entry.url,
          :published_at => entry.published,
          :guid         => entry.id
        )
      end
    end
  end
end

<!-- articles/index.html.erb -->
<div id="recent_episodes">
  <h3>Recent Railscasts Episodes</h3>
  <ul>
  <% for entry in FeedEntry.all(:limit => 10, :order => "published_at desc") %>
    <li><%= link_to h(entry.name), entry.url %></li>
  <% end %>
  </ul>
</div>

< Previous Episode

Next Episode >

34 comments

1. sthapit Jun 29, 2009 at 00:53

always enjoy your screencasts. thanks!

Report as Spam

2. Kieran P Jun 29, 2009 at 04:32

Been using FeedZirra for some time now. It's been really great. But long/complex urls on Ruby Enterprise Edition have caused problems recently.

Report as Spam

3. Michael Jun 29, 2009 at 04:59

Thanks again mate!

Report as Spam

4. Ricky Jun 29, 2009 at 06:26

Thanks Ryan

any idea how to combine multiple feeds into one master feed?

Report as Spam

5. Nils R Jun 29, 2009 at 06:52

It is starting to become kinda creepy … it's as if you were reading my mind. Always coming with stuff I need just in time!

Love your screencasts!

Report as Spam

6. RailsCasts Fan Jun 29, 2009 at 10:21

Thank you Ryan!
That's exactly what I needed!
So far, I used Nokogiri. The performance of XML processing is really better than Nokogiri and than Hpricot?

Report as Spam

7. Tim Matheson Jun 29, 2009 at 10:51

I second the notion that Ryan has ESP. His screencast's always seem to be right on target with what I am working on. We joke that he has bugged our office space :D.
Thanks Ryan another great episode.

Report as Spam

8. Gavin Jun 29, 2009 at 11:16

Great! Thanks for this Ryan

ps - I think feedzirra is supposed to rhyme with Godzilla... it's a joke

Report as Spam

9. Julien GENESTOUX Jun 29, 2009 at 13:06

Anybody dealing with RSS parsing for a lot of feeds and who don't want to have the hassle of polling should check out http://superfeedr.com

Let me know what you think! (You included Ryan ;))

Report as Spam

10. Orion Engleton Jun 30, 2009 at 00:48

This is awesome .. I had to do this a couple weeks ago.. I had the same moving parts as you but not as clear and concise!! Thanks for totally showing me the right way to do this!
I agree with Nils.. when I first saw the title I had deja vu!

Report as Spam

11. Nacho Jun 30, 2009 at 06:43

I'm getting errors installing it on a Windows box because of curl. Is there a way to install curl on windows?

Report as Spam

12. wrax Jun 30, 2009 at 22:54

@ryan what problems were you having with updating? I cant seem to get it to work either. It fetches and parses fine. Anyone else getting this?

http://pastie.org/530375

Report as Spam

13. bob d Jul 03, 2009 at 17:57

I found that this plugin works very well for any xml, but the documentation is not available. Just do the following:
- freeze the feedzirra gem
- in the parser directory clone the rss and rss_entry files, rename them and change the mappings to match you xml structure
- locate feedzirra.rb and add references to your new parser files
- in feed.rb update @feed_classes array with your new parser

In general you should be able to use the example files as reference. Also make sure to update the specs.

Report as Spam

14. Pooja Gupta Jul 07, 2009 at 04:30

Is there a way to install curl on windows? does feedzirra work with windows or not ?

Report as Spam

15. Jason Cohen Jul 07, 2009 at 07:47

Can you give guidelines for how you would unit-test this code?

For example, can you use a local file URL to ensure it properly parses and stuffs in the database various test data?

Thanks.

Report as Spam

16. Joris Jul 26, 2009 at 12:27

Feedzirra needs information from the stored feed entries to get the update working. You can read more on this here: http://groups.google.com/group/feedzirra/browse_thread/thread/6eb16d9a6d4d168e#

Report as Spam

17. Michael Aug 14, 2009 at 18:27

Does anyone have a code sample for a daemonized version of this? I'm not quite sure how to get the initial statement to run only once within daemon_generator's while clause.

I used the code as is, but it will never terminate since it will never complete the while($running) loop because of the update loop.

Report as Spam

18. anon_anon Dec 02, 2009 at 17:26

You might want to look at vtd-xml, the latest and most advanced xml processing model, far better than DOM or SAX

Report as Spam

19. Valentin Jan 06, 2010 at 03:26

Might be a bit late to post here, but I can't get this to work at all...

I followed step by step but get this error:

NoMethodError: undefined method `update_from_feed' for #<Class:0x103804fb8>

I don't really see where I am going wrong, as I have that method defined in the model ...

Thanks!

Report as Spam

20. uggs online Jan 07, 2010 at 01:28

Good sharing!
I don’t know whetheryou can read this message, but I think I should tell you something about my attitude for you! I am so grateful for your sharing.

Report as Spam

21. brand clothing Jan 07, 2010 at 21:41

Great! Thanks for this Ryan

Report as Spam

22. Metaxalone online Jan 10, 2010 at 15:05

fef trefrtgr

Report as Spam

23. order skelaxin from canadian pharmacy Jan 10, 2010 at 15:11

f ef erfew fwefwefew order skelaxin from canadian pharmacy

Report as Spam

24. wholesale nike shoes Jan 13, 2010 at 23:06

A very good article, I will always come in.