Railscasts - Thinking Sphinx

#120

Jul 28, 2008

34 comments

Thinking Sphinx

If you need a full text search engine, Thinking Sphinx is a great solution. See why in this episode.

Tags: active-record plugins

Download (27.5 MB, 12:32)

alternative download for iPod & Apple TV (16.9 MB, 12:32)

Resources

script/plugin install git://github.com/freelancing-god/thinking-sphinx.git
rake thinking_sphinx:index
rake thinking_sphinx:start

# models/author.rb
define_index do
  indexes content
  indexes :name, :sortable => true
  indexes comments.content, :as => :comment_content
  indexes [author.first_name, author.last_name], :as => :author_name
  
  has author_id, created_at
end

# articles_controller.rb
@articles = Article.search params[:search]
# :include => :author 
# :conditions => { :created_at => 1.week.ago.to_i..Time.now.to_i }
# :order => :name
# :field_weights => { :name => 20, :content => 10, :author_name => 5 }
# :match_mode => :boolean
# :page => 1, :per_page => 20

Full Source Code

< Previous Episode

Next Episode >

34 comments

1. RSL Jul 28, 2008 at 05:17

Outstanding job, Ryan. I think this is your best Railscast yet! Great job showing simultaneously how simple it is to use Thinking Sphinx and how much power you can leverage by drilling down into it. I'll be watching this one again. :)

2. Morten Jul 28, 2008 at 05:42

As always you cover something real applications actually need instead of some obscure and useless function.

3. Swami Jul 28, 2008 at 06:34

Long time fan here. Thanks for the continued outstanding quality of your railscasts.

Ferret used to be the de-facto choice. Any reasons not to use Ferret? Why having chosen Sphinx over other solutions?

Is it possible to perform searches across several models?

4. andrej Jul 28, 2008 at 07:11

Thanks for the great screen casts. Keep them coming!

Is it possible to do result highlighting with Sphinx to show the search terms?

5. CaseyJ Jul 28, 2008 at 08:13

Crazy timing! Just implemented Thinking Sphinx into a major project in place of Solr.

6. Ryan Bates Jul 28, 2008 at 08:39

@Swami, I haven't tried Ferret, so I can't comment too much about that. I have heard it has some stability issues, but that's all I know.

As for searching across multiple models, I believe it's possible with "ThinkingSphinx::Search.search", but I haven't tried it.

@andrej, I don't think Sphinx gives an easy way to highlight the search terms. However you could use the "highlight" helper method Rails provides.

7. MikeInAZ Jul 28, 2008 at 08:48

Thanks for the screencast Ryan, I just implemented TS last week.

Also, instead of thinking_sphinx:index command, you can just type ts:in

ts:start
ts:stop

8. Ryan Bates Jul 28, 2008 at 10:13

@MikeInAZ, thanks for pointing that out. I normally use auto-completion for rake tasks as you may have noticed in this episode. If anyone's interested in that, see my dotfiles:

http://github.com/ryanb/dotfiles

9. Evan Dorn Jul 28, 2008 at 11:01

Thanks for the 'cast, Ryan.

I just converted a large application from Ferret to Sphinx last week, and ran into a whole bunch of problems. I wrote a long <a href="http://blog.lrdesign.com/2008/07/fixing-problems-with-sphinx-search/">blog post</a> about the trials and tribulations I went through to get it working, including the fixes.

Though we were using ultrasphinx instead of thinking_sphinx, some of the struggles will be relevant to everyone using sphinx, for example the fact that foxy fixtures generates IDs that are often too large for sphinx's indexing strategy.

10. Evan Dorn Jul 28, 2008 at 11:03

Whoops, sorry about that, didn't realize your comments don't allow links. (suggestion: a preview button)

Here's the bare URL I mentioned in my previous comment:
http://blog.lrdesign.com/2008/07/fixing-problems-with-sphinx-search/

11. jblanche Jul 28, 2008 at 11:13

Outstanding screencast !

Just one question, how would you manage a more complicated search form.
Let's say that the user can say if he wants to search only by title, author or comments.

Is there an easy way to filter the indexes to use ?

12. Lenart Rudel Jul 28, 2008 at 11:22

@Swami: I just read this post (http://freelancing-gods.com/posts/thinking_sphinx_reborn)

Multi-model searching is (obviously) supported:
ThinkingSphinx::Search.search "help"

Haven't tried it though. Hope it helps!

13. Ryan Bates Jul 28, 2008 at 11:36

@Evan, preview button is coming very soon. :)

@jblanche, you can use the :conditions hash to search by specific fields. I think that will do what you need.

14. Jose Jul 28, 2008 at 21:04

How about a episode showing how to use you dotfiles...looks nice on this episode :P

15. Rafael Jul 29, 2008 at 05:00

Hey, just a suggestion, what about put the last comment in the top of the page. So, we can read from the more recent to the oldest. =P

16. Xavier Noria Jul 29, 2008 at 07:27

My English listening is really weak :-), can you please write somewhere what provides instant indexing?

17. Ryan Bates Jul 29, 2008 at 07:48

@Rafael, thanks for the suggestion. Most blogs do the earlier comments first, so I think it's best to follow that convention. But I'll consider it.

@Xavier, are you referring to delta indexing? See the bottom of the Thinking Sphinx Usage page for more info on that.

http://ts.freelancing-gods.com/usage.html

18. Lenart Rudel Jul 29, 2008 at 11:07

@Ryan, I'd agree with Rafel's idea.

Maybe the comment form could also be moved up (and made collapsable to save space?). This might encourage users to post (even) more comments :)

19. Michael Jul 29, 2008 at 12:59

First of all... thanks for the amazing episode again! Thanks for you r help with all the episodes you are making.

I have one thing what i dont understand. How can you manage this on a webhost server you dont maintain yourself? You have to restart it with ssh every time somebody wrote an article or smt else?

20. Ryan Bates Jul 29, 2008 at 17:07

@Michael, it's best to set up a "cron" job to trigger the rake index task. If you aren't able to do this then you may be able to get your Rails app to spawn a background process to redo the index at a regular interval. I'm not exactly sure how that would work though.

21. shuaib Jul 29, 2008 at 18:52

Well done, I really learnt a lot from this cast and so from others. You have made it very simple to me to understand it without reading a single line on thinking sphinx. Thank you, waiting more interesting topics from you

22. Jeff Cole Jul 30, 2008 at 09:59

Thanks for the podcast Ryan, I've been looking into the various search technologies for a while. As far as I can tell, one advantage to Ferret is it supports a "fuzzy" string match for search parameters. Any suggestions on how to handle this with Sphinx?

23. Ryan Bates Jul 30, 2008 at 11:39

@Jeff, sphinx has some "fuzzy" matching for phrases, where there need to be a certain number of words a given distance a part. See the "extended" match mode for this behavior.

But I don't think this is what you're looking for. If you want fuzzy searching on a given word, where similar words are also found, then I don't think Sphinx supports this.

You may want to look into Xapian. It has some powerful features regarding this. Including stemming which can find variations of a given word (run, runs, running, etc.). it also supports spelling correction to find terms which are similar to a given word.

24. Michael Jul 30, 2008 at 14:25

Is there anybody lucky with installing sphinx on a Windows computer?

if i use the install link here above it says that vendor/plugin/sphinx is removed again. And after that it wont start sphinx

25. Felipe Giotto Jul 31, 2008 at 05:02

Hi, Ryan!
I always liked all your episodes! I'm buildind a Rails programming team at my company and I'm showing some of your episodes a day. Congratulations for the great work! :D
Best regards,
Felipe Giotto.

26. rob Jul 31, 2008 at 10:35

hi Ryan....is there any possibility that you can zip and publish this project on your site (and all future projects)...

27. Pat Allan Jul 31, 2008 at 18:54

Jeff: Sphinx has support for fuzzy matching, but it's not enabled by default.

If you're using Thinking Sphinx, you can enable this by adding allow_star: true for every environment in config/sphinx.yml (ie: in a similar format to database.yml).

28. Ryan Bates Aug 01, 2008 at 16:43

@rob you can find the full code by clicking the "Full Source Code" link at the bottom of the code samples.

29. Sean Hess Aug 04, 2008 at 05:33

@andrej: Don't you guys use Javascript for anything? :) Thanks for the great episode Ryan!

30. Neil Aug 09, 2008 at 14:53

If anyone does work out the best route for highlighting search terms, please post back. I intend on giving them a solid background colour rather than using the highlight visual effect.

Sean Hess, I'm a new to js and Rails, are you suggesting an application.js script that takes the params and highlights the DOM elements that match?

Indexing is taking ages on my MBP...

p.s. thanks again, Ryan.

31. Loqi Aug 22, 2008 at 17:32

Is there an error in the video?

I have been having trouble getting the :field_weights parameter to behave. I can't find it mentioned in the thinking_sphinx API entries on the various search methods:
http://ts.freelancing-gods.com/rdoc/

The best I can find is this:
http://ts.freelancing-gods.com/rdoc/classes/ThinkingSphinx/Index/Builder.html
Where it says :field_weights is an argument to the set_property method.

This seems to imply that field weighting happens when the index is built, not when the search is performed. I'd rather be able to declare my relevancy preferences at search time (like the video shows), not at index time (like the API seems to say).

Am I missing something here?

32. Loqi Aug 22, 2008 at 17:38

Oops. Forgot to attach my email address to that message.

Great work, Ryan. Your generous contributions of work have saved me a great deal of aggravation ever since I discovered them.

33. Loqi Aug 23, 2008 at 21:01

Okay, I think I found the problem.

The match_mode => :boolean ignores all field weighting, but match_mode => :extended honors them. This is a bit puzzling, since Extended seems to be just like Boolean, but with even more query features. Extended lets you use boolean ORs and NOTs, without losing the relevancy rankings.

Here's where Sphinx says it does that:
http://sphinxsearch.com/doc.html#weighting

And here's a code sample:

--- In app/models/article.rb
define_index block
  ...
  # Declare a default relevancy table, which can be overridden at search
  set_property :field_weights => { :title => 3, :subtitle => 2 }
  ...
end

--- In app/controllers/article_controller.rb
  ...
  # Grab a page-load of articles, using title-centric field weights.
  @articles = Article.search query_string,
      :include => [:author, :comments], :match_mode => :extended,
      :field_weights => { :title => 10, :subtitle => 6 },
      :page => params[:page], :per_page => 20

34. Brandon Aug 24, 2008 at 15:54

I was thinking about the best way to run a daemon to reindex TS at a certain interval. Would it make sense to use the daemon gem in an initializer to kick off this process?

Thinking Sphinx

34 comments

Add your comment: