9
$\begingroup$

I am having an HTML string and want to find out if a word I supply is relevant in that string.

Relevancy could be measured based on frequency in the text.

An example to illustrate my problem:

this is an awesome bike store
bikes can be purchased online.
the bikes we own rock.
check out our bike store now

Now I want to test a few other words:

bike repairs
dog poo

bike repairs should be marked as relevant whereas dog poo should not be marked as relevant.

Questions:

  • How could this be done?
  • How to I filter out ambiguous words like in or or

Thanks for your ideas!

I guess it's something Google does to figure out what keywords are relevant to a website. I am basically trying to reproduce their on-page rankings.

Improve this question
$\endgroup$
5
  • $\begingroup$ Huge topic, have a look at Into to IR, this walks you from basic first principles how to build what you are asking about. Something to lookup is tf-idf then realise this doesn't solve everything and look at bayesian probability $\endgroup$
    – EdChum
    Sep 12, 2014 at 13:00
  • $\begingroup$ Why do you consider "repairs" as relevant? $\endgroup$
    – ffriend
    Sep 15, 2014 at 11:13
  • $\begingroup$ Was just an example. Didn't think too much about it :). I guess it is not relevant? $\endgroup$
    – Hendrik
    Sep 16, 2014 at 8:06
  • $\begingroup$ @Hendrik: please, use @<username> to address user - SE didn't notify me about your comment. Counting relevancy is the key point in search engines (though normally you compute how relevant is web page to a search query, you need it, right?). Do I understand it right that you just want to know how to compute relevancy of document to a search query when there are similar, but not exact words (e.g. "bike" and, say, "cyclist")? $\endgroup$
    – ffriend
    Sep 17, 2014 at 8:12
  • $\begingroup$ I guess you are asking for two things. One is a rather well understood search problem. You can just use Solr or Elasticsearch to do the heavy lifting for you. They both can find relevant documnents in a collection by weighting hits. However if you want some deeper semantic understanding of the text (i.e. "repair" is not mentioned but a typical activity in a bike shop) then the pure search engine might fall flat. $\endgroup$
    – eckes
    Sep 28, 2014 at 11:28

2 Answers 2

Reset to default
4
$\begingroup$
  • pre-process your documents (some of the steps may be skipped)
  • use a Vector Space model to represent documents (you may use TF, aforementioned TF-IDF or other models)
  • do the same with the query: preprocess and represent it in the vector space
  • find the most similar documents by computing the vector similarity (e.g. using the cosine similarity)

That's an outline of the Information Retrieval process

Introduction to Information Retrieval by Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze is a very good book to get started in IR.

Or just use Apache Solr to get everything you need out of the box (or Apache Lucene, that is used by Solr, to build your own application)

Improve this answer
$\endgroup$
1
$\begingroup$

I remember a long time ago playing with Elastic Search (the website is very different now from what I remember). There is some stuff about dealing with human language here.

Be warned that Elastic search is like a big bazooka to your problem. If your problem is very simple, maybe you want to go from scratch. There is some docs in the web about it.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.