17 votes

Text categorization: combining different kind of features

If I understand correctly, you essentially have two forms of features for your models. (1) Text data that you have represented as a sparse bag of words and (2) more traditional dense features. If that ...
user avatar
  • 800
15 votes
Accepted

Does click frequency account for relevance?

Depends on the user's intent, for starters. Users normally only view the first set of links, which means that unless the link is viewable, it's not getting clicks; meaning you'd have to be positive ...
user avatar
  • 1,892
11 votes
Accepted

Can we compare a word2vec vector with a doc2vec vector?

In paragraph vector, the vector tries to grasp the semantic meaning of all the words in the context by placing the vector itself in each and every context. Thus finally, the paragraph vector contains ...
user avatar
  • 1,924
10 votes

Text categorization: combining different kind of features

Linear models simply add their features multiplied by corresponding weights. If, for example, you have 1000 sparse features only 3 or 4 of which are active in each instance (and the others are zeros) ...
user avatar
  • 2,771
8 votes

How can you build a model that extracts data out from receipts?

The simplest pipeline would be to do the following: OCR Named Entity Extraction Entity Disambiguation OCR This is basically transforming your receipts into plain text. If you have scans (pictures) ...
user avatar
7 votes

Does click frequency account for relevance?

For my part I can say that I use click frequency on i.e. eCommerce products. When you combine it with the days of the year it can even bring you great suggestions. i.e.: We have historical data from ...
user avatar
  • 579
7 votes
Accepted

Difference between paragraph2vec and doc2vec

There may be differing implementations, but these two terms refer to the same thing. Both convert a generic block of text into a vector similarly to how word2vec converts a word to vector. Paragraph ...
user avatar
  • 568
5 votes
Accepted

How to create a good list of stopwords

One approach would be to use tf-idf score. The words which occur in most of the queries will be of little help in differentiating the good search queries from bad ones. But ones which occur very ...
user avatar
5 votes

semi-structured text parsing using machine learning

Without a sample of your data, it's unclear what's the structure of your data and what tool is suitable to process it. Here are some blind recommendations based on my experience: If you just need ...
user avatar
  • 236
5 votes

Does click frequency account for relevance?

Is it valid to use click frequency, then yes. Is it valid to use only the click frequency, then probably no. Search relevance is much more complicated than just one metric. There are entire books on ...
user avatar
  • 1,102
5 votes

Why do popular search engines not follow the usual AND, OR logic for queries?

Nice question! An exact answer should be given by looking in the search engine source code but here is a possible explanation. I run the queries at Google burglar 33,800,000 burglar AND burglar 29,...
user avatar
  • 2,583
4 votes

Extract canonical string from a list of noisy strings

As a naive solution I would suggest to first select the strings which contain the most frequent tokens inside the list. In this way you can get rid of irrelevant string. In the second phrase I would ...
user avatar
4 votes

What algorithm to use for a specific 'Named Entity Recognition'/'Information extraction' problem

Assuming your financial documents have a consistent structure and format and despite the algorithm kind of becoming "unfashionable" as of late due to the prevalence of deep learning, I would suggest ...
user avatar
4 votes

How can conclusions be drawn from recommendation systems evaluation?

"Good", I think, is based on the state of the art at the moment. So I would look at respected models from industry leaders and use their reported accuracies as a base line for what is "good": since ...
user avatar
  • 567
4 votes

how to evaluate top n recommendation system with movie lens dataset?

For various metrics feel free to look at various benchmarking libraries including MyMediaLite and LibRec. If you are doing a TOP N approach, then the way to evaluate this using a Movielens system is ...
user avatar
  • 141
4 votes

Why do popular search engines not follow the usual AND, OR logic for queries?

Google used to do, to some extend. For a long time, using +word could be used to require the presence of a word. So "a AND b" would be "+a +b" whereas "a OR b" would be "a b" (with a preference to ...
user avatar
4 votes
Accepted

How is "relevance" defined in information retrieval outside the context of systems with user feedback?

There is no formal definition for the concept of relevance, because relevance depends completely on the context and is therefore highly subjective. This is why the best way (some might say the only ...
user avatar
  • 21.8k
3 votes

Best way to search for a similar document given the ngram

You could use a hashing vectorizer on your documents. The result will be a list of vectors. Then vectorize your ngrams in the same way and calculate the projection of this new vector on the old ones. ...
user avatar
  • 540
3 votes

Why are there currently no content-based evaluation metrics for information retrieval?

Using a word-based metric would explicitly favor word-level retrieval methods. The theory is that (just as you suggest with dwell time), the URL-level metric measures more directly the desired result....
user avatar
  • 3,017
3 votes

Extract canonical string from a list of noisy strings

First compute the edit distance between all pairs of strings. See http://en.wikipedia.org/wiki/Edit_distance and http://web.stanford.edu/class/cs124/lec/med.pdf. Then exclude any outliers strings ...
user avatar
  • 1,840
3 votes

Information Extraction from Free-form text to create Transactions

My answer is based on couple of assumptions: user input is more or less standard, so there won't be "Ex 20000" you have at least majority of forms of input covered In every representative example of ...
user avatar
3 votes

How can conclusions be drawn from recommendation systems evaluation?

First of all, it's important to recall that RMSE has the same unit as the dependent variable (DV). It means that there is no absolute good or bad threshold. However, you can define it based on your DV....
user avatar
3 votes
Accepted

Plotting Precision Recall Curve

In short, the precision-recall curve shows the trade-off between the two values as you change the strictness of the classifier. There is a great explanation here, using the classification of images ...
user avatar
  • 220
3 votes
Accepted

Doc2Vec or Word2vec for word embedding

As the name implies, doc2vec generates vectors representing documents (sentences, paragraphs) but not single words. So training doc2vec won't give you word vectors but document vectors. This means you ...
user avatar
  • 146
3 votes

I have 50 videos. I ask a customer 10 questions. Based on their answers, I send them a set of videos. How do I do it?

Frame this as a classification problem and learn a decision tree to map question responses to video selections. EDIT: Fleshing this out a bit more: Collect appropriate data. Get members of your ...
user avatar
  • 3,008
3 votes

How "similarity" is measured in image retrieval?

There is no single best similarity metric, unless a query and found images are near identical. Similarity is not a universal concept. It is trained. Maybe it happened to you to say that a person A ...
user avatar
2 votes

non query-based document ranking

You could use Topic Modeling as described in this paper: http://faculty.chicagobooth.edu/workshops/orgs-markets/pdf/KaplanSwordWin2014.pdf They performed Topic Modeling on abstracts of patents (...
user avatar
  • 1,840
2 votes

Can we quantify how position within search results is related to click-through probability?

you might want to look at this paper Predicting Clicks: Estimating the Click-Through Rate for New Ads Whenever an ad is displayed on the search results page, it has some chance of being viewed ...
user avatar
  • 731
2 votes
Accepted

Typing error handling n-gram character index and vector space model

Trigram models can be more powerful for document retrieval than unigram models, but if you want to handle spelling errors, they will not be of much help. You need some form of fuzzy matching for that....
user avatar
  • 3,017

Only top scored, non community-wiki answers of a minimum length are eligible