17
votes
Text categorization: combining different kind of features
If I understand correctly, you essentially have two forms of features for your models. (1) Text data that you have represented as a sparse bag of words and (2) more traditional dense features. If that ...
- 800
11
votes
Accepted
Can we compare a word2vec vector with a doc2vec vector?
In paragraph vector, the vector tries to grasp the semantic meaning of all the words in the context by placing the vector itself in each and every context. Thus finally, the paragraph vector contains ...
- 1,954
8
votes
How can you build a model that extracts data out from receipts?
The simplest pipeline would be to do the following:
OCR
Named Entity Extraction
Entity Disambiguation
OCR
This is basically transforming your receipts into plain text.
If you have scans (pictures) ...
- 3,518
7
votes
Accepted
Difference between paragraph2vec and doc2vec
There may be differing implementations, but these two terms refer to the same thing.
Both convert a generic block of text into a vector similarly to how word2vec converts a word to vector. Paragraph ...
- 578
5
votes
Why do popular search engines not follow the usual AND, OR logic for queries?
Nice question!
An exact answer should be given by looking in the search engine source code but here is a possible explanation.
I run the queries at Google
burglar 33,800,000
burglar AND burglar 29,...
- 2,623
4
votes
Best way to search for a similar document given the ngram
You could use a hashing vectorizer on your documents. The result will be a list of vectors. Then vectorize your ngrams in the same way and calculate the projection of this new vector on the old ones. ...
- 550
4
votes
What algorithm to use for a specific 'Named Entity Recognition'/'Information extraction' problem
Assuming your financial documents have a consistent structure and format and despite the algorithm kind of becoming "unfashionable" as of late due to the prevalence of deep learning, I would suggest ...
4
votes
How can conclusions be drawn from recommendation systems evaluation?
"Good", I think, is based on the state of the art at the moment. So I would look at respected models from industry leaders and use their reported accuracies as a base line for what is "good": since ...
- 567
4
votes
Why do popular search engines not follow the usual AND, OR logic for queries?
Google used to do, to some extend. For a long time, using +word could be used to require the presence of a word. So "a AND b" would be "+a +b" whereas "a OR b" would be "a b" (with a preference to ...
- 7,979
4
votes
how to evaluate top n recommendation system with movie lens dataset?
For various metrics feel free to look at various benchmarking libraries including MyMediaLite and LibRec. If you are doing a TOP N approach, then the way to evaluate this using a Movielens system is ...
- 141
4
votes
Accepted
How is "relevance" defined in information retrieval outside the context of systems with user feedback?
There is no formal definition for the concept of relevance, because relevance depends completely on the context and is therefore highly subjective. This is why the best way (some might say the only ...
- 25k
3
votes
Accepted
Plotting Precision Recall Curve
In short, the precision-recall curve shows the trade-off between the two values as you change the strictness of the classifier.
There is a great explanation here, using the classification of images ...
- 240
3
votes
How can conclusions be drawn from recommendation systems evaluation?
First of all, it's important to recall that RMSE has the same unit as the dependent variable (DV). It means that there is no absolute good or bad threshold.
However, you can define it based on your DV....
- 1,959
3
votes
Information Extraction from Free-form text to create Transactions
My answer is based on couple of assumptions:
user input is more or less standard, so there won't be "Ex 20000"
you have at least majority of forms of input covered
In every representative example of ...
- 779
3
votes
Why are there currently no content-based evaluation metrics for information retrieval?
Using a word-based metric would explicitly favor word-level retrieval methods. The theory is that (just as you suggest with dwell time), the URL-level metric measures more directly the desired result....
- 3,077
3
votes
Accepted
Doc2Vec or Word2vec for word embedding
As the name implies, doc2vec generates vectors representing documents (sentences, paragraphs) but not single words. So training doc2vec won't give you word vectors but document vectors. This means you ...
- 146
3
votes
I have 50 videos. I ask a customer 10 questions. Based on their answers, I send them a set of videos. How do I do it?
Frame this as a classification problem and learn a decision tree to map question responses to video selections.
EDIT: Fleshing this out a bit more:
Collect appropriate data. Get members of your ...
- 3,208
3
votes
Accepted
Can macro F1 score be greater than micro F1 score?
Yes, it can.
Example with Precision
Class A: 1 TP and 1 FP
Class B: 10 TP and 90 FP
Class C: 1 TP and 1 FP
Class D: 1 TP and 1 FP
Here, $P_A = P_C = P_D = 0.5$, $P_B = 0.1$
Macro-F1 is: $P_M = \...
- 1,039
3
votes
How "similarity" is measured in image retrieval?
There is no single best similarity metric, unless a query and found images are near identical.
Similarity is not a universal concept. It is trained. Maybe it happened to you to say that a person A ...
- 131
2
votes
Accepted
Typing error handling n-gram character index and vector space model
Trigram models can be more powerful for document retrieval than unigram models, but if you want to handle spelling errors, they will not be of much help. You need some form of fuzzy matching for that....
- 3,077
2
votes
Accepted
Tokenizing words of length 1, what would happen if I do topic modeling?
The libraries usually exclude 1-length tokens and tokens with no alpha-numeric characters because typically they are noise and do not have any descriptive power. That is, these tokens are usually not ...
- 2,860
2
votes
How can conclusions be drawn from recommendation systems evaluation?
You've already mentioned a few metrics and I guess they work, at least from a algorithmic point of view. However, I think a lot of the initial validation would really has to be done manually.
I work ...
- 773
2
votes
Plotting Precision Recall Curve
What is the x and y axis of this scatter plot? If precision and recall are on these axes, then the range of both axes would be 0 - 1. I'm assuming one point would then represent one model, in this ...
- 462
2
votes
Accepted
Text Mining of Research Paper Abstracts
In order to train some supervised learning algorithm to identify 'Problem' and 'Solution', you need to somehow generate some data that has labels of these things, which may be your best bet. So you ...
- 170
2
votes
Document similarity: Vector embedding versus BoW performance?
Here some points on which we can focus -
1.) Averaging the words vectors lose the order of words, making it very similar to the concept of Bag of Words. That is why with less Data Bag of words ...
- 136
2
votes
Learning to rank: construct absolute ranking using pair-wise ranking approach
Creating a total ranking from pairwise comparisons that don’t necessarily follow the axioms or rational preferences would certainly require some optimization, and you would need to compute a quadratic ...
- 844
2
votes
Accepted
Does recall has different interpretation when comes to classification and information retrieval
According to wikipedia, Recall is defined as-
In information retrieval, recall is the fraction of the relevant documents that are successfully retrieved.
In your second formula, recall = p/q, where ...
- 1,821
2
votes
Industrial application(s) of LDA (latent Dirichlet allocation)?
I can't answer for the systems in already existing companies, but I can definitely share an application of LDA in NLP.
Latent Dirichlet Allocation is a popular technique use for topic modelling in ...
- 146
Only top scored, non community-wiki answers of a minimum length are eligible