17

If I understand correctly, you essentially have two forms of features for your models. (1) Text data that you have represented as a sparse bag of words and (2) more traditional dense features. If that is the case then there are 3 common approaches: Perform dimensionality reduction (such as LSA via TruncatedSVD) on your sparse data to make it dense and ...


14

Depends on the user's intent, for starters. Users normally only view the first set of links, which means that unless the link is viewable, it's not getting clicks; meaning you'd have to be positive those are the best links, otherwise the clicks are most likely going to reflect placement, not relevance. For example, here's a click and attention distribution ...


10

Linear models simply add their features multiplied by corresponding weights. If, for example, you have 1000 sparse features only 3 or 4 of which are active in each instance (and the others are zeros) and 20 dense features that are all non-zeros, then it's pretty likely that dense features will make most of the impact while sparse features will add only a ...


10

In paragraph vector, the vector tries to grasp the semantic meaning of all the words in the context by placing the vector itself in each and every context. Thus finally, the paragraph vector contains the semantic meaning of all the words in the context trained. When we compare this to word2vec, each word in word2vec preserves its own semantic meaning. Thus ...


7

For my part I can say that I use click frequency on i.e. eCommerce products. When you combine it with the days of the year it can even bring you great suggestions. i.e.: We have historical data from 1 year over 2 products (Snowboots[], Sandalettes[]) Snowboots[1024,1253,652,123,50,12,8,4,50,148,345,896] Sandalettes[23,50,73,100,534,701,1053,1503,1125,453,...


7

There may be differing implementations, but these two terms refer to the same thing. Both convert a generic block of text into a vector similarly to how word2vec converts a word to vector. Paragraph vectors don't need to refer to paragraphs as they are traditionally laid out in text. They can theoretically be applied to phrases, sentences, paragraphs, or ...


7

The simplest pipeline would be to do the following: OCR Named Entity Extraction Entity Disambiguation OCR This is basically transforming your receipts into plain text. If you have scans (pictures) of the receipts, then you need a method that can deal with images. For example, you could use tesseract. Named Entity Extraction This is detecting the parts ...


5

One approach would be to use tf-idf score. The words which occur in most of the queries will be of little help in differentiating the good search queries from bad ones. But ones which occur very frequently (high tf or term-frequency) in only few queries (high idf or inverse document frequency) as likely to be more important in distinguishing the good queries ...


5

Without a sample of your data, it's unclear what's the structure of your data and what tool is suitable to process it. Here are some blind recommendations based on my experience: If you just need some flexibilty parsing the text record, such as variable repeat number of certain field, or conditional parsing of fields, then you should check out this python ...


5

Is it valid to use click frequency, then yes. Is it valid to use only the click frequency, then probably no. Search relevance is much more complicated than just one metric. There are entire books on the subject. Extending this answer beyond a simple yes/no would likely make the answer far too broad (and opinionated)


5

Nice question! An exact answer should be given by looking in the search engine source code but here is a possible explanation. I run the queries at Google burglar 33,800,000 burglar AND burglar 29,200,000 burglar OR burglar 26,500,000 The results indeed do not respect the expected Boolean relation burglar AND burglar <= burglar OR burglar = burglar ...


4

As a naive solution I would suggest to first select the strings which contain the most frequent tokens inside the list. In this way you can get rid of irrelevant string. In the second phrase I would do a majority voting. Assuming the 3 sentences: Star Wars: Episode IV A New Hope | StarWars.com Star Wars Episode IV - A New Hope (1977) Star Wars: Episode IV -...


4

Assuming your financial documents have a consistent structure and format and despite the algorithm kind of becoming "unfashionable" as of late due to the prevalence of deep learning, I would suggest that you try using Conditional Random Fields (CRF). CRFs offer very competative performance in this space and are often used for named entity recognition, part ...


4

"Good", I think, is based on the state of the art at the moment. So I would look at respected models from industry leaders and use their reported accuracies as a base line for what is "good": since it comes down to what is possible.


4

For various metrics feel free to look at various benchmarking libraries including MyMediaLite and LibRec. If you are doing a TOP N approach, then the way to evaluate this using a Movielens system is simple convert the ratings into binary likes and dislikes based on some threshold. Essentially you would take the "likes" of a user. Find the user is the testing ...


4

Google used to do, to some extend. For a long time, using +word could be used to require the presence of a word. So "a AND b" would be "+a +b" whereas "a OR b" would be "a b" (with a preference to both occurring). But people did not use it much, so they eventually removed it. Google thinks it is more important to be able to process natural language queries ...


4

There is no formal definition for the concept of relevance, because relevance depends completely on the context and is therefore highly subjective. This is why the best way (some might say the only way) to evaluate relevance is to actually ask users what is relevant for them. For any ML-based task, one needs to design a proper evaluation framework in order ...


3

You could use a hashing vectorizer on your documents. The result will be a list of vectors. Then vectorize your ngrams in the same way and calculate the projection of this new vector on the old ones. This is equivalent to the database join on an index, but may have less overhead.


3

Using a word-based metric would explicitly favor word-level retrieval methods. The theory is that (just as you suggest with dwell time), the URL-level metric measures more directly the desired result. More concretely, consider a search of "alcohol from potatoes." Assume we have two pages: 1) A page that is simply a grocery list (containing "alcohol" ...


3

First compute the edit distance between all pairs of strings. See http://en.wikipedia.org/wiki/Edit_distance and http://web.stanford.edu/class/cs124/lec/med.pdf. Then exclude any outliers strings based on some distance threshold. With remaining strings, you can use the distance matrix to identify the most central string. Depending on the method you use, ...


3

My answer is based on couple of assumptions: user input is more or less standard, so there won't be "Ex 20000" you have at least majority of forms of input covered In every representative example of transaction description you would need to mark words of interest, be it name of holder and account number. You can start small with 10-20 examples for start, ...


3

First of all, it's important to recall that RMSE has the same unit as the dependent variable (DV). It means that there is no absolute good or bad threshold. However, you can define it based on your DV. For a datum which ranges from 0 to 1000, an RMSE of 0.7 is small, but if the range goes from 0 to 1, it is not that small anymore. However, although the ...


3

In short, the precision-recall curve shows the trade-off between the two values as you change the strictness of the classifier. There is a great explanation here, using the classification of images of airplanes and geese as an example. A good way to characterize the performance of a classifier is to look at how precision and recall change as you change ...


3

As the name implies, doc2vec generates vectors representing documents (sentences, paragraphs) but not single words. So training doc2vec won't give you word vectors but document vectors. This means you can't replace word2vec by doc2vec at all. Here's how the authors of the underlying paper describe what doc2vec does: Our algorithm represents each document ...


3

Frame this as a classification problem and learn a decision tree to map question responses to video selections. EDIT: Fleshing this out a bit more: Collect appropriate data. Get members of your target population to complete the survey and also indicate which videos they think would be appropriate to them, or alternatively have subject matter experts ...


2

You could use Topic Modeling as described in this paper: http://faculty.chicagobooth.edu/workshops/orgs-markets/pdf/KaplanSwordWin2014.pdf They performed Topic Modeling on abstracts of patents (limited to 150 words). They identified papers as "novel" if they were the first to introduce a topic, and measured degree of novelty by how many papers in the ...


2

you might want to look at this paper Predicting Clicks: Estimating the Click-Through Rate for New Ads Whenever an ad is displayed on the search results page, it has some chance of being viewed by the user. The farther down the page an ad is displayed, the less likely it is to be viewed. As a simplification, we consider the probability that an ad ...


2

Trigram models can be more powerful for document retrieval than unigram models, but if you want to handle spelling errors, they will not be of much help. You need some form of fuzzy matching for that. For example the string, "I like dosg too" would fool a unigram model because "dosg" is likely "dogs" misspelled, and it will encode it as "dosg" : 1. But ...


2

If you're just looking to rank documents according to how many appearances your words w1,..,wn contain, then there's no need for clustering or machine learning in general: Clustering your 50 results will give you a partition of these results into clusters containing results that are similar to one another and different from the results in other clusters. ...


Only top voted, non community-wiki answers of a minimum length are eligible