Hot answers tagged

13

Well, I'm not sure if it is MapReduce that solves the problem, but it surely wouldn't be MapReduce alone to solve all these questions you raised. But here are important things to take into account, and that make it feasible to have such low latency on queries from all these TBs of data in different machines: distributed computing: by being distributed does ...


11

According to the XGBoost documentation, XGboost expects: the examples of a same group to be consecutive examples, a list with the size of each group (which you can set with set_group method of DMatrix in Python).


10

MapReduce has nothing to do with real-time anything. It is a batch-oriented processing framework suitable for some offline tasks, like ETL and index building. Google has moved off of MapReduce for most jobs now, and even the Hadoop ecosystem is doing the same. The answer to low latency is generally to keep precomputed indices in memory. Anything that ...


6

It is my understanding that random sampling is a mandatory condition for making any generalization statements. IMHO, other parameters, such as sample size, just affect probability level (confidence) of generalization. Furthermore, clarifying the @ffriend's comment, I believe that you have to calculate needed sample size, based on desired values of confidence ...


5

The C parameter in SVMs doesn't have to do anything with the kernel function. C is the penalty associated to the instances which are either misclassified or violates the maximal margin. As you may know already, SVM returns the maximum margin for the linearly separable datasets (in the kernel space). It might be the case that the dataset is not linearly ...


5

set_group is very important to ranking, because only the scores in one group are comparable. You can sort data according to their scores in their own group. For easy ranking, you can use my xgboostExtension.


5

Nice question! An exact answer should be given by looking in the search engine source code but here is a possible explanation. I run the queries at Google burglar 33,800,000 burglar AND burglar 29,200,000 burglar OR burglar 26,500,000 The results indeed do not respect the expected Boolean relation burglar AND burglar <= burglar OR burglar = burglar ...


4

In A/B testing, bias is handled very well by ensuring visitors are randomly assigned to either version A or version B of the site. This creates independent samples drawn from the same population. Because the groups are independent and, on average, only differ in the version of the site seen, the test measures the effect of the design decision. Slight ...


4

From my experience only some classes of queries can be classified on lexical features (due to ambiguity of natural language). Instead you can try to use boolean search results (sites or segments of sites, not documents, without ranking) as features for classification (instead on words). This approach works well in classes where there is a big lexical ...


4

The cosine similarity metric does a good (if not perfect) job of controlling for the document length, so comparing the similarity of 2 documents or 2 queries using the cosine metric and tf idf weights for the words should work well in either case. I would also recommend doing LSA first on tf idf weights, and then computing the cosine distance\similarities. ...


4

There are two rules for generalizability: The sample must be representative. In expectation, at least, the distribution of features in your sample must match the distribution of features in the population. When you are fitting a model with a response variable, this includes features that you do not observe, but that affect any response variables in your ...


4

MapReduce is not used in searching. It was used a long time ago to build the index; but it is a batch processing framework, and most of the web does not change all the time, so the newer architectures are all incremental instead of batch oriented. Search in Google will largely work the same it works in Lucene and Elastic Search, except for a lot of fine ...


4

Google used to do, to some extend. For a long time, using +word could be used to require the presence of a word. So "a AND b" would be "+a +b" whereas "a OR b" would be "a b" (with a preference to both occurring). But people did not use it much, so they eventually removed it. Google thinks it is more important to be able to process natural language queries ...


4

To wrap up you can have the following to prepare for ML interviews: Machine Learning Engineering Book Machine Learning Systems Towards Data Science (Medium): They have a lot of interesting posts there. And don't forget Data Science Stack Exchange. On this site, you can have a lot of interesting questions and answers. If you don't know the answer for your ...


3

You could use a hashing vectorizer on your documents. The result will be a list of vectors. Then vectorize your ngrams in the same way and calculate the projection of this new vector on the old ones. This is equivalent to the database join on an index, but may have less overhead.


3

Google AdWords. That has absolute search volumes.


3

I thought of expanding a bit on the answer by Stanpol. While recommendation system is one approach of suggesting related queries, one more standard information retrieval based approach is the query expansion technique. Generally speaking, query expansion involves selecting additional terms from the top ranked documents retrieved in response to an initial ...


3

I am not working at Google, but I think it is some sort of recommendation system based on the words which millions of users searched before. So those people who search for "animals" often search for "wild animals" for example. Like in many online stores they recommend you to buy something in addition to the product you are looking for based on the previous ...


2

Google isn't going to give away their proprietary work, but we can speculate. Here's what I can gather from my limited usage: The recommendations do not seem to be user, geography, or history specific. There is never an empty recommendation (one that returns no results) There is not always a recommendation (some searches just return images) The ...


2

When using the public Google APIs to retrieve results, I was only able to collect 4-10 results per query. Here's how to get more than 10 results per query: https://support.google.com/customsearch/answer/1361951?hl=en Google Custom Search and Google Site Search return up to 10 results per query. If you want to display more than 10 results to the user, you ...


2

To answer a simpler, but related question, namely 'How well can my model generalize on the data that I have?' the method of learning curves might be applicable. This is a lecture given by Andrew Ng about them. The basic idea is to plot test set error and training set error vs. the complexity of the model you are using (this can be somewhat complicated). If ...


2

you might want to look at this paper Predicting Clicks: Estimating the Click-Through Rate for New Ads Whenever an ad is displayed on the search results page, it has some chance of being viewed by the user. The farther down the page an ad is displayed, the less likely it is to be viewed. As a simplification, we consider the probability that an ad ...


2

I would recommend regexp pattern matching. I know that usual implementations are slow but you have to study Thompson's construction algorithm for nondeterministic automaton. See the wikipedia dedicated article. However here the wikipedia fails to present this treasure properly. I would strongly recommend to study carefuly this blog article: Regular ...


2

Do you change the patterns often? If not, then you can use Aho-Corasick method, whose idea is, first, to build a finite automaton based on your patterns and, then, to make a single pass over the text with this automaton to find matches (if the automaton visits a "matching" state, then there is a match). The complexity of the automata building should be ...


2

I strongly recommend using scaling as described above because it is faster than the manual method. If for some reason, scaling/preprocessing is unavailable, please use the metric parameter to pass a custom weighting function. See the example below. import numpy as np from sklearn.neighbors import KNeighborsClassifier as KNN arr = np.random.randn(500, 10) # ...


2

Have you seen this paper? Optimizing Search Engines using Clickthrough Data I stumbled upon this the other day, and I'm still reading through it, but the author attempts to deal with the problem you describe. You may also find Improving Web Search Ranking by Incorporating User Behavior Information useful.


2

With any search engine you will be limited by number of requests and any way of outcoming those limits will be a gray zone of violation of end user agreement (and, eventually, you will get banned for some time, of course). You should be looking into Search APIs of known search engines, for example, Bing gives you 5000 searches per month for free which - for ...


2

There are two main paths: Load all vectors into memory. If you are able to load vectors into memory, then you might be able to search the space with "clever" brute force. One such method is found in this paper. Keep vectors on disk. If you follow this path, then you have to index the vectors. You are basically building a search engine. Common open source ...


2

That is problem is call identification, mapping a percept to a specific entity. One common option is hashing, take a percept and map it to a specific, unique integer. If two different percepts map to the same integer, they are the same entity. If two different percepts do not map to the same integer, they are different entities. Hashing takes constant time ...


2

This problem is called record linkage and there are methods to avoid iterating the whole cartesian product. The main method I know was called "blocking" and consists in doing a first "rough" pass to create groups of matching candidates (the "blocks"). For example you can create groups which contain at least X n-grams in common. This can be done through one ...


Only top voted, non community-wiki answers of a minimum length are eligible