16 votes
Accepted

How can I ensure anonymity with queries to small datasets?

This is a difficult problem, regardless of how you look at it. But some solutions might be good enough. Here, you have the advantage that you are returning aggregate statistics, and not individual ...
amon's user avatar
  • 276
12 votes
Accepted

How fit pairwise ranking models in XGBoost?

According to the XGBoost documentation, XGboost expects: the examples of a same group to be consecutive examples, a list with the size of each group (which you can set with ...
amyrit's user avatar
  • 256
7 votes

How can I ensure anonymity with queries to small datasets?

As the other answer points out, total anonymity is difficult to accomplish especially when considering an adversary who is trying to break your data privacy scheme. But, that might not be what you're ...
David's user avatar
  • 171
6 votes

How fit pairwise ranking models in XGBoost?

set_group is very important to ranking, because only the scores in one group are comparable. You can sort data according to their scores in their own group. For ...
bigdong's user avatar
  • 61
5 votes

Why do popular search engines not follow the usual AND, OR logic for queries?

Nice question! An exact answer should be given by looking in the search engine source code but here is a possible explanation. I run the queries at Google burglar 33,800,000 burglar AND burglar 29,...
DaL's user avatar
  • 2,633
4 votes

Why do popular search engines not follow the usual AND, OR logic for queries?

Google used to do, to some extend. For a long time, using +word could be used to require the presence of a word. So "a AND b" would be "+a +b" whereas "a OR b" would be "a b" (with a preference to ...
Has QUIT--Anony-Mousse's user avatar
4 votes
Accepted

Preparing for a Machine Learning Design Interview

To wrap up you can have the following to prepare for ML interviews: Machine Learning Engineering Book Machine Learning Systems Towards Data Science (Medium): They have a lot of interesting posts ...
Cuca's user avatar
  • 68
2 votes

Weighted k nearest neighbor search

I strongly recommend using scaling as described above because it is faster than the manual method. If for some reason, scaling/preprocessing is unavailable, please use the ...
kate-melnykova's user avatar
2 votes
Accepted

Gathering the number of Google results from a large amount of searches.

With any search engine you will be limited by number of requests and any way of outcoming those limits will be a gray zone of violation of end user agreement (and, eventually, you will get banned for ...
chewpakabra's user avatar
2 votes

Is the search for a specific n-gram the same like a string search?

Well, as you state the problem, it is true that the search for a certain sequence of strings/words is the same as looking for the corresponding n-gram. However, keep in mind that an n-gram, when you ...
Peter's user avatar
  • 7,456
2 votes
Accepted

How to efficiently iterate a supervised model over the Cartesian product of very large number of records?

This problem is called record linkage and there are methods to avoid iterating the whole cartesian product. The main method I know was called "blocking" and consists in doing a first "rough" pass to ...
Erwan's user avatar
  • 25.3k
2 votes

How can one quickly look up people from a large database?

That is problem is call identification, mapping a percept to a specific entity. One common option is hashing, take a percept and map it to a specific, unique integer. If two different percepts map to ...
Brian Spiering's user avatar
2 votes

Preparing for a Machine Learning Design Interview

Besides the amazing book suggested by @Carlos Mougan, you might also have a look at this one: https://github.com/chiphuyen/machine-learning-systems-design/blob/master/build/build1/consolidated.pdf
user1825567's user avatar
  • 1,396
2 votes

Best method for similarity searching on 10,000 data points with 8,000 features each in Python?

There are two main paths: Load all vectors into memory. If you are able to load vectors into memory, then you might be able to search the space with "clever" brute force. One such method is found in ...
Brian Spiering's user avatar
2 votes
Accepted

How to combine two vector embeddings into one?

How do you plan on using these embeddings? You can definitely use concatenated embeddings for similarity/retrieval, but only when comparing concat embeddings to other concat embeddings. Your point ...
Karl's user avatar
  • 611
1 vote

What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?

This is a combination of text similarity measures and a large database of popular queries. It's quite easy in the case of small closed sets, like git commands: there are only a few possible commands, ...
Erwan's user avatar
  • 25.3k
1 vote

Loading a Keyword and Evaluating the Information

You ask for quite a bit. The webpage is dynamic, so you would need to run an automated browser, navigate to the search page, enter the service number, press "search" and see if there is any item ...
Peter's user avatar
  • 7,456
1 vote

Is the search space of Hyperparameters Continuous or Discrete?

Continuous means only you have continuous variables. It can be convex or concave. It might not even be differentiable. Gradient descent only applies to differentiable convex problems (or convex ...
Piotr Rarus's user avatar
1 vote

Is there an algorithm for sampling shortest paths?

I don't know whether this answers your question. But have a look in these algorithms: The Shortest Path Faster Algorithm (SPFA) is an improvement of the Bellman–Ford algorithm which computes single-...
Pluviophile's user avatar
  • 3,848
1 vote

Need some info regarding string matching algorithms?

Try the simplest approach first - deterministic check looking for intersection overlap between the set of fruit names and the set of items bought. Set comparisons are scalable because the look-up ...
Brian Spiering's user avatar
1 vote

How to dual encode two sentences to show similarity score

How can I create a dual encoder though? Do I use two different neural networks? Or does output just contain one neuron that outputs similarity? You don't need to create two different neural ...
Caxton's user avatar
  • 161
1 vote

Is Elastic Search recommended if attribute getting search is not a huge text document?

As you mentioned you are searching just names, emails or ids, etc which is not large text. So consider a case where you have 6 documents/records having names as below then you can understand better ...
Aman Tandon's user avatar
1 vote

how to improve searching index in dataframe

This probably doesn't take into account that dates are sorted and thus performs as O(n). Try using binary-search on dates, that ...
user58692's user avatar
1 vote

How to deal with position bias in search?

What are your output (prediction) and inputs? The bias in itself is very important in predicting the click-through rate, but definitely less useful if you want to assess your ad copy/ page title in ...
The Lyrist's user avatar
1 vote

Algorithms/services to know an "iPhone case" is not an "iPhone", in the context of complex item descriptions?

The problem you are describing is called "query rewriting", taking a user's string literal and processing it to find the best items in the search index. You can always start with rule-based logic to ...
Brian Spiering's user avatar
1 vote
Accepted

Grid Search and High Variance

As mentioned it can be a good idea to repeat CV a few times and average the results to obtain a more reliable estimate If you find many parameter constellations that are within one standard deviation ...
oW_'s user avatar
  • 6,347
1 vote

Grid Search and High Variance

Averaging might help. You can optimize your hyper parameter tuning time using Bayesian Hyper-parameter tuning approach. Try to reduce the number of features or try algorithms like Random Forest with ...
Ankit's user avatar
  • 406
1 vote

Grid Search and High Variance

I'm afraid you have to repeat the K-fold CV a few times (with different seeds each time) and average the results. I'd guess that the high variance comes from the small size of the dataset.
Stergios's user avatar
  • 300
1 vote

How to search collection of podcasts (.mp3 files)?

Transcribe then topic model the recordings. This will let you know which podcast talks about the subject of interest, then you can search the transcript. If you really need it to pinpoint the moment ...
Emre's user avatar
  • 10.5k
1 vote

Why keep vocabulary and posting list separate in a search engine

It has many reasons to be (performance, design, storage, compression, evaluation of data structures). The principal reason is that all structures are verified in the practice, but you can make you own ...
Intruso's user avatar
  • 111

Only top scored, non community-wiki answers of a minimum length are eligible