Skip to main content

Questions tagged [search]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Search for documents with similar texts

I have a document with three attributes: tags, location, and text. Currently, I am indexing all of them using LangChain/pgvector/embeddings. I have satisfactory results, but I want to know if there is ...
Skhaz's user avatar
  • 101
0 votes
1 answer
21 views

Searching ChatGPT transcripts?

Maybe I just don't understand the functionality of the web version. Using GPT4 here. When I do technical queries (usually TensorFlow applications) I use similar prompts until I get the understanding I ...
PracticalKat's user avatar
0 votes
1 answer
137 views

How to combine two vector embeddings into one?

I want to use OpenCLIP for generating embeddings for each slide in an array of pptx presentations. To improve the quality of the results, I want to vectorize both slide text content and preview images....
Olek Gornostal's user avatar
0 votes
0 answers
25 views

What is the state of art production search algorithm right now for semantic search? LSH? or other clustering method?

Trying to implement semantic search for high cardinality embedding for my own learning purpose, so far LSH seems promising, but I am wondering what is the state of algorithm big tech company are using ...
progr's user avatar
  • 1
0 votes
0 answers
7 views

How to label a dataset of text pairs to use it as a universal one for calculating the precision@k metric for different models?

I am facing a semantic search problem. I am fine tuning different NLU models and i want to use precision@k as my main metric. Is it possible to label a dataset of text pairs to use it as a universal ...
Ir8_mind's user avatar
  • 183
0 votes
0 answers
18 views

Purpose of Azure Cognitive Search

Azure provides a service called Cognitive Search which is an intelligent AI-based search service based on advanced NLP. I tried this feature. And to make the search as efficient as possible, it ...
Apoorva's user avatar
  • 307
0 votes
1 answer
145 views

Fastest way to do Maximum Inner Product Search?

There have lots of $d-$dimension vectors, they combine a set $X$. We use an input query $q$. We need to find the $p$, which has the maximum inner product from the set $X$. $$ p = \arg \max_{x \in ...
jackson's user avatar
  • 25
0 votes
0 answers
21 views

Semantic search with pretrained BERT models giving irrelevant results with high similarity

I'm trying to create a Semantic search system and have experimented with multiple pretrained models from the SentenceTransformers library: LaBSE, MS-MARCO etc. The system is working well in returning ...
Aftaab Zia's user avatar
0 votes
0 answers
62 views

Combining Textual, Categorical and Numerical data for Semantic Search using SentenceTransformers model

I'm building a food semantic search model and I want to use a pre-trained SentenceTransformers model with cosine similarity. I'm using Epicurious dataset for the corpus which consists of textual (&...
Alex's user avatar
  • 1
0 votes
0 answers
13 views

Search recall optimization - what appropriate loss function to use?

I am studying machine learning and wanted to work on a project of my own so that I have better chances after graduating college. I'm studying the application of ML to improve searches using a toy ...
user9343456's user avatar
3 votes
2 answers
2k views

Semantic search - combine text and image embedding

I have a question regarding combining text and image embeddings for semantic search. The use case is product search on a (B2B) marketplace, we have image(s) and title&description of the products. ...
Steven's user avatar
  • 31
1 vote
0 answers
26 views

What algorithm should I use when trying to find closest record matches when records contain both categorical and discrete attributes?

I have 100000 records that have discrete features like topic (analytics etc) and categorical features like ticket details (eg: I need help with analytics for my business). When creating a new record, ...
Jack Smith's user avatar
15 votes
5 answers
2k views

How can I ensure anonymity with queries to small datasets?

I'm building a service that will contain personal data relating to real people. Initially the dataset will be quite small, and as such it may be possible to identify individuals if the search ...
mal's user avatar
  • 253
1 vote
1 answer
1k views

Fine tuning BERT without pre-training it on domain specific corpus

I'm building an internal semantic search engine using BERT/SBERT + ElasticSearch 8 where answers are retrieved based on their cosine similarity with a query. The documents to be searched are somewhat ...
ruslaniv's user avatar
  • 163
1 vote
1 answer
142 views

Clustering by using Locality sensitive hashing *after* Random projection

It is well known that Random Projection (RP) is tightly linked to Locality Sensitive Hashing (LSH). My goal is to cluster a large number of points lying in a d-dimensional Euclidean space, where $d$ ...
Penelope Benenati's user avatar
1 vote
1 answer
123 views

What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?

For eg. https://stackoverflow.com/questions/307291/how-does-the-google-did-you-mean-algorithm-work this is the logic behind google's did you mean algorithm - used for spell correction suggestion. What ...
jarvis's user avatar
  • 11
1 vote
0 answers
112 views

Can pre-trained transformers (I.e., BERT) handle numerical/spatial data

I’m curious to know if pre-trained transformers could handle search queries that include numerical data or make references to spatial relationships. Take an example dataset of a list of restaurants, ...
Ellio's user avatar
  • 93
2 votes
1 answer
200 views

Size of datasets over years

I am looking for statistics, to understand the evolution of the size of the (public) dataset over the years. I just found the following statistics: The poll of KDnuggets that actually shows that over ...
asdf's user avatar
  • 133
1 vote
1 answer
48 views

Loading a Keyword and Evaluating the Information

I am an Ex Service Veteran and need assistance with a Small Program to use with my Rats of Tobruk Project, for the purpose of evaluating Archives Information, which I normally do by Manual Means, but ...
user222642's user avatar
1 vote
0 answers
23 views

Item position in Gravity Search Algorithm

According to this article titled Efficient clustering in collaborative filtering recommender system: Hybrid method based on genetic algorithm and gravitational emulation local search algorithm This ...
Andre Hrs's user avatar
6 votes
2 answers
4k views

Preparing for a Machine Learning Design Interview

I am not sure if this is a relevant post here but: I made it to the final round for the Machine Learing Engineer position at Facebook. The final round interview is virtual (thanks to Corona) and will ...
Wolfy's user avatar
  • 237
1 vote
2 answers
686 views

Is the search space of Hyperparameters Continuous or Discrete?

I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete? My understanding of both those cases: 1. Continuous would make it 'easier'...
loulours's user avatar
1 vote
0 answers
84 views

Options to find the most similar question in a dataset of question-answer pairs?

I am building a chatbot that will only handle FAQs, but these FAQs are very specific to an organisation, so I cannot use any existing off-the-shelf solutions, or connect to question-answering APIs. I ...
KOB's user avatar
  • 189
2 votes
1 answer
111 views

Is there an algorithm for sampling shortest paths?

I have a triangle matrix NxN of distances between vertices (vertex i connected only with vectices j>i) and I'd like to sample path from first to the last and use it as a training sample. Is there an ...
rx303's user avatar
  • 21
1 vote
3 answers
141 views

Need some info regarding string matching algorithms?

Let me explain a scenario to better explain my question, Assume I am working in a credit-card related company in which people uploads their receipts every month, I want to check if that person bought ...
user_12's user avatar
  • 347
1 vote
1 answer
167 views

Is the search for a specific n-gram the same like a string search?

Is the result of a search for a specific n-gram like sherlock+holmes equal to the result of a regex search for "sherlock holmes" in the same document corpus? So if i read about n-grams for certain ...
bartman99's user avatar
2 votes
1 answer
461 views

How to efficiently iterate a supervised model over the Cartesian product of very large number of records?

The problem: Two large databases, with ~1M records each, "old customer data" and "new customer data". The data came from different sources and was ingested at different times, so there are many ...
Alex S Kinman's user avatar
4 votes
3 answers
329 views

How can one quickly look up people from a large database?

Vocabulary Face detection: Finding all faces in an image. Face representation: The simplest way to represent a face is as an image (pixels / color values). This is not very space efficient and likely ...
Martin Thoma's user avatar
1 vote
2 answers
673 views

How to dual encode two sentences to show similarity score

I've been trying to grasp the concept of Google's semantic experiences. By using it, I'm planning to implement a semantic query tool. With universal sentence encoder I can first pre-encode all ...
ShellRox's user avatar
  • 409
1 vote
1 answer
30 views

Is Elastic Search recommended if attribute getting search is not a huge text document?

We are currently developing a system with MEAN stack with Mongodb at backend. We have employees name, and Ids in our system and our client wants to get pretty good (Read: Google Like) search in our ...
user3422929's user avatar
2 votes
1 answer
366 views

how to improve searching index in dataframe

Given a pandas dataframe with a timestamp index, sorted. I have a label and I need to find the closest index to that label. Also, I need to find a smaller timestamp, so the search should be computed ...
Federico Caccia's user avatar
2 votes
0 answers
140 views

CNN combined with a competitive search algorithm [closed]

I'm reading some papers about Deep Neural Networks applied for board games, like for Go with AlphaGo, AlphaGo Zero and some other games, like Othello and Chess. Most of the works are using CNN's as a ...
Matheus Prandini's user avatar
1 vote
0 answers
35 views

Search Query Sample Size Determination for validation set

While designing a search system, which searches in N identifiable categories, how many search queries does one need in each category to validate the target metric (DCG) scores accurately (balanced ...
D.S.'s user avatar
  • 111
5 votes
2 answers
1k views

How to deal with position bias in search?

In search, position of the search result affects the click-through rate a great deal. How do people usually deal with this ? In practice how to remove such bias to create unbiased training data for ...
Jing's user avatar
  • 171
-3 votes
1 answer
40 views

Algorithms/services to know an "iPhone case" is not an "iPhone", in the context of complex item descriptions? [closed]

We are trying to implement a highly accurate search, based on user-entered search terms, into a large product database. For example, if the user searches for "iPhone", then one of these is ...
spechter's user avatar
-2 votes
1 answer
74 views

What is ElasticSearch and how can it be used? [closed]

I am really quite new to this whole world but I keep hearing lots of talk about ElasticSearch and Kibana. I have been to their website but would just like a really simple, plain English explanation ...
Taylrl's user avatar
  • 488
1 vote
3 answers
412 views

Grid Search and High Variance

I am currently trying to optimise some parameters on my model (15000 samples). What I am finding is a relatively large variance in the loss function 2%-10% which makes it hard to identify which ...
simeon's user avatar
  • 173
3 votes
1 answer
244 views

Best method for similarity searching on 10,000 data points with 8,000 features each in Python?

As mentioned in the title I am attempting to search through 10,000 vectors with 8000 features each, all in Python. Currently I have the vectors saved in their own directories as pickled numpy arrays. ...
Michael Vander Meiden's user avatar
6 votes
2 answers
981 views

Why do popular search engines not follow the usual AND, OR logic for queries?

I am teaching myself Information Retrieval from Christopher Manning's book (PDF link: http://nlp.stanford.edu/IR-book/pdf/01bool.pdf). I tried Exercise 1.13: "Try using the Boolean search features on ...
user avatar
0 votes
1 answer
48 views

How to search collection of podcasts (.mp3 files)?

I have a collection of podcasts, in particular the "Talkingmachines" Podcasts ("Human conversation about machine learning"), see http://www.thetalkingmachines.com/blog/ There are 22 episodes, each 1 ...
knb's user avatar
  • 602
4 votes
1 answer
558 views

Why keep vocabulary and posting list separate in a search engine

I am taking a class in information retrieval. We learned that the index of a search engine has (possibly among other things): A vocabulary mapping terms to their statistics (frequency, type, ...) and ...
icehawk's user avatar
  • 141
1 vote
2 answers
118 views

Gathering the number of Google results from a large amount of searches.

I am trying to build a simple dataset using Google, mainly because it seems like the best option for what I want. I want to measure fame for a large group of scientists. The quick method is to ...
error_null_pointer's user avatar
26 votes
2 answers
28k views

How fit pairwise ranking models in XGBoost?

As far as I know, to train learning to rank models, you need to have three things in the dataset: label or relevance group or query id feature vector For example, the Microsoft Learning to Rank ...
tokestermw's user avatar
8 votes
5 answers
2k views

Best way to search for a similar document given the ngram

I have a database of about 200 documents who's ngrams I have extracted. I want to find the document in my database that is most similar to a query document. In otherwords, I want to find the document ...
okebz's user avatar
  • 113
0 votes
1 answer
73 views

Looking for a 'CITY, STATE' within a body of text (from a CITY-STATE database)

I'm looking for an optimal way to search a large body of text for a combination of words that resemble any CITY, STATE combination I have in a separate CITY-STATE database. My only idea would be to ...
Collarbone's user avatar
4 votes
2 answers
2k views

Weighted k nearest neighbor search

I've searched quite a bit and haven't landed on any useful results. The problem statement is: Given a set of vectors, I wish to find its approximate k-nearest neighbors. The caveat here is that each ...
sushant-hiray's user avatar
3 votes
1 answer
317 views

Ranking Bias in Learning to Rank

Users tend to click on results ranked highly by search engines much more often than those ranked lower. How do you train a search engine using click data / search logs without this bias? I.e. you don'...
Ben McCann's user avatar
5 votes
1 answer
6k views

Where is the cost parameter C in the RBF kernel in SVM?

RBF kernel using SVM depends on two parameters C and gamma. If the equation of the kernel RBF as the following: $K(X,X')= \exp(\gamma||X-X'||^2)$ In the equation I can see where can I use gamma, but ...
Weam's user avatar
  • 51
1 vote
0 answers
31 views

large database operation. check for relatedness between entities

I have one small list of entities, such as: Russia Vladimir Moscow Then I have a massive database of JSON indices. For each entry there are multiple alpha-...
smatthewenglish's user avatar
0 votes
1 answer
61 views

What approaches are there to not display a search result that a user has no permission to see? [closed]

Question What approaches are there to not display a search result that a user is not supposed to see? Example Suppose we have the following situation. High-level view In an enterprise, there are ...
user avatar