Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [search-engine]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

Unsupervised metrics for search engine

I'm the new one to ML, currently writing search engine with different models (tried LSTM, BERT, sentence-transformers) to get sentence embeddings, there are about 2k documents, and don't know how to ...
speedy's user avatar
  • 1
0 votes
0 answers
8 views

Scientific literature on de-indexing algorithms

I would like to know more about the topic of de-indexing algorithms, i.e., removing results from the search engines' index. I am interested in the theoretical basics, before moving on to more complex (...
sato's user avatar
  • 101
1 vote
1 answer
31 views

Why don't search engines filter out unethical/illegal searches?

(Not sure if this question is appropriate to this SE) I'm studying the LLMs course on Coursera. One topic they deal with is how to get the LLM to not respond with unethical/illegal things, e.g. if you ...
Allure's user avatar
  • 275
0 votes
1 answer
92 views

Semantic Search on numeric data

I have a dataset in csv format. Most columns have numbers in them. The data is mostly number based, with little text. A sample of my dataset: I want to build a semantic search engine that can answer ...
Apoorva's user avatar
  • 307
0 votes
1 answer
115 views

Purely extractive Language Model

Given an email thread, I am trying to extract the body of the most recent email. I used to do that with rules. Now I am testing Large Language Models (LLM) to see if I they provide a less ad hoc ...
mirix's user avatar
  • 103
0 votes
0 answers
21 views

Semantic search with pretrained BERT models giving irrelevant results with high similarity

I'm trying to create a Semantic search system and have experimented with multiple pretrained models from the SentenceTransformers library: LaBSE, MS-MARCO etc. The system is working well in returning ...
Aftaab Zia's user avatar
0 votes
0 answers
68 views

Finding the suitable semantic search engine tool for querying an excel/csv file containing data rich in numbers

I have an excel dataset that has the columns: name of the test, user name, data source, total number of records, number of failed records, number of passed records, date of test execution etc. I want ...
Apoorva's user avatar
  • 307
0 votes
0 answers
22 views

Query modification for elastic search using machine learning

I have a problem statement that I'm struggling to formulate as a machine learning framework. There is a huge client database of documents - we're trying to come up with an efficient way of querying ...
user9343456's user avatar
0 votes
1 answer
11 views

How to handle using input feature (clicks) when it is used in target too?

I am trying to create a ranking model, where I am thinking about creating ground truth based on clicks by user. But at same time past clicks made by users seems like a vital input feature too. Any ...
dcusmeb's user avatar
0 votes
1 answer
2k views

How do I use FAISS searching with Haystack?

I have a private SQL database with text entries in it. I want to create a search engine to query these entries. I have been looking into haystack (https://haystack.deepset.ai/overview/intro) with the ...
Mondo Duke's user avatar
1 vote
0 answers
380 views

How can I decide the threshold value for relevance score in a search problem?

I am using a LSA/TF-IDF/BM25/Ensemble models for text search and finally calculating similarity score to rank my search. I would like to decide a threshold value for the score, below which I would not ...
Prateek Coder's user avatar
1 vote
0 answers
73 views

How does google indexes text documents?

Unfortunately, I didn't find much information surrounding this, no pdfs, no textbooks that discuss this in just enough detail. And I didn't see any forums posts about this. I just want to learn an ...
jewloa's user avatar
  • 11
2 votes
1 answer
205 views

Learning to Rank with Unlabelled Dataset

I have folder of about 60k PDF documents that I would like to learn to rank based on queries to surface the most relevant results. The goal is to surface and rank relevant documents, very much like a ...
amber's user avatar
  • 51
6 votes
3 answers
3k views

Best way to vectorise names and addresses for similarity searching?

I have a large dataset of around 9 million people with names and addresses. Given quirks of the process used to get the data it is highly likely that a person is in the dataset more than once, with ...
Sandy Lee's user avatar
  • 247
0 votes
0 answers
32 views

Best methods to choose between different searching models?

My question here is in regards to best practices and current methods for selecting search models on the fly based on a users query. Lets say I have four searching models, each optimized for their ...
Pythoner's user avatar
  • 123
1 vote
1 answer
278 views

Why do we calculate the vector of a document by averaging the vectors of all the words?

I am trying to build a search engine to query a folder of documents. Tutorials online suggest that we should obtain the vector of a document by averaging the vectors of all the words, then compare ...
amber's user avatar
  • 51
1 vote
1 answer
32 views

How does Google's 'showing results for' work? [closed]

If I search 'I love to eate my food' on Google then Google will 'show results for' I love to eat my food.... How does this algorithm work?
google's user avatar
  • 19
0 votes
1 answer
24 views

Could you generate search queries to poison data analysis by a search engine?

A simple problem with search engines is that you have to trust that they will not build a profile of search queries you submit. (Without Tor or e.g. homomorphic encryption, that is.) Suppose we put ...
Christopher Done's user avatar
1 vote
0 answers
33 views

About Natural Question (NQ) benchmark in NLP [closed]

I recently learned that there is a benchmark called NQ. https://ai.google.com/research/NaturalQuestions/visualization Unlike other QA benchmarks which relevant document is povided with query, it has ...
giniper's user avatar
  • 11
1 vote
1 answer
536 views

What is the difference between Okapi bm25 and NMSLIB?

I was trying to make a search system and then I got to know about Okapi bm25 which is a ranking function like tf-idf. You can make an index of your corpus and later ...
coderina's user avatar
  • 113
1 vote
1 answer
1k views

What is the formula and log base for idf?

To calculate tf-idf, we do: tf*idf tf=number of times word occurs in document What is formula for idf and log base: Log(number of documents/number of documents containing the word) Log((1+number ...
variable's user avatar
  • 187
1 vote
1 answer
94 views

Measuring quality of answers from QnA systems

I am having a question answering system which is using Seq2Seq kind of architecture. Actually it is a transformer architecture. When a question is asked it gives startposition and endposition of ...
Sandeep Bhutani's user avatar
1 vote
1 answer
31 views

An exhaustive, representative test database in phrase search algorithm

For a phrase searching algorithm, imagine the goal is to search for a name phrase and return matched results based on a pre-defined threshold. For example, searching for "Jon Smith" could return "Jon ...
Xiaohan Du's user avatar
1 vote
2 answers
2k views

How can I improve the recall of a certain class in a multiclass-classification result

I am working on a multiclass classification which is to assign medical related queries of web search to certain departments of hospital.My classifier is based on the fastText. I found for most ...
leakey's user avatar
  • 13
1 vote
1 answer
30 views

Is Elastic Search recommended if attribute getting search is not a huge text document?

We are currently developing a system with MEAN stack with Mongodb at backend. We have employees name, and Ids in our system and our client wants to get pretty good (Read: Google Like) search in our ...
user3422929's user avatar
2 votes
1 answer
449 views

scalable tools to build kNN graph over sparse data

I'm looking for scalable tools to build kNN graph over sparse data points. The dimension and number of data points can be both up to millions. What I have tried already: sklearn.neighbors....
xiaohan2012's user avatar
2 votes
0 answers
87 views

Can Google really bring back billions of results in a blink of an eye (almost) [closed]

I was always fascinated by Google's search ability, a great achievement by Google and other search engine providers also, but more so a collective human talent and ability that makes me appreciate our ...
Saleh's user avatar
  • 21
1 vote
0 answers
35 views

Search Query Sample Size Determination for validation set

While designing a search system, which searches in N identifiable categories, how many search queries does one need in each category to validate the target metric (DCG) scores accurately (balanced ...
D.S.'s user avatar
  • 111
3 votes
2 answers
272 views

What ML/DL techniques power Youtube/Netflix search systems?

Video platforms like YouTube, Netflix, Amazon prime have an excellent search system - given a search string, find most relevant videos. Which Machine Learning /Deep Learning techniques used for this? ...
Anuj Gupta's user avatar
0 votes
1 answer
608 views

Finding similar articles in realtime [closed]

I want to build a large document (news article) searchable database, such as when adding a new article I will be able to quickly find X most similar articles from it. What is the right tech/algorithm/...
rubmz's user avatar
  • 103
3 votes
1 answer
244 views

Best method for similarity searching on 10,000 data points with 8,000 features each in Python?

As mentioned in the title I am attempting to search through 10,000 vectors with 8000 features each, all in Python. Currently I have the vectors saved in their own directories as pickled numpy arrays. ...
Michael Vander Meiden's user avatar
1 vote
0 answers
65 views

What is search engine life cycle from information retrieval perspective

I am already aware of usage of information retrieval for search engine. But still trying to understand what can be possible life-cycle phases for a search engine. I mean for a search engine is it only ...
StatguyUser's user avatar
6 votes
2 answers
979 views

Why do popular search engines not follow the usual AND, OR logic for queries?

I am teaching myself Information Retrieval from Christopher Manning's book (PDF link: http://nlp.stanford.edu/IR-book/pdf/01bool.pdf). I tried Exercise 1.13: "Try using the Boolean search features on ...
user avatar