Questions tagged [information-retrieval]
Information Retrieval is an area of study concerning with retrieving documents, information or metadata from a collection of unstructured or semi-structured data.
135
questions
0
votes
0
answers
8
views
Given a Query set, find elements in a pool that are similar to the elements in the query set
I had some curiosity and I was wondering if anyone could shed some light on this. So, let's say I have a query set $Q$ that consists of the following sets of words: {...
- 21
0
votes
1
answer
16
views
Survey on image retrieval datasets
I am on a survey about image retrieval datasets. I have found some, such as:
NUS-WIDE
Oxford5k
Oxford105k
Paris6k
MSCOCO
I have been way too confused about the detection metrics and the metrics they ...
- 3
0
votes
0
answers
25
views
Can BM25 be used as an embedding algorithm?
I'v studied about BM25 algorithm. Untill now, I couldn't find an implementation of BM25 to give me an embedding of a text like TfidfTransformer and ...
- 33
0
votes
0
answers
17
views
How to rank terms with bm25 and bo1 pipeline
In pyTerrier I have list of single terms. For example (I choose those tokens to be as relevant as possible and as irrelevant as possible to enhance the effect):
...
- 101
1
vote
1
answer
25
views
Empirical indications regarding demanded skills and tasks of data science jobs?
I am wondering if there are is any information about the current (and prospected) shares in skills required for advertised/existing data science jobs. This includes of course also the concrete tasks ...
- 111
1
vote
1
answer
58
views
How to add significance weighting in user based collaborative filtering
I have been learning about recommender systems these past days. More specifically about the collaborative filtering. While exploring I found that it can be useful to use "significance weighting&...
- 31
2
votes
1
answer
61
views
Why is accuracy not a useful measure for information retrieval problems?
I have been studying about information retrieval and recommender systems. While reading about it I found that accuracy not a useful measure in information retrieval.
I understand that, accuracy might ...
- 31
0
votes
0
answers
12
views
Recommendation System | Collaborative Filtering | User-User Filtering
Apologies in advance if this question is broad or basic for data science community.
Given:
Dataset containing thousands of lines with Apache HTTP Server log file produced in Common Log Format (CLF) ...
- 151
0
votes
0
answers
15
views
Help using Bm25 to rank sentences
I'm starting to study how to rank words/sents and after using PageRank im advancing to bm25 using this tutorial enter link description here
. I have a question regarding the query part. Is it possible ...
- 1
1
vote
0
answers
39
views
information retrieval vs recommendation system
Apologizes in advance, if this question is so basic,
Problem:
I have read this paper and noticed that Information Retrieval can be identified as a field of study whereas Recommender Systems are a ...
- 151
1
vote
0
answers
19
views
Optimizing Rank Aggregation of Two Different Methods in Information Retrieval
As the title suggests, I would like to train a rank-aggregating model. My target problem is to rank text2s from a database as best as possible to a given query, <...
- 11
1
vote
1
answer
26
views
Effectiveness of tf-idf on documents with repeated keywords
I was doing some ML reading and came upon tf-idf. The tf portion counts the relative frequency of a relevant word in a document, while idf measures how common or rare a word is across the corpus.
The ...
0
votes
1
answer
15
views
What method/algorithm to use to extract information from project documents about objectives, activities, and other variables?
I'm more-or-less new to NLP so assume little existing knowledge! But I have strong coding skills in R and to a lesser extent Python.
We're interested in extracting key information about objectives, ...
- 1
1
vote
2
answers
91
views
How to interpret rank output from BM25?
I'm using the FTS5 functionality in Sqlite3 to compute a search rank, which uses BM25. Maybe I'm just not seeing it, but I can't find any description of how to actually interpret BM25's output ranking....
- 121
0
votes
1
answer
110
views
What did Sentence-Bert return here?
I used sentence bert to embed sentences from this tutorial https://www.sbert.net/docs/pretrained_models.html
...
- 165
0
votes
0
answers
16
views
Arguments of OpenIE to extract fewer event triples
I'm new to NLP and I'm trying to using OpenIE to extract event triples from texts.
I looked into its documents but quite don't understand its arguments. For example, ...
- 165
1
vote
1
answer
28
views
A way to perform voting and select a candidate based on nearest neighbours
I'm working on a project where I use FAISS to retrieve n neighbouring vectors based on a query vector. The data in question is textual and is being embedded by using a machine learning model to create ...
- 111
1
vote
1
answer
322
views
How can I implement lambda-mart with lightgbm?
I have a learning to rank task at hand and I want to use the lightgbm implementation of LambdaMART. I'm also following this notebook.
...
- 674
1
vote
1
answer
91
views
Can I use Sentence-Bert to embed event triples?
I extracted event triples from sentences using OpenIE. Can I concatenate the components in the event triple to make it a sentence and use Sentence-Bert to embed it?
It seems no one has done this way ...
- 165
2
votes
1
answer
65
views
How "similarity" is measured in image retrieval?
I know what content based image retireval is. I have read this and this
as one of them says: "given a query images, get a rank list that are most similar to the query image, based on the content ...
- 121
2
votes
1
answer
75
views
How to determine the "total number of relevant documents" in calculatiion of Recall in Precision and Recall if it's not known? Can it be estimated?
On Wikipedia there is a practical example of calculating Precision and Recall:
When a search engine returns 30 pages, only 20 of which are relevant,
while failing to return 40 additional relevant ...
- 31
1
vote
0
answers
20
views
Extracting information from bills, tax statements, etc: What ML model to use?
I have a bunch of documents such as bank statements, utilities bills, personal expenditure invoices, etc. The document types range is very broad. Some of these files are saved as pictures, others as ...
2
votes
0
answers
288
views
Is there a Mean Average Recall for Item Retrieval/ Recommendation Systems?
Mean Average Precision for Information retrieval is computed using Average Precision @ k (AP@k).
AP@k is measured by first computing Precision @ k (P@k) and then averaging the P@k only for the k's ...
- 21
1
vote
0
answers
38
views
Difference between the architectures of semantic and instance segmentation
My question is about the difference between the architectures of semantic segmentation and instance segmentation models. So, as far as I understand, a semantic segmentation model is making pixel-wise ...
- 111
0
votes
1
answer
3k
views
How to extract details (educational details, exp details etc.) from a resume?
I am trying to build a resume parser which can extract details such as Name, Address, Education details (degree name, college name, university name, course duration), Experience details (designation, ...
- 197
1
vote
1
answer
86
views
Document ranking on a web scraped dataset without any labelled data
I want to create a document ranking model which returns similar rows in the dataset for a sample query. The text in this corpus is standard english but without any labels (ie no query-related ...
- 11
0
votes
0
answers
27
views
How do I verify and test a machine learning model against reality during time?
As a software engineers we familiar with a concept of testing (unit, integration, e2e) Tests give us a level of confidence about the code and changes in our code.
Looks like for ML the "code"...
1
vote
0
answers
31
views
Approximate maximum dot product between a vector and set of vectors using only a single vector representation for the latter
If we have a vector $q$ and a set of vectors $D = \{d_1, d_2, ..., d_l\}$ is there a way to create functions $QF$ and $DF$ such that $QF(q)^TDF(D) \approx \max_i(q^Td_i)$ ?
Use case:
I want to build ...
- 11
0
votes
0
answers
20
views
How is a textual search engine able to recognize subwords from words?
I am interested to know how information retrieval systems are able to consider relevant subwords from a main search word when performing a keyword search. For example, the word ...
- 1
3
votes
1
answer
101
views
Origin of the Boolean Model of Information Retrieval
Simple question, but I can't really find the answer to that: Who "invented" Boolean Retrieval? Of course, I assume that the concept grew over time, but is there a paper or publication that ...
- 81
1
vote
1
answer
309
views
What is the difference between Okapi bm25 and NMSLIB?
I was trying to make a search system and then I got to know about Okapi bm25 which is a ranking function like tf-idf. You can make an index of your corpus and later ...
- 113
0
votes
1
answer
33
views
How to stem plural words properly?
I'm looking for a way to avoid removing ending s when s isn't a suffix. In order to do that, I first check if a word exists in ...
- 101
1
vote
1
answer
40
views
Evaluation of recommendation systems
I have developed a content-based recommendation system and it is working fine. The input is a set of documents={d1,d2,d3,...,dn} and the output will be Top N similar documents for a given document ...
- 159
3
votes
1
answer
191
views
Evaluation metric for Information retrieval system
I am currently reading Semantic Product Search paper published by Amazon. They are using two evaluation subtasks matching and ranking. In matching, they tune the model hyperparameters to
maximize ...
- 1,971
-1
votes
1
answer
28
views
How to find correlated knowledge among different documents? [closed]
Say I have a sequence of documents clicked by a user, how can I mine the identical or semanticly similar word/knowledge/phrases shared among different documents?
Maybe someone can give a paper or ...
- 103
1
vote
1
answer
294
views
Using BM25 to rank words
How effective is it to use BM25 to rank words, to be more specific i have a dictionary of words and i want to rank only words in a document that are also in my dictionary. I want to rank all words in ...
- 41
1
vote
0
answers
416
views
Merging (intersecting) more than two posting list in linear time
The intersecting algorithm for two posting lists implemented below:
...
- 141
1
vote
0
answers
22
views
Effecient Feature Searching
I have got multiple features(descriptor, vector with elements of natural or real number) from a single image, which need to be searched against many image with multiple features. It is a problem from ...
- 111
1
vote
0
answers
67
views
Using transformers for information extraction
Task
I am trying to do some information extraction on earnings reports. I am trying to extract certain metrics, e.g. net sales for quarters. The earnings reports differ quite a lot in how they are ...
- 111
4
votes
2
answers
645
views
Building a tag-based recommendation engine given a set of user tags?
Basically, the idea is to have users following tags on the site, so each users has a set of tags they are following. And then there is a document collection where each document in the collection has a ...
1
vote
1
answer
23
views
Domain scoring based on ranking
I am a computer science student working on a small information retrieval project. I have a dictionary with a domain as a key and it's ranking as value.
Based on that ranking, I need to score every ...
- 11
4
votes
1
answer
120
views
Visualizing F-score differences in information extraction
I have several corpora and NLP systems (including a few merge ensembles of output of these systems combined in unions and intersections) with which I have extracted the annotation span sets {(begin, ...
- 191
1
vote
0
answers
213
views
Meaningful Information retrieval and question answering for unstructured data - Is it even possible?
Hello good NLP people,
I am working on a task that gradually seems not solvable for me. My data-set consists of long, messy, unstructured documents (pdfs, doc, docx, scans with tables, graphs, text, ...
- 111
3
votes
1
answer
224
views
Evaluating a IR system (Precision and Recall)
I am studying by now IR system, in the field of valuation of IR system outputs related to a specific query but I need some help to understand it properly.
My book states that when an IR system has ...
9
votes
1
answer
2k
views
How can you build a model that extracts data out from receipts?
I'm trying to build a model that is capable of identifying information on receipts and invoices.
I have used google cloud vision api for text extraction from the receipt but the problem is it just ...
- 347
4
votes
1
answer
70
views
How is "relevance" defined in information retrieval outside the context of systems with user feedback?
I've seen information retrieval systems that return some results from a query, and then the user rates these results as either "relevant" or "not relevant".
What can you do if you do not have user ...
- 53
6
votes
1
answer
101
views
Evaluating the performance of a machine learned recommendation system
I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes ...
- 447
1
vote
0
answers
21
views
Populating Knowledge Base - Stanford DeepMind Alternatives
I am dealing with the task to extract structured information from domain-specific unstructured documents. The end goal is to obtain a reliable, queryable system, i.e. in the form of a chat-bot or ...
- 111
1
vote
0
answers
1k
views
Connecting to IEX with Pandas Datareader
My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
- 181
5
votes
1
answer
818
views
TS-SS and Cosine similarity among text documents using TF-IDF in Python
A common way of calculating the cosine similarity between text based documents is to calculate tf-idf and then calculating the linear kernel of the tf-idf matrix.
TF-IDF matrix is calculated using ...
- 153