Questions tagged [information-retrieval]

Information Retrieval is an area of study concerning with retrieving documents, information or metadata from a collection of unstructured or semi-structured data.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Can a term weighting function used in text retrieval be compared to one used in text classification?

I came up with a modified version of TF-IDF function for text retrieval task. I want to do retrieval experiments using Vector Space Model and compare my function to some of those proposed in the ...
1
vote
1answer
20 views

Evaluation of recommendation systems

I have developed a content-based recommendation system and it is working fine. The input is a set of documents={d1,d2,d3,...,dn} and the output will be Top N similar documents for a given document ...
3
votes
1answer
31 views

Evaluation metric for Information retrieval system

I am currently reading Semantic Product Search paper published by Amazon. They are using two evaluation subtasks matching and ranking. In matching, they tune the model hyperparameters to maximize ...
-1
votes
1answer
19 views

How to find correlated knowledge among different documents? [closed]

Say I have a sequence of documents clicked by a user, how can I mine the identical or semanticly similar word/knowledge/phrases shared among different documents? Maybe someone can give a paper or ...
0
votes
0answers
19 views

Learning to Rank vs Reinforcement Learning in Information Retrieval - which one is preferable and why?

I am trying to create an information retrieval system which can benefit from user feedback (either implicit, through e.g., click-through data) or explicit (e.g., binary feedback on irrelevant ...
0
votes
0answers
9 views

Retrieving information from 2 or more approaches

I'm trying to extract information from documents. There are two approaches currently which produces the following cases where both approaches ...
1
vote
1answer
43 views

Using BM25 to rank words

How effective is it to use BM25 to rank words, to be more specific i have a dictionary of words and i want to rank only words in a document that are also in my dictionary. I want to rank all words in ...
0
votes
0answers
18 views

vector space tf-idf weighting vs IBM

What are the differences between standard vector space tf-idf weighting and the BIM probabilistic retrieval model (in the case where no document relevance information is available)? BIM refers to ...
0
votes
0answers
27 views

How to extract main headings from paragraph using NLP?

I don't have any prior knowledge on NLP. I have many tables and I need to extract headings from paragraph above table which tells what data is stored in these tables. I looked on dependency parsing ...
0
votes
0answers
17 views

Benchmark queries for Benchmark dataset with ranked list of documents

I aim to evaluate the ranking of an information-retrieval system. For a benchmark dataset like TREC, I have followed the qrels file which has list of documents for a particular topic (query) with ...
0
votes
0answers
10 views

Merging (intersecting) more than two posting list in linear time

The intersecting algorithm for two posting lists implemented below: ...
0
votes
0answers
19 views

Effecient Feature Searching

I have got multiple features(descriptor, vector with elements of natural or real number) from a single image, which need to be searched against many image with multiple features. It is a problem from ...
0
votes
0answers
22 views

Can we use TF-IDF along with Weighted Frequency for Text Summarization?

I've been working on a text summarization problem. After studying this blog, I used weighted-frequencies of words present in a document for summarization. I would compute the weighted frequency ...
1
vote
0answers
22 views

Using transformers for information extraction

Task I am trying to do some information extraction on earnings reports. I am trying to extract certain metrics, e.g. net sales for quarters. The earnings reports differ quite a lot in how they are ...
3
votes
2answers
424 views

Building a tag-based recommendation engine given a set of user tags?

Basically, the idea is to have users following tags on the site, so each users has a set of tags they are following. And then there is a document collection where each document in the collection has a ...
1
vote
1answer
14 views

Domain scoring based on ranking

I am a computer science student working on a small information retrieval project. I have a dictionary with a domain as a key and it's ranking as value. Based on that ranking, I need to score every ...
4
votes
1answer
50 views

Visualizing F-score differences in information extraction

I have several corpora and NLP systems (including a few merge ensembles of output of these systems combined in unions and intersections) with which I have extracted the annotation span sets {(begin, ...
1
vote
0answers
113 views

Meaningful Information retrieval and question answering for unstructured data - Is it even possible?

Hello good NLP people, I am working on a task that gradually seems not solvable for me. My data-set consists of long, messy, unstructured documents (pdfs, doc, docx, scans with tables, graphs, text, ...
3
votes
1answer
52 views

Evaluating a IR system (Precision and Recall)

I am studying by now IR system, in the field of valuation of IR system outputs related to a specific query but I need some help to understand it properly. My book states that when an IR system has ...
9
votes
1answer
942 views

How can you build a model that extracts data out from receipts?

I'm trying to build a model that is capable of identifying information on receipts and invoices. I have used google cloud vision api for text extraction from the receipt but the problem is it just ...
4
votes
1answer
49 views

How is “relevance” defined in information retrieval outside the context of systems with user feedback?

I've seen information retrieval systems that return some results from a query, and then the user rates these results as either "relevant" or "not relevant". What can you do if you do not have user ...
5
votes
1answer
74 views

Evaluating the performance of a machine learned recommendation system

I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes ...
1
vote
0answers
17 views

Populating Knowledge Base - Stanford DeepMind Alternatives

I am dealing with the task to extract structured information from domain-specific unstructured documents. The end goal is to obtain a reliable, queryable system, i.e. in the form of a chat-bot or ...
1
vote
0answers
776 views

Connecting to IEX with Pandas Datareader

My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
4
votes
1answer
371 views

TS-SS and Cosine similarity among text documents using TF-IDF in Python

A common way of calculating the cosine similarity between text based documents is to calculate tf-idf and then calculating the linear kernel of the tf-idf matrix. TF-IDF matrix is calculated using ...
0
votes
1answer
60 views

Information Extraction/Semantic Search for long, unstructured documents

I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store ...
0
votes
1answer
46 views

Is it possible to create a rule-based algorithm to compute the relevance score of question-answer pair?

In information retrieval or question answering system, we use TD-IDF or BM25 to compute the similarity score of question-question pair as the baseline or coarse ranking for deep learning. In ...
2
votes
1answer
123 views

Getting answers to bullets (numbered items) from text via NLP

This is related to information extraction. In real world data, documents are written in bullets/numbered items form. For example, ...
1
vote
1answer
60 views

Why TREC set two task: document ranking and passage ranking

TREC is https://microsoft.github.io/TREC-2019-Deep-Learning/ I am new to text retrieval. Still can not understand why set the two similar task. Thank you very much.
1
vote
2answers
79 views

Intuition behind the entropy definition

The definition of information entropy is defined below: This looks fine but I got no intuition why it is defined this way. Could any one share their ideas on this? Thanks!
1
vote
1answer
220 views

Find specific topics with topic modelling

I am looking for a way to classifiy text automatically by specific topics, i don´t have labeled data. Is this a possible/usual method of achieving this? If not, what would be better? Topic Modelling ...
0
votes
2answers
70 views

Targeted information extraction / focused extractive summarization

I have a large collection of project manuals, each with a large number of pages. Each manual contains some form of summary paragraphs, although these are not necessarily similar in structure or format ...
2
votes
0answers
76 views

Text Mining with Pubmed Widget Orange

When I was running the text mining I did not have an issue for 57 different searches. I was able to retrieve all of the records regardless of how many there were. Until these 2 errors popped up. I ...
3
votes
4answers
139 views

Best way to combine two similar document

I have f.ex.: two news-articles that report the same event. However, these two text are similar BUT not the same. I would like to combine these two texts creating one text that contains only the most "...
0
votes
1answer
25 views

Dissimilarity Matrix of non-metric proximity data

we currently have a coding exercise, where we are asked to implement Constant Shift Embedding (Paper). This in itself is not a big problem. For the algorithm, all you need is a symmetric non-zero ...
2
votes
0answers
220 views

How to implement Semantic Search in R or Python

I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-...
1
vote
0answers
411 views

Doc2vec most similar document to a query string

I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title ...
2
votes
1answer
695 views

Detect sensitive data from unstructured text documents

I know this question is broad, but I need an advice to know if it's possible to achieve what I want to do. The problem is that I have around 2500 documents with sensitive data being replaced by four ...
2
votes
0answers
17 views

Why use standard tf weight over tf-idf? [closed]

Why you would use this over tf-idf? Are results better for this one?
1
vote
1answer
636 views

Can macro F1 score be greater than micro F1 score?

I am reading about evaluation metrics, and it seems that micro scores are more useful. But I was wondering about scenarios where macro F1 score is greater than micro F1 score, and if this is at all ...
1
vote
1answer
1k views

Word embeddings for Information Retrieval - Document search?

What are good ways to find for single sentence (query) the most similiar document (text). I asked myself if word vectors (weighted average of the documents) are suitable to map a single sentence to a ...
1
vote
1answer
107 views

Algorithm for document retrieval in QA system

I am working with question answering and machine reading comprehension system. I want to match questions and documents (around 100,000 docs) in database. I've used tf-idf but it accuracy is about 55% ...
1
vote
0answers
39 views

How to approach training a machine to read a form

After many rewrites I'm still not entirely thrilled with how I've presented this.... please delete if inappropriate. Background As part of my job, we record observations on underwater paper and then ...
2
votes
1answer
366 views

What is NLP technique to generalize manually created rules in text?

Let's say we have a free text containing key-value entities. Example: "... patient's tumour has width 6 cm and height 5 cm" Then an expert comes, marks it as important, thus we do have the rule for ...
0
votes
1answer
18 views

TextInformationRetrieval content based

I need to know how to avoid spam document(file with repeated keywords) weighting while ranking the top k documents.
2
votes
0answers
42 views

Reconciling time-based data when data source clock drifts

How can I reconcile time-based data when the clock on the data source tends to drift and the data may be infrequently retrieved? I measured the clock to be about 30 minutes behind after 15 hours. ...
2
votes
1answer
80 views

Capturing movement importance - logistic regression output

I'm studying some event for a set of objects that can be plotted on a square $[0, 100] ^ 2$. I have used logistic regression to calculate probabilities that event occur for different objects and the ...
2
votes
0answers
84 views

Using ontology to infer labels for process model

I'm trying to implement a specific type of process mining, that has been presented in this thesis [link]. It is based on HMMs and generates a process model in form of a directed graph, where: Nodes ...
1
vote
0answers
29 views

Efficient search for a Triples data [closed]

I have a text file which consists of ~10k triples in the format, Let say I need to extract all the relevant triples for the query "Tom and Jerry likes to play football", with this I am extracting NP(...
1
vote
0answers
13 views

Understanfing OpenIE 5 output

I used your Open IE 5 to extract the triples and got the following result, Text Input By the algorithmic approach known as LevenbergMarquardt backpropagation algorithm, the error is decreased ...