Questions tagged [information-retrieval]

Information Retrieval is an area of study concerning with retrieving documents, information or metadata from a collection of unstructured or semi-structured data.

Filter by
Sorted by
Tagged with
0
votes
1answer
14 views

Information Extraction/Semantic Search for long, unstructured documents

I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store ...
0
votes
0answers
26 views

Collecting structured data from HTML source code: A generalized way

I am working on a task to build a generic function to extract some specific fields from HTML source code. The fields we want are such as product title, price, quantity and shipment The generic ...
0
votes
1answer
17 views

Is it possible to create a rule-based algorithm to compute the relevance score of question-answer pair?

In information retrieval or question answering system, we use TD-IDF or BM25 to compute the similarity score of question-question pair as the baseline or coarse ranking for deep learning. In ...
0
votes
0answers
7 views

Testing Setup in Temporal Tensor Factorization for Recommendation

I am trying to wrap my head around Temporal Tensor Factorization for Collaborative Filtering, as described in the paper: Temporal Collaborative Filtering with Bayesian Probabilistic Tensor ...
0
votes
1answer
16 views

Getting answers to bullets (numbered items) from text via NLP

This is related to information extraction. In real world data, documents are written in bullets/numbered items form. For example, ...
0
votes
0answers
37 views

Transform data into English, then predict an answer using BERT?

I'm looking for research/examples of closed domain QA systems that utilise pre-trained ML models such as BERT, to perform question-answering on structured data (eg: CSV, JSON) that has been ...
1
vote
1answer
17 views

Why TREC set two task: document ranking and passage ranking

TREC is https://microsoft.github.io/TREC-2019-Deep-Learning/ I am new to text retrieval. Still can not understand why set the two similar task. Thank you very much.
1
vote
2answers
58 views

Intuition behind the entropy definition

The definition of information entropy is defined below: This looks fine but I got no intuition why it is defined this way. Could any one share their ideas on this? Thanks!
1
vote
1answer
58 views

Find specific topics with topic modelling

I am looking for a way to classifiy text automatically by specific topics, i don´t have labeled data. Is this a possible/usual method of achieving this? If not, what would be better? Topic Modelling ...
0
votes
2answers
43 views

Targeted information extraction / focused extractive summarization

I have a large collection of project manuals, each with a large number of pages. Each manual contains some form of summary paragraphs, although these are not necessarily similar in structure or format ...
2
votes
0answers
36 views

Text Mining with Pubmed Widget Orange

When I was running the text mining I did not have an issue for 57 different searches. I was able to retrieve all of the records regardless of how many there were. Until these 2 errors popped up. I ...
0
votes
0answers
109 views

Extraction of skills from Resume Using Machine Learning

I have gone through the previous questions regarding 'Resume Parsing' and 'Extraction of skills'. They didn't help me as my data is not structured. The resumes I am dealing are neither properly ...
3
votes
4answers
55 views

Best way to combine two similar document

I have f.ex.: two news-articles that report the same event. However, these two text are similar BUT not the same. I would like to combine these two texts creating one text that contains only the most "...
0
votes
1answer
16 views

Dissimilarity Matrix of non-metric proximity data

we currently have a coding exercise, where we are asked to implement Constant Shift Embedding (Paper). This in itself is not a big problem. For the algorithm, all you need is a symmetric non-zero ...
1
vote
0answers
129 views

How to implement Semantic Search in R or Python

I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-...
1
vote
0answers
160 views

Doc2vec most similar document to a query string

I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title ...
2
votes
1answer
212 views

Detect sensitive data from unstructured text documents

I know this question is broad, but I need an advice to know if it's possible to achieve what I want to do. The problem is that I have around 2500 documents with sensitive data being replaced by four ...
2
votes
0answers
17 views

Why use standard tf weight over tf-idf? [closed]

Why you would use this over tf-idf? Are results better for this one?
1
vote
1answer
314 views

Can macro F1 score be greater than micro F1 score?

I am reading about evaluation metrics, and it seems that micro scores are more useful. But I was wondering about scenarios where macro F1 score is greater than micro F1 score, and if this is at all ...
0
votes
0answers
16 views

Extarct information of of text such as font from pdf using python

I need to extract the information of text from the raw PDF,such as 1.Font size of text 2.Font used Is there any predefined libraries in python to extract the above information
1
vote
1answer
597 views

Word embeddings for Information Retrieval - Document search?

What are good ways to find for single sentence (query) the most similiar document (text). I asked myself if word vectors (weighted average of the documents) are suitable to map a single sentence to a ...
1
vote
1answer
60 views

Algorithm for document retrieval in QA system

I am working with question answering and machine reading comprehension system. I want to match questions and documents (around 100,000 docs) in database. I've used tf-idf but it accuracy is about 55% ...
1
vote
0answers
37 views

How to approach training a machine to read a form

After many rewrites I'm still not entirely thrilled with how I've presented this.... please delete if inappropriate. Background As part of my job, we record observations on underwater paper and then ...
1
vote
1answer
190 views

What is NLP technique to generalize manually created rules in text?

Let's say we have a free text containing key-value entities. Example: "... patient's tumour has width 6 cm and height 5 cm" Then an expert comes, marks it as important, thus we do have the rule for ...
0
votes
1answer
15 views

TextInformationRetrieval content based

I need to know how to avoid spam document(file with repeated keywords) weighting while ranking the top k documents.
1
vote
0answers
38 views

Reconciling time-based data when data source clock drifts

How can I reconcile time-based data when the clock on the data source tends to drift and the data may be infrequently retrieved? I measured the clock to be about 30 minutes behind after 15 hours. ...
2
votes
1answer
74 views

Capturing movement importance - logistic regression output

I'm studying some event for a set of objects that can be plotted on a square $[0, 100] ^ 2$. I have used logistic regression to calculate probabilities that event occur for different objects and the ...
2
votes
0answers
67 views

Using ontology to infer labels for process model

I'm trying to implement a specific type of process mining, that has been presented in this thesis [link]. It is based on HMMs and generates a process model in form of a directed graph, where: Nodes ...
1
vote
0answers
25 views

Efficient search for a Triples data [closed]

I have a text file which consists of ~10k triples in the format, Let say I need to extract all the relevant triples for the query "Tom and Jerry likes to play football", with this I am extracting NP(...
1
vote
0answers
10 views

Understanfing OpenIE 5 output

I used your Open IE 5 to extract the triples and got the following result, Text Input By the algorithmic approach known as LevenbergMarquardt backpropagation algorithm, the error is decreased ...
1
vote
1answer
40 views

mathematical accurate definition of the binary independence model

I have a hard time understanding the exact mathematical meaning behind the binary independence model. On wikipedia we can see the following definition or similarly in the book from Manning and ...
0
votes
1answer
56 views

Classifying whether a comment or review is a complaint or appreciation of product and extracting the Topic?

I need to classify whether a given review or comment is a complaint or appreciation. This is planned to be used in multiple places, product review pages of own site as well as facebook and twitter. ...
2
votes
1answer
148 views

Two definitions of DCG measure

I wanted to check the definition of Discounted Cumulative Gain (DCG) measure in the original paper Jarvelin and it seems it differs from the one given in the later literature Wang. Originally, for $n$ ...
3
votes
1answer
74 views

Sparse IR with user feedback

I'm considering a problem framing within an information retrieval context. I have a sequence of documents that feature different attributes. In the web context, these would be webpages. One ...
1
vote
1answer
256 views

Industrial application(s) of LDA (latent Dirichlet allocation)?

LDA ( Latent Dirichlet allocation) - is quite a popular topic in data-mining. Question What are the industrial systems using LDA or may be some related models ? (May be Google/Amazon/ ... ? ) PS I ...
0
votes
1answer
26 views

Does recall has different interpretation when comes to classification and information retrieval

Recall Definition In terms of classification The recall is defined as no of positive instances that are correctly detected by the classifier. $$ TP = \frac{TP}{(TP+FN)} $$ In terms of ...
1
vote
0answers
47 views

How can I train a model to modify a vector by rewarding the model based on the modified vectors nearest neighbors?

I am experimenting with a document retrieval system in which I have documents represented as vectors. When queries come in, they are turned to vectors by the same method as used for the documents. The ...
0
votes
1answer
346 views

Information retrieval / slot filling / NLP

Excuse if this has been answered before. I need to extract features and parse from a piece of text and run some analysis. For e.g. "Plot the past 5-year sales of Apple" should give me the following ...
6
votes
1answer
1k views

How can you build a model that reads out receipts and invoices?

The objective is to build a model that is capable of identifying information on receipts and invoices that can look completely different. I've had a discussion with my brother about the right ...
2
votes
0answers
57 views

How to combine heterogeneous image features extracted with different algorithms for similar image retrieval?

Say I have access to several pre-trained CNNs (e.g. AlexNet, VGG, GoogleLeNet, ResNet, DenseNet, etc.) which I can use to extract features from an image by saving the activations of some hidden layer ...
1
vote
0answers
158 views

Extracting date, relation and noun phrase from text

A sentence (Segmented from a document) as below: ...
1
vote
1answer
321 views

Is recall more important than precision for mass mailings?

Say for example, I built a classification model for a mailing campaign that will be applied to 1M records. The positive class for the model would be customers and the negative records would be non-...
1
vote
0answers
43 views

Efficient filtering of intersecting sets by cardinality

Suppose we have a set of documents with integer identifiers and an inverted index, which maps words to skip lists of document identifiers with the interface: ...
1
vote
1answer
51 views

I have 50 videos. I ask a customer 10 questions. Based on their answers, I send them a set of videos. How do I do it?

This might make you feel like I am looking for a recommender engine, but I am not. A recommender engine works well if accuracy isn't an issue, but in my case, it is. What I have proposed is to ...
1
vote
0answers
127 views

ElasticSearch for data scientists [closed]

This is to seek career advice for a data scientist. What pertains within the role of data scientist and what does not regarding ElasticSearch. Does backend development for ElasticSearch using ...
2
votes
1answer
99 views

Peformance evaluation of ranking algorithms

I have three questions: How can we assess (or measure) the performance of the ranking algorithms? Are there any specific measures, or performance metrics, for this? More specifically, how can we ...
1
vote
1answer
153 views

How to evaluate multi label image retrieval model

I'm using a deep hashing model to search most similar images in a database (most similar to the image given as a query). I'm doing this on the coco dataset which has multiple labels per image. I'd ...
1
vote
1answer
834 views

How to extract NER from a Spanish language text file?

I am trying to extract various Named Entities from a Spanish language text file. I tried using nltk but no success. I am working on python2 with nltk 3.x
2
votes
2answers
470 views

Confusion with cosine similarity

In information retrieval when we calculate the cosine similarity between the query features vector and the document features vector we penalize the unseen words in the query. Example if we have two ...
1
vote
1answer
3k views

Cosine similarity between query and document confusion

I am going through the Manning book for Information retrieval. Currently I am at the part about cosine similarity. One thing is not clear for me. Let's say that I have the tf idf vectors for the ...