Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [information-retrieval]

The tag has no usage guidance.

1
vote
1answer
41 views

What is NLP technique to generalize manually created rules in text?

Let's say we have a free text containing key-value entities. Example: "... patient's tumour has width 6 cm and height 5 cm" Then an expert comes, marks it as important, thus we do have the rule for ...
0
votes
1answer
5 views

TextInformationRetrieval content based

I need to know how to avoid spam document(file with repeated keywords) weighting while ranking the top k documents.
1
vote
0answers
37 views

Reconciling time-based data when data source clock drifts

How can I reconcile time-based data when the clock on the data source tends to drift and the data may be infrequently retrieved? I measured the clock to be about 30 minutes behind after 15 hours. ...
1
vote
1answer
63 views

Capturing movement importance - logistic regression output

I'm studying some event for a set of objects that can be plotted on a square $[0, 100] ^ 2$. I have used logistic regression to calculate probabilities that event occur for different objects and the ...
0
votes
0answers
98 views

Extract variable names of formula from a document

How can I extract all of the variable names from a given research paper? For example Some applications of algebraic curves to computational vision For example, below is the text and I need to ...
2
votes
0answers
32 views

Using ontology to infer labels for process model

I'm trying to implement a specific type of process mining, that has been presented in this thesis [link]. It is based on HMMs and generates a process model in form of a directed graph, where: Nodes ...
1
vote
0answers
14 views

Efficient search for a Triples data [closed]

I have a text file which consists of ~10k triples in the format, Let say I need to extract all the relevant triples for the query "Tom and Jerry likes to play football", with this I am extracting NP(...
1
vote
0answers
8 views

Understanfing OpenIE 5 output

I used your Open IE 5 to extract the triples and got the following result, Text Input By the algorithmic approach known as LevenbergMarquardt backpropagation algorithm, the error is decreased ...
1
vote
1answer
30 views

mathematical accurate definition of the binary independence model

I have a hard time understanding the exact mathematical meaning behind the binary independence model. On wikipedia we can see the following definition or similarly in the book from Manning and ...
0
votes
1answer
28 views

Classifying whether a comment or review is a complaint or appreciation of product and extracting the Topic?

I need to classify whether a given review or comment is a complaint or appreciation. This is planned to be used in multiple places, product review pages of own site as well as facebook and twitter. ...
2
votes
1answer
21 views

Two definitions of DCG measure

I wanted to check the definition of Discounted Cumulative Gain (DCG) measure in the original paper Jarvelin and it seems it differs from the one given in the later literature Wang. Originally, for $n$ ...
3
votes
1answer
71 views

Sparse IR with user feedback

I'm considering a problem framing within an information retrieval context. I have a sequence of documents that feature different attributes. In the web context, these would be webpages. One ...
1
vote
1answer
70 views

Industrial application(s) of LDA (latent Dirichlet allocation)?

LDA ( Latent Dirichlet allocation) - is quite a popular topic in data-mining. Question What are the industrial systems using LDA or may be some related models ? (May be Google/Amazon/ ... ? ) PS I ...
0
votes
1answer
19 views

Does recall has different interpretation when comes to classification and information retrieval

Recall Definition In terms of classification The recall is defined as no of positive instances that are correctly detected by the classifier. $$ TP = \frac{TP}{(TP+FN)} $$ In terms of ...
1
vote
0answers
46 views

How can I train a model to modify a vector by rewarding the model based on the modified vectors nearest neighbors?

I am experimenting with a document retrieval system in which I have documents represented as vectors. When queries come in, they are turned to vectors by the same method as used for the documents. The ...
0
votes
1answer
60 views

Information retrieval / slot filling / NLP

Excuse if this has been answered before. I need to extract features and parse from a piece of text and run some analysis. For e.g. "Plot the past 5-year sales of Apple" should give me the following ...
5
votes
1answer
129 views

How can you build a model that reads out receipts and invoices?

The objective is to build a model that is capable of identifying information on receipts and invoices that can look completely different. I've had a discussion with my brother about the right ...
2
votes
0answers
22 views

How to combine heterogeneous image features extracted with different algorithms for similar image retrieval?

Say I have access to several pre-trained CNNs (e.g. AlexNet, VGG, GoogleLeNet, ResNet, DenseNet, etc.) which I can use to extract features from an image by saving the activations of some hidden layer ...
1
vote
0answers
77 views

Extracting date, relation and noun phrase from text

A sentence (Segmented from a document) as below: ...
0
votes
0answers
75 views

How to deal with misspelling in text clustering?

I'm performing a clustering in a text datasets. The content of each text is obtanied from blogs, so they contain several misspelling. How to deal with that in an efficient way?
1
vote
1answer
125 views

Is recall more important than precision for mass mailings?

Say for example, I built a classification model for a mailing campaign that will be applied to 1M records. The positive class for the model would be customers and the negative records would be non-...
1
vote
0answers
34 views

Efficient filtering of intersecting sets by cardinality

Suppose we have a set of documents with integer identifiers and an inverted index, which maps words to skip lists of document identifiers with the interface: ...
0
votes
0answers
48 views

Data fingerprinting with machine learning

Hi does anyone have a idea on how the below white paper were able to generate a unique column signature from data profiling techniques ? go.waterlinedata.com/hubfs/a/data-fingerprinting-white-paper....
0
votes
0answers
12 views

how to use text data to extract variables to supplement a dataset using multiple files?

I am trying to do a political science large-N research where the cases are the countries , However , some variables were never coded in structured data sets . So in order to test them and due to the ...
1
vote
1answer
47 views

I have 50 videos. I ask a customer 10 questions. Based on their answers, I send them a set of videos. How do I do it?

This might make you feel like I am looking for a recommender engine, but I am not. A recommender engine works well if accuracy isn't an issue, but in my case, it is. What I have proposed is to ...
1
vote
0answers
94 views

ElasticSearch for data scientists [closed]

This is to seek career advice for a data scientist. What pertains within the role of data scientist and what does not regarding ElasticSearch. Does backend development for ElasticSearch using ...
2
votes
1answer
78 views

Peformance evaluation of ranking algorithms

I have three questions: How can we assess (or measure) the performance of the ranking algorithms? Are there any specific measures, or performance metrics, for this? More specifically, how can we ...
0
votes
0answers
23 views

Benchmark when evaluating performance of a similar documents retrieval?

In what metrics does Stack Overflow, Quora and other QnA sites use to tune their "Questions that may already have your answer" model? Do they use precision/recall described in this. What is a ...
1
vote
1answer
84 views

How to evaluate multi label image retrieval model

I'm using a deep hashing model to search most similar images in a database (most similar to the image given as a query). I'm doing this on the coco dataset which has multiple labels per image. I'd ...
1
vote
1answer
274 views

How to extract NER from a Spanish language text file?

I am trying to extract various Named Entities from a Spanish language text file. I tried using nltk but no success. I am working on python2 with nltk 3.x
2
votes
2answers
335 views

Confusion with cosine similarity

In information retrieval when we calculate the cosine similarity between the query features vector and the document features vector we penalize the unseen words in the query. Example if we have two ...
1
vote
1answer
2k views

Cosine similarity between query and document confusion

I am going through the Manning book for Information retrieval. Currently I am at the part about cosine similarity. One thing is not clear for me. Let's say that I have the tf idf vectors for the ...
1
vote
1answer
85 views

Learning to rank: construct absolute ranking using pair-wise ranking approach

I am learning about the "pairwise approach" for learning to rank. As far as I understood, the training output is a partial ranking function $r$ that: given given some query $q$ and two document $d_i$ ...
2
votes
1answer
57 views

What model to use for matching two datasets

I've got 500 images of paper receipts scanned and OCR as one dataset. I also have a dataset of transactions from my credit card statement including amount and date. What model is best to match the ...
5
votes
1answer
102 views

In what data science applications has the stack exchange dump been used?

Anonymized dumps of the stack exchange data are available here. Do you know projects or article that have been using these data (for social network analysis or information retrieval) ? My little ...
-1
votes
2answers
68 views

Extract person's academic degree from text

I have a corpus of free form text (emails) and am trying to extract the highest degree (eg. High School, Bachelor, Master's, phD) from each of them. Is stemming the way to go? Or lemmatization? Note ...
1
vote
1answer
64 views

ADHoc Information Retrieval

I want to extract the total bill from image receipts. I could extract the entire data present in the image but now I am struck with the problem of extracting only the information that I need. This is ...
0
votes
0answers
451 views

Extract numeric values from text snippets using machine learning

I have text snippets where I want to extract a numeric value. Is there a way to train a model for such a use case (using python): Here is how my data look like (I have about 1000 samples): "| 3 ...
1
vote
0answers
40 views

Analyzing Web page structure

I'm looking for an algorithm, that classifies a webpage. But not the content. I know the readability algorithm (by arc90) to extract a main text from an webpage. That for instance can decide, wether a ...
1
vote
1answer
774 views

Doc2Vec or Word2vec for word embedding

Is there any benefits from using Doc2vec for word embedding ( replacing word2vec ) ? in other hand if I train word2vec and doc2vec with the same dataset will I have the same word vectors ?
2
votes
2answers
2k views

Document similarity: Vector embedding versus BoW performance?

I have a collection of documents, where each document is rapidly growing with time. The task is to find similar documents at any fixed time. I have two potential approaches: A vector embedding (...
4
votes
1answer
4k views

Difference between paragraph2vec and doc2vec

Is paragraph2vec the same as Doc2vec or is every approach different?
6
votes
2answers
331 views

Why do popular search engines not follow the usual AND, OR logic for queries?

I am teaching myself Information Retrieval from Christopher Manning's book (PDF link: http://nlp.stanford.edu/IR-book/pdf/01bool.pdf). I tried Exercise 1.13: "Try using the Boolean search features on ...
0
votes
1answer
120 views

Text Mining of Research Paper Abstracts

I'm trying to build a system which tackles the following problem. Given a list of research papers L, I would like to extract the following information from each paper: The problem addressed by the ...
0
votes
1answer
364 views

State of the art approaches for Information retrieval tasks based on deep learning

I am interested in the state of the art approaches for information retrieval (IR) tasks, where you have a single query and a set of documents and the IR model will give you the best matched document. ...
1
vote
1answer
565 views

extract names in a list of names

I have been provided with a text cleaning task and I am assuming this involves some amount of natural language processing. I have a collection of names which does not have any specific pattern and ...
0
votes
4answers
1k views

Plotting Precision Recall Curve

I was wondering mathematically how Precision Recall curve is plotted. How is this curve useful?
2
votes
1answer
653 views

how to evaluate top n recommendation system with movie lens dataset?

Based on my research a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And I'm currently ...
4
votes
4answers
163 views

Extract 2 pieces of information from a string - what to use?

First of all, I am a complete newbie in regard to data science and I am not asking for the complete solution but some guidance as to what I should read up to achieve my task (what algorithms, ...
3
votes
4answers
175 views

How can conclusions be drawn from recommendation systems evaluation?

From my research, a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And basically exists many ...