A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

# Tag Info

## New answers tagged text-mining

0

As u said that one has to train a model in order to calculate Hellinger distance. I am not sure that which model u are talking about but for now I assume that u might have thinking about the latent model so according to me there is not mandatory to train a latent model to produce a document vector. one can create a document vector from the directly word-doc ...

2

Chatbots and Q&A systems differ in their complexity as well as use cases. Let's discuss each of them separately. Chatbots: They can answer various questions asked during an interactive conversation. Interactive conversion means the system keeps a track of questions asked earlier and can engage in longer conversations. They have a sought of memory which ...

2

Question-answering (QA) is sometimes used to refer to the task where the input to the system is a question and a list of possible answers (normally only a handful) or a paragraph where the answer is supposed to be found, and the expected answer is the index of the correct answer or the start/end positions where the answer located within the text. In ...

0

Update: One thing that worked the best with my data was converting the words into tf-idf vectors per document and applying Naive bayes on it to predict the probability per document or word.

1

A direct way to find the words which are the most representative of a class is to calculate the probability of the class given a word: $$p(c|w)=\frac{\#\{\ d\ |\ label(d)=c\ \land w\in d\}}{\#\{\ d\ |\ w\in d\ \}}$$ Ranking the words according to their probability $p(c|w)$ gives: highest values: the most correlated words for the class lowest values: the ...

1

I think here you must maintain the actual tf-idf and create corpus over it.. Assuming you already have lables for documents available. You can rum classification over it. Best classification I am anticipating for this problem would be naive bayes..

0

You could take pretrained embedder on multiple languages, then check distances between the encodings. There's unofficial pypi port of Facebook's LASER. It's langauge-agnostic and pretrained on both en and fr. from laserembeddings import Laser laser = Laser() sentence_en = 'My name is Hendrik' sentence_fr = 'Je suis Hendrik' en_embedding = laser....

0

Use the segment embedding (idea from BERT) for the origin text classification model. For example: input ["apple", "Peking", "in summer"] += segment emb [1,2,3,3,0] input ["tomato", "New York", "in winter"] += segment emb [1,2,2,3,3] where 1,2,3 are something like the data source type for input. Another improvement: check out PCNN or PCNN+ATT

2

You can leverage word-vector similarity in embedding models. TL;DR similiar vectors of words (for example fruits) will be clustered together in this high (vector) dimensional space. For every possible class-set you will have a class-set representative (centroid) that is actually a key (so in your case fruit, vegetable etc) all you need to do is train/find a ...

2

The problem with misspellings is usually less that they exist than that they make it unclear how specific entries should be classified. It depends somewhat on how bad the misspellings are-- someone typing StackExchaneg probably meant StackExchange, and I would be comfortable classifying the former as an alias for the latter. If the spelling mistakes are ...

Top 50 recent answers are included