Hot answers tagged

2

For data, you can search on IEEE Dataport , Kaggle. For detecting medicine names and other info trained deep learning models like CNN or you can also perform fine-tuning from the existing model.


2

You could train a character-level language model, e.g. an LSTM, on the real short texts, and use the perplexity as the signal to know whether a piece of text is real or not. In order to find an appropriate perplexity threshold, you can have a look at the distribution of perplexities over a validation holdout dataset. UPDATE: There are multiple ...


2

First, let's clarify the issue with one-hot vectors: most NLP neural models nowadays don't use one-hot encodings for the model input; instead, they use (non contextual) embedding layers. While theoretically you get the same result multiplying a one-hot vector with a matrix, it is more practical just to index the position in the table directly, which is what ...


1

Assuming that the "human readable" texts are more likely to contain actual words, you could count the number of dictionary words that occur in each. You could use Wordnet for example. The number or proportion of word hits, and their length, could be features for a model or maybe it would be enough with a simple cutoff rule. You might want to ...


1

At training time, the input to the decoder is the target sentence tokens, which are indeed unknown at the test time. What you call the second input are the desired outputs, which are not usually referred to as an input to the decoder, 1. for clarity, 2. they are technically input to the loss function. At test time, we do not need the loss function, but we ...


1

What you are describing is one of the "standard" NLP problems faced in NLP and it usually referred to as "natural language inference" (NLI), or sometimes also as "textual entailment". There is plenty of research in this kind of task, and its variants, like cross-lingual NLI (XNLI). I suggest you have a look at nlpprogress (link) ...


1

Word2vec as the name suggests will create an embedding for each word in your sentence. In order to get a sentence level embedding you would need to average (or combine in some other way) the individual embeddings together. An example of a model to generate sentence level embedding would be the Universal Sentence Encoder (USE). You may want to try it out and ...


1

Out of the box, something like Google's Universal Sentence Encoder (USE) may work for your use-case. Many of the common NLP embedding techniques nowadays work on individual words and so creating sentence-level embeddings means averaging multiple word-level vectors together. USE was built to operate at the sentence level, so you may find it better. The ...


1

Both approaches are reasonable. Updating the BERT weights will train for longer period of time, but should give more accurate results.


1

The encode method only works with single sentences as strings, i.e., you need to call it for each sentence independently: embeddings = [model.encode(s) for s in sentences]


1

1 and 3 would be nice. Separating "well-known" to "well" and "known" would not be a good idea because you lost an information and/or have an erroneous/unuseful counts.


1

I agree with Nicholas' answer, a few more thoughts: you could use a standard English tokenizer (e.g. nltk, Spacy), if only to see how they process hyphenated words. Similarly you could check how it's done in a pre-tokenized dataset, but be aware that the tokenization conventions followed might differ from one dataset to the other. Imho the choice depends on ...


1

They all sound like interesting approaches. The first one is better I think because it allows for unseen hyphenated words to be somewhat understood (as e.g. well + known ~= well-known). For a tfidf BOW model, you might get good performance from any of the above. For a model that is sensitive to word order I would certainly go with the first option and might ...


1

First, about interpreting these confusion matrices: the sum of every row is 1, which implies that every value is a conditional probability p( predicted label | true label ), i.e. the probability of a given true label to be a particular predicted label. Example: the top left cell in both matrices is 0.01, which means that when the true label is 5 the ...


1

Well, in general case, machines do not understand the text, but they understand the numbers. Thus, we always tokenize the text followed by converting them to some form of numbers. We build a vocabulary of words from the given document, where each word can be assumed as a number corresponding to its index in the vocabulary. Further, this number is converted ...


1

The trick is that you do not need masking at inference time. The purpose of masking is that you prevent the decoder state from attending to positions that correspond to tokens "in the future", i.e., those that will not be known at the inference time, because they will not have been generated yet. At inference time, it is no longer a problem because ...


1

The Python Library Python-tesseract is an optical character recognition (OCR) tool for python. That is, it will recognize and “read” the text embedded in images. Python-tesseract is a wrapper for Google's Tesseract-OCR Engin and please tell me where you find the medicine wrapper ( Packaging ) of different medicines.


Only top voted, non community-wiki answers of a minimum length are eligible