New answers tagged nlp
0
One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.
The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel&...
0
Assuming that the "human readable" texts are more likely to contain actual words, you could count the number of dictionary words that occur in each.
You could use Wordnet for example.
The number or proportion of word hits, and their length, could be features for a model or maybe it would be enough with a simple cutoff rule.
You might want to ...
1
You could train a character-level language model, e.g. an LSTM, on the real short texts, and use the perplexity as the signal to know whether a piece of text is real or not.
In order to find an appropriate perplexity threshold, you can have a look at the distribution of perplexities over a validation holdout dataset.
UPDATE: There are multiple ...
0
Try making features like vowel_count, consonant_count, digitcount , vowel_density(vowel_count/total_length_of_words)
Another wild thing -
split the strigns with numbers and _ using regex and try to see if they are english words or not, use a pretrained model like spacy.english or nltk.words to check, make a column representing english words count if any.
...
0
I am not sure what you are asking exactly, so if you are looking to determine the overall sentiment of the car throughout the whole text you have to deal with "Anaphora resolution" first, because the first obstacle you will encounter is how to know what the "it, its, she, her..." referring to, maybe the car, maybe something else. another ...
0
you can use scispacy. I Think it will suit your purpose.
https://allenai.github.io/scispacy/
1
What you are describing is one of the "standard" NLP problems faced in NLP and it usually referred to as "natural language inference" (NLI), or sometimes also as "textual entailment".
There is plenty of research in this kind of task, and its variants, like cross-lingual NLI (XNLI). I suggest you have a look at nlpprogress (link) ...
2
For data, you can search on IEEE Dataport , Kaggle.
For detecting medicine names and other info trained deep learning models like CNN or you can also perform fine-tuning from the existing model.
1
The Python Library
Python-tesseract is an optical character recognition (OCR) tool for python. That is, it will recognize and “read” the text embedded in images. Python-tesseract is a wrapper for Google's Tesseract-OCR Engin
and please tell me where you find the medicine wrapper ( Packaging ) of different medicines.
0
From an NLP perspective, the main issue is to identify and extract the terms/relations of interest from the text.
It may be easy if the sentences always appear like these simple examples: one can use pattern matching "the <entity> is <property1> [and <property2>]" (no need for ML).
However with general text it's a quite complex ...
0
What makes you think your model is overfitting? Are you concerned about the difference between the training loss and validation loss?
If so, this is not overfitting. Overfitting is when the weights learned from training fail to generalize to data unseen during model training.
In the case of the plot shown here, your validation loss continues to go down, so ...
0
Depends on data distribution. Some consumers might speak in mixed ways and they might be very few who do this and your Word Embedding might not learn anything from those data. It might be helpful to have a feature that tells or approximately tells that a conversation has mixed, Pure English or Pure French language.
1
1 and 3 would be nice. Separating "well-known" to "well" and "known" would not be a good idea because you lost an information and/or have an erroneous/unuseful counts.
0
The trick is that you do not need masking at inference time. The purpose of masking is that you prevent the decoder state from attending to positions that correspond to tokens "in the future", i.e., those that will not be known at the inference time, because they will not have been generated yet.
At inference time, it is no longer a problem because ...
0
Here are a list of tools you can look into:
https://tika.apache.org/
https://jsoup.org/
https://poi.apache.org/
This was a neat read detailing the steps. The author was doing something similar to what you are trying.
https://towardsdatascience.com/how-to-build-a-resume-parsing-tool-ae19c062e377
1
Word2vec as the name suggests will create an embedding for each word in your sentence. In order to get a sentence level embedding you would need to average (or combine in some other way) the individual embeddings together.
An example of a model to generate sentence level embedding would be the Universal Sentence Encoder (USE). You may want to try it out and ...
1
Out of the box, something like Google's Universal Sentence Encoder (USE) may work for your use-case. Many of the common NLP embedding techniques nowadays work on individual words and so creating sentence-level embeddings means averaging multiple word-level vectors together. USE was built to operate at the sentence level, so you may find it better.
The ...
0
pyresparser is useful for extracting information from resumes. I believe this should work in your case.
Check out the more details on the same here https://pypi.org/project/pyresparser/
Let me know if it works!
0
Here is a great answer to this question. I'll summarize:
The code example was taken from a "buggy" repository on GitHub and is not typical of robust solutions.
Robust solutions actually do use the first word as a target word. If the context window is length 10, then the method uses the next 5 words as the context and the first word as the target (...
0
I'm not sure I completely get the idea, but it looks to me like what you're actually interested in is the sentiment of a word in a particular context: a content word like "car" might not carry a stable sentiment by itself, but its usage in a specific context might.
So I'd suggest a method like this: for any target word you extract either the ...
1
Both approaches are reasonable. Updating the BERT weights will train for longer period of time, but should give more accurate results.
0
BERT uses transformers archtecture of neural network so parallelization can be very helpful whereas the other (ELMO and ULMfit) uses LSTM .BERT has state-of-art preformance in many of the NLP tasks .
But i've heard that araBERT is less performant than hULMounA when it comes to arabic sentiment analysis ,correct me if i'm wrong pls
0
I finally solved this problem.
My dataframe looks like this:
index text class
0 ["Hello i live in berlin", 'I'm xxx'] 1
1 ["My name is xx", "I have a cat", "Love is life"] 0
Text column ...
1
First, about interpreting these confusion matrices: the sum of every row is 1, which implies that every value is a conditional probability p( predicted label | true label ), i.e. the probability of a given true label to be a particular predicted label. Example: the top left cell in both matrices is 0.01, which means that when the true label is 5 the ...
1
The encode method only works with single sentences as strings, i.e., you need to call it for each sentence independently:
embeddings = [model.encode(s) for s in sentences]
0
Do it in two steps: replace the ones with 's' first, then do the rest of them.
0
Is it recommended to discard this numerics before creating a
vectorizer(bow/tf-idf) for any model(classification/regression)
development?
It depends on the problem statement for example year could be significant if you want to find the trend and year has many unique value but if it's constant then you can remove it.
To add to that if you are doing sentiment ...
1
I agree with Nicholas' answer, a few more thoughts:
you could use a standard English tokenizer (e.g. nltk, Spacy), if only to see how they process hyphenated words. Similarly you could check how it's done in a pre-tokenized dataset, but be aware that the tokenization conventions followed might differ from one dataset to the other.
Imho the choice depends on ...
1
They all sound like interesting approaches. The first one is better I think because it allows for unseen hyphenated words to be somewhat understood (as e.g. well + known ~= well-known).
For a tfidf BOW model, you might get good performance from any of the above.
For a model that is sensitive to word order I would certainly go with the first option and might ...
0
You should try FastText, which is open source library by Facebook research. https://fasttext.cc/docs/en/supervised-tutorial.html
You need to create a file format needed by Fasttext algorithm.
Also following suggestions for cleaning text
Change the case to lower
Remove hyper links
Try to remove typo words
Fasttext automatically converts words into n-grams. ...
1
Well, in general case, machines do not understand the text, but they understand the numbers. Thus, we always tokenize the text followed by converting them to some form of numbers. We build a vocabulary of words from the given document, where each word can be assumed as a number corresponding to its index in the vocabulary. Further, this number is converted ...
1
I could think of 2 solutions:
Since you mention stripping of the words why not make it a 2 step program where in the first classifier is a binary where in 1-3 is one class of Actions performed and the second class is 4 where there is No Action performed. If the word happens to be in the first category you can further run it for classification in between the ...
1
Just imagine it practically, If class A data is 90% and class B data is 10% ,then if you just randomly classify the label you prediction as class A then your accuracy will be 90%.
So biased data will lead your model to be biased over the class which has more data as it will give better predictions in your model.
3
Formally the problem of topic modelling is a clustering problem: given a collection of text documents, group together the documents which are topically similar.
So technically it can indeed be done with a TF-IDF representation of documents as follows:
Collect the global vocabulary across all the documents and calculate the IDF for every word.
Represent ...
0
On the basis that difficult words are difficult because they are not commonly used, I think something as simple as TF-IDF would work well.
1
The link you provided of Siamese Bert is an instance of a Bert or Roberta finetuned on STS or NLI data. Which can have the format sentence 1 is similar 3 out of 5 to sentence 2 (STS). Hence, is supervised, it does not fit your purpose.
Nonetheless, do not despair, there are some that do not require training, although may not perform as good as the supervised ...
1
Since many of your questions were answered already, I may only share my personal experience with your last question:
7) Is it a good idea to use BERT embeddings to get features for documents that can be clustered in order to find similar groups of documents? Or is there some other way that is better?
I think what a good idea would be is to start with simpler ...
1
I cannot be sure whether this graphical representation is fully correct or wrong, but your misunderstanding looks like it revolves around what is an LSTM cell and LSTM neurons.
In the second figure, the nodes depicted should be LSTM cells. A cell is in essence a forward neural network consisting of neurons and so an analytic representation of it may look ...
0
For your training, it looks like you want to combine features that do not necessarily refer to the same context. In essence, you have a questions feature and a responses feature. Those two happen to be text data but it could be numerical as age or categorical as gender. For this reason you may need a model able to combine different inputs, like a multi layer ...
0
You could alternatively use a pretrained embedder like word2vec or glove to vectorize your data into fixed length vectors.
Top 50 recent answers are included
