It all depends on your definition of what a common word is in your domain. You are using an NLTK corpus which likely doesn't fit your domain very well. Either you have a corpus containing the domain you want and you do a simple lookup. Or you don't know in advance and you need to compute these common words from your documents (your short phrases). In that ...
Maybe I'm having trouble formulating the inherent difference between NLP and NLU, when do we draw the line between the two?
There is a confusion here: NLP is the whole domain of AI which deals with natural language. It includes virtually any task related to processing language data (usually mostly written data, but that's not the point). Topic modeling is ...
Some common approaches to this problem are:
Keep only the n- most common words in a corpus (automatically done in scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer).
Keep all words, but downweight uninformative words using a transformation ...
Named Entity Recognition (NER) would extract names of people, organizations and such. Example:
"Penalty missed! Bad penalty by <person>Felipe Brisola</person> - <organization>Riga FC</organization> - shot with right foot is very close to the goal. <person>Felipe Brisola</person> should be disappointed."
So it could be ...
Yes, this is feasible.
One-class classification is a thing, but it is usually used in a context where it is hard or impossible to get negative samples. In your case, I would argue, you can quite easily get tweets that are not about activism, therefore you can render it as a binary classification, because you have data points of two classes or labels: 1 for ...
Yes. one-class SVM is actually designed for your problem. The question it answers is "how similar a new sample point (unlabeled tweet) is to my training data (hash-tagged tweets)?"
Regardless of what is a good answer to this question, I can share my brainstorming. Try to find the answer of "How can I model my data in a way that activism tweets stick together ...
I think Valentin's answer is great. I just wanted to +1 his remark that you really seem interested in how to find important words rather than filtering uncommon ones (bodybuilding might actually turn out to be not very common, but I understand even in that case it would still be irrelevant for your task).
If this assumption is correct, I think what you need ...
What you describe is called padding and is indeed used frequently in language modeling. For instance if one represents the sequence "A B C" with trigrams:
# # A
# A B
A B C
B C #
C # #
The advantages of padding:
it makes every word/symbol appear the same number of times whether it appears in the middle of the sequence or not.
it marks the beginning and ...
If you are training the document-embedding model, then split the data before you convert the text into embeddings.
If you are using a pre-trained document-embedding model, then it won't matter and it is pre-processing step that it doesn't matter when you execute it.
Pipeline when training your own document-embedding model
Split your text data into ...
By using SpaCy's pretrained XLNet's model, I got some interesting similarities. I used this model because it has been trained on a large scale corpus which has a decent probability to contain these domain-specific terms in the first place. But as @yohanes-alfredo points out, the only way the similarities will be meaningful is if the data the model was ...
There will not be any pre-trained models to cluster these words.
In fact, in order to build your own clustering model you will need more metadata about each observation/word in your array.
At the moment any model would only be able to "see" the name of the package/software in your array. So the best you could hope for is a model that clusters these words ...
Why are the vector similarities so high for unrelated words for the embedding?
For the specific example you give, I would argue that it makes sense that car and plant have high similarity. This is likely due to phrases such as car manufacturing plant
Also I am able to get vectors for non-words like "asdfasfdasfd" or "zzz123Y!/§zzzZz", and they differ ...
From OP's comment:
I want to find out if an unlabelled tweet has to categorized as activism or not according to the labelled data I already have (the ones containing activism hashtags)
This could correspond to a semi-supervised learning setting along the lines of:
Train a model on a labelled sample of data, e.g. taking tweets with #activism as positive ...
You just stumble over one big problem in the NLP field : finding the perfect metric..
Most traditional metrics (BLEU, ROUGE, ...) simply does not take into account the distance in terms of semantics between barking and crying.
So according to these metrics, The dog is crying is as similar as The dog is salmon to the reference, the dog is barking.
From a ...
WordNet is certainly an interesting resource to explore for this task. It might not cover all your vocabulary but I can't think of any other way to capture fine-grained semantic relationship between words.
You can email the authors to ask them if they could share their code with you, but maybe they can't for IP reasons or don't want to share it.
Papers like these are not unusual in experimental research. In theory you should be able to reproduce their system following the explanations in the paper.
However there are other tools available for biomedical NER: ...
As far as I know you don't have a lot of options, you're probably stuck with heuristics:
Regular expressions (e.g. for dates)
List of predefined entities (e.g. from Wikipedia) stored in a dictionary
That would probably be related to textual entailment and also relation extraction.
I'm not aware of any specific work but I would check in the biomedical domain, because there are resources such as SemRep and I wouldn't be surprised if people tried to use it for similar purposes.
It is easy. You need to tag a phrase using B (Begin), I (Interior), and E (End). For example, you want to tag "United States of America" as the name of a country. You will tag likes:
United(B_Country) States(I_Country) of(I_Country) America(E_Country)
In the same text if you find "Islamic Republic of Iran", you will tag likes: