What you need is perhaps Named Entity Recognition with custom entity dictionary. See this example:
Many packages like NLTK or Spacy have a large dictionary of such entities that enable models to identify them without using regular expressions. However, those pre-defined entities often is not applicable to one's application, thus won't recognize what you are ...
I'll suggest to test the sentence or the tweet for polarity. This can be done using the textblob library. It can be installed as pip install -U textblob. Once the text data polarity is found, it can be assigned as a separate column in the dataframe. Subsequently, the sentence polarity can then be used for further analysis.
Polarity and Subjectivity are ...
The problem you face is part of what is called in literature grammar learning or grammar inference which is part of both Natural Language Processing and Machine Learning and in general is a very difficult problem.
However for certain cases like regular grammars/languages (ie learning regular expressions / DFA learning) there are satisfactory solutions up to ...
Here are a few ideas:
If the number of strings is not too high, you could consider taking a formal approach and use a finite automata determinization algorithm (I'm very rusty about this stuff but I clearly remember that there is such a thing). The idea is to start from a big automaton made of the union of all the strings, then use the algorithm to find the ...
First about the features i think you could add some such as :
the time when the letter is received,
number of links in the email,
the whole structure (does it follow typical structure for email),
number of words that contains numbers in it,
what is the whole mood of the email (sales,threats,info,...-for this purpose you can use sentiment analysis),
There is at least one way:
Create/Acquire a grammar model for the language spoken (there are several such models for various languages used in NLP)
Test the transcripts for beign grammaticaly/syntacticaly correct.
This assesment will at least rule out gibberish and most of transcripts that do not correspond to valid sentences of the language spoken
First I think it's worth mentioning that in the context of an exploratory study with a small dataset, manual analysis is certainly as useful as applying NLP methods (if not more) since:
Small size is an advantage for manual study and a disadvantage for automatic methods.
There's no particular goal other than uncovering general patterns or insights, so it's ...
Translation as a pre-processing step is usually sufficient for many tasks (e.g. sentiment classification), but naturally undesirable for other tasks e.g. grading someone in written Dutch fluency.
Hence, for these tasks, the objective is:
Be able to train a language model for your specific language
However, you want to be able to do this with minimal ...
The NER model performance on a particular text depends on which data it was trained with originally, and naturally the standard models (like en_core_web_sm) are trained with English data which doesn't contain a lot of names from non-US/UK origin (same for other kinds of entities like organizations or locations).
Better performance can be achieved by training ...
I think the closest standard NLP task would be relationship extraction. In general it's a quite complex task which involves NER, syntactic analysis and semantic role labeling.
Note that there are various works using the term "event extraction" (for example this), but as far as I know there is no clear definition of the task. It's often related to ...
You could use clustering with a more basic similarity measure, for example cosine or even simply the proportion of words in common (e.g. Jaccard, overlap coefficient). This should gives you groups of sentences which are "quite similar" against each other, whereas sentences in different clusters are supposed to be very different. This way you would ...
This is essentially information retrieval: usually there is a collection of documents and the goal is to find the document which is the most similar to a given query (what you call the "semantic concept").
The traditional way to do that is to convert the collection of documents as vectors, typically with TFIDF weight but there are many options (I ...
This is usually done by carefully choosing two things:
The sentence representation. Word count is the most simple option but there can be many others: TFIDF weights, with/without removing stop words, with/without lemmatization, etc. In a DL approach the sentence would be represented as a sentence embedding.
The similarity measure between two sentences. ...
Rule of thumb: if a human with a lot of time wouldn't be able to do it, there's little chance a machine could. In this case if you ask somebody to classify whether a place is a park or not, they just can't do it correctly if the place name doesn't contain any indication that it is a park.
In a case like this you would need to use external resources in order ...
As mentioned in the comment you can use regex but you would need to define a set of rules for it. You can try with LexNLP which is trained on legal documents and use it to extract data type like address, companies and persons.
My first thought was that it's difficult to formally define this concept:
"b" is likely to contain "a" with some "error"
On the one hand, there is the idea that a is a substring of b. This question should have a boolean answer: either it does contain it, or it doesn't.
On the other hand there is the idea of approximate ...
I can think of two approaches.
You could use term-frequency/inverse-document-frequency (tf-idf) to cluster the vectors. Personally, I would start by first clustering the full text of the original job vacancies and then use this to assign clusters to the vectors. I have the feeling that it will outperform clustering directly the vectors.
There are ...
Yes, the parameter is available in the vanilla K-Means too.
The algorithm supports sample weights, which can be given by a parameter sample_weight. This allows assigning more weight to some samples when computing cluster centers and values of inertia. For example, assigning a weight of 2 to a sample is equivalent to adding a duplicate of that sample to the ...
Yes, its called graph-api.
You should be aware that scraping data from facebook not using this api its
violation of the regulations of fb laws and could you be ended with ban. So be careful.
To add onto @Nicholas James Bailey's answer:
tidytext provides functionality for two different main operations: text mining and text modeling.
I think the text mining part of it where we tokenize, tidy and prep text data is a bit more unique. As pointed out there are several model alternatives for text data, some of which are arguably better.
In terms of ...
If the image will always be the same and your ROI will always be in the same location, then an RNN with CV2 is not necessary, you can mark the areas that you need to extract the texts. I even recommend you more like JaidedAI
Here is the google colab of the papers you sent: Information extraction
I understand that you are trying to derive new informative feature from the available tweet texts. And you do it in two steps: first you calculate dummy binary features, next you want to aggregate all binary features into one numerical feature.
Several aggregation rules come to mind:
simply calculate the sum of all binary features (and multiply by -5 if you ...
Manually assigning a value to a feature level can be done. However, it is often better to allow the machine learning algorithm to learn the importance of different features during the training process.
The general machine learning process starts with labeled data. If the labels are numeric, it is a regression problem. In the specific case of fake tweets, a ...