Hot answers tagged

7

You have several good tutorials on the web : https://www.kaggle.com/poonaml/text-classification-using-spacy Basically, you have to : Import the data in python, here POSITIVE is the variable to predict and 0 and 1 are the 2 encoded classes. TRAIN_DATA = [(Text1, {'cats': {'POSITIVE': 1}}), (Text2, {'cats': {'POSITIVE': 0}})] Initialize a textcat pipe in ...


6

SOTA is changing so rapidly in NLP that even Data Science professionists struggle to cope with it. I have two main sources that I constantly check to gain some insights on SOTA: NLP Progress from Sebastian Ruder. It contains updates on NLP on a whole lot of subfields, NER and POST included. Paper with code contains a section on NLP. That's a great website ...


6

Please look at my comment to add more information to your post. Based on the information you provided, here are my remarks: SpaCy is trained to find locations, not addresses per se If you use a "common" language, SpaCy is trained using WikiNER data, where locations aren't addresses but more like geographical places like city names, country names etc. So it'...


6

For pretrained models, spaCy has a few in different languages. You can find them in their official documentation https://spacy.io/models The available models are: English German French Spanish Portuguese Italian Dutch Greek Multi-language If you want support for extra labels in NER, you could train a model in your own dataset. Again, this is ...


3

Spacy's Sentencizer is very simple. However, Spacy 3.0 includes Sentencerecognizer which basically is a trainable sentence tagger and should behave better. Here is the issue with the details of its inception. You can train it if you have segmented sentence data. Another option is using NLTK's sent_tokenize, which should give better results than Spacy's ...


3

Can use a package that relies on a spellchecker to find the best way to split, like this one: https://pypi.org/project/compound-word-splitter/


2

spaCy used to recommended (archive link) that you use spaCy when you want production-grade performance but don't need to customize your architecture. They recommended that you use allenNLP when you want to explore different architectures or use the state-of-the-art models. They recommended against using allenNLP for production, though. Since spaCy 3.0, ...


2

Named Entity Recognition (NER) would extract names of people, organizations and such. Example: "Penalty missed! Bad penalty by <person>Felipe Brisola</person> - <organization>Riga FC</organization> - shot with right foot is very close to the goal. <person>Felipe Brisola</person> should be disappointed." So it could be ...


2

Spacy uses a hash function that assigns an integer to any Unicode string, it is not an index in vocabulary it just a random integer that is used internally for better efficiency. It is a hash function, so it means conflicts are indeed possible, but very unlikely and too rare to have a negative influence on the accuracy of the library, so the efficiency gains ...


2

Neural tools trained on Universal Dependencies corpora use learned models for tokenization and sentence-spliting. Two I know of are: UDPipe – developed at Charles University in Prague. Gets very good results (at least for parsing), but has a little unintuitive API. Stanza – developed at Stanford University. The API is quite similar to Spacy. However, they ...


2

Generally, XML is first parsed. Then, the contents can be analyzed with something like spaCy. xml.etree.ElementTree is the most common way to parse XML in Python.


2

There is nothing in SpaCy that you can use out-of-the-box. However, they allow you to use custom components To solve your problem, I see at least three ways to do it. NTLK NLTK allows you to add known abbreviations as exceptions. See this StackOverflow post. Use a regular expression Since your problem is that you have some example of dots that shouldn't ...


2

I don' know Spacy custom NER but it's unlikely that the model is optimized on recall, otherwise it would label absolutely everything as an entity in order to reach perfect recall. Your model happens to have a good recall, but it doesn't meant that the algorithm optimizes for this. There might be some technical parameters but it's very likely that the ...


2

Following with the idea of building a classifier, one option is to use nltk library together with Keras-Tensorflow once you have a labeled dataset with the desired process categories. You can go on two main approaches: bag-of-words sequence-modeling As a quick resume of the steps to implement in a text classifier with the first approach, you could follow ...


2

The values for LOSS TOK2VEC and LOSS NER are the loss values for the token-to-vector and named entity recognition steps in your pipeline. The ENTS_F, ENTS_P, and ENTS_R column indicate the values for the F-score, precision, and recall for the named entities task (see also the items under the 'Accuracy Evaluation' block on this link. The score column shows ...


1

The NER model performance on a particular text depends on which data it was trained with originally, and naturally the standard models (like en_core_web_sm) are trained with English data which doesn't contain a lot of names from non-US/UK origin (same for other kinds of entities like organizations or locations). Better performance can be achieved by training ...


1

Yes, it matters. A lot. You need to label every entity you encounter in each sentence. As long as they are non-overlapping, you can add as many entity types and entities per document as you'd like. First of all, it matters because, in your example, your model receives one example of "Germany" that is a LOC and one that isn't. So, it must mean that ...


1

Interesting task :) I think even with a good amount of training data it will be difficult for a regular NER model to perform well with new books titles and authors: The book may contain persons names which are not authors. The book titles are difficult to identify as such in general. For example "the Republic" might or might not be about the book, ...


1

You have to give your training set to the model to be trained _= pipe.fit(triningSet.data, triningSet.target) I don't see any training dataset here. you have to fit the CountVectoriser to your data set.


1

So, your task is to detect the passive voice from sentences. Currently, you have defined some rules to detecting the passive voice and you have noticed that there some exceptions to your defined rules. Therefore, it would be a good idea to develop a model to predict the probability of a sentence being passive (or active). You can do this by encoding the ...


1

Accuracy is not the best measure for imbalanced data. Prefer precision and recall. Do undersampling/oversampling to get equal samples for each class and try XGBoost. Or else you can use SVC with class weights, give lower class weight to classes with more samples and vice versa.


1

News sentences will have more unique tokens than normal conversations. Conversations have more stop words than news articles. I think you can use bert or normal wordvect classification to train a baseline model here. I would play aroud the pipeline of fake news classifier and news-conversation classifier. like passing the text to news classifier first and ...


1

What you are doing seems fine in terms of preprocessing. Removing less informative words like stopwords, punctuation etc. is a very common technique. Here are some of my notes: probably best for speed to load your "nlp" object outside of the function call "-PRON-" must be the lemma for "you're" in this case. So you shouldn't remove it as it is following the ...


1

Using dependency parsing alone will not give you what you need. You may be able to get your answer by interpreting the dependency tree. For instance, in this case ABC-EFG GROUP Inc. is a pobj of between, which could infer that it is party 1. In this particular case, the dependency parsing isn't fully correct and party 2 (Rob Cummins) is difficult to find. I ...


1

You could just test whether the tuples entities has any elements: for sent in list(doc.sents): if len(sent.ents) > 0: nb = nb+1 Edit: For the purpose of evaluating boolean expressions (as in case of if statements), empty lists are evaluated as false, even though they don't explicitly equal False. Expanding your example with some print ...


1

I am not sure you can neatly fit the above tools and libraries into your schematic. What even is the meaning of "feature" here, is it an output (like "text classification" is an output of the algorithm)? In which case I challenge the usefulness of this schematic. Nevertheless nltk, sklearn, etc. are libraries that contain multiple and diverse tools to help ...


1

Why are the vector similarities so high for unrelated words for the embedding? For the specific example you give, I would argue that it makes sense that car and plant have high similarity. This is likely due to phrases such as car manufacturing plant Also I am able to get vectors for non-words like "asdfasfdasfd" or "zzz123Y!/§zzzZz", and they differ ...


1

Use nlp.pipe() to process texts in larger batches, which is much faster, especially for a lot of short texts: for doc in nlp.pipe(sent_list): # averaged doc vector print(doc.vector) # token vectors print([token.vector for token in doc])


1

This is still a problem undergoing active research. I think it's commonly referred to as unsupervised labeling. If the dataset is not too large, it's probably more accurate and time efficient to either do it by hand, or hire someone on Mechanical Turk to do it. If you're dead set on doing it automatically, trying some form of semi-supervised learning where ...


1

For your first question, I would try to use Regex to identify the reference numbers since they seem unique comparing to normal words. I assume they are: All capital Start with one or more letters After follow numbers Any other pattern may follow (i.e. letters, numbers, hyphens) Do not include spaces Could be something like this: \b[A-Z]+\d+([A-Z]|\d|-)*\b ...


Only top voted, non community-wiki answers of a minimum length are eligible