Hot answers tagged

7

You have several good tutorials on the web : https://www.kaggle.com/poonaml/text-classification-using-spacy Basically, you have to : Import the data in python, here POSITIVE is the variable to predict and 0 and 1 are the 2 encoded classes. TRAIN_DATA = [(Text1, {'cats': {'POSITIVE': 1}}), (Text2, {'cats': {'POSITIVE': 0}})] Initialize a textcat pipe in ...


6

SOTA is changing so rapidly in NLP that even Data Science professionists struggle to cope with it. I have two main sources that I constantly check to gain some insights on SOTA: NLP Progress from Sebastian Ruder. It contains updates on NLP on a whole lot of subfields, NER and POST included. Paper with code contains a section on NLP. That's a great website ...


4

For pretrained models, spaCy has a few in different languages. You can find them in their official documentation https://spacy.io/models The available models are: English German French Spanish Portuguese Italian Dutch Greek Multi-language If you want support for extra labels in NER, you could train a model in your own dataset. Again, this is ...


3

Please look at my comment to add more information to your post. Based on the information you provided, here are my remarks: SpaCy is trained to find locations, not addresses per se If you use a "common" language, SpaCy is trained using WikiNER data, where locations aren't addresses but more like geographical places like city names, country names etc. So it'...


2

Named Entity Recognition (NER) would extract names of people, organizations and such. Example: "Penalty missed! Bad penalty by <person>Felipe Brisola</person> - <organization>Riga FC</organization> - shot with right foot is very close to the goal. <person>Felipe Brisola</person> should be disappointed." So it could be ...


2

Neural tools trained on Universal Dependencies corpora use learned models for tokenization and sentence-spliting. Two I know of are: UDPipe – developed at Charles University in Prague. Gets very good results (at least for parsing), but has a little unintuitive API. Stanza – developed at Stanford University. The API is quite similar to Spacy. However, they ...


2

Generally, XML is first parsed. Then, the contents can be analyzed with something like spaCy. xml.etree.ElementTree is the most common way to parse XML in Python.


2

Spacy's Sentencizer is very simple. However, Spacy 3.0 includes Sentencerecognizer which basically is a trainable sentence tagger and should behave better. Here is the issue with the details of its inception. You can train it if you have segmented sentence data. Another option is using NLTK's sent_tokenize, which should give better results than Spacy's ...


2

There is nothing in SpaCy that you can use out-of-the-box. However, they allow you to use custom components To solve your problem, I see at least three ways to do it. NTLK NLTK allows you to add known abbreviations as exceptions. See this StackOverflow post. Use a regular expression Since your problem is that you have some example of dots that shouldn't ...


1

The NER model performance on a particular text depends on which data it was trained with originally, and naturally the standard models (like en_core_web_sm) are trained with English data which doesn't contain a lot of names from non-US/UK origin (same for other kinds of entities like organizations or locations). Better performance can be achieved by training ...


1

Yes, it matters. A lot. You need to label every entity you encounter in each sentence. As long as they are non-overlapping, you can add as many entity types and entities per document as you'd like. First of all, it matters because, in your example, your model receives one example of "Germany" that is a LOC and one that isn't. So, it must mean that ...


1

Interesting task :) I think even with a good amount of training data it will be difficult for a regular NER model to perform well with new books titles and authors: The book may contain persons names which are not authors. The book titles are difficult to identify as such in general. For example "the Republic" might or might not be about the book, ...


1

You have to give your training set to the model to be trained _= pipe.fit(triningSet.data, triningSet.target) I don't see any training dataset here. you have to fit the CountVectoriser to your data set.


1

So, your task is to detect the passive voice from sentences. Currently, you have defined some rules to detecting the passive voice and you have noticed that there some exceptions to your defined rules. Therefore, it would be a good idea to develop a model to predict the probability of a sentence being passive (or active). You can do this by encoding the ...


1

Accuracy is not the best measure for imbalanced data. Prefer precision and recall. Do undersampling/oversampling to get equal samples for each class and try XGBoost. Or else you can use SVC with class weights, give lower class weight to classes with more samples and vice versa.


1

News sentences will have more unique tokens than normal conversations. Conversations have more stop words than news articles. I think you can use bert or normal wordvect classification to train a baseline model here. I would play aroud the pipeline of fake news classifier and news-conversation classifier. like passing the text to news classifier first and ...


1

What you are doing seems fine in terms of preprocessing. Removing less informative words like stopwords, punctuation etc. is a very common technique. Here are some of my notes: probably best for speed to load your "nlp" object outside of the function call "-PRON-" must be the lemma for "you're" in this case. So you shouldn't remove it as it is following the ...


1

Using dependency parsing alone will not give you what you need. You may be able to get your answer by interpreting the dependency tree. For instance, in this case ABC-EFG GROUP Inc. is a pobj of between, which could infer that it is party 1. In this particular case, the dependency parsing isn't fully correct and party 2 (Rob Cummins) is difficult to find. I ...


1

Spacy uses a hash function that assigns an integer to any Unicode string, it is not an index in vocabulary it just a random integer that is used internally for better efficiency. It is a hash function, so it means conflicts are indeed possible, but very unlikely and too rare to have a negative influence on the accuracy of the library, so the efficiency gains ...


1

You could just test whether the tuples entities has any elements: for sent in list(doc.sents): if len(sent.ents) > 0: nb = nb+1 Edit: For the purpose of evaluating boolean expressions (as in case of if statements), empty lists are evaluated as false, even though they don't explicitly equal False. Expanding your example with some print ...


1

Why are the vector similarities so high for unrelated words for the embedding? For the specific example you give, I would argue that it makes sense that car and plant have high similarity. This is likely due to phrases such as car manufacturing plant Also I am able to get vectors for non-words like "asdfasfdasfd" or "zzz123Y!/§zzzZz", and they differ ...


1

Use nlp.pipe() to process texts in larger batches, which is much faster, especially for a lot of short texts: for doc in nlp.pipe(sent_list): # averaged doc vector print(doc.vector) # token vectors print([token.vector for token in doc])


1

This is still a problem undergoing active research. I think it's commonly referred to as unsupervised labeling. If the dataset is not too large, it's probably more accurate and time efficient to either do it by hand, or hire someone on Mechanical Turk to do it. If you're dead set on doing it automatically, trying some form of semi-supervised learning where ...


1

For your first question, I would try to use Regex to identify the reference numbers since they seem unique comparing to normal words. I assume they are: All capital Start with one or more letters After follow numbers Any other pattern may follow (i.e. letters, numbers, hyphens) Do not include spaces Could be something like this: \b[A-Z]+\d+([A-Z]|\d|-)*\b ...


1

More gold data of any sort (especially at this small scale) usually leads to improvement because the model improves after seeing more examples in different contexts, but you especially want to add cases that the model got wrong (after correcting the annotation, of course) to see improvement.


1

To construct sentence embeddings Spacy just averages the word embeddings. I do not have access to Spacy right now, else would have give a demonstration but you can try: spacy_nlp('hello I').vector == (spacy_nlp('hello').vector + spacy_nlp('I').vector) / 2 If this also gives False, it will be because the float values might not be exactly equal after the ...


1

spaCy recommends that you use spaCy when you want production-grade performance but don't need to customize your architecture. It recommends that you use allenNLP when you want to explore different architectures or use the state-of-the-art models. It recommends against using allenNLP for production.


1

You can not replace tags in a sentence because Python strings are immutable. You can make another string with just the tags: >>> import spacy >>> nlp = spacy.load('en_core_web_sm') >>> doc = nlp("Apple is looking at buying U.K. startup for $1 billion") >>> " ".join(token.tag_ for token in doc) 'NNP VBZ VBG IN VBG NNP NN ...


1

If your data is always as clean as this, you might be able to solve the problem using simple regular expressions. You could simply look for the first number that occurs before the string of interest. As to the dependency parsing using spacy, this is the output that your sentence would get There expl are VERB [] are ROOT are VERB [There, people, now, .] 119 ...


Only top voted, non community-wiki answers of a minimum length are eligible