Hot answers tagged

8 votes
Accepted

Address parsing using spaCy

Please look at my comment to add more information to your post. Based on the information you provided, here are my remarks: SpaCy is trained to find locations, not addresses per se If you use a "...
user avatar
7 votes

How to train a spacy model for text classification?

You have several good tutorials on the web : https://www.kaggle.com/poonaml/text-classification-using-spacy Basically, you have to : Import the data in python, here POSITIVE is the variable to ...
user avatar
6 votes

Twitter POS and NER: What is state-of-the-art?

SOTA is changing so rapidly in NLP that even Data Science professionists struggle to cope with it. I have two main sources that I constantly check to gain some insights on SOTA: NLP Progress from ...
user avatar
  • 5,687
6 votes
Accepted

Is there any way to define custom entities in Spacy

For pretrained models, spaCy has a few in different languages. You can find them in their official documentation https://spacy.io/models The available models are: English German French Spanish ...
user avatar
  • 3,730
6 votes
Accepted

Spacy custom POS tagging for medical concepts

Based on the example, it looks like you need more than simple POS tagging. Thankfully there is a full subdomain of NLP devoted to biomedical data, and there are many tools available which can help ...
user avatar
  • 21.8k
3 votes
Accepted

Converting paragraphs into sentences

Spacy's Sentencizer is very simple. However, Spacy 3.0 includes Sentencerecognizer which basically is a trainable sentence ...
user avatar
  • 15.4k
3 votes
Accepted

How can i extract words from a single concatenated word?

Can use a package that relies on a spellchecker to find the best way to split, like this one: https://pypi.org/project/compound-word-splitter/
user avatar
  • 56
2 votes
Accepted

SpaCy vs AllenNLP?

spaCy used to recommended (archive link) that you use spaCy when you want production-grade performance but don't need to customize your architecture. They recommended that you use allenNLP when you ...
user avatar
  • 136
2 votes
Accepted

Chunking Sentences with Spacy

Named Entity Recognition (NER) would extract names of people, organizations and such. Example: ...
user avatar
  • 21.8k
2 votes

SpaCy string store

Spacy uses a hash function that assigns an integer to any Unicode string, it is not an index in vocabulary it just a random integer that is used internally for better efficiency. It is a hash function,...
user avatar
  • 1,388
2 votes

Smart sentence segmentation not splitting on abbreviations

Neural tools trained on Universal Dependencies corpora use learned models for tokenization and sentence-spliting. Two I know of are: UDPipe – developed at Charles University in Prague. Gets very good ...
user avatar
  • 1,388
2 votes

Spacy tokenizer that strips HTML/XML keeping the positions

Generally, XML is first parsed. Then, the contents can be analyzed with something like spaCy. xml.etree.ElementTree is the most common way to parse XML in Python.
user avatar
2 votes

Converting paragraphs into sentences

There is nothing in SpaCy that you can use out-of-the-box. However, they allow you to use custom components To solve your problem, I see at least three ways to do it. NTLK NLTK allows you to add ...
user avatar
2 votes

How to make the spacy 3.0 custom NER model training to optimize on precision rather than recall?

I don' know Spacy custom NER but it's unlikely that the model is optimized on recall, otherwise it would label absolutely everything as an entity in order to reach perfect recall. Your model happens ...
user avatar
  • 21.8k
2 votes
Accepted

NLP approaches to infer Processes from Text

Following with the idea of building a classifier, one option is to use nltk library together with Keras-Tensorflow once you have a labeled dataset with the desired process categories. You can go on ...
user avatar
  • 2,399
2 votes
Accepted

Does spaCy support multiple GPUs?

I think I have figured out how to do this: The key is to use spawn not fork, and use cupy to ...
user avatar
2 votes

Phone number tagging with spaCy

Currently you're using using a pre-trained NER model to tag a single sentence. The pre-trained model is not especially trained for phone numbers, it performs general NER. This is why it will also tag ...
user avatar
  • 21.8k
2 votes
Accepted

Meaning of NER Training values using Spacy

The values for LOSS TOK2VEC and LOSS NER are the loss values for the token-to-vector and named entity recognition steps in your ...
user avatar
  • 5,844
2 votes

How to extract details (educational details, exp details etc.) from a resume?

Try using the spacy-transformers for NER. You can even Fine-Tune it as per the project requirements. Refer to this link: https://towardsdatascience.com/how-to-fine-tune-bert-transformer-with-spacy-3-...
user avatar
1 vote

Testing Spacy NER model

You should use the evaluate command to evaluate the test set. It would look like this: spacy evaluate ./my-model ./test-data.spacy
user avatar
  • 313
1 vote
Accepted

Extracting Names using NER | Spacy

The NER model performance on a particular text depends on which data it was trained with originally, and naturally the standard models (like en_core_web_sm) are ...
user avatar
  • 21.8k
1 vote

Spacy - Ignored (essential) entities during labeling - how important is this aspect?

Yes, it matters. A lot. You need to label every entity you encounter in each sentence. As long as they are non-overlapping, you can add as many entity types and entities per document as you'd like. ...
user avatar
1 vote

How to go about training a NER model to extract book citations in free-form?

Interesting task :) I think even with a good amount of training data it will be difficult for a regular NER model to perform well with new books titles and authors: The book may contain persons names ...
user avatar
  • 21.8k
1 vote

Facing this issue while predicting "CountVectorizer - Vocabulary wasn't fitted"

You have to give your training set to the model to be trained _= pipe.fit(triningSet.data, triningSet.target) I don't see any training dataset here. you have to ...
user avatar
1 vote

Detect passive voice in headlines

So, your task is to detect the passive voice from sentences. Currently, you have defined some rules to detecting the passive voice and you have noticed that there some exceptions to your defined rules....
user avatar
  • 1,321
1 vote

Text Classification on a very small data set with a lot of classes

Accuracy is not the best measure for imbalanced data. Prefer precision and recall. Do undersampling/oversampling to get equal samples for each class and try XGBoost. Or else you can use SVC with ...
user avatar
1 vote

How do you distinguish between conversational text and possible news article?

News sentences will have more unique tokens than normal conversations. Conversations have more stop words than news articles. I think you can use bert or normal wordvect classification to train a ...
user avatar
1 vote
Accepted

spaCy - Text Preprocessing - Keeping "Pronouns" in text

What you are doing seems fine in terms of preprocessing. Removing less informative words like stopwords, punctuation etc. is a very common technique. Here are some of my notes: probably best for ...
user avatar
1 vote
Accepted

Organization finder in spaCy

Using dependency parsing alone will not give you what you need. You may be able to get your answer by interpreting the dependency tree. For instance, in this case ...
user avatar
1 vote
Accepted

What should return doc.ents if the doc have no entities, in spacy?

You could just test whether the tuples entities has any elements: for sent in list(doc.sents): if len(sent.ents) > 0: nb = nb+1 Edit: For the ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible