This is my first time building a custom model with SPACY NER.


Define a function to create spaCy DocBin objects from the annotated data

def get_spacy_doc(file, data):

Create a blank spaCy pipeline

nlp = spacy.blank('en') db = DocBin()

Iterate through the data

for text, annot in tqdm(data): doc = nlp.make_doc(text) annot = annot['entities']

ents = []
entity_indices = []

# Extract entities from the annotations
for start, end, label in annot:
  skip_entity = False
  for idx in range(start, end):
    if idx in entity_indices:
      skip_entity = True
  if skip_entity:

  entity_indices = entity_indices + list(range(start, end))
    span = doc.char_span(start, end, label=label, alignment_mode='strict')

  if span is None:
    # Log errors for annotations that couldn't be processed
    err_data = str([start, end]) + "    " + str(text) + "\n"

  doc.ents = ents

return db '''


Split the annotated data into training and testing sets

from sklearn.model_selection import train_test_split train, test = train_test_split(cv_data, test_size=0.2)

Display the number of items in the training and testing sets

len(train), len(test)

Open a file to log errors during annotation processing

file = open('/content/drive/MyDrive/trial_domain_extraction/trained_models/train_file.txt','w')

Create spaCy DocBin objects for training and testing data

db = get_spacy_doc(file, train) db.to_disk('/content/drive/MyDrive/trial_domain_extraction/trained_models/train_data.spacy')

db = get_spacy_doc(file, test) db.to_disk('/content/drive/MyDrive/trial_domain_extraction/trained_models/test_data.spacy')

Close the error log file

file.close() '''


Train a spaCy NER model using the provided configuration and data

!python -m spacy train /content/drive/MyDrive/trial_domain_extraction/config/config.cfg --output /content/drive/MyDrive/trial_domain_extraction/trained_models/output --paths.train /content/drive/MyDrive/trial_domain_extraction/trained_models/train_data.spacy --paths.dev /content/drive/MyDrive/trial_domain_extraction/trained_models/test_data.spacy --gpu-id 0 ''' Output pipeline:

============================= Training pipeline ============================= ℹ Pipeline: [] ℹ Initial learn rate: 0.001 E # SCORE

0 0 0.00 100 200 0.00 200 400 0.00 300 600 0.00 400 800 0.00 500 1000 0.00 600 1200 0.00 700 1400 0.00 800 1600 0.00

I have followed the following article to the T.Text

My pipeline looks empty(please refer above.)Is it because there was too little training data? I have used 13 entities. Would you suggest training with more amount of annotated data would resolve this issue? If yes, how much quantitatively?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.