0
$\begingroup$

I'm using the entire text book data by scraping the information of each chapter.

How do I highlight the spacy spancat NER or Bert Q&A based models to train multiple comma separated values in the text as important. For each chapter this behavior is recurring so how do I train the model to detect that it is important and that section is the important part which discusses different topics for each chapter.

Eg: After scraping the chapter 1: There is 1 paragraph that describes what topics will be covered in this chapter like x,y,z,a,b,c,d,e.

Similarly in chapter 2, There is 1 paragraph that describes what topics will be covered in this chapter like f,g,h,i,j,k.

How do I train this model in such a way that if I move to next chapter or even take the next book, It'll recognize these patters as the topics in that chapter or get all important topics discussed in the entire book? SO, it'll be the sum of all such comma separated values in the book.

$\endgroup$
2
  • $\begingroup$ Is it a paragraph classification problem? Is the aim to recognize which paragraphs have x,y,z,.. and which ones haven't? $\endgroup$ Dec 30, 2022 at 9:00
  • $\begingroup$ @NicolasMartin We can make it a 2 step problem. 1. Detect which paragraph has multiple comma or slash seperated values. 2. Detect those values with the paragraphs from the above step 1. $\endgroup$
    – jason
    Dec 30, 2022 at 14:35

1 Answer 1

1
$\begingroup$

If the paragraphs containing plenty comma separated values are easy to detect, I wouldn't search for a very complex algorithm:

  • After separating each paragraph from each chapter, look for the ones that have higher relative stats of commas, slashes, and/or return chariots. You can compare them easily to other paragraphs. Normally, topic paragraphs should have a much higher quantity of commas than other ones.
  • Then, you can extract content between separators using the split function.

Here is an example:

s = "element1,element2,element3,element4"

# Split the string on the ',' character
elements = s.split(',')

print(elements)

If it is more complex than that, you can use doc2vec to classify according to targets [normal, topic], but you have to train with many samples, so that the model can differentiate normal and topic paragraphs correctly.

doc2vec_embs = Doc2VecEmbeddings()
x_train_tokens = doc2vec_embs.build_vocab(documents=x_train)
doc2vec_embs.train(x_train_tokens)

x_train_t = doc2vec_embs.encode(documents=x_train)
x_test_t = doc2vec_embs.encode(documents=x_test)

from sklearn.linear_model import LogisticRegression
model = LogisticRegression(solver='newton-cg', max_iter=1000)
model.fit(x_train_t, y_train)
y_pred = model.predict(x_test_t)
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
print('Accuracy:%.2f%%' % (accuracy_score(y_test, y_pred)*100))
print('Classification Report:')
print(classification_report(y_test, y_pred))

Source

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.