Description : I have dataset of categorised articles and to extract specific values from respective categorised article I have regex created for each category.

Aiming for:

  1. Nlp techniques which learns the context of the content and avoids/minimizes the use of regex
  2. If some new (similar) article comes up, depending on the learning (from 1) it tries to give the specific values.

Steps taken:

  1. Created a dataframe with various features like : 'Name of the author', 'published date', etc. and got the values from the dataset by using regex


I was considering these options ahead this stage :

  1. Using CNN : it will classify new articles depending on the feature values it learnt on and then use regexs for entity extraction. (It wont achieve the first aim)
  2. Using CRF (medium_article): making use of POS+IOB tagging

Is there any other way around ? cons of above stated methods?


1 Answer 1


CRF is a standard for such case but bi-LSTM + CRF are said to be even better (e.g. https://arxiv.org/pdf/1508.01991.pdf). Not sure if you need POS as this is usually solved using the same techniques - may not help in the main task of entity extraction.

Depending how different are articles in specific categories and how much data you have for each of them, you may need to train each category separately and have some classifier in the beginning to decide category - like in your point 1. but word2vec-like classifier may be more robust here.

  • $\begingroup$ thanks for the reply. As specific categories donot have that prominent differences. (for ex:if I will take top 50 most frequent words, it'll be same for each article) Earlier I was able to create several categories by applying (multiple -iteration of randomly seeded) kmeans Though there was some noise, but I got enough data to create regex on. Will word2vec will be effective in classifying still? Also how about this : Creating clusters again by using topic modelling (LDA or NMF) - this will group more articles in specific cluster(which were segregated earlier by kmeans because less simila $\endgroup$ Commented Sep 3, 2018 at 18:55
  • $\begingroup$ If all categories have articles of similar nature (especially entities to be extracted and phrases 'around' them), I wouldn't bother for any classification nor clustering. Just prepare BIO tagging for each article and feed them all to one model. Depending on the variatey of articles, vocabulary and article size you will need probably around 100 000 articles. $\endgroup$
    – MkL
    Commented Sep 3, 2018 at 20:48
  • $\begingroup$ @MkL Details of the dataset : 1- entities and phrases around them are more or less same(i.e some article miss those values which are contained by others) 2- there is slight variation in articles content 3- vocab is not huge 4- article size is of mostly 2-3 lines 5- there are 50k data points a- is going for bilstm+crf is still recommendable also have you tried :[github.com/Hironsan/anago]. b- 50k data points wud be sufficient? $\endgroup$ Commented Sep 7, 2018 at 8:40
  • $\begingroup$ Depends on variety of the content, number of entities to be recognized. Roughly guessing it's on the edge of some meaningful quality. Pretrained embeddings may help a bit, possibly best from Wikipedia, e.g. from fasttext fasttext.cc/docs/en/english-vectors.html $\endgroup$
    – MkL
    Commented Sep 8, 2018 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.