I have a huge data set(More than 1 million data points).My dataset is text. i am doing NER on it to identify few entities. if i randomly choose 100 data points from the total data set and train my model(LSTM), will this yield good results? i will be running for 20k random batches. Does this approximate the data properly or do i need to run for more number of batches than the total number of datapoints?


1 Answer 1


Depends entirely on your data, if your variables are mostly numerical then you can get by with small samples. If however you have a lot of categorical variables you need to make sure that each category of every variable is well represented in the subsample. If they are all numerical I would go for 1000 data points repeated 1000 times.

  • $\begingroup$ i updated my question. my data is mostly text data $\endgroup$
    – InAFlash
    Sep 18, 2018 at 9:36
  • $\begingroup$ Then it depends on your number of entities and how un/balanced they are, I would go with number of entities * 100 for sample size. $\endgroup$ Sep 18, 2018 at 9:40
  • $\begingroup$ are you sure.... $\endgroup$
    – InAFlash
    Sep 18, 2018 at 9:44
  • $\begingroup$ Without more information on your data it's anybodies guess how big the sample needs to be. $\endgroup$ Sep 18, 2018 at 9:53
  • $\begingroup$ No, i meant (entities * 100), this is not a category classification. its kind of sequence prediction. $\endgroup$
    – InAFlash
    Sep 18, 2018 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.