I have a huge data set(More than 1 million data points).My dataset is text. i am doing NER on it to identify few entities. if i randomly choose 100 data points from the total data set and train my model(LSTM), will this yield good results? i will be running for 20k random batches. Does this approximate the data properly or do i need to run for more number of batches than the total number of datapoints?


Depends entirely on your data, if your variables are mostly numerical then you can get by with small samples. If however you have a lot of categorical variables you need to make sure that each category of every variable is well represented in the subsample. If they are all numerical I would go for 1000 data points repeated 1000 times.

| improve this answer | |
  • $\begingroup$ i updated my question. my data is mostly text data $\endgroup$ – InAFlash Sep 18 '18 at 9:36
  • $\begingroup$ Then it depends on your number of entities and how un/balanced they are, I would go with number of entities * 100 for sample size. $\endgroup$ – user2974951 Sep 18 '18 at 9:40
  • $\begingroup$ are you sure.... $\endgroup$ – InAFlash Sep 18 '18 at 9:44
  • $\begingroup$ Without more information on your data it's anybodies guess how big the sample needs to be. $\endgroup$ – user2974951 Sep 18 '18 at 9:53
  • $\begingroup$ No, i meant (entities * 100), this is not a category classification. its kind of sequence prediction. $\endgroup$ – InAFlash Sep 18 '18 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.