I have a huge data set(More than 1 million data points).My dataset is text. i am doing NER on it to identify few entities. if i randomly choose 100 data points from the total data set and train my model(LSTM), will this yield good results? i will be running for 20k random batches. Does this approximate the data properly or do i need to run for more number of batches than the total number of datapoints?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
Depends entirely on your data, if your variables are mostly numerical then you can get by with small samples. If however you have a lot of categorical variables you need to make sure that each category of every variable is well represented in the subsample. If they are all numerical I would go for 1000 data points repeated 1000 times.
-
$\begingroup$ i updated my question. my data is mostly text data $\endgroup$– InAFlashSep 18, 2018 at 9:36
-
$\begingroup$ Then it depends on your number of entities and how un/balanced they are, I would go with
number of entities * 100for sample size. $\endgroup$ Sep 18, 2018 at 9:40 -
-
$\begingroup$ Without more information on your data it's anybodies guess how big the sample needs to be. $\endgroup$ Sep 18, 2018 at 9:53
-
$\begingroup$ No, i meant (entities * 100), this is not a category classification. its kind of sequence prediction. $\endgroup$– InAFlashSep 18, 2018 at 9:56