0
$\begingroup$

I am working with lots of data (we have a table that produces 30 million rows daily). What is the best way to explore it (do on EDA)? Take a frictional slicing of the data randomly (100000 rows) or select the first 100000 rows from the entire dataset or should i take all the dataset WHAT SHOULD I DO?

thanks!!!!

$\endgroup$
1
  • $\begingroup$ I think randomly selected downsampled data is the best option to do EDA on such occasions. $\endgroup$ May 19 at 15:15

2 Answers 2

1
$\begingroup$

You mentioned data is added daily. A lot of this has to do with how your data is structured and if recent data is more important than older data. It might be easier to take a random sample from recent data. But if you are looking over all of the dates, you could sample different periods. but the statistical answer also has to do with how many variables you are looking at. Practically, you might want to start with a 'reasonable' number of rows that are easy to get, do basic EDA for missing values, rules of thumb like insuring you have a minimum count for performing things like a regression. Then increase the number to the level that you would need to have a recognizable distribution for all of the variables you are interested in. What you often miss when taking random samples are outliers, so it is always useful to ask the business what they expect the upper and lower ranges to be

$\endgroup$
1
  • $\begingroup$ thank you very mush!!! $\endgroup$
    – MAS
    Jun 13 at 12:23
0
$\begingroup$

Generally it's easier to manipulate a subset of the data, but it's important to take a random sample in order to have a representative sample.

$\endgroup$
1
  • $\begingroup$ thank you very mush!!! $\endgroup$
    – MAS
    Jun 13 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.