1
$\begingroup$

Basically I am taking a small course in data science. And one of our first assignments are that we have gotten a pretty huge data set (hundred of thousands of samples, and roughly 150 independent variables). There is no response variable in this, but the assignment is pretty much: "Find something that are interesting". Which I obviously kind of broad.

My questions is: How would/should you/I approach this ?

Up until now I have just cleaned the data set as good as I can. Removed variables I don't think make much sense, made sure there are no NaN variables, and stuff like that. But what would be the next step in order to "find something useful" in a data set that may or may not contain anything interesting?

$\endgroup$
  • $\begingroup$ 1) Do you have meaningful variable names (like age, gender, etc) or is it rather anonymized (like x1, x2,...). 2) Does it have temporal variables, e.g. date, time stamps? 3) what do you mean by variables that don’t make much sense? 4) Provide a sample of data if possible. $\endgroup$ – aivanov Apr 23 at 16:40
2
$\begingroup$

Your assignment is basically the process we call EDA - Explorative Data Analysis.

So what should you do? Simply explore!

  • What is the shape of your dataset?
  • How do variables behave, do they have a factor structure, correlate, etc.
  • What are the main descriptives of your dataset, to they tell an interesting story, etc.

And once you start doing this you will find something that might be interesting to explore a bit deeper depending on your dataset. Do not just use summary functions like mean, median, etc. but also try to build simple graphs and comment everything in a neat notebook!

My tip:

Look at some EDA notebooks on Kaggle for inspiration or watch this superior video by a master at work:

https://www.youtube.com/watch?v=go5Au01Jrvs

Also here is a beginner guide as well:

https://towardsdatascience.com/exploratory-data-analysis-eda-a-practical-guide-and-template-for-structured-data-abfbf3ee3bd9

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.