1
$\begingroup$

TL:DR - Do you know of good automated tools to explore a dataset?

Long version: I have a few different datasets to work with from most various areas of business. I wonder if there are good software/scrips that could automatically round up the following answers for me: Things I would be looking for:

  • datatype of a column?
  • dimension or measure?
  • histogram or count of values in a column
  • correlation between measures
  • unique key auto-find
  • import into a strongly typed data structure

Obviously I'm looking for something more than basic summary(cars). Something that goes through each column and spits out detailed analysis report

Improve this question
$\endgroup$

4 Answers 4

Reset to default
2
$\begingroup$

There is a very cool active Python package called pandas-profiling, is exactly what you want. With a simple pandas_profiling.ProfileReport(df) it returns a lot of important statistical information about your data, the official documentation says:

For each column the following statistics - if relevant for the column type - are presented in an interactive HTML report:

  • Essentials: type, unique values, missing values
  • Quantile statistics like minimum value, Q1, median, Q3, maximum, range, interquartile range
  • Descriptive statistics like mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis,
    skewness Most frequent values
  • Histogram
  • Correlations highlighting of highly correlated variables, Spearman and Pearson matrixes

Look at their demo here. I personally used it few times and it quite nice, BUT it dataset is large (mostly I mean in variable space) it takes a long time to give the statistics. I think there should be an option to even return all these statistics for a subset of features (columns) in case you do not want for all and it is very high dimensional!

Improve this answer
$\endgroup$
2
  • $\begingroup$ But people don't like so much information in a single line of code... $\endgroup$
    – Aditya
    Jun 1, 2018 at 12:44
  • 2
    $\begingroup$ this is actually quite amazing and I will use it! Thank you. Accepted as answer, but cannot upvote. $\endgroup$
    – donkz
    Jun 6, 2018 at 15:02
1
$\begingroup$

You can probably google and find tons of similar tools, and many of them with a Freemium model.

Most tools would have most of the features you want maybe except: 1) Dimension vs. Measures This one would be a bit tricky; An integer will most likely be a measure, but it could go either way 2) Unique key auto-find: again, it could be a bit tricky; It should be too much work for you to specify as needed.

These two should be relatively easy for a human to label accordingly as needed.

I have tried the following tool and it looks quite promising. (note: I am not associated with the tool in any way)

Exploratory.io

Improve this answer
$\endgroup$
1
$\begingroup$

Welcome to DS-SE :) !

A very convenient way of doing this is by utilizing the pandas.DataFrame class in Python. It includes methods (and attributes) that perform what you asked:

"Something that goes through each column and spits out detailed analysis report"

such as:

  1. DataFrame.corr() for correlation analysis
  2. DataFrame.data.shape for dimensions
  3. DataFrame.drop_duplicates() to keep the unique elements
  4. DataFrame.hist() for histogram representation of the data
  5. etc..
Improve this answer
$\endgroup$
1
  • $\begingroup$ sorry, but that's exactly what I'm not asking $\endgroup$
    – donkz
    Jun 6, 2018 at 14:26
0
$\begingroup$

I know Facets https://github.com/PAIR-code/facets - which should be nice for a preliminary in-browser preview.

enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.