# Tools to explore various datasets

TL:DR - Do you know of good automated tools to explore a dataset?

Long version: I have a few different datasets to work with from most various areas of business. I wonder if there are good software/scrips that could automatically round up the following answers for me: Things I would be looking for:

• datatype of a column?
• dimension or measure?
• histogram or count of values in a column
• correlation between measures
• unique key auto-find
• import into a strongly typed data structure

Obviously I'm looking for something more than basic summary(cars). Something that goes through each column and spits out detailed analysis report

There is a very cool active Python package called pandas-profiling, is exactly what you want. With a simple pandas_profiling.ProfileReport(df) it returns a lot of important statistical information about your data, the official documentation says:

For each column the following statistics - if relevant for the column type - are presented in an interactive HTML report:

• Essentials: type, unique values, missing values
• Quantile statistics like minimum value, Q1, median, Q3, maximum, range, interquartile range
• Descriptive statistics like mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis,
skewness Most frequent values
• Histogram
• Correlations highlighting of highly correlated variables, Spearman and Pearson matrixes

Look at their demo here. I personally used it few times and it quite nice, BUT it dataset is large (mostly I mean in variable space) it takes a long time to give the statistics. I think there should be an option to even return all these statistics for a subset of features (columns) in case you do not want for all and it is very high dimensional!

• But people don't like so much information in a single line of code... – Aditya Jun 1 '18 at 12:44
• this is actually quite amazing and I will use it! Thank you. Accepted as answer, but cannot upvote. – donkz Jun 6 '18 at 15:02

You can probably google and find tons of similar tools, and many of them with a Freemium model.

Most tools would have most of the features you want maybe except: 1) Dimension vs. Measures This one would be a bit tricky; An integer will most likely be a measure, but it could go either way 2) Unique key auto-find: again, it could be a bit tricky; It should be too much work for you to specify as needed.

These two should be relatively easy for a human to label accordingly as needed.

I have tried the following tool and it looks quite promising. (note: I am not associated with the tool in any way)

Exploratory.io

Welcome to DS-SE :) !

A very convenient way of doing this is by utilizing the pandas.DataFrame class in Python. It includes methods (and attributes) that perform what you asked:

"Something that goes through each column and spits out detailed analysis report"

such as:

1. DataFrame.corr() for correlation analysis
2. DataFrame.data.shape for dimensions
3. DataFrame.drop_duplicates() to keep the unique elements
4. DataFrame.hist() for histogram representation of the data
5. etc..
• sorry, but that's exactly what I'm not asking – donkz Jun 6 '18 at 14:26

I know Facets https://github.com/PAIR-code/facets - which should be nice for a preliminary in-browser preview.