Tool for analyzing a Python matrix and generating a report on the contents (column types, NaN counts, means, etc.)

Question

I'm looking for a tool/library that will take a numpy or pandas matrix and generate a list of statistics for the matrix and columns. Specifically, for each column, I'd want info like the following:

• Assumed data type (numerical vs string)

Assuming it's determined to be a numerical column:

• Mean and Std Deviation
• Max and min values
• Number of NaN's
• % of NaN's

Assuming it's determined to be a string column:

• Number of distinct string values

So the tool takes the matrix and outputs some kind of report along these lines. The goal is to do some sanity checking on my input data set as well as checking data transformations along the way. For example, if I'm doing some kind of transformation and values in column A go from 1% NaN to 60% NaN, then I did something really bad.

Does such a tool exist?

Answer 1

You should take a look at pandas_profiling, I don't think it works with numpy arrays but it does exactly what you want for Pandas dataframes. It can output to PDF or to a nice looking HTML format within your Jupyter Notebooks.

https://github.com/JosPolfliet/pandas-profiling

Answer 2

You could use pandas.describe.

import pandas as pd
import numpy as np
df  = pd.DataFrame({'first': [1,2,3,4,np.NaN], 'second': [100,200,np.NaN, np.NaN,4]})
df.describe(include='all')

Regarding the percentage of NAs, you could use just a small formula for that:

count_nan = len(df) - df.count()
count_nan / df.shape[1]