I'm looking for a tool/library that will take a numpy or pandas matrix and generate a list of statistics for the matrix and columns. Specifically, for each column, I'd want info like the following:
- Assumed data type (numerical vs string)
Assuming it's determined to be a numerical column:
- Mean and Std Deviation
- Max and min values
- Number of NaN's
- % of NaN's
Assuming it's determined to be a string column:
- Number of distinct string values
So the tool takes the matrix and outputs some kind of report along these lines. The goal is to do some sanity checking on my input data set as well as checking data transformations along the way. For example, if I'm doing some kind of transformation and values in column A go from 1% NaN to 60% NaN, then I did something really bad.
Does such a tool exist?