# Tools for ML on csv files and jsons

So much of what we export is in CSV and JSON files.

Is there any useful tools you know of that can automatically perform data analysis on flat file formats. For example

• Basic statistics if numeric types: avg, stdev, mode, median
• Column type and Cardinality detection
• Find relationships between columns, if column A is X, then column B is always Y

Any of these things would be useful, even if there'd be an intermediary step of just loading the flat file into some software...

In R you can load the csv file easily by using the method read.csv and get the summary of the data using the method summary, you will get the most of the basic statistics of each columns like Min, 1st Qu, Median, Mean, 3rd Qu, Max and NA counts for numeric/integer rows and for string columns you will also get the some statics of count of different type of string if it has repeated multiple i.e factors.

Sample code:

• Loading the csv file and check the few datasets:

> df <- read.csv("SampleData.csv")
name age       Type
1    A  34     Active
2    B  56 Cancelled
3    C  12     Active
4    D  32 Cancelled
5   Z   34     Active

• Basic statistics if numeric types: avg, stdev, mode, median

> summary(df)
name        age               Type
A :1   Min.   :12.0   Active    :3
B :1   1st Qu.:32.0   Cancelled :2
C :1   Median :34.0
D :1   Mean   :33.6
Z :1   3rd Qu.:34.0
Max.   :56.0
>

• Column type and Cardinality detection

> sapply(df, class)
name       age      Type
"factor" "integer"  "factor"
>
> nrow(df)
[1] 5
> ncol(df)
[1] 3
>

• Find relationships between columns, if column A is X, then column B is always Y

For this you have to do some Basic calculation and you can go through this cookbook: http://www.cookbook-r.com/

I have attached Sample CSV Image here