Automatic statistical analysis / statistical "diff" tool?

Question

I'm new to data science and wondered if there are ways to have a tool figure out relations between variables that may be relevant to a problem.

Imagine I have a log file that I have pre-processed to the following tsv file (600k rows):

time           req  ip               latency  status  length
|------------- |--  |--------------- |-----   |-----  |-----
11:59:59.8048  Foo  111.208.52.116   1728     200 OK  2422
11:59:59.9454  Goo  11.88.226.84     1567     200 OK  2422
11:59:59.9611  Bar  111.38.177.130   26460    200 OK  1068
11:59:59.9923  Foo  11.88.226.84     1795     200 OK  2422
12:00:00.0079  Foo  11.110.239.63    1816     200 OK  2422
12:00:00.0548  Bar  111.147.192.171  15265    200 OK  1024
12:00:00.0704  Baz  111.130.195.74   1801     200 OK  9876
12:00:00.0704  Foo  111.130.195.74   1803     200 OK  2422
12:00:00.1173  Bar  11.139.142.176   10527    200 OK  1024
...
14:59:54.3229  Faz  111.136.253.236  1317386  200 OK  2417
14:59:54.3229  Foo  11.12.5.128      4319     200 OK  2418
14:59:55.4792  Far  111.100.110.47   25120    200 OK  5432
14:59:55.4792  Zoo  111.86.217.168   23236    200 OK  2417
14:59:55.4949  Rar  111.65.9.93      137184   200 OK  2417

Now I would like to know if this data can perhaps tell me what variables (or combinations of variables) are related to the CPUs being pegged at 100% (which was the case from 12:50 throughout 13:05) versus the CPU usage being around 50% (all other times).

I can of course manually try to plot and aggregate variables against each other that I think are relevant, but that takes time and experience (although I found doing it with R is fast and fun, compared to spreadsheet apps), and I wondered if there is a way to just throw 2 sets of data at R or some other tool, and it will tell me what the differences of those sets are, statistically speaking.

Of course variables can be combined and aggregated in a myriad of ways, but as Occam's Razor would suggest, this diff should start as simple as possible and only get more complicated over time, e.g. until I'm satisfied and tell it to stop.

Answer 1

If I understand correctly you don't want a statistically unbiased evaluation about how much the different variables influence the CPU usage, but only if they do that in a significant way. If this is your goal, it's quite easy to achieve it in R with a multiple linear regression.

# Assuming you saved data in a .csv file
data = read.csv('file.csv',sep=',', header=TRUE)
attach(data)
library(leaps) # Needed library
regfit = regsubsets(data$CPU_percentage~., data=data, nvmax= n)
# Assuming the column of the dataframe containing the percentage
# is called CPU_percentage
# "n" is the number of variables considered
summ = summary(regfit) # Useful info about your regression model
coef(regfit, witch.max(summ$adjr2))
# R will print the variables useful for the regression, in other words
# the variables that have some influence in the CPU usage, just add those
# variables in the next line of code in the place of var1, var 2 etc.
fit = lm(data$CPU_percentage~var1+var2+var3, data=data)
summary(fit)

The coefficients showed by the last function (summary) quantify the relation between the CPU percentage and the variables themselves.

It's important to say that I wrote the code as simple as possible and only with the necessary informations for you, the obtained results can't be used to make statistical inference and the coefficients obtained are useful just to get an idea of the variables that are related to the CPU usage (both in a positive and a negative way). You need way more code and statistics knowledge than the one I've written here, and I'll be happy to explain it better if you want, but to reach your goal, you don't need more than that.