I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I am trying to conduct a (modified) student's t-test on these models. I am able to conduct one (according to THIS link) where I compare only ONE variable common to only TWO models. I want to perform a (or multiple) t-tests with MULTIPLE variables and MULTIPLE models at once.

As mentioned, I can only perform the test with one variable (let's say F-measure) among two models (let's say decision table and neural net).

Here's the code for that. I am performing a Kolmogorov-Smirnov test (modified t):

from matplotlib import pyplot
from pandas import read_csv, DataFrame
from scipy.stats import ks_2samp

results = DataFrame()
results['A'] = read_csv('LMT (f-measure).csv', header=None).values[:, 0]
results['B'] = read_csv('LWL (f-measure).csv', header=None).values[:, 0]

value, pvalue = ks_2samp(results['A'], results['B'])
alpha = 0.05
print(value, pvalue)
if pvalue > alpha:
    print('Samples are likely drawn from the same distributions (fail to reject H0)')
    print('Samples are likely drawn from different distributions (reject H0)')

Any ideas?

  • $\begingroup$ I'm having trouble imagining any scenario where this would be a good idea - t-tests are useful and meaningful for a very specific set of statistical assumptions and interpretations, and this doesn't sound like one of them. I think you have an X-Y problem - perhaps you could explain what it is you are wanting to accomplish with this, so that someone might be able to suggest what sort of procedure you might want to try instead? $\endgroup$ – BrianH Apr 4 '19 at 0:24
  • $\begingroup$ I separate ML into two sections: making models and analyzing them. I am in the analysis stage. Having made 16 different models, I want to see which ones are the best. One approach is to simply look at raw metrics outputted by the program and compare it between the models. For instance, I could look for which model was the "best" by looking for the one with the highest "Mathew's Correlation" (as an example). However, I don't know if the differences are statistically significant (that's why we have these other tests (like t-tests)). I want to do these tests, however more efficiently: thus my Q. $\endgroup$ – Shounak Ray Apr 4 '19 at 1:21
  • $\begingroup$ Found a great solution! Check out my answer! $\endgroup$ – Shounak Ray Apr 4 '19 at 6:24

This is a simple solution to my question. It only deals with two models and two variables, but you could easily have lists with the names of the classifiers and the metrics you want to analyze. For my purposes, I just change the values of COI, ROI_1, and ROI_2 respectively.

NOTE: This solution is also generalizable. How? Just change the values of COI, ROI_1, and ROI_2 and load any chosen dataset in df = pandas.read_csv("FILENAME.csv, ...). If you want another visualization, just change the pyplot settings near the end.

The key was assigning a new DataFrame to the original DataFrame and implementing the .loc["SOMESTRING"] method. It removes all the rows in the data, EXCEPT for the one specified as a parameter.

Remember, however, to include index_col=0 when you read the file OR use some other method to set the index of the DataFrame. Without doing this, your row values will just be indexes, from 0 to MAX_INDEX.

# Written: April 4, 2019

import pandas                       # for visualizations
from matplotlib import pyplot       # for visualizations
from scipy.stats import ks_2samp    # for 2-sample Kolmogorov-Smirnov test
import os                           # for deleting CSV files

# Functions which isolates DataFrame
def removeColumns(DataFrame, typeArray, stringOfInterest):
    for i in range(0, len(typeArray)):
        if typeArray[i].find(stringOfInterest) != -1:
            DataFrame.drop(typeArray[i], axis = 1, inplace = True)

# Get the whole DataFrame
df = pandas.read_csv("ExperimentResultsCondensed.csv", index_col=0)
dfCopy = df

# Specified metrics and models for comparison
COI = "Area_under_PRC"
ROI_1 = "weka.classifiers.meta.AdaBoostM1[DecisionTable]"
ROI_2 = "weka.classifiers.meta.AdaBoostM1[DecisionStump]"

# Lists of header and row in dataFrame
#  `rows` may act strangely
headers = list(df.dtypes.index)
rows = list(df.index)

# remove irrelevant rows
df1 = dfCopy.loc[ROI_1]
df2 = dfCopy.loc[ROI_2]

# remove irrelevant columns
removeColumns(df1, headers, COI)
removeColumns(df2, headers, COI)

# Make CSV files
df1.to_csv(str(ROI_1 + "-" + COI + ".csv"), index=False)
df2.to_csv(str(ROI_2 + "-" + COI) + ".csv", index=False)

results = pandas.DataFrame()
# Read CSV files
# The CSV files can be of any netric/measure, F-measure is used as an example
results[ROI_1] = pandas.read_csv(str(ROI_1 + "-" + COI + ".csv"), header=None).values[:, 0]
results[ROI_2] = pandas.read_csv(str(ROI_2 + "-" + COI + ".csv"), header=None).values[:, 0]

# Kolmogorov-Smirnov test since we have Non-Gaussian, independent, distinctive variance datasets
# Test configurations
value, pvalue = ks_2samp(results[ROI_1], results[ROI_2])
# Corresponding confidence level: 95%
alpha = 0.05

# Output the results
print('\033[1m' + '>>>TEST STATISTIC: ')
print(">>>P-VALUE: ")
if pvalue > alpha:
    print('\t>>Samples are likely drawn from the same distributions (fail to reject H0 - NOT SIGNIFICANT)')
    print('\t>>Samples are likely drawn from different distributions (reject H0 - SIGNIFICANT)')

# Plot files
pyplot.xlabel(str(COI + " Values"))
pyplot.title(str(COI + " Density Distribution of " + ROI_1))

pyplot.xlabel(str(COI + " Values"))
pyplot.title(str(COI + " Density Distribution of " + ROI_2))

# Delete Files
os.remove(str(ROI_1 + "-" + COI + ".csv"))
os.remove(str(ROI_2 + "-" + COI + ".csv"))
  • $\begingroup$ The KS test gives just about zero (probably zero) into notions of superiority. $\endgroup$ – Dave Sep 3 '20 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.