How to use Chi-square test in dataset with negative values

Question

I could not fully explain the title. In order to use the Chi-square test in my dataset, I am finding the smallest value and add each cell with that value. (for example, the range of data here is [-8,11] so I added +8 to each cell and the range turned to [0,19]).

dataValues variable is DataFrame type that holds my all data and contains ~2000 features, ~1000 rows, dataTargetEncoded variable is Array type that contains results as 0 and 1.

for i in range(len(dataValues.index)):
    for j in range(len(dataValues.columns)):
        dataValues.iat[i, j] += 8

#fit chi2 and get results
bestfeatures = SelectKBest(score_func=chi2, k="all")
fit = bestfeatures.fit(dataValues, dataTargetEncoded)
feat_importances = pd.Series(fit.scores_, index=dataValues.columns)

#print top 10 feature
print(feat_importances.nlargest(10).index.values)

# back to normal
for i in range(len(dataValues.index)):
    for j in range(len(dataValues.columns)):
        dataValues.iat[i, j] -= 8

But this causes performance problems. Another solution I'm thinking of is to normalize it. I wrote a function that looks like this:

def normalization(df):
    from sklearn import preprocessing

    x = df.values  # returns a numpy array
    min_max_scaler = preprocessing.MinMaxScaler()
    x_scaled = min_max_scaler.fit_transform(x)
    df = pd.DataFrame(x_scaled, columns=df.columns)

return df

My program has accelerated lots, but this time my accuracy has decreased. The feature selection process I have done with the first method produces 0.85 accuracy results, this time I am producing 0.70 accuracy.

I want to get rid of this primitive method, but I also want accuracy to remain constant. How do I proceed? Thank you in advance

Answer 1

First and foremost, it does not matter to the chi-square test whether your data is positive, negative, string or any other type, as long as it is discrete (or nicely binned). This is due to the fact that the chi-square test calculations are based on a contingency table and not your raw data. The documentation of sklearn.feature_selection.chi2 and the related usage example are not clear on that at all. Not only that, but the two are not in concord regarding the type of input data (documentation says booleans or frequencies, whereas the example uses the raw iris dataset, which has quantities in centimeters), so this causes even more confusion. The reason why sklearn's chi-squared expects only non-negative features is most likely the implementation: the authors are relying on a row-by-row sum, which means that allowing negative values will produce the wrong result. Some hard-to-understand optimization is happening internally as well, so for the purposes of simple feature selection I would personally go with scipy's implementation.

Since your data is not discrete, you will have to bin every feature into some number of nominal categories in order to perform the chi-squared test. Be aware that information loss takes place during this step regardless of your technique; your aim is to minimize it by finding an approach that best suits your data. You must also understand that the results cannot be taken as the absolute truth since the test is not designed for data of continuous nature. Another massive problem that will definitely mess with your feature selection process in general is that the number of features is larger than the number of observations. I would definitely recommend taking a look at sklearn's decomposition methods such as PCA to reduce the number of features, and if your features come in groups, you can try Multiple Factor Analysis (Python implementation available via prince).

Now that that's out of the way, let's go through an example of simple feature selection using the iris dataset. We will add a useless normally distributed variable to the constructed dataframe for comparison.

import numpy as np
import scipy as sp
import pandas as pd

from sklearn import datasets, preprocessing as prep

iris = datasets.load_iris()

X, y = iris['data'], iris['target']
df = pd.DataFrame(X, columns= iris['feature_names'])
df['useless_feature'] = np.random.normal(0, 5, len(df))

Now we have to bin the data. For value-based and quantile-based binning, you can use pd.cut and pd.qcut, respectively (this great answer explains the difference between the two), but sklearn's KBinsDiscretizer provides even more options. Here I'm using it for one-dimensional k-means clustering to create the bins (separate calculation for each feature):

def bin_by_kmeans(pd_series, n_bins):
    binner = prep.KBinsDiscretizer(n_bins= n_bins, encode= 'ordinal', strategy= 'kmeans')
    binner.fit(pd_series.values.reshape(-1, 1))
    bin_edges = [
        '({:.2f} .. {:.2f})'.format(left_edge, right_edge)
        for left_edge, right_edge in zip(
            binner.bin_edges_[0][:-1],
            binner.bin_edges_[0][1:]
        )
    ]
    return list(map(lambda b: bin_edges[int(b)], binner.transform(pd_series.values.reshape(-1, 1))))

df_binned = df.copy()
for f in df.columns:
    df_binned[f] = bin_by_kmeans(df_binned[f], 5)

A good way to investigate how well your individual features are binned is by counting the number of data points in each bin (df_binned['feature_name_here'].value_counts()) and by printing out a pd.crosstab (contingency table) of the given feature and label columns.

An often quoted guideline for the validity of this calculation is that the test should be used only if the observed and expected frequencies in each cell are at least 5.

So the more zeroes you see in the contingency table, the less accurate the chi-squared results will be. This will require a bit of manual tuning.

Next comes the function that performs the chi-squared test for independence on two variables (this tutorial has very useful explanations, highly recommended read, code is pulled from there):

def get_chi_squared_results(series_A, series_B):
    contingency_table = pd.crosstab(series_A, series_B)
    chi2_stat, p_value, dof, expected_table = sp.stats.chi2_contingency(contingency_table)
    threshold = sp.stats.chi2.ppf(0.95, dof)
    return chi2_stat, threshold, p_value

The values to focus on are the statistic itself, the threshold, and its p-value. The threshold is obtained from a quantile function. You can use these three to make the final assessment of individual feature-label tests:

print('{:<20} {:>12} {:>12}\t{:<10} {:<3}'.format('Feature', 'Chi2', 'Threshold', 'P-value', 'Is dependent?'))
for f in df.columns:
    chi2_stat, threshold, p_value = get_chi_squared_results(df[f], y)
    is_over_threshold = chi2_stat >= threshold
    is_result_significant = p_value <= 0.05
    print('{:<20} {:>12.2f} {:>12.2f}\t{:<10.2f} {}'.format(
        f, chi2_stat, threshold, p_value, (is_over_threshold and is_result_significant)
    ))

In my case, the output looks like this:

Feature                      Chi2    Threshold  P-value    Is dependent?
sepal length (cm)          156.27        88.25  0.00       True
sepal width (cm)            89.55        60.48  0.00       True
petal length (cm)          271.80       106.39  0.00       True
petal width (cm)           271.75        58.12  0.00       True
useless_feature            300.00       339.26  0.46       False

In order to claim dependence between the two variables, the resulting statistic should be larger than the threshold value and the p-value should be lower than 0.05. You can choose smaller p-values for higher confidence (you'd have to calculate the threshold from sp.stats.chi2.ppf accordingly), but 0.05 is the "largest" value needed for your results to be considered significant. As far as ordering of useful features goes, consider looking at the relative magnitude of the difference between the calculated statistic and the threshold for each feature.

Answer 2

Chi-Sqaured statistic is a square of Z-statistic so I don't understand why it would matter if you have negative values in there.