How to measure correlation between several categorical features and a numerical label in Python?

Question

I have for a few weeks measured the time it takes for a product to be released through a automated release pipeline. I have several different categorical features such as "Product Category", "Product Owner". Then I also have some numerical categories such as "Hour of day when started the job", "Number of sub products" and more. In total I have around 16 different categories, where each category can take around 10 different values.

I will now try and train a regression model to see if I can predict the lead time(time it takes for the product to go through the pipeline) based on these features.

I would want to see if there are any of these features which are more correlated with the lead time than others. As I have understod it I have to seperate the numerical and categorical features and perform tests seperately on them. For numerical variables I have read about pearsonr and for correlating categorical and numerical variables I have read about ANOVA but I can't seem to find any way of implementing ANOVA in Python. Has anyone any experience with this? And if not, any other tips on how to do this?

Example data frame

        product_category     product_owner    #sub_products     hour     lead_Time_min
              A                  Bill             5             13         123
              B                  Lisa             14            19          40
              B                  Lisa             2             16          20
              D                  Eric             1             11          10
              C                  Ben              7             11          4
              C                  Lisa             14            10          25
              B                  Lisa             2             19          252

Answer 1

If you want to do an ANOVA test, you can do it with scipy and stats package. Link to documentation

You can do it like that

def anova(data):
    if len(data.groupby(level=1)) <= 2:
        raise Exception('ANOVA requires a secondary index with three or more values')

    return pd.DataFrame(
        [f_oneway(*[v for k, v in data[col].groupby(level=1)]) for col in data.columns],
        columns=['f', 'p'],
        index=data.columns) 

Source

or just choose the two columns you want to test

statistic, pvalue = stats.f_oneway(data['col1'], data['col2'])

However, I would advise you to take a different path. You need to test how important a feature is in your dataset to predict the lead_time. You can train a simple Decision Tree with the whole dataset and get the feature importance for each of the features. Then you remove those that are below a threshold.

from sklearn.tree import DecisionTreeRegressor

tree = DecisionTreeRegressor().fit(X, y)
tree.feature_importances_

Answer 2

If you perform linear regression, encoding the categorical variables by dummy numerical variables, the p-value of the corresponding coefficients will show you whether they significantly affect the lead time or not.

Bear in mind, however, that each possible value of a categorical variable translates into a separate dummy variable. Hence, you'll likely have some 160 ($= 10 \cdot 16$) dummy variables. Your set of observations would need to be huge to counter the curse of dimensionality.