# Correlate an array of categorical features to binary outcome

I have a data set that looks like this:

target,items
1,[i1,i3]
1,[i4,i5,i9]
0,[i1]
...


The variable target is 0-1 outcome. The feature "items" is a set of items (variable length). Each item is a categorical variable (one of: i1, i2, .., i_N). There's no order/relationship between the items. A business example would be "set of products in a cart, outcome whether the customer abandons cart".

The size of data is approx. 1,000,000 by 5,000 (I have ~1 million examples, and N is approximately 5,000)

I want to do the following analysis. I want to find the items that influence (or lead to) target = 1. I don't have extra features to add. What is the type of statistical analysis or machine learning modelling technique that I should use?

How large is N? Can you reshape your data into something like:

target   i1  i2  i3  i4  i5 ... i9 ... iN
1    1   0   1   0   0 ...  0 ...  0
1    0   0   0   1   1 ...  1 ...  0
0    1   0   0   0   0 ...  0 ...  0


Once you have fit everything into a data frame, you can use any two-class supervised classification algorithm to build your model. There is no "best" model in general, but try a few to see which one works the best with your data.

Sorry for posting as an answer; comment doesn't allow preformatted text.

• Yes, I can format the data this way. It's approximately 1,000,000 example by 5,000 features. After finding a good model, how does I answer my original question: here are the items that has the most influence on outcome=1? – Edward B. Jun 14 '18 at 16:57
• Is it correct to compute correlation(target, i1), correlation(target, i2) etc and take the columns with top values of correlation? Is that a correct approach from statistics point of view? – Edward B. Jun 14 '18 at 16:59
• Run a Gradient Boosting Algorithms (whatever you like XGBoost,Catboost, LightGBM) on the dataframe suggested by @The Lyrics. Once trained, you can easily look at feature importance and you will see which variable(s) (product(s) in your case) has been most informative on your target prediction. if you want to take it further you can look at shap values (github.com/slundberg/shap) instead of feature importance which gives you many more insights including dependency plots (or correlation if you wish). – TwinPenguins Jun 14 '18 at 18:25