Empirical papers for ML methods in large categorical datasets?

Question

Just wanting to see if anyone has any additional resources regarding appropriate machine learning methods for large categorical datasets (150,000 observations, health survey, ~10-15 features). Need help because it takes so long to run these algorithms, so would be good to have a handle on some general indicators to move towards. It's hard finding the specifics of my concerns (and there are many). Some things I've looked into and considered:

1. We have a massive amount of missing data, sometimes 90%. Had looked into forms of missingness, looks like our result will be biased regardless. Best imputation methods for this, if this is indicated? Or does evidence show KNN more appropriate, perhaps CCA on its own? Have mainly used MICE.

2. Most appropriate ML method for this form of data? All categorical, some multi-level, others binary. From what I've seen, RF is likely to be most appropriate, but happy to be shown alternative papers/evidence on better methods.

3. Balancing vs not-balancing the data? Outcome is 1% of the data. I've seen reasons for both, we've presently used under-sampling, as it was just computationally efficient. Not sure if other methods induce less bias in this instance.

4. Whether or not we perform some form of variable selection (using something like VSURF). Again have heard that this is obviously good for predictive ability and reducing error, but I've also seen this may essentially reflect P-hacking (correct me if this is incorrect).

Thanks!

Answer 1

A few comments:

• You are focusing on the technical characteristics of the data. Sure it's important, but the semantic nature of the task is also very important. There are a lot of studies based on health surveys, so it's likely that you could find some of your answers in previous research work on a similar topic.
• "it takes so long to run these algorithms" -> I'm confused, do you mean the running time or the designing and implementing time? Because this dataset doesn't look very large to me, so I would expect a reasonable running time. If not, maybe you need to explore other hardware options (e.g. cloud).
• I would certainly not impute the missing values, it's likely to corrupt your data imho.
• The most appropriate method depends on what is the task. Apparently it's some supervised classification, so yes, RF is a good method in general imho.
• Generally I would say that resampling should only be done for some specific reason (e.g. favouring recall over precision). In any case the non-resampling should be tried as a baseline and of course the test set is never resampled.