0
$\begingroup$

Just wanting to see if anyone has any additional resources regarding appropriate machine learning methods for large categorical datasets (150,000 observations, health survey, ~10-15 features). Need help because it takes so long to run these algorithms, so would be good to have a handle on some general indicators to move towards. It's hard finding the specifics of my concerns (and there are many). Some things I've looked into and considered:

  1. We have a massive amount of missing data, sometimes 90%. Had looked into forms of missingness, looks like our result will be biased regardless. Best imputation methods for this, if this is indicated? Or does evidence show KNN more appropriate, perhaps CCA on its own? Have mainly used MICE.

  2. Most appropriate ML method for this form of data? All categorical, some multi-level, others binary. From what I've seen, RF is likely to be most appropriate, but happy to be shown alternative papers/evidence on better methods.

  3. Balancing vs not-balancing the data? Outcome is 1% of the data. I've seen reasons for both, we've presently used under-sampling, as it was just computationally efficient. Not sure if other methods induce less bias in this instance.

  4. Whether or not we perform some form of variable selection (using something like VSURF). Again have heard that this is obviously good for predictive ability and reducing error, but I've also seen this may essentially reflect P-hacking (correct me if this is incorrect).

Thanks!

$\endgroup$

1 Answer 1

1
$\begingroup$

A few comments:

  • You are focusing on the technical characteristics of the data. Sure it's important, but the semantic nature of the task is also very important. There are a lot of studies based on health surveys, so it's likely that you could find some of your answers in previous research work on a similar topic.
  • "it takes so long to run these algorithms" -> I'm confused, do you mean the running time or the designing and implementing time? Because this dataset doesn't look very large to me, so I would expect a reasonable running time. If not, maybe you need to explore other hardware options (e.g. cloud).
  • I would certainly not impute the missing values, it's likely to corrupt your data imho.
  • The most appropriate method depends on what is the task. Apparently it's some supervised classification, so yes, RF is a good method in general imho.
  • Generally I would say that resampling should only be done for some specific reason (e.g. favouring recall over precision). In any case the non-resampling should be tried as a baseline and of course the test set is never resampled.
$\endgroup$
1
  • $\begingroup$ I'll start looking more into ML methods and survey data. By "long" I meant a couple hours - but that was primarily the imputation that took some time (missForest never finished after ~3hrs). I'll look into prediction methods without imputation, or perhaps some inference based on other variable outcomes. Thanks for your answer! $\endgroup$
    – MJay
    Jan 8, 2023 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.