I am working on a biology related dataset with over 300K features, and I only have about 5K samples. I want my model to classify many classes. For this problem in particular the class is age. Each age such as 10 or 35 will be individual classes. So roughly 80 classes (range from 10 to 90) are needed for this problem.

I immediately know that regularization is needed to shrink the number of features to prevent overfitting. I just don't know whether such a dataset can be treated as a multiclass classification problem with many number of classes. If I need more data, how many data will be enough for the model to learn? Or are there any clever ways I can do for this problem?

  • $\begingroup$ Your data set is wildly past the point where it would saturate a regression model. p < n is the broadest rule of thumb I know, and 300,000 features : 5,000 observations is a problem. It's not even an overfitting issue, it's a fundamental math issue: there aren't enough observations to look at all of those features at once. Do you really need 300k features? Do you need your model to be easily interpretable, or is prediction accuracy all that matters? $\endgroup$ – Upper_Case Jun 3 '19 at 20:54
  • $\begingroup$ that's why I point out I am going to implement regularization to reduce dimension. $\endgroup$ – James Liu Jun 4 '19 at 2:21
  • $\begingroup$ I'm not sure that will be enough. A regularized model (like a LASSO) won't work properly with p >> n like this. That's why I asked about how interpretable you need your model to be-- something like PCA may help for dimension reduction, but will make your model a lot more difficult to interpret. $\endgroup$ – Upper_Case Jun 4 '19 at 14:20

You can try dimensionality reduction techniques such as PCA, which will reduce your numbers of features and maximizing the information describing your objects.

Moreover, if you want to classify the age which is a continuous variable, you may want to try regression instead of classification

| improve this answer | |
  • $\begingroup$ I did use PCA. Although, I dont see why regression would be a good idea. Isnt that equivalent to having a multiclass classification where each age is a class? $\endgroup$ – James Liu Jun 5 '19 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.