As a newbie, I am a little confused. I have a dataset for binary classification with 11 features and 102 sample data. I have seen in most places (e.g., kaggle competions), the dataset may have hundreds of thousands of data samples for tens of features. On the other hand, this paper says (at least for LDA classifier) optimal number of features is n-1 for a sample size n. My question is, if small no. of samples is enough (or even optimal), why care for larger samples? What am I missing here?
Bounds on the needed amount of samples are very common in PAC learning. When you define a concept class you can compute a minimal size set of sample that will enable learning. However,
- More samples will allow improving accuracy
- More samples will enable learning more complex concepts, that might fit your data better.
- As @Emre wrote, real life data sets usually are not clean as in PAC learning. The concept class is not given to you, the data has noise and a given distribution is not guaranteed.
Showing that a classifier can be learnt with a small amount of data is great. It is a big advantage of the learner. However, more data usually helps and if it helps more than expected for such a classifier, it is possible that the classifier requirements doesn't hold.
Just to amplify @Dal's excellent answer, you certainly can fit models with small sample sizes. That's exactly what classical statistics attempts to do, often with success. But there is a price in terms of quality of the data , simplicity of the model and experimental design. For example, to estimate interaction terms efficiently and credibly, you want to randomly apply the treatment effects. Machine learning typically occurs in observational data where none of these assumptions are correct.
That said, 102 samples is way to small in a problem with 11 variables and a binary outcome. Classifications typically take more data than regression problems (continuous outcome). That's why you often hear pollsters using 1000 respondents to predict a categorical outcome on the basis of one or two features (questions).
There is an optimistic rule of thumb that one needs 10 variables for each parameter that one wants to estimate. I have always thought this was a bit thin, but even on that measure, your sample is too small.