Impossible to increase model accuracy [closed]

Question

I'm building binary classification models on my company's dataset. The problem I'm having is that I haven't been able to increase the accuracy of my models. I have trained, tuned, cross validated models such as logistic regression, knn, neural networks and they all have results with the same accuracy. I feel that I have tried everything. The dataset is about 100 data points with 21 features. I'm aware of the curse of dimensionality so I have tried using only subsets of the features but the accuracy is the same no matter which subsets of features I use.

Is it possible that with this dataset, no further improvements can be done? Should I just tell my boss that this is the maximum accuracy possible given the amount of data we have?

Answer 1

100 data points is a very very low number of points. I am quite surprised that the algo managed to learn something meaningful (well unless you have very obvious relationships between the features and the target). I'll suggest you look at what the logistic regression learned exactly (what coefficients are deemed significant). There is an important risk that you are overfitting on your very very small dataset. And you can't really catch that so your are just left with checking if the rationale of your model make sense.

Yes, you should probably tell your boss that this is not enough data points.

Answer 2

Like @lcrmorin said, this is not a lot of data points. For some problems and algorithms, it is enough data. Some problems are easy to predict and some are hard. Some algorithms (neural nets, gbm) are data hungry and may not do well with this small amount of data. That is the start.

Second, whatever metric(s) you are using - you mentioned accuracy but is that the metric - may not be able to be improved. Sometimes, no matter how much data, not matter what tweaks to algorithms, the best accuracy, log-loss, auroc, etc you get is the best you can get. All models cannot be continuously improved. If the metric that matches the business need does not meet the business requirements, then need to choose a different way. Perhaps rules based can do better.

If this is all the data you can get and you make it known that this may not be enough, next is to optimize the model usage the best you can. For example, if there is a cutoff value, choose wisely. Perhaps choose multiple cut-off values.