What model is suitable for classification of a small data set?

Question

I have a dataset that consists of 365 records, and I want to apply a classification model on it (binary classification).

As an output, in addition to the classification labels, I want to retrieve the classification confidence for each instance.

I don't know how to deal with such a case. Can I use, for example, linear classifiers (SVM, logistic regression) with this small dataset? Because, I want to retrieve the classification confidence as well.

I read that decision trees can be a good classifier for small datasets, but how can I retrieve the classification confidence with it?

The dataset consists of tweets, each classified as positive or negative (from a sentiment perspective), and my feature vector consists of 2400 features (combination between word2vec embeddings and other features).

Also, do you recommend me to use word2vec embeddings with such a small dataset? I think the classifier can't learn something from them using small dataset.

Answer 1

The question whether to use a linear classifier depends less on the number of samples you have in your dataset and more whether your dataset is linearly separable (by the way, SVMs can be non-linear with the kernel trick).

Now with regards to confidence in the classification, In SVMs there is a method that calculates the probability that a given sample belongs to a particular class using Platt scaling ("Original Paper"). This is the approach that is used in sklearn's SVM confidence implementation. You can read more about it in the following link:

How To Compute Confidence Measure For SVM Classifiers

In both SVMs and linear regression models you can calculate the distance of a sample from the border and treat it as a confidence measurement (but it is not exactly that).

With decision trees I'm not an expert but a similar question was posted and answered in the following link:

Decision tree, how to understand or calculate the probability/confidence of prediction result

I would strongly recommend using some known embedding method like the word2vec, since as you mentioned, your dataset is too small for your model to be able to properly learn an encoding of context and vocabulary from.

Answer 2

The only way to know if a classifier is suitable for your data set is to try it and test it. All classifiers you've mentioned have a way to give confidences of their predictions. Logistic regression and decision trees will give you the probability that a sample is the positive class. SVM's will give you the distance to the decision hyperplane which can be used as a confidence measure; with some additional computations, you can also get a probability with SVM's, but I won't detail that here.

One concern for such a small data set is overfitting when the number of features is much larger than the number of samples. You should address that by using some form of regularization.