4
$\begingroup$

I am dealing with a dataset that has a total of 81 records. Out of them I divided it to 54 records for the training and the rest for testing. I noticed a few things:

  • Accuracy is low for given test data set (~ 30%).
  • Accuracy remains same no matter whether I use SVM or Naive Bayes.
  • If I test against 2-3 values accuracy shoots up to 75%.

Is this normal or I am doing something wrong?

$\endgroup$
6
$\begingroup$

It is completely normal in some circumstances. If you consider the learning problem from a statistical perspective, learning is done by trying to estimate the conditional estimate of your output variable given input variables. When you are doing learning you basically have to main components: sample data from training and the learning algorithm, and you use both to build an estimation of the aforementioned conditional probability. Now consider the relation of each component with the desired estimated probability.

Training sample

Training data is only a sample from all the possible data. You do not have all the possible data since if you would have it you would not need to do learning, right? So a sample is just a fraction of the data. This sample could cover more or less the whole domain of the possible data. An immediate thought is that a bigger sample is better than a smaller one. As an extreme case to illustrate the idea is to have only one observation. But the size of the sample is not the whole story. It is possible that the probability you want to estimate is very complex. The complexity of that space requires also more data than a simple probability space. To give an example consider the problem of learning the mean of the heights of males from USA. Having a small random sample often would be enough if you assume a normal distribution. One the other hand take the example of learning to recognize speech. Since pronouncing is not the same against individuals you would need a pretty big sample to unveil the useful relation needed to predict and throw away things which are not useful in the data.

Model/learning algorithm

Not any model is appropriate to express any true model of the signal you learn. For example if your true signal resembles a second degree polynomial would be impossible to fit properly with a linear model, other than some particular cases. So there is a complexity in the model too, which could be compared with the complexity of the conditional probability you want to estimate during learning. Even if you know the true model and you learning model is able to describe it, sometimes you need more data to make the model able to fit. As another example consider a random forest which estimates a simple line. Random forest is very flexible since the underlying statistical model is basically a constant given a local region in the input space. Considering that, even if the model would be capable of estimating the true model, since the model is a local approximation, you need plenty of data in every useful region of the space, to learn a good approximation.

Conclusion

We considered the individual component comparisons with probability you want to estimate. But you have both. This makes even harder to understand what is happening behind the walls of learning. Thus there are no universal ways of telling when data is enough for a given model. So this is why is often the case that one algorithm can have different accuracies for different training set size and for different algorithms the required training set size to have approximate equal accuracies.

One thing I tried often in practice is given a learning model to check how accuracy evolves as the learning sample size increase. If your training sample is large enough you have typically a region in a small sample size where error is large due to not enough data. After that you would have a region where errors stays approximately the same and this perhaps is where your model do it's best. Also is possible for even larger sample sizes to have a decrease in performance for some models, maybe do to inability of the model to describe what is there or due to overfitting or some other cases.

You can trace this kind of behavior using some sort of bootstrap validation, where the sample used for training is taken repeatedly for increasing lengths. Something like 20 samples with 5% size, 20 samples with 10% size, and so on. This kind of information can give you an idea between the relation of the sample size with your learning problem for a given algorithm.

| improve this answer | |
$\endgroup$
4
$\begingroup$

Your training and test errors are affected by the size of the training. Take a look to this plot, usually known as a learning curve:

enter image description here

In this example, we compute the training score and the test score (cross validation score) of a Naive Bayes model as we increase the number of examples in the training dataset. The higher the score is, the better the model is performing.

A larger training set decreases the score because it is more difficult for the learning algorithm to learn a model that correctly represents all the training data. However, as we increase the size of the training set, the test score also increases, due to an increase in the model's ability to generalise.

As you can see, both lines in the plot reach a limit on the right size. This means that in order to learn properly, an algorithm requires enough data, just enough for us to get to the right side of this plot. Once we reach that asymptote, we can not improve the test score by using more training data.

Now, notice that this plot only refers to the size of the training set, and not to the size of the test set, but inferring from it, I may assume that when you only use 2-3 examples in your test set you don't have enough test data to determine your test score. You may even get completely different results for other 2-3 different test examples. Therefore, I would assume that the 30% test error of your larger test set is a most feasible value.

It seems that, in order to increase the performance of your model, you will have to look for other reasons: are your features well defined? Do you have enough training data?

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.