From my understanding, support vector machines run on the premise of minimizing some error function, usually with the goal of maximizing accuracy overall. However, there are a lot of contexts, particularly in the social sciences, where people don't aim for 99%+ accuracy, but instead aim for 95% accuracy (e.g., 95% CI, p-values, etc). I'm wondering if there is a way to train SVMs with the goal of developing the best model that achieves 95% accuracy. Obviously, depending on the context, 95% accuracy might be out of reach and researchers are just aiming for the best predictive models they can get; but when an SVM can train on some dataset and repeatedly get above 95% accuracy, I'm wondering if there is a way to trade off some of that accuracy for generalizability, namely by say reducing the complexity of the decision boundary (e.g., fewer folds) or by maximizing another function, say achieving 95% accuracy on a medical test while also trying to maximize the number of true positives.
-
$\begingroup$ I am afraid you sound confused; the target of the highest accuracy possible has absolutely nothing to do with 95% CI, neither it is typical to aim for 99% accuracy, as you claim. You question is based upon wrong premises. $\endgroup$– desertnautOct 23, 2022 at 23:41
1 Answer
There is no deterministic way to trade accuracy for generalizability.
Integral part of machine learning is splitting your available data into a training set and test set (or even an intermediate validation set, depending on the amount of available data). With that said you train the model on the training set and then apply the model to the 'unknown' test set. In case it performs (much) worse on the test set your model likely overfits the training data, so you would generally try to achieve similar results on the test set as on the training set. By repeating this process of model building you achieve the 'best possible model' while guaranteeing generalizability on data that the model hasn't seen before. Of course there's more to it, but this would be the general approach to balance accuracy and generalization.
As it seems to be a classification problem you are trying to solve, you can also optimize your model in regard to precision/recall or sensitivity and specificity, respectively (ROC curve etc. might help here). Quantifying those metrics would be part of model evaluation on the path to an optimal model outcome.
