I am comparing the classification accuracy between Naive Bayes (NBC), SVM and a Neural Network. I am using a Dataset of ~18K and 26 Labels.
In the current state the Neural Network get always an accuracy of >80%, but the NBC and SVM fluctuate between 15% and 80%. They mostly end up near one of the two extrema. The only difference for each run is the splitting of the data in Training/Testing with the model_selection.train_test_split() function of sklearn
For the implementation of the classifiers I am also using the classes and functions of sklearn.
I highly suspect the problem in my data but I am already doing the basic preprocessing with stop words, lowercasing, etc.