I have a biological unbalanced dataset on which I have applied deep learning, Support Vector Machine (all the kernel functions) and Artificial Neural network for multiclass classification (size: 139 samples , 5 attributes) in python. Unfortunately the accuracy is not exceeding 55%. What can be done to increase the accuracy? If a dataset cannot ever go beyond such an average accuracy, what is the solution?
A shallow neural network is the wrong approach for a problem with a small training set. Deep learning is even worse for small training sets. 139 samples is severely insufficient to train any deep learning model, or even a shallow neural network.
As a very general rule of thumb I use 100 examples for each feature in my dataset for deep learning. This then increases exponentially with every single different class you expect.
I suggest you use a machine learning technique such as SVM. This will likely result in better results given the size of your dataset. Try these techniques instead and see what results you get: k-NN, kernel SVM, k-means clustering.
If you have an unbalanced dataset then you would want to use anomaly detection algorithms which can be trained on a single distribution. You can learn the distribution of each output class you want. From there, novel examples can be classified based on the likelihood they fit within a given distribution.
There are many possible explanations. Maybe just mistakes in the code or maybe there is just no better separation possible. When usign SVM the features are extremely important and you should most of your time invest in choosing/designing them. The statement "I have applied Deep learning" ist just to vague and therefore I can not give any tip. Third when dealing with unbalanced data you have many possible actions. I recommend reading: https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis