If you have 4,500 examples of each category, you’re doing better than random guessing. This sounds like a hard problem where the examples will have only subtle differences, so that’s an accomplishment. (I assume you’re doing some kind of out-of-sample testing.) Consider how you do in classifying perhaps 20 examples of each. I would be curious to hear how you do. This is quite a different problem if you get 38/40 than 20/40.
The way to apply a CNN, though, would be to convert to a spectrogram and then run the CNN on the spectrogram array. Converting to a literal picture is unnecessary and might harm performance. There are many ways to convert your signal to two dimensions of time-frequency space. You’ve probably tried a Fourier transform. Check out wavelets.
Since you have time series data, consider recurrent neural nets and long short-term memory. These can be combined with CNN. You will find examples of CNN/LSTM code on GitHub, I’d imagine.
You have a fairly small data set, however, for deep learning approaches. Consider simpler models like logistic regression. Your in-sample performance may suffer while improving out-of-sample performance.
Finally, consider a proper scoring rule like Brier score. Frank Harrell has written much about this on the statistics Stack, Cross Validated. Shamelessly, I will mention that you may be interested in a question of mine on CV where I somewhat challenge the idea of proper scoring rules when there is a hard decision to be made: https://stats.stackexchange.com/questions/464636/proper-scoring-rule-when-there-is-a-decision-to-make-e-g-spam-vs-ham-email. My example comes from NLP, but there could be a speech situation (say text dictation, where, at some point, you have to decide to print “dad” or “bad” or “bicycle”).
