I am working on hybrid CNN-SVM for classification task, where I aim to use CNN for feature extraction and SVM for classification. So after the training of my CNN model as below:

import os
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import keras
from keras.layers import Dense, Conv2D, InputLayer, Flatten, MaxPool2D
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

print('Training data: {}, {}'.format(x_train.shape, y_train.shape))
print('Test data: {}, {}'.format(x_test.shape, y_test.shape))
x_train, x_test = x_train / 255.0, x_test / 255.0
x_train = np.expand_dims(x_train, axis=3)
x_test = np.expand_dims(x_test, axis=3)
model = keras.models.Sequential([
    InputLayer(input_shape=(28, 28, 1), name='input_data'),
    Conv2D(32, 3, activation='relu'),
    Conv2D(32, 3, activation='relu'),
    Conv2D(32, 3, activation='relu'),
    Dense(128, activation='relu'),
    Dense(10, activation='softmax', name='output_logits')

I used the following code to extract the last features from the last fully connected layer before softmax to be used as inputs for SVM classifier:

getFeature = K.function([model.layers[0].input, K.learning_phase()],
exTrain = getFeature([x_train[0:5000], 0])[0]
exTest = getFeature([x_train[5000:10000], 0])[0]
y_train = y_train[0:5000].reshape(y_train[0:5000].shape[0],)             
y_test = y_train[5000:10000] 

And then I train the SVM classifier using these datatset:

from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

parameters = {'kernel':['rbf'], 
              'C':[1, 10, 100, 1000],
              'gamma':[1e-3, 1e-4]}
clf = GridSearchCV(SVC(), parameters)
clf.fit(exTrain, y_train)
svmclf = clf.best_estimator_
svmclf.fit(exTrain, y_train)

I have the following questions:

1- Are these steps correct?

2- I have read that the shape of the The last layer should be (1,10) (for mnist classification where there are 10 classes); The last layer should have the same number of nodes as the number of classes we wish to predict for. In my case the shape of dense layer which is (1,128) which is the sahpe of exTrain the input of SVM classifier. Am confused so any clarifications please?



1 Answer 1


SVM is NOT needed at end.

Last dense layer of CNN network should be dense layer with 10 neurons as there can be 10 logits outputs corresponding to 10 MNIST classes.

Then apply a direct Softmax activation at end of above CNN dense layer with categorical cross entropy loss minimization to predict 10 probabilities for 10 possible digit classes.

Many many code examples of this online by googling it. You can also find jumpstart projects on MNIST classification in DL and Vision on popular code repositories like Github and others.

  • $\begingroup$ Actually I have found many papers using this approach of hybrid CNN-SVM for classification. My confusion is about the shape of the layer before softmax. For instance, this work github.com/shibuiwilliam/Keras_Sklearn/blob/master/… where he used CNN for CIFAR -10 ,the shape of this layer was 1024 $\endgroup$
    – root
    Feb 2, 2022 at 7:37
  • $\begingroup$ No. Even that notebook adds a DENSE layer of no of classes (i.e. 10) and then applies to Softmax on it. $\endgroup$ Feb 3, 2022 at 8:21
  • $\begingroup$ Check softmax line classificationLayer=[Dense(num_classes), Activation('softmax')] - Dense layer with num_classes is being added and then Softmax applied on it. The 1024 unit Dense layer is PENULTIMATE layer and not the final layer. In that model SVM has NOT been applied. Directly software is outputting 10 probabalities on last Dense layer also called LOGITS layer. Hope this helps. $\endgroup$ Feb 3, 2022 at 8:22
  • $\begingroup$ as to SVM, Random Forest, KNN Classifier, these SKLEARN algos are being tried in that notebook as ALTERNATIVE techniques and confusion matrix being printed for each. These are competing standalone techniques not to be combined with Keras Neural network normally. $\endgroup$ Feb 3, 2022 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.