Keras: How to connect a CNN model with a decision tree

Question

I want to train a model to predict one's emotion from the physical signals. I have a physical signal and using it as input feature;

ecg(Electrocardiography)

I want to use the CNN architecture to extract features from the data, and then use these extracted features to feed a classical "Decision Tree Classifier". Below, you can see my CNN aproach without the decision tree;

model = Sequential()
model.add(Conv1D(15,60,padding='valid', activation='relu',input_shape=(18000,1), strides = 1,  kernel_regularizer=regularizers.l1_l2(l1=0.1, l2=0.1)))
model.add(MaxPooling1D(2,data_format='channels_last'))
model.add(Dropout(0.6))
model.add(BatchNormalization())
model.add(Conv1D(30, 60, padding='valid', activation='relu',kernel_regularizer = regularizers.l1_l2(l1=0.1, l2=0.1), strides=1))
model.add(MaxPooling1D(4,data_format='channels_last'))
model.add(Dropout(0.6))
model.add(BatchNormalization())
model.add(Flatten())
model.add(Dense(3, activation = 'softmax'))

I want to edit this code so that, in the output layer there will be working decision tree instead of model.add(Dense(3, activation = 'softmax')). I have tried to save the outputs of the last convolutional layer like this;

output = model.layers[-6].output

And when I printed out the output variable, result was this;

THE OUTPUT: Tensor("conv1d_56/Relu:0", shape=(?, 8971, 30), dtype=float32)

I guess, the output variable holds the extracted features. Now, how can I feed my decision tree classifier model with this data which is stored in the output variable? Here is the decision tree from scikit learn;

from sklearn.tree import DecisionTreeClassifier

dtc = DecisionTreeClassifier(criterion = 'entropy')
dtc.fit()

How should I feed the fit() method? Thanks in advance.

Answer 1

You can make the feature extraction to an intermediate model in keras. It could look something like this:

# Build a model for only the feature extraction layers
feature_extractor = Sequential()
feature_extractor.add(Conv1D(15,60,padding='valid', activation='relu',input_shape=(18000,1), strides = 1,  kernel_regularizer=regularizers.l1_l2(l1=0.1, l2=0.1)))
feature_extractor.add(MaxPooling1D(2,data_format='channels_last'))
feature_extractor.add(Dropout(0.6))
feature_extractor.add(BatchNormalization())
feature_extractor.add(Conv1D(30, 60, padding='valid', activation='relu',kernel_regularizer = regularizers.l1_l2(l1=0.1, l2=0.1), strides=1))

# Keep adding new layers for prediciton outside of feature extraction model
x = feature_extractor.output
x = MaxPooling1D(4,data_format='channels_last')(x)
x = Dropout(0.6)(x)
x = BatchNormalization()(x)
x = Flatten()(x)
prediction_layer = Dense(3, activation = 'softmax')(x)

# Make a new model combining both
cnn_model=Model(inputs=feature_extractor.input, outputs=prediction_layer)
cnn_model.compile(optimizer=opt,loss=loss)

Then you train using the full cnn_model but you call predict only on your feature extraction model with feature_extractor.predict(X). Then you can use the output of the prediction to train your decision tree like this:

# Train full network, both feature extractor and softmax part
cnn_model.fit(X, y_one_hot) # y needs to be one hot for keras

# Predict only the output of the feature extraction model
X_ext = feature_extractor.predict(X)

dtc = DecisionTreeClassifier(criterion = 'entropy')

# Train the decision tree on the extracted features
dtc.fit(X_ext, y) # y should be one-dimensional for sklearn