Convolutional neural network low performance

Question

Problem description

I'm doing Signal Modulation Classification using a Convolutional Neural Network, but performances are very low (around 15% accuracy) and I can't find out why.

Data

Dataset is composed by 220.000 rows like these. Data is perfectly balanced: I have 20.000 datapoints for each label.

Dataset column	Type	Range	Form	Notes
Signal	i=real, q=real		[i_0, i_1, ..., i_n], [q_0, q_1, ..., q_n]	n=127
SNR	s=integer	[-18, 20]	s
Label	l=string		l	They are 11 labels

Neural Network

Neural Network is a Convolutional composed sequentially by 2 convolutional layers, and 3 fully connected.

DROPOUT_RATE = 0.5

iq_in = keras.Input(shape=in_shp, name="IQ")

conv_1 = Convolution1D(128, 7, input_shape=(1, 2, 128), padding="same", activation="relu")(iq_in)
dr_1 = Dropout(DROPOUT_RATE)(conv_1)
conv_2 = Convolution1D(128, 5, padding="same", activation="relu")(dr_1)
max_pool = MaxPooling1D(padding='same')(conv_2)

fc1 = Dense(256, name="fc1")(max_pool)
dr_2 = Dropout(DROPOUT_RATE)(fc1)
fc2 = Dense(128, name="fc2")(dr_2)
out_flatten = Flatten()(fc2)
output = Dense(11, name="output")(out_flatten)

model = keras.Model(inputs=[iq_in], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam')

model.summary()

Training

Training is being done splitting the data in 70% as Training set, 30% as Test set.

NB_EPOCH = 100     # number of epochs to train on
BATCH_SIZE = 1024  # training batch size

filepath = NEURAL_NETWORK_FILENAME

history = model.fit(
    X_train,
    Y_train,
    batch_size=BATCH_SIZE,
    epochs=NB_EPOCH,
    validation_data=(X_test, Y_test),
    callbacks = [
        keras.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=True, mode='auto'),
        keras.callbacks.EarlyStopping(monitor='val_loss', patience=5, verbose=0, mode='auto')
    ])

# we re-load the best weights once training is finished
model.load_weights(filepath)

Results

This is the confusion matrix outputted by my evaluation system.

Question

How to improve performance? Can someone criticize my Neural Network?

Thanks.

Answer 1

There are a couple of things I would suggest:

1. Reshape the input data: It looks to me that you want to analyse a time series if IQ-values and each time series is 128 datapoints. In this case you probably want to treat I and Q as the channels respectively and convolve over ther 128 points. To do this the input data needs to be of shape (128, 2). Right now you treat your data points as 128-dimensional vectors and convolve over the two channels I and Q.

2. Flatten between convolutional and dense part: Usually you have the Convolution1D and MaxPooling1D layers extract some spatial features. The fully connected layers have no notion of spatial properties, they just "understand" vectors. So most models have the Flatten layer just before the Dense layers.

3. Activation functions of the dense layers: Unless an activation is specified in the Dense layers, they use a linear activation, which does not really harvest the expressive power of the layer (see here). So fc1 and fc2 should probably get an activation="relu" as well.

4. Output activation: I would also use a softmax activation for the output, otherwise you can not interpret the output as probabilities for class membership.

With those points fixed I would expect your model to work, at least in principle. A few more points you might want to check:

5. Number of filters: Right now you use 128 filters for all the Convolution1D layers. That seems like a lot, I would start with maybe 16 and see how far you get. You can increase the number later.

6. Filter size: That is just an intuition on my part, but you might try smaller filters. In computer vision filter sizes of 7 and 5 respectively would seem oddly large. But since you probably understand the data better, you might have your reasons.

7. Not sure if this will help, but if you want to reduce the number of trainable parameters you could also try and insert another MaxPooling1D layer between the two convolutions. Usually one tries to compound spatial information with e.g. pooling while learning more features, i.e. increase the number of filters.

Try a model e.g. like this one (untested code, this is just as an example):

DROPOUT_RATE = 0.5

in_shp = (128, 2)
iq_in = keras.Input(shape=in_shp, name="IQ")

conv_1 = Convolution1D(16, 7, padding="same", activation="relu")(iq_in)
dr_1 = Dropout(DROPOUT_RATE)(conv_1)
conv_2 = Convolution1D(16, 5, padding="same", activation="relu")(dr_1)
max_pool = MaxPooling1D(padding='same')(conv_2)

out_flatten = Flatten()(max_pool)

fc1 = Dense(256, name="fc1", activation="relu")(out_flatten)
dr_2 = Dropout(DROPOUT_RATE)(fc1)
fc2 = Dense(128, name="fc2", activation="relu")(dr_2)
output = Dense(11, name="output", activation="softmax")(fc2)

model = keras.Model(inputs=[iq_in], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam')

model.summary()

With the following output:

Layer (type)                 Output Shape              Param #   
=================================================================
IQ (InputLayer)              [(None, 128, 2)]          0         
_________________________________________________________________
conv1d (Conv1D)              (None, 128, 16)           240       
_________________________________________________________________
dropout (Dropout)            (None, 128, 16)           0         
_________________________________________________________________
conv1d_1 (Conv1D)            (None, 128, 16)           1296      
_________________________________________________________________
max_pooling1d (MaxPooling1D) (None, 64, 16)            0         
_________________________________________________________________
flatten (Flatten)            (None, 1024)              0         
_________________________________________________________________
fc1 (Dense)                  (None, 256)               262400    
_________________________________________________________________
dropout_1 (Dropout)          (None, 256)               0         
_________________________________________________________________
fc2 (Dense)                  (None, 128)               32896     
_________________________________________________________________
output (Dense)               (None, 11)                1419      
=================================================================
Total params: 298,251
Trainable params: 298,251
Non-trainable params: 0
_________________________________________________________________