Problem description

I'm doing Signal Modulation Classification using a Convolutional Neural Network, but performances are very low (around 15% accuracy) and I can't find out why.

Data

Dataset is composed by 220.000 rows like these. Data is perfectly balanced: I have 20.000 datapoints for each label.

Dataset column Type Range Form Notes
Signal i=real, q=real [i_0, i_1, ..., i_n], [q_0, q_1, ..., q_n] n=127
SNR s=integer [-18, 20] s
Label l=string l They are 11 labels

Neural Network

Neural Network is a Convolutional composed sequentially by 2 convolutional layers, and 3 fully connected.

DROPOUT_RATE = 0.5

iq_in = keras.Input(shape=in_shp, name="IQ")

conv_1 = Convolution1D(128, 7, input_shape=(1, 2, 128), padding="same", activation="relu")(iq_in)
dr_1 = Dropout(DROPOUT_RATE)(conv_1)
conv_2 = Convolution1D(128, 5, padding="same", activation="relu")(dr_1)

fc1 = Dense(256, name="fc1")(max_pool)
dr_2 = Dropout(DROPOUT_RATE)(fc1)
fc2 = Dense(128, name="fc2")(dr_2)
out_flatten = Flatten()(fc2)
output = Dense(11, name="output")(out_flatten)

model = keras.Model(inputs=[iq_in], outputs=[output])

model.summary()


Training

Training is being done splitting the data in 70% as Training set, 30% as Test set.

NB_EPOCH = 100     # number of epochs to train on
BATCH_SIZE = 1024  # training batch size

filepath = NEURAL_NETWORK_FILENAME

history = model.fit(
X_train,
Y_train,
batch_size=BATCH_SIZE,
epochs=NB_EPOCH,
validation_data=(X_test, Y_test),
callbacks = [
keras.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=True, mode='auto'),
keras.callbacks.EarlyStopping(monitor='val_loss', patience=5, verbose=0, mode='auto')
])

# we re-load the best weights once training is finished


Results

This is the confusion matrix outputted by my evaluation system.

Question

How to improve performance? Can someone criticize my Neural Network?

Thanks.

There are a couple of things I would suggest:

1. Reshape the input data: It looks to me that you want to analyse a time series if IQ-values and each time series is 128 datapoints. In this case you probably want to treat I and Q as the channels respectively and convolve over ther 128 points. To do this the input data needs to be of shape (128, 2). Right now you treat your data points as 128-dimensional vectors and convolve over the two channels I and Q.

2. Flatten between convolutional and dense part: Usually you have the Convolution1D and MaxPooling1D layers extract some spatial features. The fully connected layers have no notion of spatial properties, they just "understand" vectors. So most models have the Flatten layer just before the Dense layers.

3. Activation functions of the dense layers: Unless an activation is specified in the Dense layers, they use a linear activation, which does not really harvest the expressive power of the layer (see here). So fc1 and fc2 should probably get an activation="relu" as well.

4. Output activation: I would also use a softmax activation for the output, otherwise you can not interpret the output as probabilities for class membership.

With those points fixed I would expect your model to work, at least in principle. A few more points you might want to check:

1. Number of filters: Right now you use 128 filters for all the Convolution1D layers. That seems like a lot, I would start with maybe 16 and see how far you get. You can increase the number later.

2. Filter size: That is just an intuition on my part, but you might try smaller filters. In computer vision filter sizes of 7 and 5 respectively would seem oddly large. But since you probably understand the data better, you might have your reasons.

3. Not sure if this will help, but if you want to reduce the number of trainable parameters you could also try and insert another MaxPooling1D layer between the two convolutions. Usually one tries to compound spatial information with e.g. pooling while learning more features, i.e. increase the number of filters.

Try a model e.g. like this one (untested code, this is just as an example):

DROPOUT_RATE = 0.5

in_shp = (128, 2)
iq_in = keras.Input(shape=in_shp, name="IQ")

conv_1 = Convolution1D(16, 7, padding="same", activation="relu")(iq_in)
dr_1 = Dropout(DROPOUT_RATE)(conv_1)
conv_2 = Convolution1D(16, 5, padding="same", activation="relu")(dr_1)

out_flatten = Flatten()(max_pool)

fc1 = Dense(256, name="fc1", activation="relu")(out_flatten)
dr_2 = Dropout(DROPOUT_RATE)(fc1)
fc2 = Dense(128, name="fc2", activation="relu")(dr_2)
output = Dense(11, name="output", activation="softmax")(fc2)

model = keras.Model(inputs=[iq_in], outputs=[output])

model.summary()


With the following output:

Layer (type)                 Output Shape              Param #
=================================================================
IQ (InputLayer)              [(None, 128, 2)]          0
_________________________________________________________________
conv1d (Conv1D)              (None, 128, 16)           240
_________________________________________________________________
dropout (Dropout)            (None, 128, 16)           0
_________________________________________________________________
conv1d_1 (Conv1D)            (None, 128, 16)           1296
_________________________________________________________________
max_pooling1d (MaxPooling1D) (None, 64, 16)            0
_________________________________________________________________
flatten (Flatten)            (None, 1024)              0
_________________________________________________________________
fc1 (Dense)                  (None, 256)               262400
_________________________________________________________________
dropout_1 (Dropout)          (None, 256)               0
_________________________________________________________________
fc2 (Dense)                  (None, 128)               32896
_________________________________________________________________
output (Dense)               (None, 11)                1419
=================================================================
Total params: 298,251
Trainable params: 298,251
Non-trainable params: 0
_________________________________________________________________

• Your suggestions significantly improved performances, thank you so much for the detailed answer. There's just a thing that I don't understand (and I couldn't use): why changing the input shape? Input is composed by two vectors composed by 128 elements each one, so input shape should be '(2, 128)', am I wrong? Sep 1 '21 at 6:30
• It was more a guess, but I thought the signal is made up of two values, I and Q, and you have measurments at 128 points in time. In that case for the convolution and pooling to make sense, you have to use the shape (128, 2). Sep 1 '21 at 10:50
• Signal is composed by 2 train of 128 decimal values, so (2, 128) is the right shape. Thank you again, you was helpful. Sep 1 '21 at 11:02