2
$\begingroup$

I recently posted another question and this question is the evolution of that one.

By the way I will resume all the problem below, like if the previous question didn't ever exist.

Problem description

I'm doing Signal Modulation Classification using a Convolutional Neural Network and I want to improve performance.

Data

Dataset is composed by 220.000 rows like these. Data is perfectly balanced: I have 20.000 datapoints for each label.

Dataset column Type Range Form Notes
Signal i=real, q=real [i_0, i_1, ..., i_n], [q_0, q_1, ..., q_n] n=127
SNR s=integer [-18, 20] s
Label l=string l They are 11 labels

Lower is the SNR value, and noisier is the signal: classify low SNR signals is not that easy.

Neural Network

Neural Network is a Convolutional Neural Network coded as below:

DROPOUT_RATE = 0.5

iq_in = keras.Input(shape=in_shp, name="IQ")
reshape = Reshape(in_shp + [1])(iq_in)
batch_normalization = BatchNormalization()(reshape)

conv_1 = Convolution2D(16, 4, padding="same", activation="relu")(batch_normalization)
max_pool = MaxPooling2D(padding='same')(conv_1)
batch_normalization_2 = BatchNormalization()(max_pool)
fc1 = Dense(256, activation="relu")(batch_normalization_2)
conv_2 = Convolution2D(32, 2, padding="same", activation="relu")(fc1)
batch_normalization_3 = BatchNormalization()(conv_2)
max_pool_2 = MaxPooling2D(padding='same')(batch_normalization_3)

out_flatten = Flatten()(max_pool_2)
dr = Dropout(DROPOUT_RATE)(out_flatten)
fc2 = Dense(256, activation="relu")(dr)
batch_normalization_4 = BatchNormalization()(fc2)
fc3 = Dense(128, activation="relu")(batch_normalization_4)
output = Dense(11, name="output", activation="softmax")(fc3)

model = keras.Model(inputs=[iq_in], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam')

model.summary()

NeuralNetwork

Training

Training is being done splitting the data in 75% as Training set, 25% as Test set.

NB_EPOCH = 100     # number of epochs to train on
BATCH_SIZE = 1024  # training batch size

filepath = NEURAL_NETWORK_FILENAME

history = model.fit(
    X_train,
    Y_train,
    batch_size=BATCH_SIZE,
    epochs=NB_EPOCH,
    validation_data=(X_test, Y_test),
    callbacks = [
        keras.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=True, mode='auto'),
        keras.callbacks.EarlyStopping(monitor='val_loss', patience=5, verbose=0, mode='auto')
    ])

# we re-load the best weights once training is finished
model.load_weights(filepath)

Results

ConfusionMatrix

My evaluation system evaluate how accurate is my Neural Network for classifying signals with different SNR.

Accuracy_2_SNR

What did I try?

Thisis a list of things that I tried and I'm sure that are modifying performances in worse:

  • Reducing batch size (only increases training time without improving test accuracy)
  • Training without too noisy signals (lowers accuracy)
  • Moving the Dropout layer before the Flatten layer

Questions

Any suggestion to get better performances?

Thanks in advance!

$\endgroup$
2
  • $\begingroup$ experiment with various architectures of CNN, eg using hyperparameters search. But in any case one should not expect very high accuracy as SNR tends low $\endgroup$
    – Nikos M.
    Sep 21 at 11:32
  • $\begingroup$ Yes, in fact I'm just considering to increase accuracy for classifying signals with positive SNR. Thanks for the hint of using Hyperparameters Search, that could be useful! $\endgroup$ Sep 21 at 13:30
2
$\begingroup$

There is relatively little data for a deep learning solution - 220 total data points and 20 data points for each of the 11 labels.

Increasing the amount of data would probably have the greatest impact on model performance. The best option would be to collect more data. Another option would be data augmentation.

$\endgroup$
2
  • $\begingroup$ I have a given dataset for this problem, and I can't get more datapoints unluckily $\endgroup$ Sep 23 at 17:59
  • $\begingroup$ I tested with data augmentation. According to this paper, I tried to expand the dataset by flipping, rotating and adding gaussian noise to signals. I reached 440k datapoints, and I used only the original ones for testing. There wasn't a massive increase of performances in terms of accuracy. I just had a 1-2% of accuracy increase. Thanks anyway, that was a really good hint! $\endgroup$ Oct 1 at 8:48
1
$\begingroup$

Since you have less data, please provide more data for training as try split like 80% 20%.
If training accuracy is 100% then try increasing the dropout percentage. If training accuracy is still less than 100% then try decreasing the dropout percentage and add more convolution layer. Thanks

$\endgroup$
1
  • $\begingroup$ Splitting Training and Test set using 80/20 worked to get a little performance increase. Thanks $\endgroup$ Oct 3 at 7:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.