# Optimizing CNN network

I am currently trying to recreate the result of this paper, in which they do feature extraction from a "spectogram" of log-melfilter energies..

Since the paper doesn't state what kind of feature I am seeking, i am currently trying to extract features, and match them to MFCC features. The paper states a technique called LWS (Limited weight sharing) in which the spectogram frequency axis will be divided into section, and each section don't share their weight with others.

So i've divided the my input image into 13 section to receive 1 output features from a (6,3,3) input image. 6 for the number of rows, 3 as each column represent [static delta delta_delta] data of the given log melfilter energi, and the last 3 is the color channels.

If i'd used 13 filterbanks, and made the plot, will the result of this be that each (1,3,3) matrix would result in one feature, but that seemed a bit too good to be true, so i decided to use 78 filterbanks and divide it into 13 section which should result in one feature can be extracted from a matrix of size (6,3,3)

I am training the network with this model structure:

def create_model(init_mode='normal',activation_mode='softsign',optimizer_mode="Adamax", activation_mode_conv = 'softsign'):
model = Sequential()

print model.output_shape
print model.output_shape
print model.output_shape
print model.output_shape
print model.output_shape
print model.output_shape
#print model.summary()
model.compile(loss='mean_squared_error',optimizer=optimizer_mode)

return model


This model keeps for some reason providing me very bad results.. I seem to keep getting an loss of 216, which is nearly 3 times the data range...

I did a grid seach to find out which parameter (activation function, init_mode, epochs and batch_size would be best, which are those chosen in the function above (eventhough there wasn't much change in the outcome..)

What can i do to get better results? Is the CNN network poorly designed?

• I'd prefer to leave this as a comment, because this isn't a direct answer to your question re. optimization, but SE-DataScience says I need more rep to leave a comment, so.... You might gain some insight into how to approach your problem by looking at the compact_cnn code & paper(s) by @keunwoochoi: github.com/keunwoochoi/music-auto_tagging-keras In particular, adding Batch Normalization layers should help, and replacing some of your dense layers with convolutional layers may be a useful approach. – sh37211 Mar 17 '17 at 13:48

A suggestion I give is to change the layers of CNN. You have 3 CNN layers, all sequentially added acting on the same shape and outputting the same output shape with same filter size.

What you can try is to change the filter size and using multiple filter sizes to catch different features of different sizes. For this, try this model:

main_input = Input(shape=input_shape, name="main input")
flattened_outputs = []
for i in filter_sizes:
conv_filter_i = Convolution1D(no_of_filters, i, border_mode='same', activation='relu', W_constraint=maxnorm(3))(main_input)
pooling_i = MaxPooling1D(pool_length=2)(conv_filter_i)
flattened_i = Flatten()(pooling_i)
flattened_outputs.append(flattened_i)
merged_conv_outputs = merge(flattened_outputs, mode="concat")
softmax = Dense(output_shape, activation="softmax")(merged_conv_outputs)
model = Model(input=main_input, output=softmax)
model.compile(loss='mean_squared_error',optimizer=optimizer_mode)


Note: change the dimensions as necessary.

1. Apart from this, I suggest you to use Dropout layer.
2. It helps a lot according to my personal experience. Also, do use adam optimizer.
3. Experiment with various pooling mechanisms. Remember that there is a lot of experimentation that has to go on, so try various other parameters and filter sizees too.
• what is dropout_i? – Carlton Banks Mar 23 '17 at 7:17
• Oh that, I actually had a dropout layer before. I removed it but you can try that too. Let me just edit it. – Hima Varsha Mar 23 '17 at 7:35
• @CarltonBanks - My understanding of Dropout is that it is another form of regularization. It involves randomly squashing half of the activation values. About half of the values are converted to zero. – Nanda Mar 24 '17 at 19:26

There are some suggestions which I think can improve CNN performance:

1. Input size is (6, 3) so it is not preferable to use MaxPooling layers.
2. Use padding = 'same' in all the convolution layers which output convolved image of the same dimension as input.
3. Use Relu or LeakyRelu as activation function.
4. Use Adam optimizer and tune the learning rate.
5. Convert some of the Dense layers to 1x1 Convolution with no. of filters = no. of units in dense layer as suggested by sh37211

e.g.

no_of_filters = [32, 64, 64, 32]
kernel_size = [3, 3, 3, 1]

input = Input(shape = (6,3,3), name = "input")
layer_output = [input]
for i in range(4):
convolution = Convolution2D(no_of_filters[i], kernel_size[i], kernel_size[i], padding = 'same')(layer_output[-1])
activation = LeakyReLU()(convolution)
layer_output.append(activation)

flatten = Flatten()(layer_output[-1])
flatten_dropout = Dropout(0.5)(flatten)
fc = Dense(output_dim = 13)
activation = LeakyReLU()(fc)
layer_output.append(activation)
fc = Dense(output_dim = 1, activation = 'tanh')


• @CarltonBanks Sorry for late reply. There was very small typo in example architecture. I ran for loop for only 3 iterations which left last 1x1 convolution. – Bhagyesh Vikani Mar 24 '17 at 11:21