1
$\begingroup$

I have trained a CNN on one dimensional data that is the power spectral density (PSD) of a $N$ different classes of signals ($N=4$). Each of the $N$ signals has a different spectral shape (not shown here). For illustrative purposes, the plots shown below are from the same signal class. The idea is to treat this as an image classification problem.

The model performs exceptionally well when the training data has all the examples centered around the same frequency (i.e., within a few hundred Hertz): All training examples centered on or around 0 Hz

The CNN fails to properly classify new examples that are outside the frequency range of the training data:

Invalid classification, new data outside frequency range of trained data

Model Details and Assumptions:

The CNN model is implemented in PyTorch using the following layers:

    model = nn.Sequential(
            nn.Conv1d(in_channels=1, out_channels=64, kernel_size=128, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=16, stride=2),
            nn.Dropout(p=0.25),
            nn.Flatten(),
            nn.Linear(257664, n_classes)
    )

The optimizer is torch.optim.Adam. Batch size has been varied from 8 to 128, epochs varied from 10 to 50. The input data is normalized to $[0,1]$. Training examples varied from [2000, 8000], where 20% are used for validation and 20% are used for test. I have also tried adding additional convolutional layers, varying the kernal sizes, neurons, layers, etc.

Questions:

  1. Shouldn't the CNN model generalize such that new examples that are not within the same frequency range (i.e., centered around the middle) should be identified as the correct class?
  2. Are there other steps I need to take, whether in the model or training data?
$\endgroup$

2 Answers 2

3
+50
$\begingroup$

Since the two spectra you show belong to the same class, it sounds like you want your model to be translation invariant.

Convolutions are translation equivariant, and so can get you part of the way there. So you might expect your model to generalize to new spectra like the one you showed if the model is purely convolutional, but your model is not purely convolutional.

Your model is ( conv - maxpool - linear ), and linear layers are neither translation invariant nor equivariant, so there isn't a great reason to expect it to generalize in the way you showed.

Two things you can try:

  1. Make your model fully (or more) convolutional by making it deeper, e.g. something like (conv - maxpool - conv - maxpool - conv - maxpool - conv - maxpool - linear )
  2. Use data augmentation to try to learn the kinds of transformations you want your model to be invariant to

Edit: As @Dan commented, global max pooling will remove dependence on position and so it's output will be translation invariant.

$\endgroup$
4
  • $\begingroup$ I am trying this but the model is not performing well on the training data. Do you have any suggestions for the number of output channels or kernal sizes if the input data is 1x8192? $\endgroup$ Commented Jan 9, 2023 at 1:43
  • $\begingroup$ Do the Relu layers need to be dropped? $\endgroup$ Commented Jan 9, 2023 at 1:47
  • $\begingroup$ I also recommend using a global maxpool layer to further train the network to be translationally invariant $\endgroup$
    – Dan
    Commented Jan 10, 2023 at 6:08
  • $\begingroup$ Dan is correct. Global pooling was critical in getting translation invariance if the training data did not contain off center examples. @bogovicj - can you update your answer to reflect that? $\endgroup$ Commented Jan 17, 2023 at 12:43
1
$\begingroup$

Thanks for @bogovicj for his answer, it was helpful in getting towards the right path.

  1. Improve model architecture. Larger strides for the large kernal sizes for this particular case.
  2. Include global pooling at final layer
  3. More conv layers (use 3-4) with RELU and max pooling if model does not perform well before offsets are introduced
  4. For best performance, train with data that includes off centered offsets, but not necessarily required if model is properly designed

See answer posted here and the comments for additional details:

https://ai.stackexchange.com/questions/38649/should-a-cnn-generalize-to-arbitrary-positions-in-the-data

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.