0
$\begingroup$

I'm really a novice working with these technologies and I'm struggling to design a neural network that is powerful enough to model a spectrogram. For a personal project, I'm working on a spectrogram VAE but the convolutional networks that I've seen online seem very suboptimal at reconstructing these images. To illustrate my point, I'm trained it on a single example for about 100 epochs to see if it was able to overfit as I expected. enter image description here

The results are very underwhelming, and it seems barely able to model even the simple features, like the empty space and straight lines. Now, I'm wondering, is this an issue with my code/network, or are there layers besides convolution that would work better at this type of data structure? On a spectrogram, there is generally high correlation between features along the x axis and along the y axis. What do you think about using more rectangular convolutions along the x and y axis, to try to capture those relationships? Do recurrent layers like GRU work well at capturing these dependencies?

I might be really misunderstanding how image construction tasks should be approached, but it seems to me that the features of this spectrogram should be easy to model with the right network.

Please ask if I should give more details. I'm really just looking for some advice in what direction to continue.

Here's an image of the reconstruction after 40 more epochs (it still hasn't figured out how to draw a vertical line): enter image description here

Here's the code defining my network. The block function is just a conv/transconv layer. The spectrograms are 128x126. enter image description here

Improve this question
$\endgroup$
2
  • $\begingroup$ Can you describe what a block(...) in your network architecture is? $\endgroup$
    – Luca Anzalone
    Commented Jul 21, 2023 at 19:30
  • 1
    $\begingroup$ I said "The block function is just a conv/transconv layer". I put LeakyRelu and Batch Normalization after each one. $\endgroup$
    – BOBONA
    Commented Jul 21, 2023 at 20:02

1 Answer 1

Reset to default
0
$\begingroup$

I think you need a more powerful architecture: try implementing both encoder and decoder as residual networks. In you specific application you may want to also consider:

  • Increasing the receptive field of each convolution, to better capture the correlations along $x$ and $y$ axes, by either increasing the kernel size (at the cost of more computation and parameters) or by using dilated convolutions.
  • If the reconstructions exhibit artifacts, try substitute a transpose conv with (bilinear) upsampling and a regular convolution.
  • Using instance normalization instead of batch normalization.
  • Learning a more complex latent space by modelling it with a mixture of Gaussians or a multivariate Gaussian with triangular covariance (instead of diagonal covariance as usually done.)
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the advice! Should I just make the current layers residual or can I also add additional layers that do not reduce the input size (using stride=1)? Would there be a benefit of doing layers with a mix of kernel sizes, like (16, 4) and (4, 16) or is just making a larger kernel like (16, 16) preferable? I guess I could try both too. Also why prefer instance over batch normalization? $\endgroup$
    – BOBONA
    Commented Jul 23, 2023 at 19:38
  • $\begingroup$ @BOBONA Yes, you should make the current layers residual and also add more layer in a single block: the additional conv that you put in the block should not reduce size, since you want to reduce it across blocks. In general, consider that 2 conv layers with kernel size 3 is equivalent to a single conv with kernel 5. So you can put more conv instead of directly having a large kernel which is expensive and slow. IN may work better than BN: just try what works best. $\endgroup$
    – Luca Anzalone
    Commented Jul 24, 2023 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.