I am currently studying this model speech generation known as WaveNet model by Google. https://arxiv.org/pdf/1609.03499.pdf using the linked original paper and this implementation.

I find the model very confusing in the input it takes and the output it generates, and some of the layer dimensions didn't seem to match based on what I understood from the WaveNet paper, or am I misunderstanding something?

  1. What is the input to the WaveNet, isn't this a mel-spectrum input and not just 1 floating point value for raw audio? E.g. the input kernel layer shows as shaped 1x1x128. Isn't the input to the input_convolution layer the mel-spectrum frames, which are 80 float values * 10,000 max_decoder_steps, so the in_channels for this conv1d layer should be 80 instead of 1?

inference/input_convolution/kernel:0 (float32_ref 1x1x128) [128, bytes: 512]

  1. Is there reason for upsampling stride values to be [11, 25], like are the specific numbers 11 and 25 special or relevant in affecting other shapes/dimensions?
inference/ConvTranspose1D_layer_0/kernel:0 (float32_ref 1x11x80x80) [70400, bytes: 281600]
inference/ConvTranspose1D_layer_1/kernel:0 (float32_ref 1x25x80x80) [160000, bytes: 640000]
  1. Why is the input-channels in residual_block_causal_conv 128 and residual_block_cin_conv 80? What exactly is their inputs? (e.g. is it mel-spectrum or just a raw floating point value?) Is the wavenet-vocoder generating just 1 float value per 1 input mel-spectrum frame of 80 floats?
inference/ResidualConv1DGLU_0/residual_block_causal_conv_ResidualConv1DGLU_0/kernel:0 (float32_ref 3x128x256) [98304, bytes: 393216]
inference/ResidualConv1DGLU_0/residual_block_cin_conv_ResidualConv1DGLU_0/kernel:0 (float32_ref 1x80x256) [20480, bytes: 81920]

I was able to print the whole WaveNet network using the print(tf.trainable_variables()), but the model still seems very confusing.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.