How to generate structured parameters from a spectrogram?

Question

Say I have an algorithm that accepts as input structured parameters of the following format, generates an audio clip and then a 512x512 spectrogram out of it:

[ param1 = numeric_value, param2 = numeric_value, ..., param100 = numeric_value ]

How can I do the opposite? That is, provide a 512x512 spectrogram and get a set of random candidate parameter values that would yield a similar but random spectrogram if fed into the algorithm?

In terms of text-to-image models, I see this as the opposite problem. Instead of using a prompt to generate a random matching image, I would like to use an image to obtain a random matching "prompt" that's not natural language (i.e. structured and numeric).

Regarding the algorithm, we can assume that the amount of changes in a resulting spectrogram is proportional to the amount of changes in the parameter values. That is, close to no parameter changes will yield a very similar spectrogram than the previous, making training somewhat possible. The algorithm is also deterministic and will always produce the same output for a given input.

Is this possible? GANs seemed to be a nice architecture for this knowing that I can generate as many "real" training data as I want using the algorithm. The generator would generate a random list of structured parameters from a spectrogram, whereas the discriminator would check whether the parameter list is real or fake (i.e. coming from my training set or the generator). In practice though, I'm not sure how I would implement any of this knowing that GANs are usually not used that way (they usually produce images, not the other way around). There might also be a better architecture for this use case that I'm not aware of (e.g. latent space encoder).

Answer 1

This can be done with supervised learning. First create a dataset of spectrogram patches by systematically varying all the parameters - and keep these two associated, for example use a database of parameters, and an ID for the generated spectrogram. Then train a neural network, by letting the input X be the spectrogram and the target Y be the parameters. A Convolutional and/or Recurrent Neural Network is the most relevant here. This network will learn to output the parameter setting whenever it gets an input spectrogram. As a side-effect the network will also internally create a latent representation. In order to explore variations, you can start with the latent corresponding to an input (spectrogram) and then add noise to this latent vector to get new outputs (parameters).