I am trying to explore the use of neural network based models to replace a cumbersome challenge with a faster more approximate method. The problem involves learning a distribution termed the "total image," which represents a 2D target distribution. However, I do not have access to the "total image" directly, and I must find a way to learn it in a self-supervised manner. [I realise the appropriate answer to that is "why?", but this is for pure curiosity and exploration so please try to go with it :') ] Instead, I possess observed samples that are transformed via a linear operator, and sum to the "total image."

To formally define the problem, let:

The "total image" be denoted as T(x, y) in 2D space, where x and y are the coordinates.

The observed samples be S(u, v), which are samples of the transformed T(x, y). Further: O(S) denotes the iterative application of some linear operator on S(u, v), resulting in an approximation of T(x, y) when summed and the process converges (this is very similar to the FFT, hence my choice of u and v for the transformed coordinates).

The key constraints and characteristics of the problem are as follows:

The "total image" T(x, y) is an unknown distribution, and it cannot be used directly during the model training process.

The observed samples S(u, v) represent inputs that once transformed by a linear operator O(S) sum to an approximation of T(x, y).

Due to the large volume of observed samples, it is essential to use a small subset of samples as inputs to the model for computational efficiency.

In evaluation mode, the model should generate a learned "total image" T_learned(x, y) from a specific input, e.g. where the inputs are represented as vectors with ones, zeroes, or similar values. This as a sort of proxy for sampling inside a VAE for example, where you want to essentially learn your T(x, y) as a prior which exists in the complete absence (or perhaps complete presence) of knowledge passed as input to the model.

In short, the model's objective is to converge to an approximation of T(x, y) when generating images from samples of input coordinates S(u, v) with their associated values.

Here is my attempt at a pictorial representation of the problem:

enter image description here

Could anyone suggest insights, recommendations, or guidance on what to search for to tackle this problem. To be clear, I'm not asking for a specific model here, I'm more interested in knowing if there are current approaches that are sort of in the ballpark. e.g. "you should look into GANs for this because they solve this" or "self-attention can help with this step specifically", if that makes sense. Thank you!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.