Algorithms, techniques, papers for regression with vector output

Question

I have a regression problem that has relatively low dimensional input (say 8 initial relevant features, without the engineered ones), but very high dimensional output vector (not a single value for the output, a vector of reals). Basically for each sample during training, a low dimension of features and 2500 reals as the output go into the model, then testing samples take the same types of input features in to predict the 2500 reals (a curve).

              inputs                               outputs
 _________________________________________ _______________________________________
| a | b | c | ... | (engineered features) | 2500 reals as output (y's of a curve) |

Since we are not machine learning experts, we had some trouble in the beginning even finding an algorithm that supported output vectors as opposed to a single point.

I'm not looking for generic data science 101 suggestions like: cleaning the data, picking the relevant features, understanding the data, etc. We've done that well enough I believe, and we've even had some limited success already learning these curves with about a predication rate of ~70% (which sounds rough, but isn't too bad given the nature of this data). However, we are not experts as I said, and I'm still convinced that we are not even considering all of our options in terms of algorithms, frameworks, feature engineering techniques, etc. for this specific type of problem; I'd like to squeeze everything we can to get the best possible predictions. Given this data, there definitely isn't a 100% relationship between the input and output. But I do think we can get it predicting at a rate of higher than 70%. Maybe. Sometimes the predictions are near perfect, other times they have the "general shape" but are sometimes warped in places we can't explain.

We are using Python/SciPy/scikit-learn. We are doing 20-fold cross validation and our dataset has ~22000 samples.

How would you approach this type of problem - which algorithm, or which papers, have you come across / used / are relevant for this type of problem? What should we be keeping in mind for this specific type of problem?

We are very happy with this:

Not bad:

Not good:

Interesting:

Answer 1

Perhaps you could frame this as predicting a distribution (I hope I haven't mis-understood your question!).

You could try using Markov Chain Monte Carlo (don't be scared by the name!). What this essentially does, it to sample a posterior distribution from your input data and see how likely it was that is is related to your prior distribution (your data). If it thinks the sample fits with your data, it keeps puts it records that sample, otherwise it discards it. It keeps searching like this until some stop-criterion is reached. All the accepted points that were recorded then represent your target distribution, which can be N-dimensional (and so hopefully matches your use-case).

Here is a quick video introduction to the topic in general, and here is a deeper video explanation. Here is a more thorough text-based introduction.

You can get started doing this with packages like:

• PyMC3 (code and docs) if you use Python. There are interactive examples here, e.g. the PyMC3_tips_and_heuristic notebook too.
• If you use R, then there is the mcmc package.