I have a regression problem that has relatively low dimensional input (say 8 initial relevant features, without the engineered ones), but very high dimensional output vector (not a single value for the output, a vector of reals). Basically for each sample during training, a low dimension of features and 2500 reals as the output go into the model, then testing samples take the same types of input features in to predict the 2500 reals (a curve).
inputs outputs
_________________________________________ _______________________________________
| a | b | c | ... | (engineered features) | 2500 reals as output (y's of a curve) |
Since we are not machine learning experts, we had some trouble in the beginning even finding an algorithm that supported output vectors as opposed to a single point.
I'm not looking for generic data science 101 suggestions like: cleaning the data, picking the relevant features, understanding the data, etc. We've done that well enough I believe, and we've even had some limited success already learning these curves with about a predication rate of ~70% (which sounds rough, but isn't too bad given the nature of this data). However, we are not experts as I said, and I'm still convinced that we are not even considering all of our options in terms of algorithms, frameworks, feature engineering techniques, etc. for this specific type of problem; I'd like to squeeze everything we can to get the best possible predictions. Given this data, there definitely isn't a 100% relationship between the input and output. But I do think we can get it predicting at a rate of higher than 70%. Maybe. Sometimes the predictions are near perfect, other times they have the "general shape" but are sometimes warped in places we can't explain.
We are using Python/SciPy/scikit-learn. We are doing 20-fold cross validation and our dataset has ~22000 samples.
How would you approach this type of problem - which algorithm, or which papers, have you come across / used / are relevant for this type of problem? What should we be keeping in mind for this specific type of problem?