I'm getting my feet wet with data science and machine learning. Please bear with me as I try to explain my problem, I haven't been able to find anything about this method but I suspect there's a simple name for what I'm trying to do. To give some idea about my level of knowledge, I've had some basic statistics training in my university studies (social sciences), and I work as a programmer at the moment. So when it comes to statistics I'm not a complete layperson, but my understanding leaves a lot to be desired.

I need to make predictions on a certain dataset, all features and the target are continuous variables (interval/ratio level), and I'm using a regression multi-layer perceptron with a single hidden layer. The $R^{\textrm{2}}$ is relatively low when fitting on the entire dataset, so I was hoping to improve the predictive power by doing cluster analysis and fitting multiple regressors, on each cluster separately. This didn't work for me, the highest cluster $R^{\textrm{2}}$ was the same as for the entire set, so I dropped that approach. I've tried multiple clustering algorithms, haven't noticed much of a difference among them, but I haven't done an exhaustive search, so I may have missed something.

What I ended up doing is, I fit the model on the training subset, identify the data point for which the prediction error in the test subset was greatest, and remove that one data point. The train-test split is done randomly, so the probability of some "bad" data point remaining by chance is pretty low. I repeat this process until I get an $R^{\textrm{2}}$ I'm satisfied with, after which I designate the remaining data points as belonging to e.g. "Group 1". This whole process is then repeated on the entire dataset minus Group 1, and ultimately all of the data should be divided into groups inside which reasonably reliable predictions can be made. To give some idea about the data: $R^{\textrm{2}}$ on the entire set of about 11000 data points hovers around 0.7. In Group 1, I kept 7000 data points for which I can get up to 0.9. The remaining groups also have acceptable (for my standards) $R^{\textrm{2}}$s.

After all of the data is split into groups in this way, I expect to be able to train a classifier on the features, to predict the group that a certain data point belongs to, and use the appropriate regression model for that group to predict the final target variable. My question is, is there some methodological flaw in this approach, specifically the removal of certain data points based on prediction errors? Am I introducing some artefacts into the data, or something like that? As far as I can tell there's no leakage of information about the target. What this looks like to me is, some roundabout way of doing cluster analysis, as it seems to exclude "outliers" in a certain sense, but I couldn't be more precise, and that may not be the case at all.

Note: all mentions of $R^{\textrm{2}}$ refer to predictions, i.e. the score I get on the test set, not training.

EDIT: I remove data points only from test subset (which changes randomly in each iteration), but now that I think about it I guess there's no reason to limit exclusions to the test subset, as a worse prediction may happen in the training subset. I'll update when I try it. Also, I haven't yet tried fitting the classifier, so I suppose I may not get any better results with this final model; regardless of the results I get, I'm interested in the validity of this approach. Also, if someone knows of a theoretical limitation here, if there's a reason why I couldn't get better results in principle with this approach, I'd like to know.

  • 1
    $\begingroup$ "I fit the model on the training subset, identify the data point for which the prediction error in the test subset was greatest, and remove that one data point" do you only remove points from the test set?! $\endgroup$ – oW_ Apr 23 '18 at 17:17
  • $\begingroup$ @oW_ I do remove points from what is currently the test set, but the train-test split changes every time a new regression is run, so there isn't a fixed subset that is always removed from; instead, the test subset is randomly chosen on each fitting so I'm reasonably sure that's not a problem, every data point has an equal chance (in the long run). $\endgroup$ – mtosic Apr 23 '18 at 17:59
  • $\begingroup$ and when you run the next regression you still use all the data including the ones labeled as group 1 in the previous run? $\endgroup$ – oW_ Apr 23 '18 at 21:22
  • $\begingroup$ not quite sure about the details or if you actually validated this by fitting this classifier but I would suspect that you are just moving your error from your regression to the "group" classifier and in the end won't get better results?! maybe you can add some more details about the whole procedure $\endgroup$ – oW_ Apr 23 '18 at 21:28
  • $\begingroup$ @oW_ After Group 1 is determined, the remaining data is further split into groups, so Group 1 is excluded. I added some more info in the edit $\endgroup$ – mtosic Apr 24 '18 at 8:39

The process you described is not valid, it is almost certainly overfitting. However the overall idea may very well be valid. What you should do instead is take a portion of your dataset as the testing which is not looked at during algorithm development, and split the rest between training and validation sets however you please.

Looking only at the training and validation sets, develop your entire end to end algorithm, including the classifier!

Once you are happy with the algorithm, try running it on the test set to get a more accurate measure of performance.

|improve this answer|||||
  • $\begingroup$ So, after finalizing the model I got some very poor results. If it's overfitting, I don't want to know just how bad it would be if I were to do it right :) $\endgroup$ – mtosic Apr 26 '18 at 10:01
  • $\begingroup$ Sorry to hear it. Hopefully you still learned some valuable skills along the way! $\endgroup$ – kbrose Apr 26 '18 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.