We have input features and features space - X and output target - Y (continous or discrete). The connection between X and Y inside small part of features space (A) is stronger than inside rest of features space (B). For example, if we already know how to divide features space on A and B, we can train regression or classifier model on A and on B. For example if it is a regression, the MSE may be equal 0.1 on A and 1.1 on B. If it is a classifier, the accuracy may be equal 0.8 on A and 0.51 on B. Our objective is to maximize the score only on A and not on the whole features space. Our task is to learn how to divide the features space into A and B and train the final model only on A.

I can invent a lot of "DIY" methods for solving this problem, but is there a mainstream approach and may be ready to use code or libraries?

Of course A - is enough big to get good statistical quality of a model training only on it, but enough small to spoil the quality of the model if we train it on a whole features space.

A real life example - is stocks trading. We have two features X1 and X2 for all stocks of the exchange. And the target - Y - stocks returns on the next day. We train a model on the whole data set, and make forecasts on the next day to buy top N stocks with highest forecast. But when we train a model only on samples inside i-th quantile (for example from 0.9 to 1.0) of the X1 and select top N stocks each day, we get much better results.

I cannot find any generic machine learning approach for model construction or samples filtration to get better results than this manual samples filtration by quantile of X1 input.

  • $\begingroup$ Welcome to DataScienceSE. I think there's a flaw in this kind of approach: the goal of a supervised ML system is normally to be able to predict the target for any instance. If you allow the separation of the feature space between an 'easy zone' and a 'hard zone', then why not restrict the easy zone to only a single instance, or a subset of instances? For example, A could be equal to exactly the training set... This works because we are sure to know the target for these instances, so the perf on A is great ;) $\endgroup$
    – Erwan
    Aug 5 at 16:40
  • $\begingroup$ Hello, Erwan! I updated my question and added a real life example. $\endgroup$
    – liberal
    Aug 5 at 18:41
  • $\begingroup$ I don't think this is the right way to design the problem: basically you're looking for the instances where the prediction is the most reliable, i.e. you're trying to predict the confidence of the prediction. So to me this is a confidence estimation problem: once you can predict confidence reliably enough, you can select the most confident predictions. I know about confidence estimation in machine translation, but there are probably other applications. $\endgroup$
    – Erwan
    Aug 11 at 11:42


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.