0
$\begingroup$

I am going to build the model (e.g. multiple linear regression) to predict the appartment cost in my city. First I have to find outliers in training data. For this task RANSAC regression algorithm looks attractive because it allows not only to detect outliers but to build the model itself. One thing that confuses me is how to test the trained model. The standart approach to check whether model has good predictive power is to split data on train and test and apply trained model on test data. For RANSAC this will not work due to the reason that test data also has outliers and they will bias the score of model.

My question is how we can check that trained model is good?

$\endgroup$

1 Answer 1

1
$\begingroup$

There are different aspects to consider here:

1. Robustness

One of the reasons to use RANSAC is its robustness towards outliers. That means that some outliers more or less in the training set will have no strong effect on the the resulting model. Robustness is an important property of a model: if your model is not robust, it might change strongly depending on how you cut your data into train and test (in the end, it might be a question of the random seed).

Cross Validation is a good way to evaluate robustness. With cross validation, you train multiple models on different training sets and compare their performance. The variance between the models is a good indicator how robust the training is.

2. Outliers in the test set.

First of all, it is important, that your handling of the test-outliers does not depend on your model. Especially, you cannot use your RANSAC-Regression to filter out outliers. Doing so would allow a model to manipulate its own evaluation.

Unfortunately, when it comes to the concrete treatment of test-outliers, I tend to give you an "It depends". Here, it depends on the concrete use case for which you build the model and how outliers influence this uses case.

Let me give you some ideas to consider when deciding for the best treatment of outliers:

  1. If you are aiming for a scientific work (i.e. a paper), evaluation standards are important. In that case, I would stick to the evaluation used in literature for the sake of comparison.
  2. If your outliers are real values (and no measurement errors) that your model is just not able to catch (e.g. due to some complex patterns or latent factors that are not visible in the data), then it might be a good idea to stick to the standard, such as MSE, because these errors will also occur when using the model.
  3. If in case of outliers the magnitude of the error does not matter, then you might consider to perform a clipping of the error (i.e. you set a threshold $T$ and all errors above $T$ are set to $T$). Here the choice of $T$ depends on the use-case an might be a bit arbitrary.#
  4. If outliers are wrong values in the dataset (e.g. due to typos when entering the data), then you might consider to use another algorithm or heuristic to remove outliers from the test-set. But be aware that in this case the impact of similar problems in the application will not be evaluated.

I hope this helps you to find a reasonable way to deal with your outliers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.