0
$\begingroup$

I have a geostatistical dataset and I've been building linear regression model, but when I plotted the data I've noticed that part of the data shows an absolute straight line trend, i.e. it is most likely not a real-world data, but it was back-calculated.

All the data is in pandas dataframe and I guess I need to find subset of the dataset that shows a perfect linear relationship, but I'm not quite sure how to approach it.

dataset 3d plot

$\endgroup$
1
$\begingroup$

Try to fit several linear regression models, for very small number of observations from the data set (for instance, $n$ or $n+1$ randomly selected points, where $n$ is the number of dimensions). If there are enough perfectly linearly aligned points and enough models, it is likely that one of the models will be built from points that were only drawn from the back-calculus.

Then compare the models predictions to the actual data. For the models which perform very well on a large number of test cases (i.e. more than $m$ points are correctly predicted with an error of $\epsilon$ or less), these observations have probably been computed.

You will have to define $m$ and $\epsilon$ based on your knowledge of the problem / by adjustment.

If I may suggest, you can then keep these observations in your dataset, but give them a lower weight for the final model training.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.