Predicting the likelihood that a prediction from a linear regression model is accurate

Question

So to set up the problem:

I have a data set that had labeled data like colour, brand and quality as independent variables and the dependent is RRP (price).

I have made a linear regression model using this data and can predict the dependent variable using the independent variables (i am using scikitlearn so just using model.predict. This is causing me siginificant problems and I'm not sure if this is the right way to deal with this and I'm not sure if this will hinder my goal of getting accurate values for predicted variables.

Is there a way to calculate the potential accuracy of the price that is predicted? It seems to me if I ask the model to predict on brand x and quality y and the model knows brand x and the quality always produces a tight range of prices the accuracy is potentially higher?

Answer 1

In case you are talking about providing certain interval to your predictions, what you might need is adding some confidence interval to your linear regression predictor, something which you can make via a resampling method like bootstrapping as a robust way to find predictions intervals. One key advantage is that it does not assume any kind of distribution, being a distribution-free method to find your predictions and, if needed, to your regression coefficients estimates.

The steps would be:

1. Draw n random samples (with replacement) from your dataset, where n is the bootstrap sample size
2. Fit a linear regression on the bootstrap sample from step 1 and predict a value
3. Take a single residual at random from the original regression fit, add it to the predicted value and save the result.
4. Repeat 1 to 3 steps several times (1k times for instance)
5. Find the desired percentiles of your interval (2.5th to 97.5th for instance) Source of info in this book

On the other hand, if you mean providing a generic confidence metric value for your model, you should find for instance a MSE or MSA on a test set.

Answer 2

You don't say what the amount of available data is and if you use a test set. When you are up to prediction, always use a test set (some randomly chosen part of the data, say 20-30% NOT used for model training) to test your model predictions. With sklearn:

import numpy as np
from sklearn.model_selection import train_test_split
X, y = np.arange(10).reshape((5, 2)), range(5)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)

Predict the test data based on your model and compare the predictions to the TRUE values in the test set (mean absolute error or mean squared error for instance):

from sklearn.metrics import mean_absolute_error
mean_absolute_error(y_true, y_pred)

So you get an idea of how well your model works.

Also note that linear regression can deliver weak predictions if you have high correlation in your $x$ variables or if there is non-linearity in your $x$ variables. In this case you could try to use random forest (or related methods), generalised additive models (GAM), lasso/ridge regression etc. (a lot of things can matter wrt good model choice). However, the takeaway is that linear regression can perform very poorly in some cases (compared to other methods).