Which metric to use to evaluate prediction problem

Question

The product manager wants to know if you can develop a model to predict the number of views a listing will receive based on the boat's features. She would consider using your model if, on average, the predictions were only 50% off of the true number of views a listing would receive.

(please interpret the bold sentence!!)

The data used in this: https://www.kaggle.com/datasets/karthikbhandary2/boat-sales

I tried using the RandomForestRegressor() model along with RandomizedSearchCV.

param = [
    {'n_estimators': [100, 200, 300, 400, 450, 500, 600, 700, 800, 900, 1000], 
     'max_depth': [3, 4, 6, 8, 10, 12,14,16,18,20], 
     'max_leaf_nodes': [15, 20, 25,30,35,40]}, 
]

rf = RandomForestRegressor(random_state=42)
rs_rf = RandomizedSearchCV(rf, param, cv = 4, n_jobs = -1, verbose = 1)
rs_rf.fit(X_train, y_train)

I tried using RMSE like this:

rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print(rmse)

I got the result as 483887.72666891955

My question is whether I am doing this right or not?

Answer 1

If you want to see how much the prediction is off from the actual value as a percentage you should look into using the mean average percentage error (MAPE). The scikit-learn library has this loss implemented in the metrics package. You can simply call the mean_absolute_percentage_error function instead of the mean_squared_error you're currently using:

mean_absolute_percentage_error(y_true, y_pred)