New answers tagged

0

If your only concern is small error values, why not simply scale the output by some constant? The idea would be to multiply all the actual values by some factor e.g. 10*y_actual Next, train your model on the scaled values. To make a prediction in the orginal rang you would have to scale back the outputs: y_scale_orginal = y_prediction / 10


0

MSE and Standard deviation Mean squared error, shows us how much error we have over all our points. Indeed the goal is to reduce it, however, in your case, the error yielded would already be small. One way to understand the relevance of your (MSE) RMSE is to compare it to the standard deviation. Imagine having a standard deviation lower than your learned ...


0

I'd use relative RMSE $\sqrt{\frac{1}{n} \sum \frac{(Preicted - True)^2}{True^2}}$. In this case, close to 0 implies a good model, regardless of the scale of the true values. Similarly, you can try relative MAE.


0

Offline evaluation is very tricky due to all kind of bias. The most prominent type of bias is position bias. I recommend the following paper (https://arxiv.org/pdf/1608.04468.pdf), which contains metrics I have used myself for monitoring and developement of recommendors for a large sport fashion company. The idea is to apply a counter-factual approach to ...


0

There are two major types of evaluation - online and offline. Online evaluation means showing the model's predictions to actual users. Since the goal of a recommender system to sell more products, the best overall best metric for a recommender system is increasing sales to actual users. This is best done by putting the model in production and A/B test if the ...


2

Essentially, the function of your testset is to evaluate the performance of your model on new data. It mimics the situation of your model being put into production. The validation set is used for optimizing your algorithm. Personally I would recommend tuning your algorithm using your validation set and using the hyperparameters of the training epoch with the ...


0

I have no knowledge at all about this kind of problem, but logically I would say: Should I remove the operations where there were no products shown (even though some of them there were clicks on products)?Example (should discard this columns?): Yes, because otherwise you could in theory end up with a rate higher than 1, this wouldn't make sense. However ...


1

Selecting the correct scoring metric depends on the business problem you are trying to solve. I would research the differences between f1 micro and macro and determine which scoring metric ultimately tracks performance of your task in a more seemly manner. For example: do you just want to maximize f1 score across all samples? Or do you care about the ...


Top 50 recent answers are included