The AIC formula is :
$AIC = 2k + n Log(RSS/n)$
So if RSS is equal to 0, it is undefined. How do I deal with this? What value should it take?
1
-
$\begingroup$ So your model has a perfect fit? $\endgroup$– PeterCommented Oct 19, 2020 at 10:49
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
7
RSS = 0 implies that the model is a perfect fit since there was no residual. The limit of the log of 0 is $-\infty$, and since lower AICs are better, and this model is perfect,it makes perfect sense that the AIC should be a negative number such that no number can be lower.
-
$\begingroup$ Right, that makes sense, but the problem is that I am evaluating the AIC fo each fit and then I want to take the mean of these values (I cannot compute the AIC of the model with all the fits together because I have a different number of datapoints for each). So how should I handle this? $\endgroup$ Commented Oct 19, 2020 at 9:53
-
$\begingroup$ I would recommend opening a new question regarding taking the mean of an AIC across many fits, and how to deal with the case that a given fit is perfect. It's not something that I've heard of before, so I think you would have to provide lots of details and particularly why you're doing it. $\endgroup$ Commented Oct 19, 2020 at 9:56
-
$\begingroup$ Maybe my approach of taking the mean is wrong. Do you know how could I compute the AIC if for every fit I have a different number of data points? $\endgroup$ Commented Oct 19, 2020 at 10:01
-
$\begingroup$ As far as I know, you can't compare the AIC between models on different datasets, that's where my understanding of what you're trying to achieve falls apart. If you have many models (or fits), there's nothing to stop you from calculating the AIC of each one. $\endgroup$ Commented Oct 19, 2020 at 10:31
-
$\begingroup$ Agreed, on different datasets you cannot (well, you actually can but it takes som transformations) $\endgroup$ Commented Oct 19, 2020 at 10:42
$\begingroup$
$\endgroup$
9
As a follow-up to @Cameron Chandler; if your RSS=0 it should trigger some alerts since it is 100% overfitting, thus evaluating that model does not make any sense as such. But yes, take the limit of $\log(x), x\rightarrow 0^+$ and notice it goes towards $-\infty$ thus you cannot find a more perfekt model - if you only look at AIC, but you can (almost) not find a worse model if you look at the application. If you evaluate multiple fits I would simply just ignore those "perfect fits"
-
$\begingroup$ As I mentioned in the comment above, I am evaluating the AIC for each fit and then I want to take the mean of these values, so if I set it to minus infinity, I will have problems with the mean whenever I have one perfect fit in my dataset $\endgroup$ Commented Oct 19, 2020 at 9:56
-
$\begingroup$ I have edited the answer accordingly $\endgroup$ Commented Oct 19, 2020 at 10:02
-
$\begingroup$ The thing is it will be a perfect fit for one of the models but not for the rest, so if I just ignore it, I would be loosing information. Maybe I should just set it to a big negative number? $\endgroup$ Commented Oct 19, 2020 at 10:54
-
$\begingroup$ How would you lose information? I would treat it as "missing" since the score provides no information what so ever (other than you are overfitting) $\endgroup$ Commented Oct 19, 2020 at 10:58
-
$\begingroup$ Ah I see your point, but I am not trying to evaluate overfitting at the moment since I am fitting the parameters for each tumor. But maybe it is the best solution, thanks. $\endgroup$ Commented Oct 19, 2020 at 11:09