3
$\begingroup$

I ask this because I am currently working with a CNN model built for diagnosis of pneumonia. Originally, I followed a notebook on kaggle to build the model and thereby learn what each bit of code is for, etc. The dataset used was rather imbalanced, with a far greater number of pneumonia cases than normal (healthy) ones. Hence the model.fit class_weight parameter was set to {0:6.0, 1:0.5}. (0 being normal, 1 being pneumonia)

Since then, whilst working on the model and making adjustments, I acquired a number of new data to add to the model such that now the dataset is fairly balanced. In fact, I ensure that the data is loaded into the model so that it is exactly balanced, the dataframes used are coded to ensure an equal number of pneumonia and normal cases in the training testing and validation dataframes.

So, accordingly, I am now trying to remove the use of the class_weights parameter as (as far as I understand it) it is not necessary and may impart some bias in the results. However, in doing so, the model no longer seems to improve in accuracy. It essentially stalls on 0.5 indefinitely. Whereas, with the weights applied, I achieve 0.90+ accuracy.

Simply put, is there some reason for this? The code is quite long, but I'm happy to post it if it is deemed required, but I feel like this may be due to my lack of understanding than error in code (as it has otherwise been working fine and as expected). Thanks in advance.

EDIT: For the sake of clarity and understanding, I performed a grid search over possible values for applied weight values. It confirmed an appropriate choice as being 0:~4.0, 1:0.4, but also suggests 0:1, 1:5.0.

EDIT 2: For further clarity, a link to a github containing the model code and output files etc. https://github.com/GeeKandaa/ML-Code

$\endgroup$
6
  • $\begingroup$ Given the edit, and neural nets being fickle, could this just be random effects of the training? $\endgroup$ Mar 21 at 23:14
  • $\begingroup$ As far as my understanding of it goes, I don't believe that's the case here. I performed a gridsearch three times which gave comparable results around the values given in the edit. Further, setting weights to default (0:1,1:1) consistently returned poor validation results. $\endgroup$
    – GeeKandaa
    Mar 22 at 0:29
  • $\begingroup$ Interesting! I don't suppose you could share the data and code? $\endgroup$ Mar 22 at 0:35
  • 1
    $\begingroup$ Sorry for the late reply, as this work is part of my dissertation I wanted to double check with my supervisor what I could and could not share at this point. I've added a link to a github containing the code (It is quite long and convoluted, likely not very optimised..) The output data from my gridsearches are contained in Support_Files under base_gridsearch_class_weights#X.json $\endgroup$
    – GeeKandaa
    Mar 22 at 20:22
  • $\begingroup$ Thanks! I still suspect NN convergence issues: in the results #3 file e.g., in each case with 0.5 accuracy, the confusion matrix has predicted everything to be the same class, and which one depends on the weights. In the first few grid points, you increase the weights towards the negative class, but after the first two suddenly the network starts predicting the positive class. $\endgroup$ Mar 22 at 20:38
0
$\begingroup$

Class Weight can be important even for balanced data if, for example, some class is more significant than others, so loss wrt this class should count more.

One can even think of class weights as unique extra hyper-parameters with their own effect on the outcome (either positive or negative) and treat them as such without interpretation.

Related: How does class_weight work in Decision Tree?

$\endgroup$
10
  • $\begingroup$ I had assumed as much, thanks for confirming. Mathematically speaking, it's seems I understand what is happening, which implies I merely don't understand why this is the case for this particular dataset. My intuition provokes me to believe that the model holding the normal cases as "more important" leads it to classify pneumonia cases as normal cases with some form of anomaly. Does that seem a reasonable justification for the behaviour? $\endgroup$
    – GeeKandaa
    Mar 21 at 12:01
  • $\begingroup$ Further, I don't suppose you would have an idea of any examples where class weights are applied on balanced data that I could study? My google searches so far have only returned inbalance as a justification for their use. $\endgroup$
    – GeeKandaa
    Mar 21 at 12:04
  • $\begingroup$ I dont have some examples handy but I agree with your conclusion that importance is what is at issue here. For example pneumonia can be deadly so it is only reasonable. However only the original authors can authenticaly explain its use $\endgroup$
    – Nikos M.
    Mar 21 at 12:08
  • $\begingroup$ That's okay, I'll keep looking. Thanks for the help! $\endgroup$
    – GeeKandaa
    Mar 21 at 12:10
  • 2
    $\begingroup$ Just to give some reference for my "the class weights should only really affect.." comment, proceedings.mlr.press/v97/byrd19a/byrd19a.pdf $\endgroup$
    – GeeKandaa
    Mar 22 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.