0
$\begingroup$

Unlike the regular MNIST which gets 97-99% with a fairly basic network, dermaMNIST gets training/validation stuck on 0.69. This tells me the model is underfitting. But, making it bigger seems to have zero effect. There is a known class imbalance issue, but it should affect generalization more than ability to fit on training data...? The data is already normalized. What obvious thing am I missing?

train_dataset = DermaMNIST(split='train', transform=train_transforms, download=True)
eval_dataset = DermaMNIST(split='val', transform=eval_transforms, download=True)


train_transforms = transforms.Compose([
    transforms.ToTensor()
])

train_loader = DataLoader(train_dataset, batch_size=100, shuffle=True)
eval_loader = DataLoader(eval_dataset, batch_size=100, shuffle=True)


accuracy_metric = Accuracy(task="multiclass", average="micro", num_classes=7).to(device)

# Loss function and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-5)

class ConvNet(nn.Module):
    """A basic convolutional neural network."""

    def __init__(self, im_width, im_height, num_classes: int = 7, input_channels=3):
        super().__init__()

        self.backbone_out_channels = 16
        self.im_width = im_width
        self.im_height = im_height

        self.backbone = nn.Sequential(
            nn.Conv2d(in_channels=input_channels, out_channels=self.backbone_out_channels, kernel_size=3, padding=1),
            nn.MaxPool2d(kernel_size=2, stride=2, padding=0),
            nn.ReLU()
        )

        self.backbone1 = nn.Sequential(
            nn.Conv2d(in_channels=self.backbone_out_channels, out_channels=self.backbone_out_channels, kernel_size=3, padding=1),
            nn.ReLU()
        )

        self.backbone2 = nn.Sequential(
            nn.Conv2d(in_channels=self.backbone_out_channels, out_channels=self.backbone_out_channels, kernel_size=3, padding=1),
            nn.ReLU()
        )

        self.backbone3 = nn.Sequential(
            nn.Conv2d(in_channels=self.backbone_out_channels, out_channels=self.backbone_out_channels, kernel_size=3, padding=1),
            nn.ReLU()
        )

        self.backbone4 = nn.Sequential(
            nn.Conv2d(in_channels=self.backbone_out_channels, out_channels=self.backbone_out_channels, kernel_size=3, padding=1),
            nn.ReLU()
        )

        self.classifier = nn.Sequential(
            nn.Linear(self.backbone_out_channels * (self.im_width // 2) * (self.im_height // 2), 1024),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(1024, num_classes),
            nn.Softmax()
        )

    def forward(self, x):
        x = self.backbone(x)
        x = self.backbone1(x)
        x = self.backbone2(x)
        x = self.backbone3(x)
        x = self.backbone4(x)
        x = x.view(-1, self.backbone_out_channels * (self.im_width // 2) * (self.im_height // 2))
        x = self.classifier(x)
        return x
$\endgroup$
3
  • $\begingroup$ You seem to be missing some lines of code (e.g. backbone2 through backbone4), but have you tried a model with more parameters? Additionally, if the training performance it similar to the testing performance with an F1 score of ~0.7 I wouldn't call it overfitting. $\endgroup$
    – Oxbowerce
    Commented Apr 5 at 18:26
  • $\begingroup$ @Oxbowerce Added the rest of the model code. Was just showing a snippet to avoid spamming. Also, I had a typo (now corrected). The model was under-fitting i.e. both accuracies would go to 0.69 at first epoch but wouldn't get higher. This was an interview question - the original model had just one CCN and one FC layer with 50 neurons and I was supposed to get 0.72 val acc. After adding several layers and increasing the number of channels a lot, acc didn't change. Thought I had a bug. ResNet gets 0.9 train acc, so I guess I was supposed to keep adding layers. Now it's a matter of val acc==0.69 $\endgroup$
    – Zwerchhau
    Commented Apr 5 at 19:15
  • $\begingroup$ The val acc might be addressed by augmentation and classbalance. Curious if anyone is able to get >0.73 on this DermaMNIST dataset since the benchmark accuracy published in Nature is only 0.73. $\endgroup$
    – Zwerchhau
    Commented Apr 5 at 19:16

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.