3
$\begingroup$

I am solving a video classification problem. There are 9 classes in total. At first I took ResNet as a feature extractor, this gave me 0.74 accuracy. Then I changed ResNet to EVA (I also tried Swin), hoping that this would increase accuracy. But this greatly worsened the result. That's my model:

class RSNAModel(nn.Module):
    def __init__(self, pretrained=True):
        super(RSNAModel, self).__init__()

        self.backbone = timm.create_model(model_name=Config['FEATURE_EXTRACTOR'], pretrained=True, num_classes=0)
        freeze_module(self.backbone)
        
        
        self.num_features = self.backbone.num_features
        self.backbone.classifier = Identity()
        self.dropout = nn.Dropout(p=Config['DR_RATE'])
        self.rnn = nn.LSTM(
            input_size=self.num_features, 
            hidden_size=self.num_features // 2, 
            num_layers=1,
            dropout=Config['RNN_DP'],
            bidirectional=True,
            batch_first=True, 
        )
        
        self.head = nn.Linear(in_features=self.num_features, out_features=Config['NUM_CLASSES'])
        
    def forward(self, inputs):
        b, f, c, h, w = inputs.shape
        inputs = inputs.reshape(b * f, c, h, w)
        embeddings = self.backbone(inputs)
        embeddings = embeddings.reshape(b, f, self.num_features)
        sequence_outputs, h_n = self.rnn(embeddings)
        features = sequence_outputs[:, -1]
        outputs = self.dropout(features)
        outputs = self.head(outputs)
        
        return outputs
==========================================================================================
Layer (type:depth-idx)                   Output Shape              Param #
==========================================================================================
RSNAModel                                [4, 9]                    --
├─Beit: 1-1                              [64, 1408]                363,264
│    └─PatchEmbed: 2-1                   [64, 256, 1408]           --
│    │    └─Conv2d: 3-1                  [64, 1408, 16, 16]        (829,312)
│    │    └─Identity: 3-2                [64, 256, 1408]           --
│    └─Dropout: 2-2                      [64, 257, 1408]           --
│    └─ModuleList: 2-3                   --                        --
│    │    └─Block: 3-3                   [64, 257, 1408]    
          ............
│    │    └─Block: 3-42                  [64, 257, 1408]           (25,248,768)
│    └─Identity: 2-4                     [64, 257, 1408]           --
│    └─LayerNorm: 2-5                    [64, 1408]                (2,816)
│    └─Identity: 2-6                     [64, 1408]                --
├─LSTM: 1-2                              [4, 16, 1408]             11,906,048
├─Dropout: 1-3                           [4, 1408]                 --
├─Linear: 1-4                            [4, 9]                    12,681
==========================================================================================
Total params: 1,023,064,841
Trainable params: 11,918,729
Non-trainable params: 1,011,146,112
Total mult-adds (G): 63.75
==========================================================================================
Input size (MB): 38.54
Forward/backward pass size (MB): 62167.32
Params size (MB): 3138.77
Estimated Total Size (MB): 65344.63
==========================================================================================

What should i do to improve results? What other tricks can be applied to improve accuracy? I believe that a good vision transformer as backbone can improve accuracy.

$\endgroup$

1 Answer 1

2
$\begingroup$

The current state-of-the-art are transformer architectures. One example is ViViT: A Video Vision Transformer which has an implementation ViViT-pytorch.

Something like this:

from vivit import * 

class Transformer(nn.Module):
    def __init__(self, dim, depth, heads, dim_head, mlp_dim, dropout = 0.):
        super().__init__()
        self.layers = nn.ModuleList([])
        self.norm = nn.LayerNorm(dim)
        for _ in range(depth):
            self.layers.append(nn.ModuleList([
                PreNorm(dim, Attention(dim, heads = heads, dim_head = dim_head, dropout = dropout)),
                PreNorm(dim, FeedForward(dim, mlp_dim, dropout = dropout))
            ]))

    def forward(self, x):
        for attn, ff in self.layers:
            x = attn(x) + x
            x = ff(x) + x
        return self.norm(x)
$\endgroup$
2
  • $\begingroup$ Thank you for feedback. I have just stopped training because val loss at the 40th epoch about 3.5 colab.research.google.com/drive/… Could you look at it and say what is wrong? $\endgroup$
    – Nikto
    Commented Dec 11, 2022 at 18:16
  • $\begingroup$ Maybe increasing the dropout will fix these problems? And is it worth increasing batch_size? Now batch_size=4, because there are only 450 training and 80 test videos in the dataset. $\endgroup$
    – Nikto
    Commented Dec 11, 2022 at 20:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.