I wrote a pytorch data loader which used to return data of shape (4,1,192,320) representing the 4 samples of single channel image, each of size 192 x 320. I then used to unfold it into shape (4,15,64,64) (Note that 192*320 = 15*64*64). Resize it to shape (4,15,64*64). And then finally apply my FFN which used to return tensor of shape (4,15,256). (FFN is just first of several neural network layer in my whole model. But lets just stick to FFN for simplicity.) This is the whole code:

import torch
import torch.nn as nn
from torchvision import transforms
from torch.utils.data import Dataset, DataLoader

class FFN(nn.Module):
    def __init__(self, in_dim, out_dim, dropout=0.1):
        super(FFN, self).__init__()
        self.linear = nn.Linear(in_dim, out_dim)
        self.dropout = nn.Dropout(dropout)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.linear(x)
        x = self.relu(x)
        x = self.dropout(x)
        return x

class DummyDataLoader(Dataset):
    def __init__(self):
        self.transforms = transforms.Compose([
                    transforms.Resize((192, 320)),
    def __len__(self):
        return 10000 # return dummy length
    def __getitem__(self, idx):
        frame = torch.randn(192,380)
        frame = self.transforms(frame)
        return frame
dataset = DummyDataLoader()
dataloader = torch.utils.data.DataLoader(dataset, batch_size=4, shuffle=False)
frames = next(iter(dataloader))
print('Raw: ', tuple(frames.shape))

unfold = torch.nn.Unfold(kernel_size=64, stride=64)
unfolded_ = unfold(frames)        
unfolded = unfolded_.view(unfolded_.size(0),-1,64,64)
print('Unfolded: ', tuple(unfolded.shape))

unfolded_reshaped = unfolded.reshape(unfolded.size(0), -1, 64*64)
ffn = FFN(64*64, 256, 0.1)
ffn_out = ffn(unfolded_reshaped)
print('FFN: ', tuple(ffn_out.shape))

This outputs:

Raw:  (4, 1, 192, 320)
Unfolded:  (4, 15, 64, 64)
FFN:  (4, 15, 256)

Now, I realized, I also need to implement sliding window. That is, In each iteration, data loader wont just return single frame but multiple frames based on sliding window size, so that the model will learn inter-frame relation. If window size is 5, it will return 5 frames. To implement this, I just changed __getitem__ from:

def __getitem__(self, idx):
    frame = torch.randn(192,380)
    frame = self.transforms(frame)
    return frame


def __getitem__(self, idx):
    frames = [torch.randn(192,380) for _ in range(5)]
    transformed_frames = [self.transforms(frame) for frame in frames]
    return torch.stack(transformed_frames)

But the code started giving me following error:

Raw:  (4, 5, 1, 192, 320)

RuntimeError                              Traceback (most recent call last)
d:\workspaces\my-project\my-project-win-stacked.ipynb Cell 19 line 6
     57 print('Raw: ', tuple(frames.shape))
     59 unfold = torch.nn.Unfold(kernel_size=64, stride=64)
---> 60 unfolded_ = unfold(frames)        
     61 unfolded = unfolded_.view(unfolded_.size(0),-1,64,64)
     62 print('Unfolded: ', tuple(unfolded.shape))

File ~\AppData\Roaming\Python\Python311\site-packages\torch\nn\modules\module.py:1511, in Module._wrapped_call_impl(self, *args, **kwargs)
   1509     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1510 else:
-> 1511     return self._call_impl(*args, **kwargs)

File ~\AppData\Roaming\Python\Python311\site-packages\torch\nn\modules\module.py:1520, in Module._call_impl(self, *args, **kwargs)
   1515 # If we don't have any hooks, we want to skip the rest of the logic in
   1516 # this function, and just call forward.
   1517 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1518         or _global_backward_pre_hooks or _global_backward_hooks
   1519         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1520     return forward_call(*args, **kwargs)
   1522 try:
   1523     result = None

File ~\AppData\Roaming\Python\Python311\site-packages\torch\nn\modules\fold.py:298, in Unfold.forward(self, input)
    297 def forward(self, input: Tensor) -> Tensor:
--> 298     return F.unfold(input, self.kernel_size, self.dilation,
    299                     self.padding, self.stride)

File ~\AppData\Roaming\Python\Python311\site-packages\torch\nn\functional.py:4790, in unfold(input, kernel_size, dilation, padding, stride)
   4786 if has_torch_function_unary(input):
   4787     return handle_torch_function(
   4788         unfold, (input,), input, kernel_size, dilation=dilation, padding=padding, stride=stride
   4789     )
-> 4790 return torch._C._nn.im2col(input, _pair(kernel_size), _pair(dilation), _pair(padding), _pair(stride))

RuntimeError: Expected 3D or 4D (batch mode) tensor with possibly 0 batch size and other non-zero dimensions for input, but got: [4, 5, 1, 192, 320]

As you can see, the data loader now returns data of shape [4, 5, 1, 192, 320] in each iteration. But it fails in next step of unfolding, as it seem to expect 4D tensor for batch mode. But data loader returned 5D tensor. I believe, each step in my model pipeline (several FFNs, encoders and decoders) will fail if I return such 5D tensor from data loader as they all be expecting 4D tensor for batch mode.

Q1. How we can combine batching and windowing without breaking / revamping existing model, or revamping is inevitable?

Q2. If I revamping model is inevitable, how do I do it, such that it will involve minimal code changes (say for example for above model, which involves unfolding and FFN)?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.