I make some variants of ResNet, originally found in TorchVision, modify them, train them and so on. What I have found is that even in .eval() mode, even if I load state right before evaluation, I receive different results. The code looks like
... imageData = imageDataset.getImage(imageNum) imageData = np.expand_dims(imageData, 0) pytImageData = torch.from_numpy(imageData).cuda() ... self.loadState(self.curEpochNum) model.eval() self.pytOptimizer.zero_grad() with torch.set_grad_enabled(False): activations = model.forward(pytImageData).cpu().numpy() self.loadState(self.curEpochNum) activations2 = model.forward(pytImageData).cpu().numpy() diff = activations2 - activations print('diff', diff.min(), diff.max())
Difference is quite high. I have found this during investigation of occlusion heatmaps, they were quite noisy (and I would even say gave strange results, but maybe this is just one more mystery of neural nets, with different nature).
ImageNet-1000 dataset, torch version 1.5.0.dev20200128, torchvision version 0.6.0.dev20200128
Another question appears given that this is caused by fractional max pooling: how to make it stable? As I suppose, only random mode is implemented in PyTorch, so random positions are selected for each convolution at each run. But the authour also invented pseudorandom mode. Is it possible for example to turn it on somehow?