I've been getting a strange error while trying to run a working trainer as part of my application that involves pytorch and it's tensor class which is confusing me.
It relates to precision.
In my jupiter notebook I define an array [0.1111,0.2222] for example. When I run torch.tensor(array) it returns a tensor with datatype float32. When I run a sequential layer on this eg. self.linear_relu(newtensor) it works fine in the notebook.
But in the Vs Code environment it is defaulting to float64 everytime !
Except in the debug console. eg.
i = float(32)
i
32.0
torch.tensor(i)
tensor(32.)
t=torch.tensor(i)
t.dtype
torch.float32
t.storage_type
<built-in method storage_type of Tensor object at 0x7fd681386080>
t.storage_type()
<class 'torch.FloatStorage'>
x[0]
tensor(0.9957681660899654341179144, dtype=torch.float64)
x[0].storage_type()
<class 'torch.DoubleStorage'>
x[0].dtype
torch.float64
After getting this error I started setting the default data type manually with.
torch.set_default_dtype(torch.float32)
I look in vscode launch and there is no mention of doubles. I don't cast any of my variable to doubles. But here we are.
Of note I am manually casting ALL of the values be placed into tensors to float eg.
float(value)
All the math values for all the functions I use in python are also returning float types as are all the values in the classes I'm storing values in minus a few integer constants.
Why is the Tensor defaulting to Double, and why is pytorch nn.Linear bombing when it gets the value ?
Meanwhile the Parameters of the network are defaulting to float32, which is the cause of the issue.
