0
$\begingroup$

The expected standard deviation of Xavier initialization of a D-dimensional tensor would be $1 / \sqrt (D)$ but it is not as below. Please help understand why.

import torch

D = 27


x = torch.zeros(1, D)
torch.nn.init.xavier_normal_(x)
torch.std(x, dim=-1)
-----
tensor([0.2820])

Expected std

1 / np.sqrt(D)
---
0.1925
Improve this question
$\endgroup$
1
  • $\begingroup$ The link you have provided does not support your expectation that the std should be 1/sqrt(D). Please elaborate. $\endgroup$
    – desertnaut
    Commented Oct 6 at 14:45

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.