This question not related to a specific method or technique, rather there is a broader concept that I'm struggling to see clearly.
Introduction
In machine learning, we have loss functions that we're trying to minimize. Gradient descent is a general solution to minimizing the output of these loss functions. I understand the basic idea there, but it's the details that I'm getting stuck on.
Suppose that I have some loss function $J_\Theta(x)$, where I have some $x$ input, where $\Theta$ is just some matrix of arbitrary parameters.
In many examples, I see $\Theta$ written something like:
$\Theta = \begin{bmatrix} \hat{x_0} \\ \hat{x_1} \\ \hat{x_2} \\ \vdots \\ \end{bmatrix} $
and each row of $\Theta$ is actually a vector.
Note that some concrete examples that I've run across include updating word embeddings in word2vec, or updating a softmax layer. I'm happy to elaborate if my explanation is too abstract.
In the examples of gradient descent that I've seen, typically the derivative of $J$ is taken w.r.t each row of $\Theta$, not individual elements.
So something like $\frac{dJ}{d\hat{x_0}}$, where each vector in $\Theta$ is updated according to the output of this gradient function.
Now for my point of confusion:
Suppose the parameters are initialized to zero, which is sometimes a thing. Wouldn't the updates be the same for each element of $\hat{x_i}$ in $\Theta$ during the update step? Wouldn't that lead to each vector having the same numbers for each element? I know that's the wrong conclusion, but I'm not able to see how each dimension would, in the end, become different values. I assume (hope) I'm missing something simple.