I am trying to understand working of LSTM networks and kind of not clear about how different neurons in a cell interact each other. I had a look at a similar question, but still not clear about few things.
Assume an NER task where each time step get embedding on one word as input and there are 10 time steps. I defined the LSTM cell with 100 units, As per my understanding each units will have 3 separate gates and memory cell as shown in the second figure (total 100 memory cells and 150 gates). From the theory of LSTM, each of these memory states stores different information, like one might store about singularity, other might store information about gender and so on (please correct me if I am wrong). With this set up, I have few doubts about the working of LSTM and the interaction between each units/neurons
1) In the figure, each cell has multiple neuron, lets say 100. Does the first input X1 (first word vector) will be input to the first neuron of the cell along with previous cell's activation function (say h0) and its output will go to the second neuron and so on..? Or same input X1 and h0 will be fed to each 100 neurons parallel (independent) and respective outputs will go to next time step (not sure whether any aggregation happens) ?
2) How different units would be able to store different features (singularity, gender etc..)? is it because they are initialized with random and different weights?
3) If neurons are working parallel, Is there any interaction happens between them to share information? will it be possible that multiple cell state store about same information (say, gender)
4) If LSTM cell 1 has 100 neurons/units and input X1, output h1 will also have 100 values (one for each neuron) ?