Is it theoretically possible to correct the contents of the cell state, and what would it result in?
Yes it is. Using back propagation, it is possible to get the gradient of any value that affects a well-defined output. For training data, that includes all current cell outputs - in fact these are necessary to calculate as an interim step in order to get the gradients of the weights. Once you have the gradients of a cost or error function, then you can perform a step of gradient descent in order to discover a value that would result in a lower error for given training data.
In usual training scenarios you do not alter neuron outputs after they have been calculated using feed-forward, because these are not parameters of the model. Typical reasons to alter neuron values (or even the input) are in order to view what the ideal state might be in a given scenario. If your state can be visualised through some decoder - maybe even another neural network - then this could allow you to see the difference between actual internal state, and a potentially better one. That could be useful in error analysis for example.
So why not to correct the value stored in the cell state?
That's because in usual training scenarios, you are creating a network that predicts an output value. You can calculate the right corrections for training data, but not when predicting, because the whole point of predicting is to estimate a label that you do not already have. As such, you want to alter your function parameters, and not interim values.
It would be very useful if we carry the cell state forward, between minibatches.
Only during training. In a prediction scenario you usually have no way of calculating the necessary gradients. What you don't want is to train a system that then requires using error values and gradients that you do not have in production.
In some scenarios, such as an online system predicting next item $x_{t+1}$ in a sequence, where you could immediately train based on error after you observed the next item and before you continued the prediction sequence for $x_{t+2}$, you could possibly use the approach. I am not sure whether it would help performance, but in principle it could. If it did help, you'd have to compare the improvement versus other simpler changes such as different hyper-parameters on a network that didn't correct internal state using gradients.
In summary, it is possible your idea would work quite well in an online system with near-immediate feedback. In that case you could think of a set of weights as being "rules to update a belief state from data", and the output of hidden layer neurons as being "a current belief state". When errors occur, it does appear to make sense to update both the rules that led to the error and the current belief that resulted from earlier faulty rules. It is perhaps worth an experiment or two. The main caveat is that the two update processes (for weights and LSTM layer state) would interact and/or adapt to each other, so it may not lead to measurably different performance than just adding more LSTM cells to the layer.
