As always in ML modelling problems: it depends.
The critical factor here is that you are predicting based on properties of a sequence. The sequence does not need to be sampled at fixed time steps, or even be time based at all. E.g. in natural language processing the sequence of letters or words is only very loosely associated with the timing of the same items when spoken or read. The sequence does need to be drawn from a self-consistent source where moving from one item to next item in the sequence has the same meaning with respect to what you are trying to predict.
If the interruptions you see would cause large differences in the expected distribution of your input vector X or output Y, then you may not be able to use them as-is. You may still be able to rescue them as input data for a RNN, and make predictions based on them, if you add whether or not there has been an interruption - and if it might be relevant to your purpose, for how long - as additional features of input items in X.
Another option is to follow the time sequence strictly but add placeholder value of X (with a different flag set) for "unkown", maybe along with some rough guess for the missing features (or even a ML prediction from a different model), which you might do if the quantity being aggregated over is still active and important where there are missing records, but the aggregation service you are using as input has failed.
Depending on how much data you have available, and whether you need to make predictions at times when your aggregation service has failed, you could do anything from discarding incomplete data to ignoring the time differences, to interpolating missing records.
The first question you need to answer is "what does it mean for a sequence item to be missing":
If the whole system that provided the aggregation data is switched off, and there is no concept of pending items that will enter the system the moment it is switched on, then you can probably safely ignore the missing records and ignore the precise timing.
If your prediction target is sensitive to elapsed time, and yo uneed to make predictions even when there have been interruptions to the aggregation system, then you will want to augment the sequence data somehow to account for the missing records, and train/predict with that added feature engineering.
If only the aggregation is switched off (or has a chance of failure causing gaps in the record), and you don't need to run any predictions when it is not available, then you may be able to make the rule that the prediction system will only function when there are 6 recent consecutive records available, and discard the data that is causing you to worry. This makes sense if interruptions are rare or the predictive model is not critical.
If you are not sure which of these things to do, then the next approach is to do all of them that seem that they could be intuitively correct, tuning each model separately, then see how each different approach performs on your test set.
