How can I reconcile time-based data when the clock on the data source tends to drift and the data may be infrequently retrieved?
I measured the clock to be about 30 minutes behind after 15 hours. UPDATE: the manufacturer confirmed the worst case drift is roughly 10% at the temperature extremes (-40c - 125C)
Due to extremely limited space on the data source device, the data is stored in buckets for each hour and there are 24 buckets, one for each hour in the day. The data is the total count for that hour. UPDATE: we were able to reduce the interval for the buckets to 15 minutes, leaving us with 96 smaller buckets. This is around the point where inaccuracies in the allocation may be acceptable
The device retrieving the data can correct the clock on the data source, but it is not always connected. Data is unlikely to be generated overnight, but may be generated in the morning before the retrieving device is connected. The retrieving device may be connected only infrequently.
One problematic case I have identified is if the retrieving device is connected only in the morning and evening, with more data generated later. In the evening, the data source would be significantly behind, maybe 30m. If the data is retrieved in the first half of an hour, the data source would still be in the previous hour, and data for the rest of the hour would be put in a different bucket after the time is corrected.
Here is a sample of the time drift over about 9 hours: (when the color changes, I stopped rounding down the drift)