1
$\begingroup$

Following a Tensorflow time series analysis tutorial, I came across a particular way of converting data timestamps into a time-of-day periodic signal, that could help the model interpret the data better than just providing the timestamp.

timestamp_s = date_time.map(pd.Timestamp.timestamp)

day = 24*60*60
year = (365.2425)*day

df['Day sin'] = np.sin(timestamp_s * (2 * np.pi / day))
df['Day cos'] = np.cos(timestamp_s * (2 * np.pi / day))
df['Year sin'] = np.sin(timestamp_s * (2 * np.pi / year))
df['Year cos'] = np.cos(timestamp_s * (2 * np.pi / year))

plt.plot(np.array(df['Day sin'])[:25])
plt.plot(np.array(df['Day cos'])[:25])
plt.xlabel('Time [h]')
plt.title('Time of day signal')

enter image description here

I am not sure I understand how time of day and day of year periodic structure was extracted from the time stamp, so I would appreciate any pointers regarding this.

Lastly, would a simple normalized time_of_day and day_of_year extra new columns from date_time column suffice?

$\endgroup$
1
  • $\begingroup$ The day and year are converted to a cyclical feature by using a sine and cosine function. Reason for this is that in the original representation there is a large distance between the start and end date of a period (e.g. the last day of the first year and the first day of the next year) when in reality we know that they are very close together (only 1 day difference in the example). A sine function is therefore used to capture this information, and in addition a cosine function is used to make the distinction between values exactly 2 * pi apart (i.e. hour 0 and hour 24 in your plot). $\endgroup$
    – Oxbowerce
    Jan 19 at 8:28

1 Answer 1

1
$\begingroup$

timestamp_s = date_time.map(pd.Timestamp.timestamp) takes a column of timestamps and converts them into seconds-since-1970 format (also called unix timestamp).

day is set to 86400 seconds. The remainder from dividing timestamp_s by day is the time of day, where 0 is midnight (in UTC timezone), and 34200 is noon, and 86359 is 23:59:59.

np.sin() takes input in radians, so that is what the multiplying by 2 * np.pi is doing. (There is no need to explicitly take the remainder, of course, because sine is a cyclic function.)

The year calculation is using the same idea, but using the number of seconds in a year. So 0 is Jan 1st, 00:00:00 UTC, 86359 is Jan 1st, 23:59:59 UTC, and er... 31557599 is 23:59:59 UTC on Dec 31st.

Well, kind of. They are using 365.25 to avoid messing around with Feb 29th and leap years. But it does mean that e.g. 10am on Dec 25th is only the same number every 4th year.

Another common one would be to use day * 7 to see if day of week is a useful predictor. E.g. if the data is supermarket sales figures.

I really like their fourier transform graph they show in that article. That clearly shows that weekday would not be at all useful. Ah, just seen it is temperature data being plotted. That makes sense!

Lastly, would a simple normalized time_of_day and day_of_year extra new columns from date_time column suffice?

As in -1.0 for 00:00:00 through to +1.0 for 23:59:59. And -1.0 for Jan 1st through to +1.0 for Dec 31st.

The nice feature the sine waves bring is you don't get that disjoint at midnight, and at new year. You could instead do -1.0 for 00:00:00 through to +1.0 for 12:00:00, then back to -1.0 for 23:59:59 (and something similar with Jun 30th). But, at the point, sine is looking both smoother and simpler to code.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.