I am dealing with bike-share data. I have 2
trips_df(subset shown), total entries =
weather_df(subset shown), total entries =
I am trying to calculate and attach the
total_precipitation for each trip, as a column. I do this by looking up the
end_timestamp datetime for each trip from
trips_df, in the
weather_df, and summing the
precipitation_amount within those times, then attaching that value back in the
trips_df under the new column. I can attach the code if it's helpful.
I ran the code on a subset of 65 entries and it took ~1.3s. (
CPU times: user 1.27 s, sys: 8.77 ms, total: 1.28 s, Wall time: 1.28 s). Extrapolating that performance to my entire data, it would take (1.3 * 1048568)/65 = 20971.36seconds or 5.8hours.
What am I supposed to do in this situation? For context, this is a Kaggle style data science project so I'll have to do further data wrangling, and data extraction then apply a predictive model.