Consider simplifying seasonality by reducing to just 7-day totals.
You have encountered a very common modeling situation
where there are multiple Generating Processes out in
the real world (such as discarding gift wrapping in December)
which all contribute with varying magnitudes to your target variable Y.
You will want to identify a subset of them, rank order their magnitudes,
and start modeling them one by one, in an effort to explain
more and more variance.
Suppose that is_working_day has the greatest predictive power,
explaining most of the variance.
Then you will want to synthesize such a column from
your raw yyyy-mm-dd datestamps,
and let random forests, or some other modeling technique,
have access to it when inferring future forecasts.
Notice that you'll have to augment the initial dataset
with things like "bank holidays", "federal holidays",
"sanitation worker holidays", which may overlap while
being non-identical.
Now you have residuals, the delta between something observed
in your input data and its corresponding model prediction.
Train a new model on the residuals,
perhaps based on day_number or daily_high_celsius
to predict seasonality over the course of a year.
You will have new residuals,
which will be smaller if your more complex model
has greater predictive power.
At this point, having de-trended the weekly and annual seasonality,
you may be ready to account for year-over-year
population growth with an ARIMA model on the residuals.
Alternatively, you may prefer to gather apparently relevant
features, including census population, highway traffic volumes,
and tax revenue, up front.
And then feed them all at once to e.g. XGBoost,
letting the model figure out which ones are truly informative.
You may find that the following (non-ARIMA) model
adequately captures the dynamics of interest.
#! /usr/bin/env python
from pathlib import Path
from pandas.tseries.holiday import USFederalHolidayCalendar
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split
import pandas as pd
CSV = Path("open_source_austin_daily_waste_2003_jan_2021_jul.csv")
def get_garbage_df() -> pd.DataFrame:
df = pd.read_csv(CSV)
df["ticket_date"] = pd.to_datetime(df.ticket_date)
df["elapsed"] = (df.ticket_date - df.ticket_date.min()).dt.days
df["month"] = df.ticket_date.dt.month
holidays = USFederalHolidayCalendar().holidays(
start=df.ticket_date.min(),
end=df.ticket_date.max(),
)
df["holiday"] = df.ticket_date.isin(holidays)
df["weekday"] = df.ticket_date.dt.weekday
return df
def main() -> None:
df = get_garbage_df()
y = df.net_weight_kg
df = df.drop(columns=["net_weight_kg", "ticket_date"])
X = df
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
reg = GradientBoostingRegressor(random_state=0)
reg.fit(X_train, y_train)
print(int(reg.predict(X_test[1:2])[0]))
print(reg.score(X_test, y_test))
if __name__ == "__main__":
main()
First, I want to fit an ARIMA model to the dataset. But it cannot capture well the values since they are following a multimodal distribution. So I tried to segment it on two datasets: one dataset for the weekends and an other one for business days so that I would have to fit two forecasting models for each. However, the segmentation criterion does not work and business days dataset is also bimodal.
