1
$\begingroup$

Background Info:

I am working with some climate data where I want to predict crop yields with my dataset containing climate- and satellite-derived features.

This is a time series regression forecasting problem and I want to put it through XGBoost and Lasso to generate my predictions. However, there is a mismatch in the sample frequency between my features and target variable; the features are recorded weekly, and the target variable is recorded yearly.

At the moment, I am using a wide-format table as my input dataset to the ML models, but I feel like my models are not generating accurate predictions when the input table is formatted with the wide-format, since there are less samples compared to if I used a long-format table.

Data Table Reference:

For reference, the wide format table looks something like this, where the suffix represents a week number on a feature:

ID Year crop_yield precip1 precip2 precip3 max_temp1 max_temp2 max_temp3
1100 2000 32.1 5.3 3.0 3.1 13.3 15.3 3.1
1100 2001 31.6 6.6 3.2 1.1 11.3 12.3 6.1
5903 2000 41.2 3.4 0.5 2.1 10.3 18.3 8.1
5903 2001 27.7 1.7 3.8 8.1 12.3 16.3 5.1

And the long format table would look something like this:

ID Year crop_yield Week precip max_temp
1100 2000 32.1 1 5.3 13.3
1100 2000 32.1 2 3.0 15.3
1100 2000 32.1 3 3.1 3.1
1100 2001 31.6 1 6.6 11.3
1100 2001 31.6 2 3.2 12.3
1100 2001 31.6 3 1.1 6.1
5903 2000 41.2 1 3.4 10.3
5903 2000 41.2 2 0.5 18.3
5903 2000 41.2 3 2.1 8.1
5903 2001 27.7 1 1.7 12.3
5903 2001 27.7 2 3.8 16.3
5903 2001 27.7 3 8.1 5.1

Question:

Would it be advisable to use the long format table as the input to my ML models? I feel like the identical crop yields for each associated ID and year would throw my models off.

In addition, is there a better way to frame my data that I haven't explored yet?

$\endgroup$
0
$\begingroup$

To deal with mismatch of timesteps, you have to find a balance between all those mismatched timesteps according to your objective. For example, if your objective is to have a prediction for the next quarter, your learning data should be adapted to quarters.

learning data < 1 quarter: get the mean values of all weeks that last quarters.

learning data > 1 quarter: repeat the yearly value to every corresponding quarter, or make a (logistic?) regression to simulate the quarterly evolution over years.

This is a simple example to explain the idea, but real cases might require further investigation to define the right timestep according to objectives.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.