Linear Regression for Time Series Prediction

Your sliding window approach for time series prediction seems promising, but the rather low MSE values suggest the need for further scrutiny. Following, I will try to provide a detailed code review, focusing on debugging, validation, and potential suggestions for improvement.

Mathematical Framework

Linear Regression

Linear regression models the relationship between predictors $X$ and a target variable $Y$ as:

$$ \hat{Y} = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \epsilon $$

where:

• $\beta_0$ is the intercept,
• $\beta_i$ are the coefficients of the predictors $X_i$,
• $p$ is the number of predictors.
• $\epsilon$ represents the error term (assumed to follow a normal distribution, $\epsilon \sim \mathcal{N}(0, \sigma^2)$).

In your problem, $X$ consists of the flattened matrices for three consecutive days ($3 \times 24 \times 25 = 1800$ features), and $Y$ represents the flattened matrix of the next day ($1 \times 24 \times 25 = 600$ features).

It is often more convenient to using matrix notation:

Matrix Form

$$ \mathbf{Y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon} $$

where:

• $\mathbf{Y}$ is the $n \times 1$ vector of observed target values,
• $\mathbf{X}$ is the $n \times (p + 1)$ matrix of predictors, including a column of ones for the intercept,
• $\boldsymbol{\beta}$ is the $(p + 1) \times 1$ vector of coefficients (including the intercept),
• $\boldsymbol{\epsilon}$ is the $n \times 1$ vector of error terms.

The goal is to find $\boldsymbol{\beta}$ such that the residual sum of squares (RSS) is minimised:

$$ \text{RSS} = (\mathbf{Y} - \mathbf{X} \boldsymbol{\beta})^\top (\mathbf{Y} - \mathbf{X} \boldsymbol{\beta}) $$

The solution is obtained via the normal equation:

$$ \boldsymbol{\beta} = (\mathbf{X}^\top \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{Y} $$

Mean Squared Error (MSE)

The model’s performance is evaluated using the MSE:

$$ \text{MSE} = \frac{1}{n} \sum_{i=1}^{n} \left( Y_i - \hat{Y}_i \right)^2 $$

where:

• $n$ is the number of observations,
• $Y_i$ and $\hat{Y}_i$ are the actual and predicted values, respectively.

Interpreting "Low MSE"

Before evaluating the results, it is important to define what constitutes "low MSE" in this context. MSE values are relative and depend on several factors:

1. Scale of the Target Variable:

   • MSE depends on the magnitude of the target values. For example, an MSE of 0.001 is small for pixel values normalised between 0 and 1 but could be very large for targets in the range of 0 to 0.01.
   • Normalising the target values or using metrics like $R^2$ can provide clearer insight: $$ R^2 = 1 - \frac{\text{MSE}}{\text{Variance of } Y} $$

2. Baseline Comparison:

   • Compare the MSE to simple benchmarks, such as:
     • Persistence Model: Predicting the next day as identical to the previous day.
     • Moving Average: Averaging the past three days.
   • If the regression model’s MSE is close to these baselines, its performance may not be exceptional.

3. Variability in the Data:

   • Data with low variability or strong temporal patterns (e.g., high autocorrelation) will naturally lead to lower MSE values, as the task is easier to predict.
   • Conversely, noisy or irregular data would make low MSE surprising.

4. Model Assumptions:

   • Linear regression assumes linear relationships and normally distributed, homoscedastic residuals. If these assumptions are violated but MSE remains low, issues like overfitting or data leakage should be considered.

Clarifying "Low MSE" in This Case

• Contextualise the MSE values by:
  1. Providing Scale Context: Assess the range of the target variable.
  2. Baseline Comparison: Compare against naive models to gauge relative performance.
  3. Evaluating Variance: Compare MSE to the variance of the target variable to determine its significance.
  4. Benchmarking: Review similar datasets or tasks for comparison.

Defining "low MSE" relative to these factors ensures the results are interpreted meaningfully and any anomalies are identified.

Debugging Suggestions

1. Alignment Check

Misaligned predictors and targets can lead to incorrect training/testing splits. Verify that data indices are properly aligned:

for i in range(3, num_days):
    print(f"Train: Days {i-3}, {i-2}, {i-1}; Predict: Day {i}")

2. Temporal Dependency Analysis

• ACF and PACF Plots: Use autocorrelation (ACF) and partial autocorrelation (PACF) to understand the temporal structure of your data:

  • ACF (Autocorrelation Function): Measures correlation between time series values at different lags.
  • PACF (Partial Autocorrelation Function): Isolates the effect of individual lags.

• How to Generate:

  import statsmodels.api as sm
  import matplotlib.pyplot as plt
  
  series = data_flattened[:, 0]  # Use the first feature across all days
  fig, axes = plt.subplots(2, 1, figsize=(10, 8))
  sm.graphics.tsa.plot_acf(series, lags=30, ax=axes[0])
  sm.graphics.tsa.plot_pacf(series, lags=30, ax=axes[1])
  plt.show()
  

• Interpretation:

  • Strong ACF correlations at short lags suggest that temporal dependencies are significant. Adding lagged variables explicitly as features may improve performance.

Validation Strategy

1. Walk-Forward Validation

Train and evaluate the model iteratively using only past data to prevent information leakage:

for i in range(3, num_days):
    X_train = data_flattened[i-3:i].flatten()
    y_train = data_flattened[i]
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)  # Predict day i+1

• Why It Matters: This ensures that the model is validated in a way that mimics real-world forecasting, where future data is unavailable during training.

Recommendations for Improvement

Verify the problem

• Ensure that you really are dealing with unexpectedly low MSE - check the Interpreting "Low MSE" section.

Validation

• Use strict chronological splits for training and testing to prevent leakage.
• Test the model on an out-of-sample dataset (eg., reserve the last few days for testing).

Benchmarking

• Compare against baselines like naive persistence or simple moving averages.
• Use MSE reductions relative to the baseline as a measure of model improvement.

Advanced Models

• CNNs: Leverage spatial structure in your 24x25 matrices for better feature extraction, particularly when relationships between neighbouring pixels are relevant. (LeCun et al., 1998)
• LSTMs: Long Short-Term Memory networks (LSTMs) are designed to capture long-term dependencies in sequential data, making them suitable for modelling temporal patterns in your dataset. (Hochreiter & Schmidhuber, 1997)
• ARIMA/SARIMA: Handle temporal dependencies like trends and seasonality explicitly. These models are well-suited for univariate time series but can be extended to multivariate settings. (Box et al., 2015)
• ARCH/GARCH Models: If your data exhibits time-varying volatility, Autoregressive Conditional Heteroskedasticity (ARCH) or Generalised ARCH (GARCH) models may be appropriate to model conditional variance over time. (Engle, 1982; Bollerslev, 1986)
• RNNs: Recurrent Neural Networks (RNNs) are ideal for sequential data but may struggle with long-term dependencies compared to LSTMs. (Rumelhart et al., 1986)

Each of these models provides unique strengths, and choosing the right one depends on the specific temporal and spatial complexities of your dataset.

Summing Up

While your implementation is a good start, the (apparently) low MSE values suggest potential issues such as data leakage or overly simplistic temporal patterns. First I would verify that the MSE is indeed low as detailed in the Interpreting "Low MSE" section. By verifying data alignment, analysing temporal dependencies with ACF/PACF, and benchmarking against baselines, you can hopefully obtain "better" results (and a better understand your model's strengths and limitations). Exploring more advanced models may further enhance predictive performance and address the spatial and temporal complexities of your data.

References:

1. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324. https://doi.org/10.1109/5.726791
2. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735
3. Box, G. E. P., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time Series Analysis: Forecasting and Control (5th ed.). Wiley.
4. Engle, R. F. (1982). Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation. Econometrica, 50(4), 987-1007. https://doi.org/10.2307/1912773
5. Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 31(3), 307-327. https://doi.org/10.1016/0304-4076(86)90063-1 Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536. https://doi.org/10.1038/323533a0