2
$\begingroup$

I'm working with panel data - every row represents a timestamp (observation) and there are multiple rows for a single timestamp (around 20 rows each). I have a total of 8719 unique timestamps.

rows_with_the_same_timestamp

Obs_temp is the target column. "1" column represents the hour. Every timestamp has 20 different observations (with different feature values but same target value).

When I randomly split the data into train & test and predict, Random forest and KNN scored 0.55 and 0.0002 MAE respectively. (Baseline MAE=1.97) Which I was expecting since 20 rows for the same timestamp could end up on both train and the test set.

When I dropped all the columns related to time, they still manage to score almost perfectly. So my question is, how does Random Forest knows that it already has some of the test observations in train set?

Edit: Sample dataset was updated with rows with the same timestamp (filtered on 2 different timestamps).

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Could you update the data example to match what you're describing? I see unique timestamp values... $\endgroup$
    – Ben Reiniger ♦
    Mar 8, 2022 at 14:37
  • $\begingroup$ @BenReiniger Thank you, I've updated the sample data accordingly. $\endgroup$
    – UgurZCifci
    Mar 10, 2022 at 9:30

1 Answer 1

Reset to default
3
$\begingroup$

My reccomendation will be to see what are the important features of the model. Probably use SHAP if you are on random forest https://github.com/slundberg/shap

You might see that one features is extremely highly relevant. This can be a data leakage.

One of the goals of explainable AI is model debugging.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.