0
$\begingroup$

I'm building a model that fills the missing values from a Dataframe that contains the number of visitors for different stores, each day:

day store_a store_b store_c
2021-01-01 100 200 300
2021-01-02 110 220 290
2021-01-03 50 110 170
2021-01-04 NAN 220 290
2021-01-05 7 16 NAN
2021-01-06 90 NAN NAN

I'm using the IterativeImputer class from scikit-learn, this method of imputation puts aside one column at each step and train an estimator on the other columns to predict the column that was put aside.

My question is: should I transpose my dataframe or not?

If I keep my dataframe (1 line = 1 day, 1 column = 1 store), this means that we can completely predict the number of visitors in a store in a particular day just by looking at the other stores.

But if I transpose my dataframe (1 line = 1 store, 1 line = 1 day), this means that we can predict the number of visitors just by looking at the history of one store.

I guess one simple way to check if I should transpose is comparing the RMSE of the two methods but I wanted some explanation instead of "method A works better, move along."

Improve this question
$\endgroup$
2
  • $\begingroup$ How many rows/columns do you have? $\endgroup$
    – WBM
    Commented Apr 15, 2021 at 16:29
  • $\begingroup$ In my training dataset I had 93 days and 300 stores, so my dataframe has 93 lines and 300 columns. $\endgroup$
    – Be Chiller Too
    Commented Apr 16, 2021 at 14:22

1 Answer 1

Reset to default
0
$\begingroup$

Just by eyeballing the data i would think the number of visitors for store A on jan 4th is around 110. Basically it is always approximately half of Store B on any given day.

It seems like the number of visitors across stores are correlated, so you could potentially use simple linear regression to get a reasonable estimate for any given day.

Improve this answer
$\endgroup$
3
  • $\begingroup$ thanks for the insight but this was just a minimal example :) The real data is much more complex, with more than a thousand stores for multiple years $\endgroup$
    – Be Chiller Too
    Commented Apr 16, 2021 at 14:21
  • $\begingroup$ Even if you had thousands of stores, you could still find pairs that are highly correlated and use them to impute the other. $\endgroup$
    – Jayaram Iyer
    Commented Apr 16, 2021 at 17:05
  • $\begingroup$ Indeed, that is what I am trying to do $\endgroup$
    – Be Chiller Too
    Commented Apr 20, 2021 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.