-1
$\begingroup$

I have a database of thermal consumption of 100 buildings. Each file has two columns, one is timestamp and the other is usage. My task is to build a prediction model for forecasting the usage for the next day. Having this in mind, now look at this chart:

visulazing the Na values

Can I put this data as-is into my models or should I drop these Na values and then try to train a model? I suppose I can impute the missing values for sites with less than 5% missing data, but is doing this for sites such as site 15 is logical? And what method do you suggest for imputation?

Thank you in advance for your help.

$\endgroup$

1 Answer 1

0
$\begingroup$

I would remove the sites with more than a threshold (say60%) na data (for eg site 32). For the other sites with na data, i would fill the na data columns with average values. This way i would be able to use the entire dataset for training. While testing, use cross-validation to ensure a good accuracy.

Sine you did not mention what methods have you already tried and dont have a query or confusion, i would suggest going through this website: https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html

It has lots of information about how to deal with na values.

$\endgroup$
2
  • $\begingroup$ You see, filling with average will not result in high accuracy in my models. I need to try more complex methods for filling in data. I'm reading some papers about using gated recurrent neural networks. I've read that for time series data it's better to use such methods. $\endgroup$ Commented Feb 12, 2020 at 10:46
  • $\begingroup$ Seems good! Do try out by removing the missing value columns too, and check the accuracy though. I hope it helps! $\endgroup$ Commented Feb 12, 2020 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.