1
$\begingroup$

I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e.g., temperature, humidity, wind, rain, sun radiation).

Some of my weather features (temperature, humidity, wind) are in hourly intervals and others in daily intervals (rain, sun radiation). My target (daily electricity consumption) is also in hourly intervals.

My goal is to predict daily consumption so I need to aggregate my input data at a daily interval (sum of 24 values for consumption, average of 24 temperatures, ....)

My question is: Do I have to aggregate the data before split train/test of after ?

If I do aggregation first, I will consider only those days with 24 values (one for each hour) and drop others in order to not introduce bias Then I will split train/test. So basically, I will clean my data before splitting.

Am I missing something?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
0
$\begingroup$

If you are training the model based on the aggregated values anyway then it wouldn't make any difference on the final dataset that is fed to the model.

You might consider aggregating earlier in your pipeline as to reduce the amount of work that has to be done afterwards (whatever steps you go through). Aggregating means less data points, so following steps will run faster.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you @n1k31t4 ! Yes the model will be trained with aggregated (daily) values. Do you think using only mean weather datas (ie mean day temperature, ....) is enough ? I'm I wondering if introducing also min/max of the day (ie min and max temperature, humidity,...) will help ? $\endgroup$
    – JaYo
    Sep 19, 2018 at 19:53
  • $\begingroup$ That is something you'll need to try out! Really depends on the way the weather works... And that's not a simple system ;-) Creating extra features like min/max temperature is a good idea. Squeeze as much information as you can out of the data. Your model will have to be able to cope with possible redundancy of information though! $\endgroup$
    – n1k31t4
    Sep 19, 2018 at 20:11
0
$\begingroup$

I think aggregation is basically a column operation and data split a row operation, so they can interchange freely.

Having said that, I don't think you should aggregate because energy consumption will likely depend on the distribution and deviation of temperatures as well as its mean or sum. E.g. cold at nighttime and hot daytime may yield more consumption than the average with low deviation.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you @Pieter21 ! Actually my datas are stored on csv file : 1 row per hour . So aggregating will also be a row operation (ie sum of 24 rows of hour consumption for the same day, ...). My goal is to predict day consumption. To go in the direction of your suggestion, I was thinking of introducing also min and max values (in addition to mean) for weather features. So to simplify : predict daily consumption based on mean/min/max temperatures. $\endgroup$
    – JaYo
    Sep 19, 2018 at 19:59
  • $\begingroup$ Alternatively, you can do the aggregation after the complete model (predict 24 consumption values for each hour of the day, and add up) $\endgroup$
    – Pieter21
    Sep 19, 2018 at 20:09
  • $\begingroup$ following your suggestion, how can I use rain and sun radiation features ? This features are measured at daily timestep ...Thanks $\endgroup$
    – JaYo
    Sep 19, 2018 at 20:19
0
$\begingroup$

As part of your data munging process, you can resample (upsample or downsample) the data into the desired intervals. When either downsampling or upsampling, the syntax is practically identical, but the methods called are different. Both use the concept of "method chaining" - df.method1().method2().method3() - to direct the output from one method call to the input of the next, and so on, as a sequence of operations, one feeding into the next.

For example, if you have hourly data and want daily data, pandas will not just guess how to throw out 23 of 24 points. You must specify this in the method. One approach could be to take the mean:

new_data_frame = df['Temperature'].resample('D').mean()

You can use other methods such as .max(), .min(), .count(), etc.

If you upsample (i.e., add more intervals, such as going from every 6 hrs to 3 hrs), you will have to fill in the missing values. You can do this using either a backfill or forward fill. For example, you could do the following:

new_data_frame.resample('3H').ffill()

Once your data is resampled accordingly to your satisfaction you can proceed as normal with your train/test split.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.