Feature Selection on Aggregated Targetdata

Question

I have a question about feature selection on a dataset where the target variable is aggregated by the sum of different data points. I want to predict the number of sales depending on a variety of features like:

• week
• price per unit
• store country
• store city
• 2-3 other categorical meta-data
• other features

I am aware that this data should be interpreted as time series but because of the lack of available historical data, no model can compete with the naive approach.

The problem I am facing is, that the target variable is the sum of grouped features like month, price, country and city. If I add or remove one of the grouped features I will get some identical data points and the dataset will be inconsistent, so I have to perform a grouped aggregation (sum) to get consistency back. This operation will change the target variable and the number of data points. I have no idea how to validate regression models trained on different subsets of the features because the underlying dataset is not equal. I know there are other feature selection techniques than wrapper methods like filter or embedded which provides workarounds but I would like to know if there are techniques to resolve this issue.

Example:

    week  price_per_unit store_country  sales
0      1             3.0            C1     30
1      1             3.0            C2     32
2      1             4.0            C1     23
3      2             3.5            C1     19
4      2             3.5            C2     27
5      2             6.5            C1     35
6      3             2.0            C1     17
7      3             3.0            C1     15
8      3             4.0            C2      7
9      4             2.0            C1     19
10     4             5.0            C1     41
11     4             5.0            C2     21

After dropping the column store_country:

    week  price_per_unit  sales
0      1             3.0     30
1      1             4.0     23
2      1             3.0     32
3      2             6.5     35
4      2             3.5     19
5      2             3.5     27
6      3             2.0     17
7      3             3.0     15
8      3             4.0      7
9      4             5.0     41
10     4             2.0     19
11     4             5.0     21

Now there are duplicate data points and the sale column is wrong because I need the sum so after aggrgation I have:

   week  price_per_unit  sales
0     1             3.0     62
1     1             4.0     23
2     2             3.5     46
3     2             6.5     35
4     3             2.0     17
5     3             3.0     15
6     3             4.0      7
7     4             2.0     19
8     4             5.0     62

Let's assume I want to perform forward or backward selection with linear regression. In every step a column will be added or removed. So the number of rows depends on the feature which is selected. I can't think of a metric to compare these regression models.

Answer 1

It seems to me that there's a bit of confusion between these two steps:

1. Designing the problem and preparing the data accordingly
2. Applying ML methods to the data (feature selection, etc.)

The two steps must be distinct: any aggregation must take place during step 1 so that at the end of step 1 the dataset is fixed.

In your example in step 1 you decided to predict the total volume of sale across stores given the week and price per unit. Note that you could also choose other options:

• average volume of sale by store as target
• keep one row for each store: depending on the goal, sometimes it's ok to have inconsistencies in the data, regression can deal with these.
• you could decide that an instance for a store spans over the N past weeks and provide the sales for these weeks, the target being the week after that (that would probably help btw).
• ...

And of course you can do this as many times as needed for studying different problems, each time obtaining a different version of the data which represents a particular problem. It's only once you have formally defined your problem and formatted your data for it that the ML part starts. For instance stepwise regression (or any other technique) cannot work if the data is modified/aggregated during the steps.