Learning from aggregated data

Question

Online and in the literature there seems to be a general consensus that training a machine learning model using aggregated data is harder and/or fundamentally different from training on raw event data. I am unable to intuit why this would be the case.

Lets say for example we have a data set from the online advertising domain:

feature_1	feature_2	feature_3	click
a	g	z	0
b	f	z	1

We could group by each of the categorical features and instead of a binary click target have a click through rate (CTR) target which is calculate as sum(clicks)/sum(displays)*100. We could even decide the threshold for CTR, for good / bad CTR and convert that aggregated data table back into a binary classification problem.

Now I am unable to understand why the two datasets differ when fed into a model, in the raw event case, the model will see each example and over many passes learn the aggregation, i.e what is the probability of a click given a set of features.

Now this question also bring me to another thought - what is a model doing differently here VS just aggregating the historical data and calculating the historical probs of a click? If we have all possible feature crosses in our dataset, is even the most complex DL model somehow able to learn something more superior, and if yes - how?

Answer 1

I would disagree with the statement that "training a model on aggregated data is harder than training a model on raw data". It is different, but I wouldn't say fundametally.

It ultimately depends on the application, or the problem that you are trying to solve.

E.g., for the small data sample you presented, I see no problem in training a logistic regression classifier on the "raw" table to predict the click variable, and on a data set that is aggregated somehow to predict, as you say, whether the click-through-rate is good/bad. It is 100% possible and legit thing to do.

Consider the following two scenarios.

Scenario 1: you want to have a real-time model that based on feature_1, feature_2, ... feature_n will predict whether a user will click on some ad. The model runs in real-time, e.g., a user is doing something on a web-site, some features are being calculated and fed into a model whose decisions influence which ads are being shown on the site. You want this model to be trained on a "raw" data set, in the format as you've presented.

Scenario 2: you want to have a daily prediction of the click-through-rate for the ads being shown on a web-site, so that you can estimate how successful a particular marketing campaign is. For this case, you need to aggregate your data on a daily level, so that your test & train data sets are from the same distribution. Some script is run daily, runs a model on the data from the previous day and compiles a report for the following day.

Scenario 1 runs as a real-time model, Scenario 2 runs the model as a batch job. Both models can be built from the same algorithm, e.g., as I mentioned logistic regression, or any other basically. The intrinsic difference is in the utility, or the application of the model. How you interpret its results is how you get some value out of it. In Scenario 1, you want a model to predict/model a single user interaction, in Scenario 2 you want to capture the complete user behaviour in a single day.

Tightly connected to the application of the model is the topic of feature engineering. I wouldn't go too much into this, as it is domain/problem-specific endeavour. If you are interested in the theory of learning, I suggest you go through this Wikipedia page and the links therein. You can also take a look at the chapter 5 about learning in this book. Its been a while since I've read it, so my knowledge on this subject is bit fuzzy.

Hope this helps you.

Answer 2

There are a few reasons why training a machine learning model on aggregated data might be different or harder than training on raw event data:

1. Loss of Information: Aggregated data loses some of the information that is present in the raw event data. For example, aggregating clicks and displays by category will give you the click-through rate for each category, but you lose information about the individual clicks and displays that went into that calculation. This loss of information can make it harder for the model to learn patterns and correlations in the data.

2. Bias: Aggregating data can introduce bias into the dataset. For example, if you aggregate clicks and displays by category, you may have categories that are overrepresented or underrepresented in the aggregated data. This can lead to bias in the model's predictions.

3. Non-linear Relationships: In some cases, the relationship between the features and the target variable may be non-linear. Aggregating the data can make it harder for the model to learn these non-linear relationships.

Regarding your second question, a machine learning model is able to learn more than just the historical probabilities of a click. Even if you have all possible feature crosses in your dataset, a complex DL model is able to learn more sophisticated relationships between the features and the target variable. For example, a deep neural network can learn hierarchical representations of the data that capture increasingly abstract features. This can lead to better performance on the task at hand. Additionally, a model can learn to generalize to new data that it has not seen before, whereas simply aggregating historical data would not allow for this kind of generalization.