# How to combine two differently scaled, but equally important "running" signals into a reward function?

I asked this question on Artificial Intelligence, but got no answer, so I am moving it here.

I have two signals that I want to use to model a reward for a reinforcement learning algorithm.

The first one is the CPU TIME: running mean from this diagram: The second one is the running mean of the MAX RESIDUAL from this diagram: Both signals are equally important, but they have very different scales. I could combine the signals linearly together like this:

$$r = w_\rho \rho + w_\tau \tau$$

where $$r$$ is the reward function, $$\tau$$ is the CPU TIME: running mean, and $$\rho$$ is the MAX RESIDUAL. The problem is, how to set the weights $$w_\tau,w_\rho$$ to make the contributions equally important if $$\rho$$ and $$\tau$$ are on very different scales?

Reinforcement learning algorithms will learn policies based on the reward, and if one signal has values that are much smaller than the other, it will influence the reward much less, which is not the behavior I would like to model.

Edit: Dataset on Kaggle

Edit: comment from Pedro

It seems that a linear combination of signals is possible for the scaled mean CPU Time (mean to get rid of oscillations) and the scaled MAX residual: • Why not move them to a same scale? Aug 13 '20 at 7:44
• try using log scale $r = log(\rho) + log(\tau)$ Aug 13 '20 at 7:45
• @NoahWeber: I tried z-normalisation, but the scales are still significantly different from each other, if z-normalization is used for a running signal. Aug 13 '20 at 7:49
• @PedroHenriqueMonforte: you can see in the diagrams, that even in the logscale, the scales are very different before 200 on the x-axis, so in this crucial period, the contributions to the reward will strongly differ, which is not good... Aug 13 '20 at 7:50
• In linear scale $\rho$ goes from 0 to 0.14 of $\tau$. In log scale it varies from 7 (at the very start) to 1 times. So in log scale, it is actually $\rho$ that takes the domain of the reward function. Actually this reward function says: it is only worth to reduce the CPU time if you don't increase the MAX_RESIDUAL in the same proportion. I can't see why this wouldn't work. Aug 13 '20 at 8:21