1
$\begingroup$

I asked this question on Artificial Intelligence, but got no answer, so I am moving it here.

I have two signals that I want to use to model a reward for a reinforcement learning algorithm.

The first one is the CPU TIME: running mean from this diagram:

enter image description here

The second one is the running mean of the MAX RESIDUAL from this diagram:

enter image description here

Both signals are equally important, but they have very different scales. I could combine the signals linearly together like this:

$r = w_\rho \rho + w_\tau \tau$

where $r$ is the reward function, $\tau$ is the CPU TIME: running mean, and $\rho$ is the MAX RESIDUAL. The problem is, how to set the weights $w_\tau,w_\rho$ to make the contributions equally important if $\rho$ and $\tau$ are on very different scales?

Reinforcement learning algorithms will learn policies based on the reward, and if one signal has values that are much smaller than the other, it will influence the reward much less, which is not the behavior I would like to model.

Edit: Dataset on Kaggle

Edit: comment from Pedro

It seems that a linear combination of signals is possible for the scaled mean CPU Time (mean to get rid of oscillations) and the scaled MAX residual:

enter image description here

$\endgroup$
7
  • $\begingroup$ Why not move them to a same scale? $\endgroup$
    – Noah Weber
    Aug 13 '20 at 7:44
  • $\begingroup$ try using log scale $ r = log(\rho) + log(\tau) $ $\endgroup$ Aug 13 '20 at 7:45
  • $\begingroup$ @NoahWeber: I tried z-normalisation, but the scales are still significantly different from each other, if z-normalization is used for a running signal. $\endgroup$
    – tmaric
    Aug 13 '20 at 7:49
  • $\begingroup$ @PedroHenriqueMonforte: you can see in the diagrams, that even in the logscale, the scales are very different before 200 on the x-axis, so in this crucial period, the contributions to the reward will strongly differ, which is not good... $\endgroup$
    – tmaric
    Aug 13 '20 at 7:50
  • 1
    $\begingroup$ In linear scale $\rho$ goes from 0 to 0.14 of $\tau$. In log scale it varies from 7 (at the very start) to 1 times. So in log scale, it is actually $\rho$ that takes the domain of the reward function. Actually this reward function says: it is only worth to reduce the CPU time if you don't increase the MAX_RESIDUAL in the same proportion. I can't see why this wouldn't work. $\endgroup$ Aug 13 '20 at 8:21
1
$\begingroup$

Using z-normalisation ensures that they have same mean and Standard Deviation, but ofcourse values will be still different since mean and Standard Deviation depend on the Distribution of the data.

Alternative is to use feature scaling, where you force the values between 0 and 1 for both signals.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.