# Issue while predicting multiple values which possess different order of magnitude (regression)

I have a system that take electric signal and output parameters (regression). However I run into an issue. The parameters I want to predict are not on the same magnitude. I would like also to use only one neural networks if possible.

Parameter 1:
Mean    = 12.53673
Minimum = 10.00461
Maximum = 14.98899

Parameter 2:
Mean    = 148656955394038
Minimum = 75029133522564
Maximum = 224934092847235

Parameter 3:
Mean    = 1.475720134278e+17
Minimum = 7.506184799345e+16
Maximum = 2.249190781380e+17

...


If I use any loss function such as RMSE, MSE, MAE, MSLE the losses of the big parameters takeover all the rest. The network therefore randomly guess all the smaller parameters

I am searching a good way to find those parameters using one network. I tried using "mean absolute percentage error". But the learning is exceptionally slow, and it often get stuck into extremely low value and extremely high value.

I am maybe thinking about creating one loss per target. Freeze all the network except the output layer and each epoch let one of the loss train the whole network. But I am not sure it is a good idea.

Currently I am normalizing the target before predicting it, but I don't like this solution very much. I would prefer having a slightly more complicated architecture than normalizing and denormalizing (afraid of data leakage) and it seems more robust to directly output the right value.

Moreover In the future I will have an issue similar except I will need to predict series that possesses different order of magnitude (and not only some distinct value). So I hope being able to reuse the technique I am learning

Any help would be very appreciated! Thanks :)

EDIT: For those who happens to have the same issue, this is the code I went with (keras)

class Model:
def __init__(self, shape_features, shape_targets, mean_targets, std_targets):
self.shape_features = shape_features
self.shape_targets = shape_targets
self.mean_targets = mean_targets
self.std_targets = std_targets

def build(self):
inputs = Input(shape=self.shape_features, name='Input')
x = Conv1D(filters=16, kernel_size=3, activation='relu', name='Convolution1D_1')(inputs)
x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_1')(x)
x = Conv1D(filters=32, kernel_size=3, activation='relu', name='Convolution1D_2')(x)
x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_2')(x)
x = Conv1D(filters=64, kernel_size=3, activation='relu', name='Convolution1D_3')(x)
x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_3')(x)
x = Flatten(name='Flatten')(x)
x = Dense(128, activation='relu', name='Dense_1')(x)
x = Dense(64, activation='relu', name='Dense_2')(x)
x = Dropout(0.2, name='Dropout')(x)
outputs = Dense(self.shape_targets, activation='linear')(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

def de_normalizing(tensor):
return tensor * self.std_targets + self.mean_targets

predictions = Lambda(de_normalizing)(outputs)
model_prediction = tf.keras.Model(inputs=inputs, outputs=predictions)

def custom_loss(y_true, y_pred):
y_true = (y_true-self.mean_targets)/self.std_targets
return K.mean(K.square(y_pred - y_true), axis=-1)

return model, model_prediction, custom_loss


One model is used for training, the other is used for predictions

I am maybe thinking about creating one loss per target. Freeze all the network except the output layer and each epoch let one of the loss train the whole network. But I am not sure it is a good idea.

This is one option. It should work but it may be slower to converge as you only make use of only a third of your losses each epoch. Also, it is not completely unlikely that your losses may work "against each other". Meaning that they could somewhat cancel each other out, which can make convergence take longer.

Currently I am normalizing the target before predicting it, but I don't like this solution very much. I would prefer having a slightly more complicated architecture than normalizing and denormalizing (afraid of data leakage) and it seems more robust to directly output the right value.

That's what I would do. I don't think you should be afraid of data leakage. Computers have enough precision in them that normalizing/denormalizing shouldn't create any issues. Also, keep in mind that computers have greater precision for floats close to 0. So normalizing your data may actually be good.

And when you say, I'd rather directly output the right value. You can do both! Your network could directly output the denormalized value, but the loss used to train the network can be computed on any layer before that.

• Thank you so much for your insights! I read some more posts and scientific papers and it seems that normalizing and denormalizing seems the way to go, so I will certainly do that. For the last paragraph you say, I'm extremely interested by the technique, however I am not sure how to implement in a clean way (keras). Maybe computing in advance the mean and std then storing them in placeholders ? If you have advice or if you know ressources that tackle that I'm also very interested. Thanks again :) – th0mash Jun 5 '20 at 8:19
• I am entirely sure of how to do this in Keras (I am more of a PyTorch man myself where this is easier to do). However, it doesn't have to be an actual layer in a network. You could wrap your network in an object where the denormalization is a postprocessing method. So your actual network doesn't denormalize, but it's wrapped in an object so you don't have to worry about it. And indeed, you'd store the mean and standard deviation as variable of that object for each parameter. This can even be part of the learning process – Valentin Calomme Jun 5 '20 at 9:18
• The wrapping method seems to be a very good way to do it, I didn't thought of that, thanks :) Last small question, if you have some time, how do you think you would you tackle this if you were using Pytorch (since you said it is easier) ? – th0mash Jun 5 '20 at 9:37
• PyTorch offers it out-the-box. You can add the (de-normalization) in the forward method. And since you need to define your training loop, you can also decide how your loss is computed. For Keras, I saw this Stackoverflow post which might help: stackoverflow.com/questions/40004706/keras-custom-scaling-layer – Valentin Calomme Jun 5 '20 at 10:46
• Ok that seems more convenient indeed. I am trying something new right now: I created a loss function and I normalize the targets inside the loss function. I thought that could have been a good solution but it seems to not work as well as I anticipated. The loss is correct and everything can run but the gradient doesn't reflect how much the weight needs to change (slowly increase but need to reach 10^17) – th0mash Jun 5 '20 at 13:26