I have a system that take electric signal and output parameters (regression). However I run into an issue. The parameters I want to predict are not on the same magnitude. I would like also to use only one neural networks if possible.
Parameter 1: Mean = 12.53673 Minimum = 10.00461 Maximum = 14.98899 Parameter 2: Mean = 148656955394038 Minimum = 75029133522564 Maximum = 224934092847235 Parameter 3: Mean = 1.475720134278e+17 Minimum = 7.506184799345e+16 Maximum = 2.249190781380e+17 ...
If I use any loss function such as RMSE, MSE, MAE, MSLE the losses of the big parameters takeover all the rest. The network therefore randomly guess all the smaller parameters
I am searching a good way to find those parameters using one network. I tried using "mean absolute percentage error". But the learning is exceptionally slow, and it often get stuck into extremely low value and extremely high value.
I am maybe thinking about creating one loss per target. Freeze all the network except the output layer and each epoch let one of the loss train the whole network. But I am not sure it is a good idea.
Currently I am normalizing the target before predicting it, but I don't like this solution very much. I would prefer having a slightly more complicated architecture than normalizing and denormalizing (afraid of data leakage) and it seems more robust to directly output the right value.
Moreover In the future I will have an issue similar except I will need to predict series that possesses different order of magnitude (and not only some distinct value). So I hope being able to reuse the technique I am learning
Any help would be very appreciated! Thanks :)
EDIT: For those who happens to have the same issue, this is the code I went with (keras)
class Model: def __init__(self, shape_features, shape_targets, mean_targets, std_targets): self.shape_features = shape_features self.shape_targets = shape_targets self.mean_targets = mean_targets self.std_targets = std_targets def build(self): inputs = Input(shape=self.shape_features, name='Input') x = Conv1D(filters=16, kernel_size=3, activation='relu', name='Convolution1D_1')(inputs) x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_1')(x) x = Conv1D(filters=32, kernel_size=3, activation='relu', name='Convolution1D_2')(x) x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_2')(x) x = Conv1D(filters=64, kernel_size=3, activation='relu', name='Convolution1D_3')(x) x = MaxPooling1D(pool_size=2, strides=2, padding='valid', name='MaxPooling1D_3')(x) x = Flatten(name='Flatten')(x) x = Dense(128, activation='relu', name='Dense_1')(x) x = Dense(64, activation='relu', name='Dense_2')(x) x = Dropout(0.2, name='Dropout')(x) outputs = Dense(self.shape_targets, activation='linear')(x) model = tf.keras.Model(inputs=inputs, outputs=outputs) def de_normalizing(tensor): return tensor * self.std_targets + self.mean_targets predictions = Lambda(de_normalizing)(outputs) model_prediction = tf.keras.Model(inputs=inputs, outputs=predictions) def custom_loss(y_true, y_pred): y_true = (y_true-self.mean_targets)/self.std_targets return K.mean(K.square(y_pred - y_true), axis=-1) return model, model_prediction, custom_loss
One model is used for training, the other is used for predictions