0
$\begingroup$

I am trying to train a simple neural network model for multiclass classification.

I have x1,x2,x3,x4 columns with 4 classes to predict.

If just train on x1,x2,x3,x4 then I get accuracy of 88%

With some domain knowledge I can create three new features which I know will definitely help the model train better.

The three new features are :-

  1. df['x12'] = abs (df['x1']-df['x2'])

  2. df['x13'] = abs (df['x1']-df['x3'])

  3. df['x14'] = abs (df['x1']-df['x4'])

If i train on x1,x2,x3,x4 and abs(x1-x2),abs(x1-x3),abs(x1-x4) then I get an accuracy of 98%

I want to get 98% accuracy without abs(x1-x2),abs(x1-x3),abs(x1-x4)

With these new manually created features I get a validation accuracy of 98% which is great.

However, When I remove these features the validation accuracy jumps down to 88%.

My question is that the function abs(x1-x2) should be very simple enough for the model to learn on its own without me manually doing feature engineering.

Then why does the accuracy drop when I remove these three (very simple) features?

Is the model not capable enough to learn it on its own?

This is what my model looks like :-


    inputs = Input(shape=input_shape)

    x = Dense(units=64)(inputs)
    x = PReLU()(x)  
    x = BatchNormalization()(x)
    x = Dropout(rate=0.3)(x)

    x = Dense(units=64)(x)
    x = PReLU()(x)  
    x = BatchNormalization()(x)
    x = Dropout(rate=0.3)(x)

    x = Dense(units=64)(x)
    x = PReLU()(x)  
    x = BatchNormalization()(x)
    x = Dropout(rate=0.3)(x)

    x = Dense(units=64)(x)
    x = PReLU()(x)  
    x = BatchNormalization()(x)
    x = Dropout(rate=0.3)(x)

    x = Dense(units=32)(x)
    x = PReLU()(x)  
    x = BatchNormalization()(x)
    x = Dropout(rate=0.3)(x)

    multiclass_output = Dense(units=num_outputs, activation='softmax')(x)

    model = Model(inputs=inputs, outputs=multiclass_output)

    model.compile(
        loss="categorical_crossentropy",
        metrics=["accuracy"],
        optimizer=Adam(learning_rate=LR)
    )

    return model
```
$\endgroup$
1
  • $\begingroup$ Are you trying to learn the abs function or are you trying to learn multiclass classification? A model trained for one cannot perform the other $\endgroup$
    – Karl
    Commented Sep 13 at 1:43

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.