I am trying to train a simple neural network model for multiclass classification.
I have x1,x2,x3,x4 columns with 4 classes to predict.
If just train on x1,x2,x3,x4 then I get accuracy of 88%
With some domain knowledge I can create three new features which I know will definitely help the model train better.
The three new features are :-
df['x12'] = abs (df['x1']-df['x2'])
df['x13'] = abs (df['x1']-df['x3'])
df['x14'] = abs (df['x1']-df['x4'])
If i train on x1,x2,x3,x4 and abs(x1-x2),abs(x1-x3),abs(x1-x4) then I get an accuracy of 98%
I want to get 98% accuracy without abs(x1-x2),abs(x1-x3),abs(x1-x4)
With these new manually created features I get a validation accuracy of 98% which is great.
However, When I remove these features the validation accuracy jumps down to 88%.
My question is that the function abs(x1-x2) should be very simple enough for the model to learn on its own without me manually doing feature engineering.
Then why does the accuracy drop when I remove these three (very simple) features?
Is the model not capable enough to learn it on its own?
This is what my model looks like :-
inputs = Input(shape=input_shape)
x = Dense(units=64)(inputs)
x = PReLU()(x)
x = BatchNormalization()(x)
x = Dropout(rate=0.3)(x)
x = Dense(units=64)(x)
x = PReLU()(x)
x = BatchNormalization()(x)
x = Dropout(rate=0.3)(x)
x = Dense(units=64)(x)
x = PReLU()(x)
x = BatchNormalization()(x)
x = Dropout(rate=0.3)(x)
x = Dense(units=64)(x)
x = PReLU()(x)
x = BatchNormalization()(x)
x = Dropout(rate=0.3)(x)
x = Dense(units=32)(x)
x = PReLU()(x)
x = BatchNormalization()(x)
x = Dropout(rate=0.3)(x)
multiclass_output = Dense(units=num_outputs, activation='softmax')(x)
model = Model(inputs=inputs, outputs=multiclass_output)
model.compile(
loss="categorical_crossentropy",
metrics=["accuracy"],
optimizer=Adam(learning_rate=LR)
)
return model
```