1
$\begingroup$

I'm trying to use a Neural Network in order to predict two different values from an input that contains 24 different features. The results I've gotten so far are not good enough, so any suggestions would be appreciated since I've been stuck for some time. This is what I've done so far:

Input data:

I have an input that contains 24 different features (the total dataset has around 150,000 instances). So I've tried to standarize my input, normalize it, log transform it, and use PCA in order to reduce the dimensionality of the problem. Out of this, PCA has proven to be the best solution (using the first 5 principal components).

To make sure that input is significant, I have done a quick fit using a Random Forest Regressor and a Extra Tress Regressor to calculate the importance of each feature (after having performed PCA). And compared to a random feature, all the features that are being used seem to be significant enough for the model.

Neural Network

For the neural network, I have tried a lot of things arriving to the following architecture:

initializer_sm = tf.keras.initializers.GlorotNormal()

model_nn = keras.Sequential([
    keras.layers.Dense(128,input_dim = xnn_train.shape[1], activation='selu', kernel_initializer=initializer),
    keras.layers.Dense(256, activation='selu', kernel_initializer=initializer),
    keras.layers.Dense(256, activation='selu', kernel_initializer=initializer),

    keras.layers.Dense(2, activation = 'softplus', kernel_initializer=initializer_sm)
    ])

def custom_loss(y_true,y_pred):
    
    return (K.abs(y_true - y_pred)/y_true)*100

epochs = 100
opt = keras.optimizers.Nadam()
model_nn.compile(optimizer = opt, loss = custom_loss)
history_nn = model_nn.fit(xnn_train, ynn_train, epochs = epochs, batch_size = 1024, validation_data = (xnn_val,ynn_val))

With the following results:

enter image description here

And these are the histogram for each output predicted w.r.t the real values:

enter image description here

I have also tried to implement 1D convolutial layers before the architecture shown before, but the only effect that they have is that the loss function converges faster.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ "The results I've gotten so far are not good enough" - what is motivating this statement? Are you comparing to baselines or other models? Are you confident that better results are achievable with the given data and features? It might be helpful to share in more detail the nature of the problem such as the features and the outputs you are trying to predict. Given that you are doing dimension reduction to 5 dimensions, the network seems a little large. Have you tried something simpler like basic linear regression on the reduced features? $\endgroup$
    – James
    Apr 24, 2021 at 2:57
  • $\begingroup$ Yes, great question. I'm comparing this solution to other models that have been used. This is basically an inverse problem, where the inputs are measured in the lab and the outputs are calculated (clearly the forward problem is to calculate the inputs here mentioned, by using the outputs -- this process is done analiticaly). So different models such as gradient descent and gauss-newton descent have been used to solve this inverse problem, I'm basically comparing my results to those obtained with such models. $\endgroup$
    – mechanics_physics
    Apr 24, 2021 at 3:01
  • $\begingroup$ I second James' suggestions: as far as I understand you don't have any baseline or model to compare against for the same exact problem, right? (my understanding is that you're comparing against the inverse problem, but maybe I'm wrong). If I'm right the first thing I'd suggest is to train and evaluate a few baseline systems using simple regression methods (e.g. decision trees, SVM...). Also it's not sure that you need to reduce the dimensionality imho, 24 features is not that many. $\endgroup$
    – Erwan
    Apr 25, 2021 at 0:25

1 Answer 1

Reset to default
1
$\begingroup$

Your network is too large for this much data. Reduce the number of units in the layers, go for simple 'relu' in the layers except the last one where you should use softmax. Consider reducing number of layers.

I would recommend using decision trees libraries like XGBoost, CatBoost, LightGBM etc. for this problem. You will get the best result.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.