0
$\begingroup$

I am trying to create an ANN with as many as 50 independent variables. Based on my intuition and business logic, I have tried to create many combination of input variables to feed the ANN, retrain it and try to predict. But the results are not as expected and although the model make very good test set predictions it fails when new data is fed. Can someone plz guide where I am doing wrong? My immediate thought is I need to select the input variables based in a better way. Statistically is there a way to decide that?

What information can I provide about the model or results which would help to solve this?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

I feel that is something more fundamentally wrong if your test scores are good but your predictions on new data are bad. I think it sounds like your new prediction data is not coming from the same distribution as your train/test data. That means it is different in some way which will cause you model to perform more poorly on the new data.

Improve this answer
$\endgroup$
1
$\begingroup$

Either your test set is very similar with your training data and you're overfitting or your new samples are fundamentally different from your training/test data (i.e. different distribution).

Improve this answer
$\endgroup$
3
  • $\begingroup$ my model accuracy is close to 85-90%. Does it mean overfitting? $\endgroup$
    – Ankur Bansal
    Commented Jan 12, 2019 at 6:35
  • $\begingroup$ also, I am first training the model, saving the model parameters and then using the saved model later to make predictions on the live data. While making predictions, I am scaling the data which is again not using the same basis as training model used but the standard scaling sample is coming from the same distribution. Does this alarm any bells? $\endgroup$
    – Ankur Bansal
    Commented Jan 12, 2019 at 6:37
  • $\begingroup$ what do you mean by "standard scaling sample is coming from the same distribution"? And, whose accuracy is it (train,test,validation)? you should use the same stdScaler you fit for the training set by the way. $\endgroup$
    – gunes
    Commented Jan 13, 2019 at 17:33
1
$\begingroup$

If you are trying to do regression model you could always try to compute correlation between values. If r-Pearson correlation between input $x_i$ and output $y$ means that your output is similar to linear function of $x_i$. For non-linear functions you can try computing $\rho$-Spearman correlations. For categorical data $\tau$-Kendall correlations sometimes do the trick.

After picking inputs related to outputs you should check which inputs are irrevelant because they are just linear-combination of the other inputs. By computing correlation between selected inputs, you can check which input signals are not needed - other 'includes' them already.

You can also try PCA - Principal Component Analysis which map you data into new space, where every signal is orthogonal to all others.

Good idea would be checking if some inputs should be delayed. Not only can you check correlations between delayed inputs and outputs, but also you could use cross-correlation function to check what value of lag is optimal.

Improve this answer
$\endgroup$
1
  • $\begingroup$ thank you for the suggestions. I am using ANN for classification with a binary outcome (using sigmoid function). My data is non-linear with relation to the dependent variable, so I will try working with ρ -Spearman correlations as you have mentioned. $\endgroup$
    – Ankur Bansal
    Commented Jan 12, 2019 at 6:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.