What should be the requirement for training data in order to obtain a good regression model using neural network?

Question

I have made a neural network regression model using the theory for the first time and would like to clarify some basic doubts, whose concrete answers I couldn't find yet.

Data:- I have 3000 samples with 7 input features (consisting of 5 normalized continuous variables and 1 binary variable recoded using one-hot encoding). I have 2 outputs per sample.

1. So I have trained my model and I am getting around 95% accuracy. I feel it's too good to be true. Is it?

2. I have normalized my training data and used the mean and standard deviation of the training data to normalize the test data. Right approach, right?

3. I have read that neural network is after all a method to do interpolation, so how would it behave on the data with features with values which are not within the range while it's being trained?

   • I ask this because when I take a $80\%$ random sample out of the available data, it's possible that for a given feature (continuous variable) $X_1$ out of $X_1, X_2, ..., X_7$, $X_1 \in [a,b]$, where $a,b$ are some scalars. However, the other $20\%$ test data might have $X_1$ values $<a$ or $>b$. Is my trained model expected to give correct results then?

4. Should I treat each output independently and create models, $i.e.$ weights separately (2 in my case) or should I find weights with the joint output? (The outputs may have different scales, so should I normalize them too in this case?)

   • How different would be the behaviours of both the models?

I believe the questions are preliminary but it would be really helpful to get my basics clear.

Answer 1

1. If this is your test accuracy (and you make sure not to use any test samples as part of the training), then I don't see any reason not to trust it. Maybe the distribution of the training data is a very good approximation of the real distribution. You can perform a quick sanity check by trying a different separation between the training and test dataset.

2. Yes, that is correct.

3. The short answer: you have no way of knowing how it will behave outside the range of trained samples. However, a case where a considerable amount of the test set is taken out of the range of all the training set with a 80-20 division is not likely.

4. Don't have any personal experience with this question, but from a quick read of several similar questions, it seems that the optimization process would be driven by the target variable which has the largest scale. So you either need to add some weights to the loss function or normalize the outputs (same-same). Regarding the multi-output vs. multi-model question, I think that the first answer in the following thread is on point: What is the difference between using single multi-output NN and multiple single-output NNs