Can anybody help me to understand the behavior of metrics (RMSE namely) when testing model? I have NN with 1 hidden layer for regression task. RMSE equal 0.07 for external test dataset. But if I break this dataset into parts that are logically and physically necessary for me (for example, part of the dataset is responsible for one large object for which "y" are predicted), then RMSE increases for parts of the test dataset. For example, I divided the external test dataset into 3 parts and separately for these parts the RMSE is equal to 0.3 0.2 0.2.
What does it mean, and why do such changes occur? what metric value really reflects what is happening in this case? The y pairs in the rmse formula do not change when moving from individual parts of a set of dates to the general one.
