How to learn steep functions using neural network?

Question

I am trying to use a neural network to learn the below function. In total, I have 25 features and 19 outputs. The above image shows the distribution of two features with respect to one of the outputs. There are close to 200k examples, and the neural network contains 7 layers with 256 neurons having leaky relu as activation. The last layer is a linear layer. The problem is that the output is too steep and the neural network has a large error on peaks. How can I modify the target variables or neural networks such that it makes a small mean absolute error on the peaks (basically flatten the below error plot)? Below is the error plot

Things which I have tried but have not worked-

1. Used log to transform target variables.
2. Increasing the size of the neural network with the hope that a large enough neural network will fit anything but the training loss is still large on peaks
3. Used mean absolute, mean squared and percentage loss I have asked the same question here https://stats.stackexchange.com/questions/589724/how-to-learn-steep-functions-using-neural-network, but would like to get opinion from this community as well.

Answer 1

The problem isn't that the loss function is steep, the problem is that it is not convex/differentiable / there are local minima. Read this answer.

Answer 2

My understanding of your question is that you have a multivariate regression problem (i.e., 19 different target variables). Given it's a multivariate regression problem and you have specific acceptance criteria for model performance, it might be useful to define a custom loss function. The custom loss function should penalize "peak" mispredictions more than "non-peak" mispredictions. Keep adjusting the weight on "peak" mispredictions in the custom loss function until the model fits how you want it to.

Answer 3

Your problems seem like solving a stock exchange problem with respect to some given values. Take it as a time series problem where the curve of your graph changes according to some low peak and high peak value, take "Mean Absolute Error or Mean Squared Error. Also, take ReLU as a Non-Linearity function and add a penalization term (L1-Regularizer). One thing to note here is that your function is a Non-Convex function for solving this kind of problem increase the width of your neural network but the depth should be low, I mean take 8 to 12 layers of Neural Network and in each layer take 16 to 32 Neurons, also give a Residual Connection after each 3 layers except last. Try all these steps, I hope the model will learn this behavior with minor data points.