Let's say I measure people hiking uphill, and it happens that their hiking speed is related to the slope so they hike slower where the hill is steeper. My input variable is the hill's elevation and my output variable is their hiking speed. I fit a neural net to this data, so that I can later predict the hiking speed given new elevation measurements. Can a neural net represent the input's first derivative and so do a good job capturing this relation? What about representing a second derivative, i.e. if hiking speed was related to the curvature of the hill?
What you are asking for is if a neural network is able to represent and/or approximate functions.
The answer is: Yes, the Neural Networks can represent derivatives of functions.
In this paper, the possibility is explained.
Other author take the same conclusion: