# How does TensorFlow compute gradients of nonelementary integrals?

I was reading about custom activation functions, autodiff etc, trying to understand it all. I think I have a glimpse of it now, but right after closing the book I thought "what about the nonelementary integrals"?

So the way I understand TensorFlow works, is it computes the derivative of the activation function by graphing it (decomposing into pieces of elementary functions), then does a forward pass, and a backward pass to determine the derivative.

So I tried this simple code:

from keras.datasets import mnist

(X_train, y_train), (X_val, y_val) = mnist.load_data()
test_model = keras.models.Sequential([
keras.layers.Flatten(input_shape = X_train.shape[1:]),
keras.layers.Dense(100, activation=tf.math.erf),
keras.layers.Dense(10, input_shape = [100], activation="softmax")
])



I have used the error function as the activation, for two reasons:

• It resembles sigmoid functions when it comes to the shape, so It just might work.
• It is a nonelementary integral, so It cannot be decomposed into simple, elementary functions.

To my amusement, calling test_model.fit did work without errors, and the training was even increasing the model's accuracy. But there is one thing - in theory it shouldn't work at all, because a nonelementary integral cannot be decomposed into elementary functions and thus cannot be graphed.

This leaves me with a question - how does TensorFlow handle this?

• Unless I've misunderstood what you're asking, I don't think the lack of integral is a problem. The error function has a perfectly well defined derivative in terms of elementary functions, just no elementary integral. Gradient descent doesn't need to know the integral of course. Is this helpful?
– htl
Commented Jul 24, 2021 at 17:11
• The derivative of erf is well-known, what's the problem? Commented Jul 24, 2021 at 19:54
• How I understand it, tensorflow first tries to decompose the functions into elementary functions (graphing), and then using this decomposition calculates the derivative using autodiff. Of course, if the derivative of erf was used by TensorFlow directly, this would be obvious. My question is though - is this the case in tensorflow? Or does it do it some other way. Commented Jul 24, 2021 at 21:50

$$\hspace{5cm}$$
$$\hspace{4cm}$$Image Credit - leonardoaraujosantos.gitbook.io/artificial-inteligence/