0
$\begingroup$

I was reading about custom activation functions, autodiff etc, trying to understand it all. I think I have a glimpse of it now, but right after closing the book I thought "what about the nonelementary integrals"?

So the way I understand TensorFlow works, is it computes the derivative of the activation function by graphing it (decomposing into pieces of elementary functions), then does a forward pass, and a backward pass to determine the derivative.

So I tried this simple code:

from keras.datasets import mnist

(X_train, y_train), (X_val, y_val) = mnist.load_data()
test_model = keras.models.Sequential([
    keras.layers.Flatten(input_shape = X_train.shape[1:]),
    keras.layers.Dense(100, activation=tf.math.erf),
    keras.layers.Dense(10, input_shape = [100], activation="softmax")
])

test_model.compile(loss = "sparse_categorical_crossentropy", optimizer="nadam", metrics=["sparse_categorical_accuracy"])

I have used the error function as the activation, for two reasons:

  • It resembles sigmoid functions when it comes to the shape, so It just might work.
  • It is a nonelementary integral, so It cannot be decomposed into simple, elementary functions.

To my amusement, calling test_model.fit did work without errors, and the training was even increasing the model's accuracy. But there is one thing - in theory it shouldn't work at all, because a nonelementary integral cannot be decomposed into elementary functions and thus cannot be graphed.

This leaves me with a question - how does TensorFlow handle this?

$\endgroup$
3
  • 1
    $\begingroup$ Unless I've misunderstood what you're asking, I don't think the lack of integral is a problem. The error function has a perfectly well defined derivative in terms of elementary functions, just no elementary integral. Gradient descent doesn't need to know the integral of course. Is this helpful? $\endgroup$
    – htl
    Jul 24, 2021 at 17:11
  • $\begingroup$ The derivative of erf is well-known, what's the problem? $\endgroup$
    – user253751
    Jul 24, 2021 at 19:54
  • $\begingroup$ How I understand it, tensorflow first tries to decompose the functions into elementary functions (graphing), and then using this decomposition calculates the derivative using autodiff. Of course, if the derivative of erf was used by TensorFlow directly, this would be obvious. My question is though - is this the case in tensorflow? Or does it do it some other way. $\endgroup$ Jul 24, 2021 at 21:50

1 Answer 1

0
$\begingroup$

The computational graph is created for the complex function. So, all the steps are just elementary steps.

For example - Below function -.

$\hspace{5cm}$ enter image description here

Becomes - enter image description here
$\hspace{4cm}$Image Credit - leonardoaraujosantos.gitbook.io/artificial-inteligence/

For further knowledge -

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.