1
$\begingroup$

In this intuitive explanation of why wide-enough shallow feedforward neural networks can satisfy the universal approximation theorem from any continuous function on a compact domain, the author uses, throughout the article, two hidden layers to demonstrate his case. However, towards the end of the article, it is stated:

"We've seen how to use networks with two hidden layers to approximate an arbitrary function. Can you find a proof showing that it's possible with just a single hidden layer? As a hint, try working in the case of just two input variables, and showing that: (a) it's possible to get step functions not just in the x or y directions, but in an arbitrary direction; (b) by adding up many of the constructions from part (a) it's possible to approximate a tower function which is circular in shape, rather than rectangular; (c) using these circular towers, it's possible to approximate an arbitrary function."

Now, working in two dimensions for the input, part "(a)" and "(c)" are easy enough. "(a)" follows by setting the first set of weights to non-binary linear combinations and "(c)" follows if we assume part "(b)" and that our target function is uniformly continuous on its compact domain (as then, it can be approximated arbitrarily closely by a finite set of neurons). However, part "(b)" itself just seems out of grasp.

No matter what fancy arrangement of step functions I take and stack together, I cannot find a construction that yields a circular tower function that decays to 0 quickly enough outside of its targeted circular domain, even if when you take the number of construction components (the if-else neuron pairs) to infinity.

The best construction I have found (see image), in the limit, yields a desired target value in the area of a circle of arbitrary size, but outside of the circle, it is unfortunately also non-zero and only decays to zero with the formula:

$$ NN(x) = h_d = \frac{2*h*sin^{-1}(\frac{r}{d(x)})}{\pi} $$

where $ d \ge r$ is the distance of the given point from the center of the construction, where $ r $ is the radius of the construction tower circle (equal to half the width of all of the bump functions), $ h $ is the target height and $ h_d $ is the resultant height of the given point $ x $.

Note that this failed circular tower function is obtained by placing infinitely many if-else bump functions (each corresponding to a neuron pair) with uniform weights of $ h/\infty $ feeding in the output layer while rotating $ \pi $ radians about a target point $ x $ (with target value $h$ from the value of the target function applied to $ x$. See the image below for more visual details. Now, my hope was that a large enough finite construction of this type would converge to a circular tower, but as stated, the values outside of the circle do not vanish, so the construction fails. However, the article claims that there is a construction that works and yields a circular tower, so what construction is that?

Clearly, if no construction exists, then the article has only demonstrated that continuous (uniformly continuous) functions on a compact domain can be approximated by feedforward neural networks with two hidden layers (not just one).

failed construction

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.