# Are there deterministic problems a neural network can't learn?

I have a software which takes 5 input numbers and outputs a number, deterministically. I want to try and mimic this software precisely with a neural network, but I am finding it very difficult. The loss function seems to hover around one value throughout training or occasionally increase, but it does not seem to minimize.

The function I want to mimic is relatively complex and I believe it involves solving differential equations. What are some things I might try to make this work better? Are there some functions that just can't be learned by a neural net?

In theory, it should be possible for the neural network to learn the function. This is what Cybenko’s Theorem tells us. Basically, if you have 5 input numbers and 1 output number determined deterministically, you have something similar to: Input: 1, 1, 1, 1, 1 Output: 5 And your NN is trying to figure out the formula to get 5 from the five ones. If we, as humans, can figure out that summing the five ones gives us the output, that means that a NN can also achieve this. Of course, this is just in theory and differential equations are much more complex than a simple sum, but still it should be possible.

"In theory there's no difference between theory and practice, but in practice there is."

-Anon

As we know, a dense neural network with a hidden layer (of an appropriate width) can approximate any function, but often directly approximating a given function is hard.

There are a few ways to make it easier, though. There's always the "throw money at it" option of getting way more data, making the layer wider, and training longer. Depending on the specific problem you're trying to solve, though, there may be much more effective methods. This list is far from exhaustive, and you may be well served to seek out survey/other papers in your field to see what others have learned.

## Data comes first

• Do you have enough samples to both properly train and confidently test it?
• Do you have any bad samples? You might be surprised by the answer.
• Are there any significant gaps in your training or testing data? There may be things about the function that the model can't directly learn with just the data provided.

## Use your domain knowledge

• Can your data be preprocessed in any way to help the model out?
• Is there a better choice of architecture for your data? Or combination of architectures? For instance, I've heard of a connection between diffusion models and differential equations (but this is not my area).
• Does the function represent a process with parts such that it makes sense to train & grow, or bake sub-networks into the architecture instead of a single stack of layers?
• Does it make sense to directly input all 5 features together, or could disjoint subsets of your features better be handled through input sub-networks?

## Tricks of the trade

• Increased learning rate at the beginning can help with pesky local minima. Just don't forget to use a scheduler to decrease it over time.
• There's no one best optimizer, try a few different ones.
• Adding nodes to a layer while simultaneously encouraging sparsity of features (e.g. with dropout) can get nice results, though it makes convergence harder.
• Loss getting low-ish but not low enough? Adding a layer could help. But also, despite what the theory states, if it's not going to help it can hurt -- good luck getting it to converge to the identity layer even with every other layer frozen.
• Residual connections can sometimes help deeper models converge on better solutions, but it's not a guarantee. Same with skip connections.

The universal approximation theorem states that a deep learning network can construct to learn a wide class of functions, including predicting one number from five other numbers.

Finding that construction can be a challenge. The three best tips are:

1. A large amount of data. Hopefully, millions of examples for complex functions.
2. A large network. For example, large language models (LLM) have billions of parameters.
3. A large amount of compute. Graphics processing unit (GPU) are useful and sometimes clusters of compute.