# Tag Info

0

A neural network is usually trained on a large set of paired example data (supervised learning). For each example in the set, the best known optimization for each weight is calculated, but it is then multiplied by the learning rate, which is a very small number. If a learning rate was not used, you would make large adjustments to each weight, only to destroy ...

0

Training a neural network involves optimizing a large set of parameters. This optimization step is commonly known as backpropgation (a.k.a backprop) via a form of gradient descent. Backprop via gradient descent allows a network to adjust its learnable parameters (i.e., weights) such that the loss (difference between the forward pass output and the actual ...

0

Solving optimisation problems is difficult, and finding a closed-form solution that finds the optimal point for the cost function is complicated. Consequently, optimisation problems are solved using iterative steps. This means people choose solutions which are guaranteed to decrease the cost or objective function with each step. This idea is somehow used in ...

0

In many applications I've seen (e.g. GANs) $\beta_1$ is set to $0$, so $m_1=g_1$, i.e. the numerator of the update rule is the same as in SGD. This leaves two main differences, both related to the MA of the second moment: $v_t:$ raw MA of the second moment serves as a gradient normalizer that divides the gradient by the square root of the moving average of ...

0

Main objective of mini-batch gradient descent is to achieve faster results over full-batch gradient descent as it will start learning weights before completion of one epoch. SGD will start learning earlier than Mini-batch, isn't it? But mini-batch reduces variance of the gradient compared to SGD. Coming to the question, you're right it's possible to compare ...

0

The regularization term (reg_term) is sometimes negative due negatative parameters. Hence S[f"dW{l}"] contains some negative values. I realize the reg_term has to be added before taking the sqrt, like this: S[f"dW{l}"] = beta2 * S[f"dW{l}"] + (1 - beta2) * np.square(gradients[f"dW{l}"] + reg_term)

0

It sounds like you’re not trying to create a model that can be used outside of the training dataset and instead you’re trying to get your network to memorise the dataset. In other words, you’re aiming to overfit. If that is the case the simplest thing that might work is to take the group of samples your network is getting wrong and upsample them (include ...

1

I agree with @Piotr Rarus (+1); as he said, the no free lunch theorem certainly applies to this. I'd like to add some exploration of the four possible states for perfectly fitting the training data: Impossible due to training function if it is only surjective (onto but not one-to-one). This is of course unlikely, but still feasible. Impossible due to ...

0

if you want to find an analytical solution for the problem I think that it doesn't exist. At this point you should apply decomposition (don't try to work on all your variables, but iteratively considers only a subset of them called working set), if (and only if) you use a working set W, |W|=2, then your subproblem has an analytical solution. Due to the large ...

0

If nothing else is possible, you could try using a genetic algorithm.

2

What your advisor is suggesting is called data leakage and it’s somewhat similar to training and testing your model on the same data. You might find it useful to read about p-hacking and backtest overfitting to get a feel for why this is a problem in quantitative finance. There’s also this excellent comic strip about the related concept of p-hacking... The ...

4

You are absolutely correct that this is an problematic approach. Your testing set should only be used at the last possible stage before deploying a model. By using your testing set to make modeling decisions you will introduce bias which will favor the observations found in your test set and may not generalize. In an ideal world, your test set would ...

Top 50 recent answers are included