The biggest advantage of ReLu is indeed non-saturation of its gradient, which greatly accelerates the convergence of stochastic gradient descent compared to the sigmoid / tanh functions (paper by ...

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document ...

Sigmoid function is a partial case of softmax, when the number of classes $K=2$. That's why the similarity of their derivatives shouldn't surprise you. Why do so many functions used in data science ...

Tensors come pretty natural in convolutionals networks. Local pixel information matters: if $e$ is a pixel in your example above, it's important to know that $a$ through $i$ are its neighbors. This ...

Theoretically, the formula with two matrices is more clear and self-evident, I think that's the reason why it's used more often. In practice, both approaches are actually used in production and hence ...

Here's a math answer for you. Neural network is an approximation function $f(\theta)$ of the joint distribution $p(X, Y)$ of input data $X$ and labels $Y$. The learning process is the process of ...

To freeze the lower layers during training, the simplest solution is to give the optimizer the list of variables to train, excluding the variables from the lower layers: train_vars = tf....

Keras Model.predict() method doesn't shuffle the data, so each row in p_valid corresponds to the row in x_valid. For instance, p_valid holds the probabilities for the image x_valid. By the way, ...

Your definition is correct. For the reference you can compare it with the probabilistic model from Tensorflow "Vector Representations of Words" tutorial:  \begin{align} P(w_t | h) &= \text{...