Maxim
  • Member for 5 years, 9 months
  • Last seen more than a month ago
Why ReLU is better than the other activation functions
Accepted answer
25 votes

The biggest advantage of ReLu is indeed non-saturation of its gradient, which greatly accelerates the convergence of stochastic gradient descent compared to the sigmoid / tanh functions (paper by ...

View answer
Word2Vec embeddings with TF-IDF
12 votes

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document ...

View answer
Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?
Accepted answer
10 votes

Sigmoid function is a partial case of softmax, when the number of classes $K=2$. That's why the similarity of their derivatives shouldn't surprise you. Why do so many functions used in data science ...

View answer
What is the point of tensors in CNNs? Why not simply reshape the data into matrices?
3 votes

Tensors come pretty natural in convolutionals networks. Local pixel information matters: if $e$ is a pixel in your example above, it's important to know that $a$ through $i$ are its neighbors. This ...

View answer
RNN: why Wx + Uh instead of W[x,h]
Accepted answer
2 votes

Theoretically, the formula with two matrices is more clear and self-evident, I think that's the reason why it's used more often. In practice, both approaches are actually used in production and hence ...

View answer
How do neural networks account for outliers?
2 votes

Here's a math answer for you. Neural network is an approximation function $f(\theta)$ of the joint distribution $p(X, Y)$ of input data $X$ and labels $Y$. The learning process is the process of ...

View answer
Transfer learning within Tensorflow's inception model
2 votes

To freeze the lower layers during training, the simplest solution is to give the optimizer the list of variables to train, excluding the variables from the lower layers: train_vars = tf....

View answer
How to associate array of class probabilities with corresponding image in keras?
Accepted answer
1 votes

Keras Model.predict() method doesn't shuffle the data, so each row in p_valid corresponds to the row in x_valid. For instance, p_valid[2] holds the probabilities for the image x_valid[2]. By the way, ...

View answer
Word2Vec, softmax function
Accepted answer
1 votes

Your definition is correct. For the reference you can compare it with the probabilistic model from Tensorflow "Vector Representations of Words" tutorial: $$ \begin{align} P(w_t | h) &= \text{...

View answer
Datasets in NLP research papers
1 votes

I have come across such a dataset in CS 20SI GitHub repo: it's a collection of abstracts from 7200 research papers. If you need even more, you can always write a simple crawler of arXiv web-site. The ...

View answer
Do numerical inaccuracies play any role in training neural networks?
1 votes

Are there publications which mention numerical problems in neural network optimization? Of course, there has been a lot of research on vanishing gradients, which is entirely a numerical problem. ...

View answer