Skip to main content
234 votes

How to set class weights for imbalanced classes in Keras?

You could simply implement the class_weight from sklearn: Let's import the module first ...
PSc's user avatar
  • 2,501
221 votes

How to set class weights for imbalanced classes in Keras?

If you are talking about the regular case, where your network produces only one output, then your assumption is correct. In order to force your algorithm to treat every instance of class 1 as 50 ...
layser's user avatar
  • 2,326
144 votes

How to draw Deep learning network architecture diagrams?

I recently found this online tool that produces publication-ready NN-architecture schematics. It is called NN-SVG and made by Alex Lenail. You can easily export these to use in, say, LaTeX for ...
Pablo Rivas's user avatar
  • 1,541
119 votes

When to use GRU over LSTM?

GRUs and LSTMs utilize different approaches toward gating information to prevent the vanishing gradient problem. Here are the main points comparing the two: The GRU unit controls the flow of ...
Abhishek's user avatar
  • 1,959
99 votes

Cross-entropy loss explanation

The cross entropy formula takes in two distributions, $p(x)$, the true distribution, and $q(x)$, the estimated distribution, defined over the discrete variable $x$ and is given by $$H(p,q) = -\sum_{\...
Neil Slater's user avatar
  • 28.9k
92 votes

How do you visualize neural network architectures?

I recently created a tool for drawing NN architectures and exporting SVG, called NN-SVG
Alex Lenail's user avatar
  • 1,021
90 votes

How to draw Deep learning network architecture diagrams?

I wrote some latex code to draw Deep networks for one of my reports. You can find it here: With this, you can draw networks like these:
Haris Iqbal's user avatar
87 votes

What is the difference between Gradient Descent and Stochastic Gradient Descent?

For a quick simple explanation: In both gradient descent (GD) and stochastic gradient descent (SGD), you update a set of parameters in an iterative manner to minimize an error function. While in GD, ...
Sociopath's user avatar
  • 1,243
84 votes

What is the difference between "equivariant to translation" and "invariant to translation"

Equivariance and invariance are sometimes used interchangeably in common speech. They have ancient roots in maths and physics. As pointed out by @Xi'an, you can find previous uses (anterior to ...
Laurent Duval's user avatar
79 votes

In softmax classifier, why use exp function to do normalization?

It is more than just numerical. A quick reminder of the softmax: $$ P(y=j | x) = \frac{e^{x_j}}{\sum_{k=1}^K e^{x_k}} $$ Where $x$ is an input vector with length equal to the number of classes $K$. ...
vega's user avatar
  • 916
77 votes

Does batch_size in Keras have any effects in results' quality?

After one and a half years, I come back to my answer because my previous answer was wrong. Batch size impacts learning significantly. What happens when you put a batch through your network is that ...
Jan van der Vegt's user avatar
73 votes

When to use GRU over LSTM?

*To complement already great answers above. From my experience, GRUs train faster and perform better than LSTMs on less training data if you are doing language modeling (not sure about other tasks). ...
minerals's user avatar
  • 2,147
73 votes

Why mini batch size is better than one single "batch" with all training data?

The key advantage of using minibatch as opposed to the full dataset goes back to the fundamental idea of stochastic gradient descent1. In batch gradient descent, you compute the gradient over the ...
horaceT's user avatar
  • 1,360
73 votes

What does Logits in machine learning mean?

Logits interpreted to be the unnormalised (or not-yet normalised) predictions (or outputs) of a model. These can give results, but we don't normally stop with logits, because interpreting their raw ...
n1k31t4's user avatar
  • 14.9k
70 votes

Should we apply normalization to test data as well?

Yes you need to apply normalisation to test data, if your algorithm works with or needs normalised training data*. That is because your model works on the representation given by its input vectors. ...
Neil Slater's user avatar
  • 28.9k
66 votes

When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization?

The normal vs uniform init seem to be rather unclear in fact. If we refer solely on the Glorot's and He's initializations papers, they both use a similar theoritical analysis: they find a good ...
tlorieul's user avatar
  • 1,009
65 votes

How to get accuracy, F1, precision and recall, for a keras model?

Metrics have been removed from Keras core. You need to calculate them manually. They removed them on 2.0 version. Those metrics are all global metrics, but Keras works in batches. As a result, it ...
Tasos's user avatar
  • 3,930
60 votes

What is the difference between "equivariant to translation" and "invariant to translation"

The terms are different: Equivariant to translation means that a translation of input features results in an equivalent translation of outputs. So if your pattern 0,3,2,0,0 on the input results in 0,...
Neil Slater's user avatar
  • 28.9k
59 votes

Adding Features To Time Series Model LSTM

For RNNs (e.g., LSTMs and GRUs), the layer input is a list of timesteps, and each timestep is a feature tensor. That means that you could have a input tensor like this (in Pythonic notation): ...
Adam Sypniewski's user avatar
57 votes

Why is ReLU used as an activation function?

In mathematics (linear algebra) a function is considered linear whenever a function$f: A \rightarrow B$ if for every $x$ and $y$ in the domain $A$ has the following property: $f(x) + f(y) = f(x+y)$. ...
Tophat's user avatar
  • 2,420
56 votes

What is the relationship between the accuracy and the loss in deep learning?

There is no relationship between these two metrics. Loss can be seen as a distance between the true values of the problem and the values predicted by the model. The larger the loss, the larger the ...
Jérémy Blain's user avatar
53 votes

Multi GPU in Keras

From the Keras FAQs, below is copy-pasted code to enable 'data parallelism'. I.e. having each of your GPUs process a different subset of your data independently. ...
weiji14's user avatar
  • 656
52 votes

How do you visualize neural network architectures?

Tensorflow, Keras, MXNet, PyTorch If the neural network is given as a Tensorflow graph, then you can visualize this graph with TensorBoard. Here is how the MNIST CNN looks like: You can add names / ...
Martin Thoma's user avatar
  • 18.9k
50 votes

Why should the data be shuffled for machine learning tasks

Based on What should we do when a question posted on DataScience is a duplicate of a question posted on CrossValidated?, I am reposting my answer to the same question asked on CrossValidated (https://...
Josh's user avatar
  • 623
49 votes

Time Series prediction using LSTMs: Importance of making time series stationary

In general time series are not really different from other machine learning problems - you want your test set to 'look like' your training set, because you want the model you learned on your training ...
tom's user avatar
  • 2,248
47 votes

Why should the data be shuffled for machine learning tasks

Shuffling data serves the purpose of reducing variance and making sure that models remain general and overfit less. The obvious case where you'd shuffle your data is if your data is sorted by their ...
Valentin Calomme's user avatar
46 votes

Does gradient descent always converge to an optimum?

Gradient Descent is an algorithm which is designed to find the optimal points, but these optimal points are not necessarily global. And yes if it happens that it diverges from a local location it may ...
Green Falcon's user avatar
  • 14.1k
44 votes

Data science related funny quotes

Q: How many machine learning specialists does it take to change a light bulb? A: Just one, but they require a million light bulbs to train properly. Q: How many machine learning specialists does it ...
43 votes

Number of parameters in an LSTM model

The LSTM has a set of 2 matrices: U and W for each of the (3) gates. The (.) in the diagram indicates multiplication of these matrices with the input $x$ and output $h$. U has dimensions $n \times m$ ...
43 votes

What is Ground Truth

The ground truth is what you measured for your target variable for the training and testing examples. Nearly all the time you can safely treat this the same as the label. In some cases it is not ...
Neil Slater's user avatar
  • 28.9k

Only top scored, non community-wiki answers of a minimum length are eligible