Questions tagged [deep-learning]
a new area of Machine Learning research concerned with the technologies used for learning hierarchical representations of data, mainly done with deep neural networks (i.e. networks with two or more hidden layers), but also with some sort of Probabilistic Graphical Models.
4,834
questions
256
votes
10
answers
415k
views
How to set class weights for imbalanced classes in Keras?
I know that there is a possibility in Keras with the class_weights parameter dictionary at fitting, but I couldn't find any example. Would somebody so kind to ...
- 8,387
194
votes
6
answers
341k
views
How to draw Deep learning network architecture diagrams?
I have built my model. Now I want to draw the network architecture diagram for my research paper. Example is shown below:
- 2,447
191
votes
5
answers
144k
views
What is the "dying ReLU" problem in neural networks?
Referring to the Stanford course notes on Convolutional Neural Networks for Visual Recognition, a paragraph says:
"Unfortunately, ReLU units can be fragile during training and can
"die". For ...
- 3,965
178
votes
6
answers
178k
views
When to use GRU over LSTM?
The key difference between a GRU and an LSTM is that a GRU has two gates (reset and update gates) whereas an LSTM has three gates (namely input, output and forget gates).
Why do we make use of GRU ...
- 2,021
175
votes
20
answers
230k
views
How do you visualize neural network architectures?
When writing a paper / making a presentation about a topic which is about neural networks, one usually visualizes the networks architecture.
What are good / simple ways to visualize common ...
- 18.7k
114
votes
10
answers
123k
views
Choosing a learning rate
I'm currently working on implementing Stochastic Gradient Descent, SGD, for neural nets using back-propagation, and while I understand its purpose I have some ...
- 1,824
87
votes
1
answer
86k
views
When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization?
I knew that Residual Network (ResNet) made He normal initialization popular. In ResNet, He normal initialization is used , while the first layer uses He uniform initialization.
I've looked through ...
- 2,186
86
votes
8
answers
64k
views
Time series prediction using ARIMA vs LSTM
The problem that I am dealing with is predicting time series values. I am looking at one time series at a time and based on for example 15% of the input data, I would like to predict its future values....
- 1,075
80
votes
5
answers
44k
views
What is the difference between "equivariant to translation" and "invariant to translation"
I'm having trouble understanding the difference between equivariant to translation and invariant to translation.
In the book Deep Learning. MIT Press, 2016 (I. Goodfellow, A. Courville, and Y. Bengio)...
- 963
75
votes
6
answers
149k
views
What is the difference between Gradient Descent and Stochastic Gradient Descent?
What is the difference between Gradient Descent and Stochastic Gradient Descent?
I am not very familiar with these, can you describe the difference with a short example?
- 1,069
73
votes
6
answers
146k
views
Cross-entropy loss explanation
Suppose I build a neural network for classification. The last layer is a dense layer with Softmax activation. I have five different classes to classify. Suppose for a single training example, the <...
- 3,011
68
votes
5
answers
49k
views
Adding Features To Time Series Model LSTM
have been reading up a bit on LSTM's and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to ...
- 1,205
67
votes
11
answers
96k
views
Why should the data be shuffled for machine learning tasks
In machine learning tasks it is common to shuffle data and normalize it. The purpose of normalization is clear (for having same range of feature values). But, after struggling a lot, I did not find ...
- 13.9k
67
votes
4
answers
74k
views
Why mini batch size is better than one single "batch" with all training data?
I often read that in case of Deep Learning models the usual practice is to apply mini batches (generally a small one, 32/64) over several training epochs. I cannot really fathom the reason behind this....
- 8,387
67
votes
5
answers
28k
views
In softmax classifier, why use exp function to do normalization?
Why use softmax as opposed to standard normalization? In the comment area of the top answer of this question, @Kilian Batzner raised 2 questions which also confuse me a lot. It seems no one gives an ...
- 773
63
votes
4
answers
77k
views
Does batch_size in Keras have any effects in results' quality?
I am about to train a big LSTM network with 2-3 million articles and am struggling with Memory Errors (I use AWS EC2 g2x2large).
I found out that one solution is to reduce the ...
- 1,160
63
votes
5
answers
187k
views
How to get accuracy, F1, precision and recall, for a keras model?
I want to compute the precision, recall and F1-score for my binary KerasClassifier model, but don't find any solution.
Here's my actual code:
...
- 1,037
59
votes
5
answers
75k
views
Number of parameters in an LSTM model
How many parameters does a single stacked LSTM have? The number of parameters imposes a lower bound on the number of training examples required and also influences the training time. Hence knowing the ...
- 1,297
59
votes
3
answers
27k
views
How to fight underfitting in a deep neural net
When I started with artificial neural networks (NN) I thought I'd have to fight overfitting as the main problem. But in practice I can't even get my NN to pass the 20% error rate barrier. I can't even ...
- 713
51
votes
7
answers
42k
views
Deep Learning vs gradient boosting: When to use what?
I have a big data problem with a large dataset (take for example 50 million rows and 200 columns). The dataset consists of about 100 numerical columns and 100 categorical columns and a response column ...
- 1,615
48
votes
3
answers
46k
views
What does Logits in machine learning mean?
"One common mistake that I would make is adding a non-linearity to my logits output."
What does the term "logit" means here or what does it represent ?
- 1,027
47
votes
3
answers
102k
views
What is Ground Truth
In the context of Machine Learning, I have seen the term Ground Truth used a lot. I have searched a lot and found the following definition in Wikipedia:
In machine learning, the term "ground ...
- 13.9k
46
votes
5
answers
65k
views
Does gradient descent always converge to an optimum?
I am wondering whether there is any scenario in which gradient descent does not converge to a minimum.
I am aware that gradient descent is not always guaranteed to converge to a global optimum. I am ...
- 563
46
votes
4
answers
31k
views
Why is ReLU used as an activation function?
Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network.
Which I am able to understand ...
- 583
46
votes
2
answers
73k
views
Merging two different models in Keras
I am trying to merge two Keras models into a single model and I am unable to accomplish this.
For example in the attached Figure, I would like to fetch the middle layer $A2$ of dimension 8, and use ...
- 1,033
45
votes
4
answers
28k
views
Early stopping on validation loss or on accuracy?
I am currently training a neural network and I cannot decide which to use to implement my Early Stopping criteria: validation loss or a metrics like accuracy/f1score/auc/whatever calculated on the ...
- 1,239
44
votes
4
answers
57k
views
Multi GPU in Keras
How we can program in the Keras library (or TensorFlow) to partition training on multiple GPUs? Let's say that you are in an Amazon ec2 instance that has 8 GPUs and you would like to use all of them ...
- 579
44
votes
5
answers
40k
views
Intuitive explanation of Noise Contrastive Estimation (NCE) loss?
I read about NCE (a form of candidate sampling) from these two sources:
Tensorflow writeup
Original Paper
Can someone help me with the following:
A simple explanation of how NCE works (I found the ...
- 3,965
43
votes
13
answers
26k
views
Data science related funny quotes [closed]
It has been customary for the users of different communities to quote funny things about their fields. It may be fun to share your funny things about Machine Learning, Deep Learning, Data Science and ...
43
votes
2
answers
64k
views
Should we apply normalization to test data as well?
I am doing a project on an author identification problem. I applied the tf-idf normalization to train data and then trained an SVM on that data.
Now when using the classifier, should I normalize test ...
- 655
42
votes
5
answers
53k
views
What is the relationship between the accuracy and the loss in deep learning?
I have created three different models using deep learning for multi-class classification and each model gave me a different accuracy and loss value. The results of the testing model as the following:
...
- 1,975
41
votes
3
answers
49k
views
Choosing between CPU and GPU for training a neural network
I've seen discussions about the 'overhead' of a GPU, and that for 'small' networks, it may actually be faster to train on a CPU (or network of CPUs) than a GPU.
What is meant by 'small'?
For ...
- 1,981
40
votes
6
answers
52k
views
Are there free cloud services to train machine learning models?
I want to train a deep model with a large amount of training data, but my desktop does not have that power to train such a deep model with these abundant data.
I'd like to know whether there are any ...
- 13.9k
40
votes
6
answers
10k
views
How to set the number of neurons and layers in neural networks
I am a beginner to neural networks and have had trouble grasping two concepts:
How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever.
How does one decide ...
- 573
39
votes
1
answer
56k
views
How does Keras calculate accuracy?
How does Keras calculate accuracy from the classwise probabilities? Say, for example we have 100 samples in the test set which can belong to one of two classes. We also have a list of the classwise ...
- 1,032
37
votes
5
answers
45k
views
In the context of Deep Learning, what is training warmup steps
I found the term "training warmup steps" in some of the papers. What exactly does this term mean? Has it got anything to do with "learning rate"? If so, how does it affect it?
- 1,049
37
votes
2
answers
100k
views
Keras difference beetween val_loss and loss during training
What is the difference between val_loss and loss during training in Keras?
E.g.
...
36
votes
1
answer
13k
views
Paper: What's the difference between Layer Normalization, Recurrent Batch Normalization (2016), and Batch Normalized RNN (2015)?
So, recently there's a Layer Normalization paper. There's also an implementation of it on Keras.
But I remember there are papers titled Recurrent Batch Normalization (Cooijmans, 2016) and Batch ...
- 2,186
35
votes
6
answers
15k
views
Why do convolutional neural networks work?
I have often heard people saying that why convolutional neural networks are still poorly understood. Is it known why convolutional neural networks always end up learning increasingly sophisticated ...
- 451
35
votes
1
answer
15k
views
Time Series prediction using LSTMs: Importance of making time series stationary
In this link on Stationarity and differencing, it has been mentioned that models like ARIMA require a stationarized time series for forecasting as it's statistical properties like mean, variance, ...
- 785
33
votes
4
answers
72k
views
How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?
How do you use LeakyRelu as an activation function in sequence DNN in keras?
If I want to write something similar to:
...
- 1,794
32
votes
2
answers
38k
views
Are there any rules for choosing the size of a mini-batch?
When training neural networks, one hyperparameter is the size of a minibatch. Common choices are 32, 64, and 128 elements per mini batch.
Are there any rules/guidelines on how big a mini-batch should ...
- 18.7k
32
votes
4
answers
15k
views
Gumbel-Softmax trick vs Softmax with temperature
From what I understand, the Gumbel-Softmax trick is a technique that enables us to sample discrete random variables, in a way that is differentiable (and therefore suited for end-to-end deep learning)....
- 421
30
votes
7
answers
18k
views
Can machine learning learn a function like finding maximum from a list?
I have an input which is a list and the output is the maximum of the elements of the input-list.
Can machine learning learn such a function which always selects the maximum of the input-elements ...
- 309
29
votes
6
answers
10k
views
Deep learning basics
I am looking for a paper detailing the very basics of deep learning. Ideally like the Andrew Ng course for deep learning. Do you know where I can find this ?
- 433
29
votes
1
answer
25k
views
What is the difference between upsampling and bi-linear upsampling in a CNN?
I am trying to understand this paper and am unsure of what bi-linear upsampling is. Can anyone explain this at a high-level?
- 503
29
votes
2
answers
29k
views
What is the difference between fit() and fit_generator() in Keras?
What is the difference between fit() and fit_generator() in Keras?
When should I use fit() ...
- 1,975
28
votes
3
answers
54k
views
What is weight and bias in deep learning?
I'm starting to learn Machine learning from Tensorflow website. I have developed a very very rudimentary understanding of the flow a deep learning program follows (this method makes me learn fast ...
- 389
28
votes
3
answers
2k
views
Why are NLP and Machine Learning communities interested in deep learning?
I hope you can help me, as I have some questions on this topic. I'm new in the field of deep learning, and while I did some tutorials, I can't relate or distinguish concepts from one another.
- 449
28
votes
2
answers
18k
views
Is there away to change the metric used by the Early Stopping callback in Keras?
When using the early stopping callback in Keras, training stops when some metric (usually validation loss) is not increasing. Is there a way to use another metric (like precision, recall, or f-measure)...
- 393