Questions tagged [neural-network]
Artificial neural networks (ANN), are composed of 'neurons' - programming constructs that mimic the properties of biological neurons. A set of weighted connections between the neurons allows information to propagate through the network to solve artificial intelligence problems without the network designer having had a model of a real system.
4,362
questions
282
votes
12
answers
273k
views
What are deconvolutional layers?
I recently read Fully Convolutional Networks for Semantic Segmentation by Jonathan Long, Evan Shelhamer, Trevor Darrell. I don't understand what "deconvolutional layers" do / how they work.
The ...
- 18.8k
196
votes
6
answers
351k
views
How to draw Deep learning network architecture diagrams?
I have built my model. Now I want to draw the network architecture diagram for my research paper. Example is shown below:
- 2,467
195
votes
5
answers
146k
views
What is the "dying ReLU" problem in neural networks?
Referring to the Stanford course notes on Convolutional Neural Networks for Visual Recognition, a paragraph says:
"Unfortunately, ReLU units can be fragile during training and can
"die". For ...
- 4,025
179
votes
6
answers
180k
views
When to use GRU over LSTM?
The key difference between a GRU and an LSTM is that a GRU has two gates (reset and update gates) whereas an LSTM has three gates (namely input, output and forget gates).
Why do we make use of GRU ...
- 2,041
178
votes
20
answers
240k
views
How do you visualize neural network architectures?
When writing a paper / making a presentation about a topic which is about neural networks, one usually visualizes the networks architecture.
What are good / simple ways to visualize common ...
- 18.8k
150
votes
17
answers
126k
views
Best python library for neural networks
I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?
114
votes
11
answers
124k
views
Choosing a learning rate
I'm currently working on implementing Stochastic Gradient Descent, SGD, for neural nets using back-propagation, and while I understand its purpose I have some ...
- 1,824
109
votes
5
answers
80k
views
Backprop Through Max-Pooling Layers?
This is a small conceptual question that's been nagging me for a while: How can we back-propagate through a max-pooling layer in a neural network?
I came across max-pooling layers while going through ...
- 1,220
87
votes
1
answer
87k
views
When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization?
I knew that Residual Network (ResNet) made He normal initialization popular. In ResNet, He normal initialization is used , while the first layer uses He uniform initialization.
I've looked through ...
- 2,186
86
votes
4
answers
50k
views
How are 1x1 convolutions the same as a fully connected layer?
I recently read Yan LeCuns comment on 1x1 convolutions:
In Convolutional Nets, there is no such thing as "fully-connected layers". There are only convolution layers with 1x1 convolution ...
- 18.8k
81
votes
5
answers
46k
views
What is the difference between "equivariant to translation" and "invariant to translation"
I'm having trouble understanding the difference between equivariant to translation and invariant to translation.
In the book Deep Learning. MIT Press, 2016 (I. Goodfellow, A. Courville, and Y. Bengio)...
- 973
78
votes
6
answers
155k
views
What is the difference between Gradient Descent and Stochastic Gradient Descent?
What is the difference between Gradient Descent and Stochastic Gradient Descent?
I am not very familiar with these, can you describe the difference with a short example?
- 1,099
74
votes
6
answers
147k
views
Cross-entropy loss explanation
Suppose I build a neural network for classification. The last layer is a dense layer with Softmax activation. I have five different classes to classify. Suppose for a single training example, the <...
- 3,021
69
votes
11
answers
98k
views
Why should the data be shuffled for machine learning tasks
In machine learning tasks it is common to shuffle data and normalize it. The purpose of normalization is clear (for having same range of feature values). But, after struggling a lot, I did not find ...
- 13.9k
68
votes
5
answers
49k
views
Adding Features To Time Series Model LSTM
have been reading up a bit on LSTM's and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to ...
- 1,205
67
votes
2
answers
63k
views
Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)
Which is better for accuracy or are they the same?
Of course, if you use categorical_crossentropy you use one hot encoding, and if you use ...
- 773
63
votes
5
answers
192k
views
How to get accuracy, F1, precision and recall, for a keras model?
I want to compute the precision, recall and F1-score for my binary KerasClassifier model, but don't find any solution.
Here's my actual code:
...
- 1,037
62
votes
5
answers
74k
views
RNN vs CNN at a high level
I've been thinking about the Recurrent Neural Networks (RNN) and their varieties and Convolutional Neural Networks (CNN) and their varieties.
Would these two points be fair to say:
Use CNNs to break ...
- 745
60
votes
2
answers
61k
views
What is the difference between LeakyReLU and PReLU?
I thought both, PReLU and Leaky ReLU are
$$f(x) = \max(x, \alpha x) \qquad \text{ with } \alpha \in (0, 1)$$
Keras, however, has both functions in the docs.
Leaky ReLU
Source of LeakyReLU:
...
- 18.8k
60
votes
3
answers
27k
views
How to fight underfitting in a deep neural net
When I started with artificial neural networks (NN) I thought I'd have to fight overfitting as the main problem. But in practice I can't even get my NN to pass the 20% error rate barrier. I can't even ...
- 723
60
votes
5
answers
53k
views
Neural networks: which cost function to use?
I am using TensorFlow for experiments mainly with neural networks. Although I have done quite some experiments (XOR-Problem, MNIST, some Regression stuff, ...) now, I struggle with choosing the "...
- 723
54
votes
5
answers
34k
views
How do subsequent convolution layers work?
This question boils down to "how do convolution layers exactly work.
Suppose I have an $n \times m$ greyscale image. So the image has one channel.
In the first layer, I apply a $3\times 3$ ...
- 18.8k
51
votes
2
answers
89k
views
Why not always use the ADAM optimization technique?
It seems the Adaptive Moment Estimation (Adam) optimizer nearly always works better (faster and more reliably reaching a global minimum) when minimising the cost function in training neural nets.
Why ...
- 1,594
49
votes
2
answers
44k
views
What loss function to use for imbalanced classes (using PyTorch)?
I have a dataset with 3 classes with the following items:
Class 1: 900 elements
Class 2: 15000 elements
Class 3: 800 elements
I need to predict class 1 and class 3, which signal important ...
- 787
48
votes
4
answers
32k
views
Why is ReLU used as an activation function?
Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network.
Which I am able to understand ...
- 603
47
votes
3
answers
102k
views
What is Ground Truth
In the context of Machine Learning, I have seen the term Ground Truth used a lot. I have searched a lot and found the following definition in Wikipedia:
In machine learning, the term "ground ...
- 13.9k
46
votes
5
answers
66k
views
Does gradient descent always converge to an optimum?
I am wondering whether there is any scenario in which gradient descent does not converge to a minimum.
I am aware that gradient descent is not always guaranteed to converge to a global optimum. I am ...
- 563
46
votes
4
answers
29k
views
Early stopping on validation loss or on accuracy?
I am currently training a neural network and I cannot decide which to use to implement my Early Stopping criteria: validation loss or a metrics like accuracy/f1score/auc/whatever calculated on the ...
- 1,259
44
votes
2
answers
65k
views
Should we apply normalization to test data as well?
I am doing a project on an author identification problem. I applied the tf-idf normalization to train data and then trained an SVM on that data.
Now when using the classifier, should I normalize test ...
- 665
43
votes
13
answers
27k
views
Data science related funny quotes [closed]
It has been customary for the users of different communities to quote funny things about their fields. It may be fun to share your funny things about Machine Learning, Deep Learning, Data Science and ...
43
votes
5
answers
56k
views
What is the relationship between the accuracy and the loss in deep learning?
I have created three different models using deep learning for multi-class classification and each model gave me a different accuracy and loss value. The results of the testing model as the following:
...
- 1,985
42
votes
1
answer
36k
views
How to decide neural network architecture?
I was wondering how do we have to decide how many nodes in hidden layers, and how many hidden layers to put when we build a neural network architecture.
I understand the input and output layer ...
- 535
41
votes
3
answers
49k
views
Choosing between CPU and GPU for training a neural network
I've seen discussions about the 'overhead' of a GPU, and that for 'small' networks, it may actually be faster to train on a CPU (or network of CPUs) than a GPU.
What is meant by 'small'?
For ...
- 1,981
41
votes
2
answers
52k
views
How to calculate mAP for detection task for the PASCAL VOC Challenge?
How to calculate the mAP (mean Average Precision) for the detection task for the Pascal VOC leaderboards?
There said - at page 11:
Average Precision (AP). For the VOC2007 challenge, the interpolated
...
- 649
41
votes
2
answers
42k
views
How to prepare/augment images for neural network?
I would like to use a neural network for image classification. I'll start with pre-trained CaffeNet and train it for my application.
How should I prepare the input images?
In this case, all the ...
- 3,142
41
votes
4
answers
50k
views
Guidelines for selecting an optimizer for training neural networks
I have been using neural networks for a while now. However, one thing that I constantly struggle with is the selection of an optimizer for training the network (using backprop). What I usually do is ...
- 511
40
votes
6
answers
52k
views
Are there free cloud services to train machine learning models?
I want to train a deep model with a large amount of training data, but my desktop does not have that power to train such a deep model with these abundant data.
I'd like to know whether there are any ...
- 13.9k
40
votes
6
answers
11k
views
How to set the number of neurons and layers in neural networks
I am a beginner to neural networks and have had trouble grasping two concepts:
How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever.
How does one decide ...
- 573
40
votes
1
answer
21k
views
The difference between `Dense` and `TimeDistributedDense` of `Keras`
I am still confused about the difference between Dense and TimeDistributedDense of Keras ...
- 503
39
votes
3
answers
35k
views
Why use both validation set and test set?
Consider a neural network:
For a given set of data, we divide it into training, validation and test set. Suppose we do it in the classic 60:20:20 ratio, then we prevent overfitting by validating the ...
- 1,346
39
votes
1
answer
56k
views
How does Keras calculate accuracy?
How does Keras calculate accuracy from the classwise probabilities? Say, for example we have 100 samples in the test set which can belong to one of two classes. We also have a list of the classwise ...
- 1,042
38
votes
4
answers
47k
views
What is the meaning of "The number of units in the LSTM cell"?
From Tensorflow code: Tensorflow. RnnCell.
num_units: int, The number of units in the LSTM cell.
I can't understand what this means. What are the units of LSTM ...
- 849
37
votes
1
answer
36k
views
RNN's with multiple features
I have a bit of self taught knowledge working with Machine Learning algorithms (the basic Random Forest and Linear Regression type stuff). I decided to branch out and begin learning RNN's with Keras. ...
- 1,205
35
votes
9
answers
13k
views
Why is it wrong to train and test a model on the same dataset?
What are the pitfalls of doing so and why is it a bad practice? Is it possible that the model starts to learn the images "by heart" instead of understanding the underlying logic?
- 461
35
votes
6
answers
16k
views
Why do convolutional neural networks work?
I have often heard people saying that why convolutional neural networks are still poorly understood. Is it known why convolutional neural networks always end up learning increasingly sophisticated ...
- 451
35
votes
1
answer
70k
views
What is the best Keras model for multi-class classification?
I am working on research, where need to classify one of three event WINNER=(win, draw, lose)
...
- 557
34
votes
4
answers
16k
views
Gumbel-Softmax trick vs Softmax with temperature
From what I understand, the Gumbel-Softmax trick is a technique that enables us to sample discrete random variables, in a way that is differentiable (and therefore suited for end-to-end deep learning)....
- 441
33
votes
4
answers
25k
views
Neural Network parse string data?
So, I'm just starting to learn how a neural network can operate to recognize patterns and categorize inputs, and I've seen how an artificial neural network can parse image data and categorize the ...
32
votes
6
answers
118k
views
Validation loss is not decreasing
I am trying to train a LSTM model. Is this model suffering from overfitting?
Here is train and validation loss graph:
- 571
32
votes
4
answers
13k
views
Role derivative of sigmoid function in neural networks
I try to understand role of derivative of sigmoid function in neural networks.
First I plot sigmoid function, and derivative of all points from definition using python. What is the role of this ...
- 467