Questions tagged [neural-network]
Artificial neural networks (ANN), are composed of 'neurons' - programming constructs that mimic the properties of biological neurons. A set of weighted connections between the neurons allows information to propagate through the network to solve artificial intelligence problems without the network designer having had a model of a real system.
275
questions
40
votes
6
answers
11k
views
How to set the number of neurons and layers in neural networks
I am a beginner to neural networks and have had trouble grasping two concepts:
How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever.
How does one decide ...
- 573
46
votes
5
answers
66k
views
Does gradient descent always converge to an optimum?
I am wondering whether there is any scenario in which gradient descent does not converge to a minimum.
I am aware that gradient descent is not always guaranteed to converge to a global optimum. I am ...
- 563
35
votes
6
answers
16k
views
Why do convolutional neural networks work?
I have often heard people saying that why convolutional neural networks are still poorly understood. Is it known why convolutional neural networks always end up learning increasingly sophisticated ...
- 451
282
votes
12
answers
273k
views
What are deconvolutional layers?
I recently read Fully Convolutional Networks for Semantic Segmentation by Jonathan Long, Evan Shelhamer, Trevor Darrell. I don't understand what "deconvolutional layers" do / how they work.
The ...
- 18.8k
195
votes
5
answers
146k
views
What is the "dying ReLU" problem in neural networks?
Referring to the Stanford course notes on Convolutional Neural Networks for Visual Recognition, a paragraph says:
"Unfortunately, ReLU units can be fragile during training and can
"die". For ...
- 4,025
68
votes
5
answers
49k
views
Adding Features To Time Series Model LSTM
have been reading up a bit on LSTM's and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to ...
- 1,205
44
votes
2
answers
65k
views
Should we apply normalization to test data as well?
I am doing a project on an author identification problem. I applied the tf-idf normalization to train data and then trained an SVM on that data.
Now when using the classifier, should I normalize test ...
- 665
69
votes
11
answers
98k
views
Why should the data be shuffled for machine learning tasks
In machine learning tasks it is common to shuffle data and normalize it. The purpose of normalization is clear (for having same range of feature values). But, after struggling a lot, I did not find ...
- 13.9k
17
votes
4
answers
6k
views
Can a neural network compute $y = x^2$?
In spirit of the famous Tensorflow Fizz Buzz joke and XOr problem I started to think, if it's possible to design a neural network that implements $y = x^2$ function?
Given some representation of a ...
- 317
81
votes
5
answers
46k
views
What is the difference between "equivariant to translation" and "invariant to translation"
I'm having trouble understanding the difference between equivariant to translation and invariant to translation.
In the book Deep Learning. MIT Press, 2016 (I. Goodfellow, A. Courville, and Y. Bengio)...
- 973
39
votes
3
answers
35k
views
Why use both validation set and test set?
Consider a neural network:
For a given set of data, we divide it into training, validation and test set. Suppose we do it in the classic 60:20:20 ratio, then we prevent overfitting by validating the ...
- 1,346
32
votes
4
answers
13k
views
Role derivative of sigmoid function in neural networks
I try to understand role of derivative of sigmoid function in neural networks.
First I plot sigmoid function, and derivative of all points from definition using python. What is the role of this ...
- 467
23
votes
2
answers
38k
views
Why ReLU is better than the other activation functions
Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
- 13.9k
6
votes
2
answers
7k
views
Does MLP always find local minimum
In linear regression we use the following cost function which is a convex function:
We Use the following cost function
in ...
- 13.9k
178
votes
20
answers
240k
views
How do you visualize neural network architectures?
When writing a paper / making a presentation about a topic which is about neural networks, one usually visualizes the networks architecture.
What are good / simple ways to visualize common ...
- 18.8k
150
votes
17
answers
126k
views
Best python library for neural networks
I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?
48
votes
4
answers
32k
views
Why is ReLU used as an activation function?
Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network.
Which I am able to understand ...
- 603
43
votes
5
answers
56k
views
What is the relationship between the accuracy and the loss in deep learning?
I have created three different models using deep learning for multi-class classification and each model gave me a different accuracy and loss value. The results of the testing model as the following:
...
- 1,985
12
votes
1
answer
5k
views
Using a pre trained CNN classifier and apply it on a different image dataset
How would you optimize a pre-trained neural network to apply it to a separate problem? Would you just add more layers to the pre-trained model and test it on your ...
- 667
9
votes
2
answers
2k
views
Transform an Autoencoder to a Variational Autoencoder?
I would like to compare the training by an Autoencoder and a variational autoencoder. I have already run the traing using AE. I would like to know if it's possible to transform this AE into a VAE and ...
- 614
6
votes
2
answers
8k
views
What are useful evaluation metrics used in machine learning
I am using CNN in order to predict codes after analyzing text. As an example, I will write "I am crazy" .. the model will predict some code " X321".
All this based on CNN.
I want to evaluate my ...
- 215
3
votes
1
answer
375
views
Understanding computations of Perceptron and Multi-Layer Perceptrons on Geometric level
I am currently watching amazing Deep Learning lecture series from Carnegie Melllon University, but I am having little bit of trouble understanding how Perceptrons and MLP are making their decisions on ...
- 716
109
votes
5
answers
80k
views
Backprop Through Max-Pooling Layers?
This is a small conceptual question that's been nagging me for a while: How can we back-propagate through a max-pooling layer in a neural network?
I came across max-pooling layers while going through ...
- 1,220
87
votes
1
answer
87k
views
When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization?
I knew that Residual Network (ResNet) made He normal initialization popular. In ResNet, He normal initialization is used , while the first layer uses He uniform initialization.
I've looked through ...
- 2,186
86
votes
4
answers
50k
views
How are 1x1 convolutions the same as a fully connected layer?
I recently read Yan LeCuns comment on 1x1 convolutions:
In Convolutional Nets, there is no such thing as "fully-connected layers". There are only convolution layers with 1x1 convolution ...
- 18.8k
74
votes
6
answers
147k
views
Cross-entropy loss explanation
Suppose I build a neural network for classification. The last layer is a dense layer with Softmax activation. I have five different classes to classify. Suppose for a single training example, the <...
- 3,021
33
votes
4
answers
25k
views
Neural Network parse string data?
So, I'm just starting to learn how a neural network can operate to recognize patterns and categorize inputs, and I've seen how an artificial neural network can parse image data and categorize the ...
32
votes
6
answers
118k
views
Validation loss is not decreasing
I am trying to train a LSTM model. Is this model suffering from overfitting?
Here is train and validation loss graph:
- 571
24
votes
2
answers
20k
views
What's the difference between the cell and hidden state in LSTM?
LSTM cells consist of two types of states, the cell state and hidden state.
How do cell and hidden states differ, in terms of their functionality? What information do they carry?
user105907
23
votes
2
answers
8k
views
Why do activation functions have to be monotonic?
I am currently preparing for an exam on neural networks. In several protocols from former exams I read that the activation functions of neurons (in multilayer perceptrons) have to be monotonic.
I ...
- 18.8k
15
votes
3
answers
61k
views
Keras Sequential model returns loss 'nan'
I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value.
I have sigmoid activation ...
- 405
14
votes
3
answers
4k
views
Why is there a $2$ at the denominator of the mean squared error function?
In the famous Deep Learning Book, in chapter 1, equation 6, the Quadratic Cost (or Mean Squared Error) in a neural network is defined as
$ C(w, b) = \frac{1}{2n}\sum_{x}||y(x)-a||^2 $
where $w$ is ...
- 161
11
votes
2
answers
10k
views
Why large weights are prohibited in neural networks?
Why weights with large values cause neural networks to be overfitted, and consequently we use approaches like regularization to neutralize weights with large values?
- 13.9k
10
votes
5
answers
16k
views
In which epoch should i stop the training to avoid overfitting
I'm working on an age estimation project trying to classify a given face in a predefined age range. For that purpose I'm training a deep NN using the keras library. The accuracy for the training and ...
- 188
9
votes
1
answer
5k
views
When does decision tree perform better than the neural network?
I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree ...
- 601
8
votes
1
answer
40k
views
How to download dynamic files created during work on Google Colab?
I have two different files and on the first, I tried to save data to file as:
np.save(open(Q1_TRAINING_DATA_FILE, 'wb'), q1_data)
On second file, i'm trying to ...
- 185
8
votes
2
answers
13k
views
How to plot cost versus number of iterations in scikit learn?
One of the recommendations in the Coursera Machine Learning course when working with gradient descent based algorithms is:
Debugging gradient descent. Make a plot with number of iterations on the x-...
- 215
5
votes
1
answer
6k
views
Is it possible to customize the activation function in scikit-learn's MLPClassifier?
Scikit-learn lists these as the implemented activation functions for it's multi-layer perceptron classifier:
...
- 91
3
votes
1
answer
458
views
When should I oversample data?
I am dealing with multi-class classifiers. My data is unbalanced. Hence, I need to apply sampling techniques before training (undersampling or oversampling). When I apply undersampling, ...
- 151
2
votes
1
answer
511
views
Compute backpropagation
I have the question which is mentioned in the above picture. It is trying to find the derivative of f with respect to weight matrix ...
- 571
2
votes
2
answers
2k
views
Explain Binary Classification with output 0.5 (True)
What is the interpretation of output 0.5 of a typical classifier?
I made a prediction and the probability of that data point being from the True class is 0.5.
- 339
2
votes
2
answers
2k
views
Finding optimal weights for models
I'm trying to implement an algorithm to find the minimal value of a function.
Before moving to sigmoid activation functions, i'm trying to understand linear regression.
Usually, a gradient descent ...
- 409
1
vote
1
answer
95
views
Does derivative of an activation function refer to process of back-propogation in ANN?
I am confused that in ANN, we use Gradient Descent to find the curve of the optimal points of predicted values.
Let's say we use MSE (Mean Squared Error) and the lost function is 1/2m * (y-y')^2
And ...
- 13
196
votes
6
answers
351k
views
How to draw Deep learning network architecture diagrams?
I have built my model. Now I want to draw the network architecture diagram for my research paper. Example is shown below:
- 2,467
42
votes
1
answer
36k
views
How to decide neural network architecture?
I was wondering how do we have to decide how many nodes in hidden layers, and how many hidden layers to put when we build a neural network architecture.
I understand the input and output layer ...
- 535
41
votes
2
answers
42k
views
How to prepare/augment images for neural network?
I would like to use a neural network for image classification. I'll start with pre-trained CaffeNet and train it for my application.
How should I prepare the input images?
In this case, all the ...
- 3,142
39
votes
1
answer
56k
views
How does Keras calculate accuracy?
How does Keras calculate accuracy from the classwise probabilities? Say, for example we have 100 samples in the test set which can belong to one of two classes. We also have a list of the classwise ...
- 1,042
37
votes
1
answer
36k
views
RNN's with multiple features
I have a bit of self taught knowledge working with Machine Learning algorithms (the basic Random Forest and Linear Regression type stuff). I decided to branch out and begin learning RNN's with Keras. ...
- 1,205
28
votes
3
answers
2k
views
Why are NLP and Machine Learning communities interested in deep learning?
I hope you can help me, as I have some questions on this topic. I'm new in the field of deep learning, and while I did some tutorials, I can't relate or distinguish concepts from one another.
- 449
28
votes
2
answers
18k
views
Is there away to change the metric used by the Early Stopping callback in Keras?
When using the early stopping callback in Keras, training stops when some metric (usually validation loss) is not increasing. Is there a way to use another metric (like precision, recall, or f-measure)...
- 393