Questions tagged [deep-learning]
a new area of Machine Learning research concerned with the technologies used for learning hierarchical representations of data, mainly done with deep neural networks (i.e. networks with two or more hidden layers), but also with some sort of Probabilistic Graphical Models.
1,743
questions with no upvoted or accepted answers
22
votes
1
answer
3k
views
Understanding Timestamps and Batchsize of Keras LSTM considering Hiddenstates and TBPTT
What I'm trying to do
What I am trying to do is predicting the next data-point $x_t$ for each point in the timeseries $[x_0, x_1, x_2,...,x_T]$ in the context of a date-stream in real-time, in theory ...
KenMarsu
9
votes
0
answers
2k
views
Why is my Keras model not learning image segmentation?
Edit: as is turns out, not even the model's initial creator could successfully fine-tune it. This is most likely a problem of implementation, or possibly related to the non-intuitive way in which the ...
- 199
8
votes
1
answer
251
views
How to predict advantage value in deep reinforcement learning
I'm currently working on a collection of reinforcement algorithms: https://github.com/lhk/rl_gym
For deep q-learning, you need to calculate the q-values that should be predicted by your network. There ...
- 181
7
votes
0
answers
1k
views
Multivariate, multistep forecasting with LSTM
I want to use an RNN with LSTM to forecast multiple steps into the future, based on multiple inputs. I have some ideas for different ways to approach this, but I'm afraid I'm missing the "right way" ...
- 171
7
votes
0
answers
2k
views
Fine tuning accuracy lower than Raw Transfer Learning Accuracy
I've used transfer learning on Inception V3 with ImageNet weights on Keras with Tensorflow backend on python 2.7 to create an image classifier. I first extracted and saved the bottleneck features from ...
- 71
6
votes
1
answer
2k
views
What is the minimum number of times a word needs to appear in word2vec training corpus for quality results?
When training a word2vec model with, eg, gensim, you can specify the minimum times a word needs to be seen (with the parameter min_count). The default value for this seems to be 5.
Are there any ...
- 203
6
votes
0
answers
372
views
Maths of Xavier initialization
The paper I read is Glorot et al (2010). And the math part is in Section 4.2.1.
Formula (5) and (10) make sense to me but I cannot derive formula (6) and (7) myself from (2) and (3).
I found many ...
- 61
6
votes
0
answers
289
views
how to propagate error from convolutional layer to previous layer？
I've been trying to implement a simple convolutional neural network. But I've been stuck at this problem for over a week.
To be specific, assume there are 3 layers in a convolutional pass, marked as ...
- 161
5
votes
0
answers
1k
views
Tensorflow, Optimizer.apply_gradient: 'NoneType' object has no attribute 'merge_call'
My program gives the following error message:
...
- 163
5
votes
4
answers
118
views
Reasoning behind using Deep Learning on non-local data
I understand the using of deep learning for data that have "local" structure, for example, images/videos/texts, as the convolutional layers reduce the amount of dimensions.
However, I saw ...
- 323
5
votes
1
answer
4k
views
Gradient flow through concatenation operation
I need help in understanding the gradient flow through a concatenation operation.
I'm implementing a network (mostly a CNN) which has a concatenation operation (in pytorch). The network is defined ...
- 151
4
votes
1
answer
1k
views
Difference Between Attention and Fully Connected Layers in Deep Learning
There have been several papers in the last few years on the so-called "Attention" mechanism in deep learning (e.g. 1 2). The concept seems to be that we want the neural network to focus on ...
- 836
4
votes
2
answers
1k
views
How is attention different from linear MLPs?
Each output for both the attention layer (as in transformers) and MLPs or feedforward layer(linear-activation) are weighted sums of previous layer. So how they are different?
4
votes
1
answer
2k
views
Self-attention mechanism did not improve the LSTM classification model
I am doing an 8-class classification using time series data.
It appears that the implementation of the self-attention mechanism has no effect on the model so I think my implementations have some ...
- 125
4
votes
2
answers
1k
views
How propagate the error delta in backpropagation in convolutional neural networks (CNN)?
My CNN has the following structure:
Output neurons: 10
Input matrix (I): 28x28
Convolutional layer (C): 3
feature maps with a 5x5 kernel (output dimension is 3x24x24)
Max pooling layer (MP): size 2x2 ...
- 99
4
votes
2
answers
936
views
How does ResNet bottleneck architecture's input size is possible to change from 56x56x64 to 56x56x356?
In ResNet papaer, First residual block's input size is 56x56x64 caused by 7x7x64 filter in first layer. But, in the paper, they showed residual block that has 56x56x256 input size. How does it is ...
- 115
4
votes
0
answers
575
views
How to use a ragged tensor with a convolutional layer?
I have textual data of various lengths for which ragged tensors seems well suited. For instance my data could look as follows :
...
4
votes
2
answers
6k
views
Saving and loading keras.callbacks.History object with np.save and np.load
I have been saving my training history in keras as follows:
...
- 51
4
votes
4
answers
658
views
Where does the "deep learning needs big data" rule come from
When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a ...
- 41
4
votes
1
answer
1k
views
Calculating saliency maps for text classification
I'm following the text classification with movie reviews TensorFlow tutorial, and wanted to extend the project by looking, for a certain input, which words influenced the classification the most.
I ...
- 41
4
votes
1
answer
409
views
Time horizon T in policy gradients (actor-critic)
I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture.
At the bottom of that slide, the gradient of the expected sum of rewards ...
4
votes
2
answers
235
views
Benefits of using Deep Learning-specific hyperparameter optimization tools vs. sklearn?
There are quite a few library for hyperparameter optimization that are specific to Keras or other Deep Learning libraries, like Hyperas or Talos.
My question is, what's the main benefit of using ...
- 161
4
votes
1
answer
276
views
Importance/intuition behind stacking RNNs
Nowadays there's a trend towards using architectures of "deep" RNNs i.e. vertically stacked RNNs. RNN chapter from Bengio's bookThese networks seem to work well in practice.
What's the intuition ...
- 1,297
4
votes
1
answer
302
views
How does Pooling Layer in CNN introduce invariance to other transformations besides translation
Here is a quote from deeplearningbook which I am trying to process. I am not sure what do they mean by this quote, can someone help me understand please?
Pooling over spatial regions produces ...
- 716
4
votes
3
answers
899
views
How to determine the number of the training images in Keras after data augmentaion?
I want to create a CNN model and I am using data augmentation. I want know the number of augmented images in Keras.
How to determine the number of the training images in Keras after data augmentation?...
- 1,985
3
votes
0
answers
103
views
Intuitively, why do Non-monotonic Activations Work?
The swish/SiLU activation is very popular, and many would argue it has dethroned ReLU. However, it is non-monotonic, which seems to go against popular intuition (at least on this site: example 1, ...
- 53
3
votes
1
answer
89
views
Reinforcement Learning applied to Optimisation Problem
Problem Statement: We are given an optimisation problem; with production centres, source airport, destination airports, transfer points and finally delivered to the customers. This is better explained ...
- 31
3
votes
0
answers
218
views
Why margin loss is used in Capsule Network instead of Cross Entropy loss?
I'm reading the Capsule Network paper proposed by Hinton. I'm not sure why the margin loss is used instead of the cross entropy loss. Any intuitive explaination for this?
- 131
3
votes
1
answer
195
views
VQ-GAN understanding
I tried to understand how VQ-GAN works, but unfortunately I have not understood it. I tried to read some articles about it and watch a video. I believe a good and simple article will help me. You ...
3
votes
1
answer
831
views
Explanation of why Neural Networks are non convex
Why having a symmetry of values for the hidden state imply that the neural network is non convex? I could not find an intuitive answer for this yet.
Also, if we consider a Fully Connected network wtih ...
- 141
3
votes
1
answer
440
views
What are the 'protos' in TF Object Detection?
I am struggling to understand what are the 'protos' in TF Object Detection?
Why do we need them here?
Also, while setting up the TF API we need to download and compile protocol buffers.
There is also ...
- 31
3
votes
3
answers
410
views
Can CNNs detect features of different images?
In lecture, we talked about “parameter sharing” as a benefit of using convolutional networks. Which of the following statements about parameter sharing in ConvNets are true? (Check all that apply.)
...
- 31
3
votes
1
answer
136
views
How to specify version for dependencies so that each one is compatible and stays within a size limit?
I am trying to deploy a web app to Heroku. The free tier is limited to 500 MB.
I am using my resnet34 model as a .pkl file.
I create model with it using the fastai ...
- 280
3
votes
0
answers
115
views
AlexNet Research Paper VS PytTorch and Tensorflow implementation
I'm making my way through Deep Learning research papers, starting with AlexNet, and I found differences in the implementation of PyTorch and Tensorflow that I can't explain.
In the research paper, ...
- 223
3
votes
1
answer
70
views
What's the best way to validate a rare event detection model during training?
When training a deep model for rare event detection (e.g. sound of an alarm in a home device audio stream), is it best to use a balanced validation set (50% alarm, 50% normal) to determine early ...
- 61
3
votes
0
answers
49
views
NN training with repetitive features
I posted the question also on ai.stackexchange but it didn't get any answers so I though I could try here.
Here is a copy paste:
Let's say you are training a NN in a RL setting where the state (i.e. ...
- 131
3
votes
1
answer
197
views
Is it possible to solve Rubik's cube using DQN?
I'm trying to solve Rubik's cube using deep learning and I came across with DQN, so I decided to give it a try. I developed all the code and started training but I got this results:
Loss goes up and ...
3
votes
0
answers
707
views
Understanding depthwise convolution vs convolution with group parameters in pytorch
So in the mobilenet-v1 network, depthwise conv layers are used. And I understand that as follows.
For a input feature map of (C_in, F_in, F_in), we take only 1 ...
- 91
3
votes
0
answers
328
views
Chess deep learning siamese network overfitting when shouldn't in theory
TLDR: My network is training with pairs so instead of 10^6 samples it has 10^12 samples (The number of samples squared) . With that large of a data set is shouldn't overfit but it does after very few ...
- 31
3
votes
1
answer
187
views
stacking features vs concatenating layers
I am trying to get to the logical intuition of differences between stacking multiple features and passing it via a final block (which could comprise multiple layers and lets say a final classification ...
- 328
3
votes
3
answers
406
views
How to perform node classification using Graph Neural Networks
I'm am trying to perform node classification using graph neural network methods. My initial plan was to convert my graphs to adjacency matrices and train my network on that, with the node features ...
- 179
3
votes
1
answer
47
views
Approach to classify blocks of time series
I am wondering if there exists an approach to classify blocks of time series, and not specifically individual time series. If so, can you point me out papers/articles/tutorials where these type of ...
- 151
3
votes
1
answer
74
views
Facial recognition architecture
Image recognition uses deep learning, and in particular CNNs to train on and recognise faces. Usually, this entails training on lots of data.
However, recently, we have seen face recognition being ...
- 1,973
3
votes
2
answers
136
views
How can we create an label, value detector?
I am trying to implement an text detector using MaskRCNN such that the model detects the label and value as shown in the image below.
Detecting the same is easier for fields like page date and order ...
- 81
3
votes
1
answer
724
views
Discriminator of a Conditional GAN with continuous labels
OK, let's say we have well-labeled images with non-discrete labels such as brightness or size or something and we want to generate images based on it. If it were done with a discrete label it could ...
- 203
3
votes
0
answers
103
views
Keras model with second to last sigmoid activated Conv1D layer followed by globalMaxPool outputs values outside [0,1]. Why?
I am trying to train a binary classifier. It is a residual network with skip layers etc. but ultimately, the bottom two layers are a 1D convolution with sigmoid activation followed by a global max ...
- 31
3
votes
2
answers
128
views
ConvNet with concatenated data
I have a basic question regarding convolutional neural network. Assume I have a set of 1000 RGB images and I train a CNN from this set. I can obviously split each of my RGB images into 3 different ...
- 31
3
votes
0
answers
39
views
Deep learning and label noise. Best practices for the real world
Unlike MNIST or other benchmark datasets collected data often come with subpar, inaccurate labels.
What are the best practices to help the neural networks to don't overfit the noise? Things that comes ...
- 191
3
votes
1
answer
88
views
What Models should i try for this problem?
I need some advice for a problem i'm working on with automobile data. The vehicles provide a series of codes at every second which are bieng stored, though it can vary how many.
For example , at time ...
- 131
3
votes
0
answers
46
views
A Deep CNN model delivering better results with standardization, when compared with normalization
I developed a deep CNN model, based on the architecture discussed in this paper, to generate predictions for time series data. My training data is shown in the figure below:
In order to train the ...
- 131