Questions tagged [neural-network]
Artificial neural networks (ANN), are composed of 'neurons' - programming constructs that mimic the properties of biological neurons. A set of weighted connections between the neurons allows information to propagate through the network to solve artificial intelligence problems without the network designer having had a model of a real system.
1,422
questions with no upvoted or accepted answers
9
votes
4
answers
3k
views
Loss Function for Probability Regression
I am trying to predict a probability with a neural network, but having trouble figuring out which loss function is best. Cross entropy was my first thought, but other resources always talk about it in ...
- 191
8
votes
0
answers
132
views
Training value neural network AlphaGo style
I have been trying to replicate the results obtained by AlphaGo following their supervise learning protocol. The papers specify that they use a network that has two heads: a value head that predicts ...
- 101
8
votes
1
answer
252
views
How to predict advantage value in deep reinforcement learning
I'm currently working on a collection of reinforcement algorithms: https://github.com/lhk/rl_gym
For deep q-learning, you need to calculate the q-values that should be predicted by your network. There ...
- 181
7
votes
0
answers
2k
views
Fine tuning accuracy lower than Raw Transfer Learning Accuracy
I've used transfer learning on Inception V3 with ImageNet weights on Keras with Tensorflow backend on python 2.7 to create an image classifier. I first extracted and saved the bottleneck features from ...
- 71
6
votes
0
answers
138
views
Unable to transform (greatly performing) Autoencoder into Variational Autoencoder
Following the procedure described in this SO question, I am trying to transform my (greatly performing) convolutional Autoencoder into a Variational version of the same Autoencoder. As explained in ...
- 61
6
votes
2
answers
325
views
Training stateful LSTM with different number of sequences
I'm using a stateful LSTM for stock market analysis, and I have varying amounts of data for each stock, ranging from 20 years to just a few weeks (i.e. for newly listed stocks).
I use 3 years of data ...
- 740
6
votes
0
answers
372
views
Maths of Xavier initialization
The paper I read is Glorot et al (2010). And the math part is in Section 4.2.1.
Formula (5) and (10) make sense to me but I cannot derive formula (6) and (7) myself from (2) and (3).
I found many ...
- 61
6
votes
0
answers
130
views
Learning a logical function with a 2 layer BDN network - manual weight setting rule question?
So I am trying to construct a 2-layer network of binary decision neurons as proposed by McCullough and Pitts (1943) to learn a logical function (a composition of AND's and OR's) such as:
$((\neg x_1\...
- 61
6
votes
0
answers
289
views
how to propagate error from convolutional layer to previous layer？
I've been trying to implement a simple convolutional neural network. But I've been stuck at this problem for over a week.
To be specific, assume there are 3 layers in a convolutional pass, marked as ...
- 161
5
votes
1
answer
98
views
What ML architecture fits fixed length signal regression?
My problem is of regression type -
How to estimate a fish weight using a fixed-length signal (80 data points) of the change in resistance when the fish swim through a gate with electrodes (basically 4 ...
- 51
5
votes
0
answers
1k
views
Tensorflow, Optimizer.apply_gradient: 'NoneType' object has no attribute 'merge_call'
My program gives the following error message:
...
- 163
5
votes
4
answers
118
views
Reasoning behind using Deep Learning on non-local data
I understand the using of deep learning for data that have "local" structure, for example, images/videos/texts, as the convolutional layers reduce the amount of dimensions.
However, I saw ...
- 323
5
votes
1
answer
163
views
LSTM Long Term Dependencies Keras
I am familiar with the LSTM unit (memory cell, forget gate, output gate etc) however I am struggling to see how this links to the LSTM implementation in Keras.
In Keras the input data structure for X ...
- 171
4
votes
2
answers
1k
views
How propagate the error delta in backpropagation in convolutional neural networks (CNN)?
My CNN has the following structure:
Output neurons: 10
Input matrix (I): 28x28
Convolutional layer (C): 3
feature maps with a 5x5 kernel (output dimension is 3x24x24)
Max pooling layer (MP): size 2x2 ...
- 99
4
votes
1
answer
87
views
Training Machine Learning Model - Neural Network - Islands Problem
I was working on the following leetcode problem:
Given a 2d grid map of '1's (land) and '0's (water), count the number
of islands. An island is surrounded by water and is formed by
connecting ...
- 41
4
votes
4
answers
404
views
Backpropagation: Relevance of the error signal of a neuron
During my quest to understand back propagation in a more rigorous approach I have come across with the definition of error signal of a neuron which is defined as follows for the $j^{\text{th}}$ neuron ...
- 41
4
votes
2
answers
6k
views
Saving and loading keras.callbacks.History object with np.save and np.load
I have been saving my training history in keras as follows:
...
- 51
4
votes
1
answer
129
views
EM-ELM Cross validation
I know that cross validation is used to find the best hyperparameters that minimize the average error. For example, the number of neurons that minimize the average error of cross-validation is ...
- 41
4
votes
1
answer
67
views
Hochreiter LSTM (p. 4): Maximal values of logistic sigmoid derivative times weight
My questions follow the below page 4 excerpt from Hochreiter's LSTM paper:
If $f_{l_{m}}$ is the logistic sigmoid function, then the maximal
value of $f^\prime_{l_{m}}$ is 0.25. If $y^{l_{m-1}}$ ...
4
votes
1
answer
355
views
Why is my U-matrix visually not separating the classes?
I am visualizing the U-matrix generated using a Self Organizing Map codebook to (visually) identify regions of similarity in the data.
Although I would like to use SOM to identify clusters in ...
- 321
4
votes
1
answer
128
views
Neural Network - Sparsity of collaborative based filtering and modelling the prediction problem
I'm fairly new to machine learning and for that matter, neural networks, but for the past couple of days I decided to take a stab at a fairly classical and practical problem of neural networks/machine ...
- 161
4
votes
1
answer
259
views
How to automatically verify official documents?
I am new to machine learning and data science. I apologise if the question seems very basic. I have a requirement where I need to verify information submitted via a form with the corresponding ...
3
votes
1
answer
174
views
How is padding masking considered in the Attention Head of a Transformer?
For purely educational purposes, my goal is to implement basic Transformer architecture from scratch. So far I focused on the encoder for classification tasks and assumed that all samples in a batch ...
- 171
3
votes
0
answers
103
views
Intuitively, why do Non-monotonic Activations Work?
The swish/SiLU activation is very popular, and many would argue it has dethroned ReLU. However, it is non-monotonic, which seems to go against popular intuition (at least on this site: example 1, ...
- 53
3
votes
0
answers
1k
views
Combining heterogeneous numerical and text features
We want to solve a regression problem of the form "given two objects $x$ and $y$, predict their score (think about it as a similarity) $w(x,y)$". We have 2 types of features:
For each ...
- 81
3
votes
0
answers
30
views
How to train a neural network where computing the loss requires multiple object values?
I want to train a function that given metadata about an image produces hyper-parameters for an algorithm which operates on the image.
My understanding is (please forgive me I'm a novice here) a neural ...
3
votes
0
answers
2k
views
Hopfield Network python implementation, Network doesn't converge to one of the learned patterns
I'm trying to implement a Hopfield Network in python using the NumPy library. The network has 2500 nodes (50 height x 50 width). The network learns 10 patterns from images of size 50x50 stored in &...
- 31
3
votes
0
answers
650
views
What exactly negative/positive value of Captum's Integrated Gradient mean?
I use Captum's Integrated Gradient to interprete my PyTorch's neural network. I know that from github and original paper mentioned that ...
Positive attribution score means that the input in that ...
- 31
3
votes
0
answers
31
views
Force Matching in Coarse Grained Molecular Dynamics with Jax - Forces do not match when neglecting energy loss
I am currently exploring force matching approaches for molecular dynamic simulations.
As I am still in an exploration state, I'd tried investigated
Force Matching Neural Network Colab Notebook
...
3
votes
0
answers
78
views
How is the input gate in the LSTM learn?
How is the input gate neural network trained what to remember by propagating the error rate from predicting the next word in the language model? How does it help it to learn if it remembered the right ...
- 131
3
votes
0
answers
850
views
PyTorch: Train without dataloader (loop trough dataframe instead)
I was wondering if it is bad practice to instead of using built in tools such as dataloader just loop trough each row in a pandas df. Lets say I am doing text classification and my training loop looks ...
- 173
3
votes
1
answer
429
views
Neural Network regression negative performance
I have a problem with the performance of a multi layer perceptron regressor (neural network) and I cannot figure out why.
Task: I am trying to improve a time series prediction. I have predictions of a ...
- 31
3
votes
0
answers
50
views
NN training with repetitive features
I posted the question also on ai.stackexchange but it didn't get any answers so I though I could try here.
Here is a copy paste:
Let's say you are training a NN in a RL setting where the state (i.e. ...
- 131
3
votes
2
answers
53
views
Backpropagation with a different sized training set?
I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m $$
and whose output is a similarly sized array:
$$\vec{h}_{\theta,...
- 31
3
votes
0
answers
711
views
Understanding depthwise convolution vs convolution with group parameters in pytorch
So in the mobilenet-v1 network, depthwise conv layers are used. And I understand that as follows.
For a input feature map of (C_in, F_in, F_in), we take only 1 ...
- 91
3
votes
0
answers
43
views
Information Extraction from image / text - approach?
I need assistance with a ML project I am currently trying to create.
I receive a lot of invoices from a lot of different suppliers - all in their own unique layout. I need to extract 3 key elements ...
- 185
3
votes
3
answers
410
views
How to perform node classification using Graph Neural Networks
I'm am trying to perform node classification using graph neural network methods. My initial plan was to convert my graphs to adjacency matrices and train my network on that, with the node features ...
- 179
3
votes
2
answers
2k
views
Keypoint detection from an image using a neural network
I am trying to design and train a neural network, which would be able to give me coordinates of certain key points in the image.
Dataset
I've got a dataset containing 1800 images similar to these:
...
- 131
3
votes
1
answer
140
views
Physical modelling with neural networks - single output + stack ensemble vs multi-output
We are trying to replace an existing physical model (8 inputs/7 outputs) with artificial neural networks. The physics behind the existing model is mainly thermodynamics of humid air for air ...
- 31
3
votes
0
answers
216
views
How to create anchor-positive and anchor-negative pair for feature X in a signature data set for training Siamese network
How to create anchor-positive and anchor-negative pair for feature X in a signature data set for training Siamese network?
Im have a cedar signature data set with 55 peoples signatures(classes) with ...
- 1,431
3
votes
1
answer
738
views
should I shift a dataset to use it for Time series regression with RNN/LSTM?
I'm seeing this tutorial to know how to use LSTM to predict time series data and I noticed that he shifted the target/labels up so that the features are all in time t but the target is t+1
so my ...
- 243
3
votes
0
answers
381
views
Training an LSTM with different time steps and number of features
I want to use an LSTM using Keras to make course grade predictions. My dataset includes student transcripts, which consist of courses taken and their respective grades of students. For each course, I ...
- 31
3
votes
1
answer
405
views
What is the meaning of an empty SHAP graph in Explainable AI?
Using Python, I created a neural network to perform predictions on a binary class dataset (e.g. will a passenger survive the Titanic?). I am using the SHAP package to explain individual predictions. ...
- 31
3
votes
1
answer
2k
views
Training an ensemble of small neural networks efficiently in TensorFlow 2
I have a bunch of small neural networks (say, 5 to 50 feed-forward neural networks with only two hidden layers with 10-100 neurons each), which differ only in the weight initialization. I want to ...
- 440
3
votes
0
answers
80
views
Neural network cost is constant never changing during training
I am trying to build a binary classifier to predict a pulsar star with Single Hidden layer Neural Network.
But the cost on training dataset after almost 100 iterations has no change, following is the ...
- 141
3
votes
1
answer
64
views
Contextual Spell Correction
I want to create a spell checker that corrects the spelling mistakes contextually.
For example,
Erroneous sentence: I want to apply for credit cart
Corrected sentence: I want to apply for credit ...
- 51
3
votes
2
answers
128
views
ConvNet with concatenated data
I have a basic question regarding convolutional neural network. Assume I have a set of 1000 RGB images and I train a CNN from this set. I can obviously split each of my RGB images into 3 different ...
- 31
3
votes
0
answers
116
views
Computing derivatives for backpropagation across a convolution step
This will be a long post, but I hope it'll be instructive to anyone else in my position. I'm trying to find how the derivatives of the loss function are calculated with respect to the kernels and ...
- 299
3
votes
0
answers
46
views
A Deep CNN model delivering better results with standardization, when compared with normalization
I developed a deep CNN model, based on the architecture discussed in this paper, to generate predictions for time series data. My training data is shown in the figure below:
In order to train the ...
- 131
3
votes
1
answer
134
views
Stateful loss function and online regression
Working in tensorflow + keras I'm trying to define a custom loss function. N.B. I'm more interested in the value of the loss rather than the actual value of the predictions (this will be used for ...
- 31