New answers tagged neural-network
0
If you use the scikit-learn GridSearchCV class (from sklearn.model_selection) together with the scikit-learn wrapper in keras, you can get your final model refit on the whole training set directly via the best_estimator_ attribute (i.e. the model instanced with the best hyperparms found in the CV process) already refit with the whole training dataset.
I have ...
0
I believe it should work if we update the last Softmax units and update the weight of the last layer accordingly.
I have tried this with MNIST digit and seems to work
#Let's assume "model" is the trained model on 10 outputs
saved_model = model.save('/content/model.h5')
model_reload = keras.models.load_model('/content/model.h5')
#Create model_2 ...
2
What your advisor is suggesting is called data leakage and it’s somewhat similar to training and testing your model on the same data. You might find it useful to read about p-hacking and backtest overfitting to get a feel for why this is a problem in quantitative finance. There’s also this excellent comic strip about the related concept of p-hacking...
The ...
0
So you are correct that the principle of backpropagation is to do the reverse of the operations. The same is true about the convolutional layer.
The forward pass of the convolutional layer can be expressed by
$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$.
Where $m$ and $n$ is the shape of the convolutional kernel that you will pass ...
4
You are absolutely correct that this is an problematic approach. Your testing set should only be used at the last possible stage before deploying a model.
By using your testing set to make modeling decisions you will introduce bias which will favor the observations found in your test set and may not generalize. In an ideal world, your test set would ...
1
"Wide and Deep network" was used in a 2016 paper.
$\hspace{5cm}$Wide & Deep Learning for Recommender Systems
You may create using keras Concatenate layer
concat = keras.layers.Concatenate()([input, hidden2])
References to read -
The paper
Google blog
Another paper
0
Unevenly spaced / irregular times series is totally different from (regular) time series analysis. In fact, in my knowledge there is no perfect statistical model for it. There are some packages in R like ust and few more.
1
It is possible to build that kind of a CNN. It is important to maintain uniform distribution for both the classes ('cat' and 'not cat'). That is you should have an almost equal number of samples for each of these classes to avoid biasing your model to the 'non-cat' class just because it has huge number of examples.
The number of non-cat examples can be ...
0
In the convolution layer of the convolutional neural network (CNN), each output value depends on a small number of input values, known as the sparsity of connections.
In neural network usage, "dense" connections connect all inputs.
By contrast, a CNN is "sparse" because only the local "patch" of pixels is connected, instead ...
1
Andrew Ng is making this point in comparison to a simple Neural network.
Let's say you have a 10x10 image,
In a dense neural network,
- We will connect every 100 neurons to the 100 in the next layer.(Dense)
- Over that, each all will have a distinct weight (No sharing)
So, total parm = 10K
In a Convolution Neural Network, the approach is as shown in this ...
1
In fact these involve different aspects of parameters in a CNN.
Parameter sharing, means one parameter may be shared by more than one input/connection. So this reduces total amount of independent parameters. Parameters shared are non-zero.
Sparsity of connections means that some parameters are simply missing (ie are zero), nothing to do with sharing same ...
0
First make sure your dataset is labelled properly in 5 distinct classes from 0 to 4 and no more.
Then add dropout layer in between each layer with prob of 0.1 and gradually increase till 0.5 until you find a good val score.
In your optimizer add weight decay term with value around 0.1.
Find some appropriate regularizer for your data type online.
0
This should work -
- Make your last layer as second last layer with activation='relu'
- Assign weights from the previous model
- Add a layer on top of it with 15 Neurons and activation='softmax'.
0
It will not
It's about using a model which was trained on thousands of Classes on Millions of images of ImageNet.
Chances are very high that most of the classes you have in your dataset is already there.
In general, if you trained a model on a super-class (e.g. vehicle), then you may reuse it to classify the Car variant(Utilizing its initial layers).
Point ...
2
Stacked LSTM is a special version of hierachical recurrent neural networks, where hard-wired memory and gating units help long-term preservation of state information.
Hierarchy and recurrence have been explored in many works.
One early example is the Neural Abstraction Pyramid, which introduced recurrent computation to hierarchical convolutional neural ...
0
Artificial Neural Networks are like a black box learning system. There is no known, or generally agreed upon, method that dictates what each weight represents or means for a given learning problem. Its internal representation of the problem is opaque to the architect.
In fact, the final trained weights are very closely tied to the neural network architecture ...
0
Maybe you can use the Concatenate layer
outputs = tf.keras.Concatenate()([model1, model2])
full_model = tf.keras.Model(inputs=inputs, outputs=outputs, name='full_model')
This will simply concatenate the two softmax output into one.
1
Have you looked into focal loss?
The idea seems to be similar to what you are describing - If predictions (~0.8) is close to GT Label of 1, it does not add to the loss value.
0
There’s a brilliant free interactive book here that explains how neural networks work if you want to understand them in more detail. The chapter I have linked to demonstrates that as long as there is at least one hidden layer, neural networks can approximate any function.
As fractalnature says above, if there were no hidden layers each of your output neurons ...
2
The simple option is to design your features so that they represent the distribution of the values: every feature $f_i$ represents a bin and its value for a particular instance is the frequency of the corresponding range for this instance.
Example: let's consider 10 bins between 0 and 1, i.e. $f_1=[0,0.1), f_2=[0.1,0.2),..., f_{10}=[0.9,1]$:
$x_1=[0.2, 0.25,...
1
X[train].shape[0] - This is the number of instances. Let's say it is M
X[train].shape[1] - This is the shape of each instance. Each instance is (1 x N)
Since input instances are of 1-D, the input data become m x N. Had it been 2-D, it would have been m x Nx x Ny
And one more question, I know that CNN required fixed input size. But I split my data into ...
0
Weights get updated based on the number of examples you feed in a batch. This is because, a full pass(forward and backward) of matrix computations has to be completed in order for the weights to be updated, after back-propagation and proceeding with next epoch, with batch type you had chose.
Moreover, If you use stochastic gradient descent, where each ...
0
I was thinking about this very question this week, and I had an idea. Forgive me if this is way off.
Suppose an n dimensional dataset to be sorted into k clusters.
If we have a 3 layer network:
input layer: n neurons
hidden layer: k neurons, softmax activation
output layer: n neurons, linear activation
If we use our dataset as both x and y (this is a typical ...
1
I would not say there is such a convention for it per se (if anyone has anything to comment on this, I would also like to know).
I think to make it clearer how the layer's input x interacts with the weights W, it might better to define the dimensions as the following:
x: (m x n)
W: (n x k)
bias term b: (k)
m remains as the number of examples. n represents ...
2
Most deep learning frameworks have APIs that are significantly similar to NumPy. I recommend you take a look at PyTorch as it will let you refactor your code reasonably intuitively to make use of your GPU via Cuda. Speaking as someone who has coded a neural network in NumPy, I would highly recommend learning a popular deep learning framework. It will be ...
0
Dimensionality reduction can be explained very easy:
Consider you have a huge matrix, all countries are listed on the horizontal and all possible features on the vertical and for each feature and each country there is a flag set in the matrix. A lot of the features share similarities from the patterns in which they appear, so that those features can be ...
1
Thanks for updating the post, this level of fluctuation in the validation set is a lot less dramatic than before and appears to be similar to regular fluctuation I have seen in my experience. Kudos that you have also managed to prevent the model from overfitting.
0
The function value is never exactly equal to those exact point because of numerical precision error.And again those functions in torch calculate left or right derivative which is defined in every case.So non-differentiability doesn't pose a problem here.
0
If $0.5$ is the threshold for declaring a class (perhaps more sensible in a binary classification than your problem, yes), there is no incentive for accuracy to regard a $1$ as a $0.95$ instead of a $0.51$.
Meanwhile your cross-entropy loss function sees that the correct answer is $1$ and wants to get the probability as close to $1$ as it can. Accuracy, ...
0
Consider loss to be something akin to implied variability of your model. When loss is extremely high, your model's output could be just about anything. As the loss lowers, your model is becoming more confident in its output and will be able to give similar output classification/regressions even if the initial weights or input data is slightly different. ...
0
My guess is that the data you provide does not have enough information to predict $a, b, c, d$ or $e$. Therefore, because $b$ is over-represented in the dataset, it will always predict $b$, because thats the safest bet. If you didn't know anything about the input or you if you wouldn't be able to extract any useful information from it, you would probably ...
0
You can optimize with non-gradient based methods. The field is called derivative-free optimization.
Local Search is one common approach for derivative-free optimization of neural networks.
0
Yes you can use neural network for any non linear function. Maybe your variables don't have a strong linear correlation but you can train it using neural network to find the best pattern possible . Although it is tough to say whether it will generalize or not
1
The vocab file contains a mapping from vocabulary strings and indices used for embedding lookup in the model.
The merges say how to split the input string into subword units. The algorithm is as follows: At the beginning of merging, a word split into characters and then you greedily search for neighboring symbols that can be merged (i.e., are in a list of ...
0
I think that if you append a token <EOS> (end of sentence) at the end of each sentence when you merge, this would not be a problem, because the RNN would learn to cut sentences and to generate independently if you shuffle your data and train with several shuffles.
However, as you say your data is heterogeneous, you might consider to first run some ...
0
From the documentation:
inputs: A 3D tensor with shape [batch, timesteps, feature].
Are you saying that you want 1 input and 1 feature, but you want to output 100 neurons? I would consider adding more timesteps.
train_X = train_X.reshape(train_X.shape[0],10,1)
test_X = test_X.reshape(test_X.shape[0],10,1)
Also, I wouldn't add regularization to a ReLU ...
0
73 millions trainable parms
- When using Transfer learning we first freeze the base model
- Train it till you reach good accuracy
- Then unfreeze it and train for just few epochs. Keep LR small
Other probable issues -
- If your labels are not One-Hot coded, please use sparse_categorical_crossentropy
- Add validation_split in fit method
- Suggest you add a ...
1
Theoretically, you could take many pictures and map these pictures to the score of each player. However, I would advise against it. First, you would need plenty of pictures and it might be infeasible to cover all possible game scenarios. Second, game scoring is discrete whereas a traditional neural net would approach it as a regression. This means that your ...
0
I think you should use a pre-trained neural network for image recognition and adjust the weights to detect the individual objects you need.
Afterwards, you will need to combine this with some good old-fashioned scripts to manually calculate the score.
Deep Learning doesn't do magic, even less with < 100 pictures of a game. If you managed to take a really ...
1
I'll refer to an answer for a similar topic. If you have enough data for classes 2 and 3, there's no reason to change your training scheme if you use standard metrics. The baseline should always be training without changing the weights, and if you see that the model does very bad on classes 2 and 3, you can change the training scheme. However, I have rarely ...
1
Class weights make sense only in the context of a loss function. When you validate your model you are making predictions and comparing to ground truth using a metric - but in that phase you aren't propagating back any changes, so weights are useless.
0
It has no value if it is same for all the training set.
Let's say, you are using a global health dataset for Life expectancy then country code can be a useful feature. It might contain hidden information.
But if you are doing the same analysis for one country e.g. India, keeping a feature country which has only one value e.g. India, will be of no use.
It ...
0
You are describing a problem of supervised learning with multiple inputs. That is not an uncommon task and you can find many tutorials about multiple inputs for neural networks out there. Using Tensorflow, I personally recommend Keras Functional API for this task, since it gives you more control on the layers while keeping the high-level simplicity.
0
Imagine you have a bunch of points that lie roughly on the line y=x. Although you could find a polynomial that passes through each and every single point you could argue that the line y=x is a better approximator because it doesn't fit to the noise of each point.
That is the point of regularization. When you have a network with smaller weights a few small ...
1
You have some wrong dimension here. Rules for Dim of weight $W^{[l]} = d^{l} * d^{l-1} $
$ W_0 = [ 4 * 2 ] $
$ W_2 = [ 2 * 4 ] $
As
$ dim (z^{[2]}) = [2 * 1] $
so is
$ \delta^{[2]}$
So
$ W_{2}^{T} \delta^{2} $ is $ [4 * 2 ] * [2 * 1] = [4 * 1 ] dimension $
0
See the docs of keras
import tensorflow as tf
model.compile( ..., metrics=[tf.keras.metrics.Precision(), tf.keras.metrics.Recall()])])
0
Changing the batch size will not change the overall training time too much. Since with any batch size you are passing almost 80K images.
One(and the best) approach will be to use transfer learning.
If you have a compelling reason to do full training, you will need a GPU powered bigger hardware. Google Colab can be an option. There are many other options ...
0
When you use keras fit, pass the value for x as a generator function which will provide (perhaps using yield) the batch of data (x, y) tuple. Also in the generator function, you can use checkpoint.
https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit
0
By scaling X but not y, and having small weights at initialization, I think you are making it hard for the NN to bring the weights up to the scale they need to be. Possibly if you let it train for more epochs it would find the right scale, but quick experimentation suggests it gets stuck into an oscillating loss that is quite large.
I had success in ...
0
When you use regularization, your loss will be larger because you add the regularization term. So, it is normal if your best loss without regularization is lower than your best loss with regularization.
However, adding regularization should not affect convergence in the long-term. Meaning that even though your overall loss might be larger, it should decrease ...
Top 50 recent answers are included
