4
The answer to your needs is called "bucketing". It consists of creating batches of sequences with similar length, to minimize the needed padding.
In tensorflow, you can do it with tf.data.experimental.bucket_by_sequence_length. Take into account that previously it was in a different python package (tf.contrib.data.bucket_by_sequence_length), so the ...
3
The Keras Documentation indicates that filename can have the epoch number in the filename. To allow each set of data to be preserved consider using a filename more like:
checkpoint_filepath = './checkpoint-{epoch:02d}.hdf5'
filepath: string or PathLike, path to save the model file.
filepath can contain named formatting options, which will be filled the ...
3
Please have a look at your weights after training. I assume your Neurons die due to relu activation as they output Zero for Input < 0.
Unfortunately, the ReLU activation function is not perfect. It suffers
from a problem known as the dying ReLUs: during training, some neurons
effectively “die,” meaning they stop outputting anything other than 0.
In some ...
3
Because the data is time series while only Dense layers are used in the model, the problem is caused by model initialization. A model with a 'bad' initialization will constantly predict zero, as you will see by running the script below.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
...
2
Think of loss function what to minimize and optimizer how to minimize the loss.
loss could be mean absolute error and in order to reduce it, weights and biases are updated after each epoch. optimizer is used to calculate and update them.
1
This should do the trick:
tf.keras.utils.plot_model(
model, to_file='model.png', show_shapes=False, show_layer_names=True,
rankdir='TB', expand_nested=False, dpi=96
)
This will generate an image like:
Example: https://www.tensorflow.org/guide/keras/functional
1
Found a solution, which is to pass a custom batch generator of type keras.utils.Sequence to the model.fit function (where one can write any logic to construct batches and to modify/augment training data) instead of passing the entire dataset in one go. Relevant code for reference:
# Must implement the __len__ function returning the number
# of batches in ...
1
You don't need that at Inference time. It's for training purposes.
You can skip these Layers while exporting to JavsScript. These layers do not have weights.
On this example [Link], which has these Layers, I removed these layers and it worked fine.
# Added this code to remove the first 4 Aug layers
input = model.input
model_export = input
for layer in ...
1
Data Augmentation is basically a technique to increase the diversity of your training set by applying random (but realistic) transformations such as image rotation. Training deep learning neural network models on more data can result in more skillful models, and the augmentation techniques can create variations of the images that can improve the ability of ...
1
This can be explained only if you specify the total number of samples in your entire data set. The 200 Gb file size might be because of a large number of features for each sample in the dataset. The is how steps_per_epoch and validation_steps work.
The training generator will yield steps_per_epoch batches.
When the epoch ends, the validation generator will ...
1
First images are turned into pixel values. If it is a color image we have RGB combination of values. It it is black and white then the pixel values for each image in the dataset are unsigned integers in the range between black and white, or 0 and 255. This is the first transformation into numbers.
After that we apply an embedding to it to convert it into an ...
1
This is because of the data type you provide to the imshow() function.
Check the documentation:
The function may scale the image, depending on its depth:
If the image is 8-bit unsigned, it is displayed as is. If the image is
16-bit unsigned or 32-bit integer, the pixels are divided by 256. That
is, the value range [0,255*256] is mapped to [0,255]. If the ...
1
Your loss is cross entropy, your variant of gradient descent is stochastic gradient descent, and your optimizer for stochastic gradient descent would seem to be the momentum optimizer according to the keras docs.
(https://keras.io/api/optimizers/sgd/).
Here is the intuition:
The loss is a way of measuring the difference between your target label(s) and your ...
1
In tf.keras.preprocessing.text (docs) you have utilities to process discrete token sequences, normally used to represent text.
In tf.keras.preprocessing.sequence (docs) you have utilities to process both continuous value sequences (normally used to represent time series) like TimeSeriesGenerator, and discrete token sequences (i.e. text), like the skipgrams ...
1
Getting reproducible results with Keras is not straightforward, see https://machinelearningmastery.com/reproducible-results-neural-networks-keras/, since TensorFlow, Numpy and the working environment itself can introduce different seeds affecting different parts of the training. In order to get reproducible training processes, try setting the following ...
1
Their is some random elements when using packages such as TensorFlow, Numpy etc. Some examples includes:
How the weights are initialized.
How the data is shuffled (if enabled) in each batch. Batches containing different data, will produce different gradients which might influence convergence.
This means, that even when you run the same code it is actually ...
1
Its the number of features that has to remain consistent not the number of timestamps.
Outputs will be batches of one row predictions, so in your case its 1500 and than 1000. Model does not care, features should remain same. And beware of dataleakage with time series.
1
You may use lambda callback and save it in a dictionary.
weights_dict = {}
weight_callback = tf.keras.callbacks.LambdaCallback \
( on_epoch_end=lambda epoch, logs: weights_dict.update({epoch:model.get_weights()}))
history = model.fit( x_train, y_train, batch_size=16, epochs=5, callbacks=weight_callback )
# retrive weights
for epoch,weights in weights_dict....
1
I have been able to find an answer in Tensorflow Warrior's answer here.
In Keras, when an LSTM(return_sequences = True) layer is followed by Dense() layer, this is equivalent to LSTM(return_sequences = True) followed by TimeDistributed(Dense()).
When return_sequences is set to False, Dense is applied to the last time step only.
Number of parameters were ...
1
The 'Attention' terminology varies before and after a landmark paper in 2018 - Attention is all you need.
Before 2018
Here 'query' is the hidden state of the decoder of the previous timestep.
'Values' - All the hidden states of the encoder
Remember - the 'query' attends to all the 'values'
So far so good. Attention mechanisms were used widely between 2014 ...
Only top voted, non community-wiki answers of a minimum length are eligible
