New answers tagged neural-network
0
You can increase the weight of the class.
model.fit(class_weight={0: 1., 1: 3.}) #weight class 0 once and class 1 three times
0
You can train the network to optimize for recall instead of accuracy.
from tensorflow.keras.metrics import Recall
model.compile(metrics=[Recall()])
0
Neural networks work as a great encoder/decoder for any task you give them. To create a good representation of data, you require sparse representations. The dead neurons actually contribute to that. ReLU actually help the process.
In fact, a recent paper actually justified that ReLU is even better than LeakyReLU which you mentioned above. They start with the ...
0
In practice, dead ReLUs connections are not a major issue. Most deep learning networks can still adequate representations with only sub-selection of possible connections. This is because deep learning networks are highly over-parameterized.
Even with the possible drawbacks backs of the dying ReLUs problem, the computational effectiveness and efficiency of ...
1
There is usually no straightforward interpretation of what cross-entropy means in the context of the given task. In practice, it is more important to follow the trend of how the cross-entropy develops during the training.
The measure comes from the information theory. It says how surprised the model is when it has some belief about the output (a distribution)...
1
You can use the below type code.
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, LSTM, Concatenate
from tensorflow.keras.models import Model
# input of first NN
input_l1 = Input(shape=(2,))
out_l1 = Dense(1)(input_l1)
# input is 2nd NN
input_l2 = Input(shape=(2,))
out_l2 = Dense(1)(input_l2)
# concat layer output shape will be (...
0
If you are using TensorFlow to write your architecture,
You can create 3 training datasets( one for one class) using tf.data and then you can use tf.data.experimental.sample_from_datasets with the same weightage to generate the batch data. Check weights parameter in the documentation.
You can initialize well in the final layer. Please check this blog from ...
0
If implemented properly, there should be no difference. The very first thing that happens with the indices is corresponding embeddings are loaded. From this perspective, there is no difference between having the pad embedding at the 0th or at the 1000th position.
When you use padding, you should always do masking on the output and other places where it is ...
0
To convert numpy array to tensor,
import tensor as tf
#Considering y variable holds numpy array
y_tensor = tf.convert_to_tensor(y, dtype=tf.int64)
#You can use any of the available datatypes that suits best - https://www.tensorflow.org/api_docs/python/tf/dtypes/DType
1
This kind of overfitting is typical when finetuning large LMs.
The usual approaches to "avoid" it are:
Early stopping: select the checkpoint with the best validation loss.
Random restarts: train multiple times from scratch, and select the model with the best validation performance.
0
Which is better single frame similarly check or sequence check?
Sequence Check. Because in the Single frame Similarity check (Consider SNN), You need to input pairwise images. So that you need so many pairwise images for your SNN for your Batch Generation.
For the Batch Generation idea is to make usuable batches for training the network. We need to create ...
0
It could be any of those factors. The root cause can be found through experimentation. Hold everything constant and change a single factor. Then systematically change each factor while holding all other factors constant.
0
You have to use 1-D CNN.
Before that you must prepare your dataset in a sequential format. e.g.
Below is a depiction of how the Convolution will work [Source - D2L],
Study these references for Conv1D:
Machine learning mastery
Dive into Deep Learning
0
Given your mask is very small, you should look at reducing your convolutions to 2x2 since that will help aggregate more information from these smaller masks. EfficientNet has 3x3 and 5x5 convolutions which may not be suited for your purpose.
It is a better idea for you to train from scratch using smaller convolutions (2x2). Also, since you will lose the edge ...
0
First, let me give you an intuitive explanation. When you drop a pebble in water, you see the ripples being formed. Imagine that in reverse. All those ripples comes together at the point from which they started.
Node embeddings are like that. You take the information of neighbourhood nodes and combine it with the information in the original node. The art ...
0
Each sample of your dataframe is a 1D vector, you need to Conv1D (1D convolutions) with filter sizes of 3 or 5.
If you want to convert each sample of yours into an image, you need to do some pre-processing. Here's a paper which does that: paper.
Also, since this a tabular data, if you want to stick to neural networks then consider TabNet. It has ...
1
These are the equation of Adam [Ref - Dive Into Deep Learning]
\begin{aligned}
\mathbf{v}_t & \leftarrow \beta_1 \mathbf{v}_{t-1} + (1 - \beta_1) \mathbf{g}_t \\
\mathbf{s}_t & \leftarrow \beta_2 \mathbf{s}_{t-1} + (1 - \beta_2) \mathbf{g}_t^2
\end{aligned}
\begin{aligned}
\hat{\mathbf{v}}_t = \frac{\mathbf{v}_t}{1 - \beta_1^t} \text{ ...
0
One approach is to have a dummy class that represents no treatment and use the accuracy score via a threshold (lower than threshold) correspond to no treatment at all.
Threshold as used above becomes a new hyper-parameter and you have new input-output pairs that are now exact (depending on threshold).
0
About the type of convolutions
Suppose the input layers has the $k_{input}$ channels, than the number of parameters to be learned by neural network is:
$$
3 \cdot 3 \cdot k_{input} \cdot k_1 + 5 \cdot 5 \cdot k_1 \cdot k_2
$$
Because each of the input channels of the image is mapped to one of the output channels. There is a separate filter to each pair $(...
1
Building on @JahKnows theoretical answer, here is what the weights of Conv2D look like in action.
from keras import *
from keras.layers.convolutional import Conv2D
model = Sequential()
model.add(Conv2D(12, kernel_size=3, input_shape=(25, 25, 1)))
#just initialized, not fit to any data.
>>> weights[0].shape
'(3, 3, 1, 12)'
So a 3x3 matrix (9 ...
0
MLP typically refers to a type of neural network called a 'Multi-Layer Perceptron'. As you can read on the Wikipedia page, these neural networks consist of neurons organized in layers. Each neuron has a (typically fixed) function that transforms its weighted input to produce the output for that particular neuron known as its activation function.
A 'network ...
1
I can think of two very good resources that may help you to acquire the foundational knowledge on graph theory and complex networks:
Network Science by Barabasi is you prefer a web tutorial: http://networksciencebook.com/
The Structure of Complex Networks by Estrada if you prefer a book: https://oxford.universitypressscholarship.com/view/10.1093/acprof:oso/...
0
According to the Cambridge Dictionary saturation means
the act or result of filling a thing or place completely so that no
more can be added
In this context, it refers to a function for which a bigger input will not lead to a relevant increase in output. So if the gradient is saturated (meaning it is extremely close to zero), a bigger upstream gradient ...
0
@Erwan probably gave you a better idea, but for a simpler problem, when each token has a separate class, this can be viewed as a Part of Speech Tagging, which is essentially a per-token multiclass problem. This assumes each token can be mapped only to one correct class. In such case the LSTM's output would be size, e.g. (1, seq_length, hidden_dim), which you ...
0
You should have a list of actual classes, e.g. classes = ['Superman', 'Batman', ...,'Gozilla']. The model outputs per-class logits, but without your dataset interface it's hard to say what your targets is. Since it's a multiclass problem, it should be an integer between 0 and 5. I assume the order of targets and the order of classes in classes list is the ...
0
Well, yes there are. One of the applications of GANs is image inpainting which is widely used in Photoshop-like applications. You can omit an obstacle that prohibits you to see the entire image and use GANs to see, generate, the original object without any other disturbing objects that may not let you see the desired object entirely.
1
In various ways suchs as
momentum: think of momentum as a means of dampening oscillations and speeding up the iterations, leading to faster convergence. This means replacing gradients with a leaky average over past gradients.
sparse features and preconditioning (Adagrad): decrease the learning rate dynamically on a per-coordinate basis. This means, use the ...
0
There are several different ways to frame the problem.
One way is multiclass classification. The goal would be to assign a single discrete label to every phrase. In order to get phrases, you'll have to build a parse tree first. You did not list of all of the labels but let's assume they are all nouns. Then you'll need a Part-Of-Speech Tagger (POS Tagger) to ...
0
If you're dealing with classification problem, then model.predict is supposed to give you logits.
outputs = net(images)
_, predicted = torch.max(outputs, 1)
for i in range(num_input):
print(classes[predicted[i]])
if you have only one input then the predicted class would be as following:
classes[predicted[0]]
In your case:
prediction = F.softmax(...
1
Technically this is sequence labeling, the most common application being Named Entity Recognition.
However it looks like in this case you're trying to solve a problem of coreference resolution, which is a quite difficult task in general. I think this usually involves a more complex model than simple sequence labeling, but I'm not an expert in this. You might ...
0
Nlp like spacy remove stop works and identify proper nouns and nouns. It does fairly well at identifying nouns but is not perfect. Try a sample of your data and see how accurate your percentages become. Use the medium model
0
This might help if what you're asking is related to merging models:
Merging two different models in Keras
2
If you're seeing performance that is much better on the validation than the unseen test data, then that is suggestive of some sort of overfitting or, if not, that the data do not come from the same distribution. That could mean that your test images are very different from the training and validation data, for example.
First, I'd double check the data to ...
1
Why is potentially hidden in your feature explanations.
Given that your Features give some indication on why, like Patient demographics data, or some other Features that you can include here, you can use them to answer the why.
Like this using shapley and or eli5 you can integrate it with your Standard predictor classes and for the explanations with user ...
1
The power to model non-linear decision boundaries comes directly from the non-linear activation function. You can understand this when you see that concatenating N linear transformations (i.e. dense layers) is equivalent to a single linear transformation. This is the mathematical proof with 2 linear layers:
$ y = (xW_1 + b_1) W_2 + b_2 = x W_1 W_2 + (b_1 W_2 ...
2
Dropout (2014 paper) is my first thought.
By effectively removing N% of your neurons on each pass through the data, you make it harder for any two neurons to work together. When its buddy disappears it is forced to find another way to learn the patterns in the data. On the next epoch, when its old buddy comes back, it has a new perspective on life, its ...
0
Yes, these type of loss functions can be optimized using backpropagation, also in Tensorflow.
The value of the loss is a scalar (same as just the cross entropy, or the MSE, otherwise you wouldn't be able to add them), which means that it doesn't really work any different from just optimizing for any other scalar loss function. As long as the operations ...
3
In principle, it is possible to reuse the special tokens as you describe.
However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain better translation quality.
Another option would be to reuse just the embeddings instead of the whole model.
1
1 - Activation functions are non-linear functions. These are added in between layers which are simply Linear transformations.
Example without activation function:
ConvLayer1(Input) -> ConvMaps1
ConvLayer2(ConvMaps2) -> ConvMaps2
Mathematically, this would be $I_{nput} \circledast K_{ernel_1} \circledast K_{ernel_2} $,
which is equivalent to $I_{nput} ...
0
Why activation functions: For this , its straight forward to add some non-linearity into the model, and this helps in defining which neuron to fire which should not.
For second one , the compute will increase slightly but not so rapidly , why because the convolution operation you are performing on the first layer will reduce the image size in the second ...
1
The choice for the number of neurons in the last two dense layers and the number of filter is somewhat arbitrary and most of the time determined by trying different configurations (using something like a hyperparameter grid search). See also this answer on stats stackexchange. If you want to change the size of the 5x5 kernels you will only have to change the ...
1
All these parameters are trainable.
Note that in normal Transformers it is typical to have fixed (non-trainable) positional embeddings, but in BERT they are learned.
Note also the "pooler" component, which is an extra projection that was not mentioned in the paper, but which the authors commented on later.
1
I'll make this as explicit and non handwavy as possible, at the risk of being boring.
Let us take a small toy example: a network with two inputs $x_1$ and $x_2$, two hidden layers with two units each and one output. For simplicity, we will assume that the activation function is the identity function (i.e. no activation) and that the objective function is the ...
1
I think that the "neurons" analogy is not very helpful to understand what is going on with artificial neural networks.
Neural networks are not comprised by "neurons", but by differentiable operations. These operations are arbitrary, e.g. convolutions, indexing (in embeddings), pooling, etc.
What you proposed is a perfectly valid building ...
0
There can be many answers for this question but probably cause it is just an unnecessary complexity.
You can achieve same result (x2) with the current architecture (i.e. using multiplication layer; not to mention just squaring your input features). Why would you use something more specific and not something more general?
Beside that why would your final ...
1
Backpropagation extends to the full model, through all decoder and encoder layers up to the embedding tables.
1
The example you cited (using x^2 instead of x) is the idea more popular outside deep learning community, called feature engineering. The trend in neural network modeling is instead to,
Play with weights (w) and fine tune them.
Not change the input vector (x) but feed it to the network directly.
If a single layer neural network is not good enough, add more ...
0
More gpu means more data in a batch. And the gradients of a batch data is averaged for back-propagation.
If the learning rate of a batch is fixed, then the learning rate of a data is smaller.
If the learning rate of a data is fixed, then the learning rate of a batch is larger.
https://github.com/guotong1988/BERT-GPU
0
Your model is very small for a convnet. 1 conv layer, 1 maxpool and 1 fc is very shallow. Try adding more layers and batchnorm2d after each conv layer followed by relu. No pool layers.
1
To me those are separate things since both models have a different cost function to be optimized.
On the other hand you could combine those models by constructing embeddings based on random forest splits and then using those embeddings as inputs for a neural network.
Toy example shows that there is a non-trivial configuration of a neural net that can get as ...
Top 50 recent answers are included
