New answers tagged

0 votes

How to optimize my CNN classification architecture

From looking at what you've provided for your problem i'd consider the following; Reduce the Number of Layers: The original model had multiple convolutional and dense layers, which increased the model'...
RegressIt's user avatar
  • 131
1 vote
Accepted

Why was the learning rate decreased for Roberta compared to LSTM?

The learning rate is a hyperparameter, and it has to be tuned for a specific neural architecture. Also, the optimal learning rate is affected by other hyperparameters, like the batch size. We don't ...
noe's user avatar
  • 25.6k
1 vote

Why does degradation occur in deep neural networks?

Why do deep networks need to learn identity functions? The authors talk about linear functions because they are thinking "If a network with fewer layers should work, why should a network with ...
Josiah Yoder's user avatar
3 votes
Accepted

What do these terms mean in the context of Roberta?

Fine-tuning means that you modify the model to adapt it to your data. Feature extraction means that you don't modify the model. In the case of RoBERTa, it means that you feed your data to the model, ...
noe's user avatar
  • 25.6k
1 vote
Accepted

What are the differences between Embedding Layer and Roberta Embedding?

An embedding layer is just a building block to be used as part of neural architectures. It is just a lookup table whose purpose is to represent tokens as vectors, and to learn these vectors as part of ...
noe's user avatar
  • 25.6k
0 votes

Extraction of name from phonetic transcription

This could be solved with a sequence-to-sequence model, but I feel this might be overkill for your use case. In your case, it seems that you are trying to label part of the phonetic transcription as a ...
Valentin Calomme's user avatar
1 vote
Accepted

What are the differences between contextual embeddings of Bidirectional-LSTM and Transformer?

Here are some differences: Computational complexity: LSTMs have linear complexity $O(n)$, because you need to process input tokens one by one, while transformers have constant $O(1)$ complexity ...
noe's user avatar
  • 25.6k
0 votes

How to learn certain Maths to understand machine Learning papers?

you can study basic maths of a competitive examination maybe take 8-9 months of 10-11 hours studying each day say the JEE, but it is a prrofoundly the most optimal way you can study and grasp machine ...
Nishchay Kumar's user avatar
1 vote

Is vision transformer (ViT) always better than CNN?

There are CNN models which can match Transformers. See ConvNext, EfficientNet, etc. ViT is also an old vision transformer architecture. Swin transformers are more performant nowadays.
Cory Fan's user avatar
5 votes
Accepted

The end-to-end Training Process for Knowledge Distillation

Correct. Nevertheless, you may see some variations, e.g. when the number of classes is very large, the size of the transfer dataset can be huge (this is the case of text generation, where the ...
noe's user avatar
  • 25.6k
1 vote
Accepted

Questions about hidden states of bidirectional LSTMs

Let's take a look at the following diagram (source): A bidirectional LSTM consists of two independent LSTMs $A$ (green) and $A'$ (darker green). The input sequence $X_0...X_i$ (blue) is passed to ...
noe's user avatar
  • 25.6k
1 vote
Accepted

Train Reward Model using Llama2:

Problem is solved. The issue is max_length. when lower value used to max_length this issue is not occurs. that means 30GB gpu not enogh to for this process.
Sandun Tharaka's user avatar
0 votes

Why do neural networks with more layers perform better?

You raise a good point, the same number of parameters in a shallow network perform less than a deeper one! My theory is that it is about the optimization methodology like Backpropagation or SGD. Also ...
Emad Ezzeldin's user avatar
0 votes

Seeking Guidance on Constrained Input Modeling for Soil Moisture Correction Using Rainfall Observations

What if you consider a multi-modal approach? From the context I assume you have images and some tabular data, then what if you use an autoencoder to extract information from the images, and then use ...
timmy1691's user avatar
4 votes
Accepted

What are the differences between BPE and byte-level BPE?

Byte-level BPE is a subtype of BPE that uses bytes instead of characters as basic token component. In the character-level BPE, the vocabulary is composed of sequences of characters that appear ...
noe's user avatar
  • 25.6k
-1 votes

Why does hyperparameter tuning occur on validation dataset and not at the very beginning?

The primary reason for this approach lies in the need to prevent overfitting and accurately assess the generalization performance of the model. Let's delve into the details: Preventing Overfitting: ...
Boby Sinha's user avatar
0 votes

Intuition behind the RNN/LSTM hidden state?

I don't think you are too far off. Here's Geoff Hinton explaining his motivation for using the phrase 'hidden' in his work on neural nets in the late 80's: The reason hidden units in neural nets are ...
Complex Manifold's user avatar
1 vote
Accepted

What is the difference between hidden states in RNN and Transformers model?

The hidden states in RNNs and Transformers share their purpose, but they work in different ways. In RNNs, the hidden state is a way for the network to maintain a memory of previous time steps. The ...
noe's user avatar
  • 25.6k
1 vote

Why not Back propagate through time in LSTM , similar to RNN

In practice you backprop through time. "Backprop through time" is just a fancy way of saying you truncate your input data to a maximum sequence length. Your hidden states backprop to ...
Karl's user avatar
  • 471
1 vote

Anomaly Detection in Log Data using LSTM

The anomalies are defined by deviations from historical patterns. I respectfully disagree. I mean, sure, we identify anomalies by observing deviations from historical patterns. But we're doing AD ...
J_H's user avatar
  • 802
0 votes

Seeking Guidance on Constrained Input Modeling for Soil Moisture Correction Using Rainfall Observations

observational data for rainfall Well, clearly temperature, cloud cover, and dew point are going to matter, as well. I assume you can readily obtain such figures. CNN regression approach Given the ...
J_H's user avatar
  • 802
0 votes

Heuristic argument for Weight decay and regularization

In L2 weights are updated by: $w' \rightarrow w(1-\frac{\eta\lambda}{n})-\frac{\eta}{m}\sum_x\frac{\partial C_x}{\partial w}$. During the first epochs of training most of the neurons in the network ...
Tsvi's user avatar
  • 1
2 votes

text extraction from bank statements from pdf format

I think the most convenient way to do that in Python is by using a tabula library. I happened to do similar task with pdf files and probably the easiest way to do so is to use tabula.read_pdf function....
Tomasz Witkowski's user avatar
0 votes

Bert model for document sentiment classification

Consider using other models that are Bert-based, those models often are more developed in terms of the final applications. For example, this is the open-source Bert-based encoder classification model ...
mikepetterson's user avatar
1 vote

Fine-tuning NLP models

Consider few-shot learning approaches, recently found guys that practice such approaches for their information extraction models, claiming you will need somewhere between 8 training examples. But it ...
mikepetterson's user avatar

Top 50 recent answers are included