Questions tagged [deep-learning]

a new area of Machine Learning research concerned with the technologies used for learning hierarchical representations of data, mainly done with deep neural networks (i.e. networks with two or more hidden layers), but also with some sort of Probabilistic Graphical Models.

Filter by
Sorted by
Tagged with
6 votes
2 answers
2k views

Why does degradation occur in deep neural networks?

It has been shown that "plain" neural networks tend to have an increased amount training error, and accompanied test error, as more layers are added. I am not quite certain as to why this occurs. In ...
0 votes
0 answers
10 views

Why was the learning rate decreased for Roberta compared to LSTM?

I'm reading the codebase of a project that uses Bidirectional-LSTM. The learning rate for it is 0.02. Later, someone improved the project by replacing LSTM with Roberta and decreased the learning rate ...
0 votes
0 answers
24 views

How to find adjacent neighbours using Python?

Full problem description at stackoverflow I need to find the adjacent neighbours (not necessarily nearest neighbours) to a given point in a multidimensional space. As shown in the screenshot below, I ...
2 votes
2 answers
72 views

Newly discovered learning rule

Does anyone know how this algorithm performs the learning process for neural networks? I've stumbled over this solution. It works, but I don't know how and why. It's neuron-local and works without ...
1 vote
1 answer
66 views

Why VAE Encoder outputs log variance and not standard deviation?

When talking about VAE (and viewing VAE implementations), the Encoder outputs: μ, log(variance) when we train the model (the ...
0 votes
1 answer
66 views

How can I explain the cause of different performances for two different LSTM models and improve the performance?

I've built two different models for Load Forecasting. Dataset has six features. The performance evaluation metric is the Mean Absolute Percentage Error(MAPE). Both models are based on LSTM. Here is ...
2 votes
1 answer
172 views

What do these terms mean in the context of Roberta?

When I read articles about Roberta, I often read the terms "transfer learning" and "fine-tuning". Additionally, they also mention "feature extraction". What are the ...
0 votes
1 answer
76 views

Predict the values of variable features over timestamps

HI i am having a dataset which contain timestamps and number of users at that timestamp. Each user has resource values which change per timestamp. How can i make predictions of number of users ...
0 votes
0 answers
29 views

Why is there a difference in Training Accuracy Output, when the training dataset is the same but the validation dataset is different?

I am looking at the output of a multi-class image segmentation deep learning model. I used U-Net to implement this. I am confused about why the training accuracies are different for a different ...
4 votes
4 answers
681 views

Where does the "deep learning needs big data" rule come from

When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a ...
2 votes
2 answers
1k views

How to remove background (watermark) logo from image

I have been scratching my head for a while. What I have is a scanned PDF document with text and water marked logo at the back as in the below image. I want to do OCR over this, which becomes very ...
1 vote
2 answers
370 views

How gradients are flown back to Network in siamese architecture? How weights of all CNN models are same even when using different models

TL;DR: Intuition behind the gradient flow in Siamese Network? How can 3 models share the same weights? And if 1 model is used, how Gradients are updated from 3 different paths? I am trying to build a ...
0 votes
0 answers
22 views

How to optimize my CNN classification architecture

I have this CNN based model architecture that takes an RGB image. Now I'm trying to change it for a color classification case on an object (10 color classes: white, black, yellow, etc). This current ...
0 votes
0 answers
18 views

Why do the Llama 2 weights have eight different files?

I downloaded the weights for Llama 2 (70B-chat). This process created a folder titled "llama-2-70b-chat," which contained 8 files titled consolidated.00.pth, consolidated.01.pth, and so on ...
0 votes
1 answer
23 views

What are the differences between Embedding Layer and Roberta Embedding?

I'm reading an article about the Embedding Layer: The Embedding Layer learns word embeddings from raw text. It is initialized with small random numbers and can be learned simultaneously with a neural ...
1 vote
1 answer
13 views

Extraction of name from phonetic transcription

I have a use case where I want to extract the name from the phonetic transcription. For example if the phonetic transcription is - “m a j n e j m ɪ z s ʌ m i ɹ z o w ʃ i”, the output should be the ...
0 votes
1 answer
97 views

Why a CNN with decreasing filter layers sizes could perform better than a "regular one" with increasing sizes?

I did dozens (or probably hundreds) of tests and the best result with less total parameters(4 times or less) was a decreasing filter layers size architecture. This is a CNN for multiclass image ...
2 votes
2 answers
126 views

Why does a filter need to be applied to the output of the input gate before cell state is added to?

In a neural network there are 4 gates: input, output, forget and a gate whose output performs element wise multiplication with the output of the input gate, which is added to the cell state (I don't ...
0 votes
1 answer
351 views

predict_classes() returning only 0 or 1 for multiclass image classification

I am trying to build a multi class image classifier but the only returns 0 or 1 . Why is it not returning "Rock" , "Paper" , "Scissor" ? and why only 0 and 1 but not 2? CODE: ...
3 votes
1 answer
2k views

Is finetuning from a pretrained model always better than training from scratch?

At the worst case scenario, we could treat the pretrained weights as a random initialization, same as what we would do for training from scratch, right? If that is the case, then wouldn't it be better ...
1 vote
2 answers
476 views

Negative examples for a Yes/No image classification neural network

I am trying to retrain a neural network using transfer learning that can classify whether an image has a certain object, say, a car. My positive sample dataset is quite small, only 2500~ images. It ...
0 votes
1 answer
51 views

What are the differences between contextual embeddings of Bidirectional-LSTM and Transformer?

A Transformer, like Roberta, can generate contextual embeddings using the encoder part, similar to a Bidirectional-LSTM that concatenates hidden states. What are the differences between them ? Are ...
2 votes
1 answer
1k views

RBF neural network python library/implementation

I want to use a Radial Basis Function Neural Network for my thesis. Is there any library that implements it? And in the negative case, which is the best library to implement it?
1 vote
3 answers
348 views

Understanding the concept vanishing gradient and exploding gradient problem in terms of training data

I'm trying to figure out the essence of the concepts "vanishing gradient and exploding gradient problem" in terms of real-world input-output training examples instead of in terms of the properties of ...
0 votes
1 answer
168 views

Unbalanced dataset on image classification, is it better to lose samples and balance it?

I am dealing with a binary image classifier. I'm using a CNN to predict if an image is positive or negative. The problem is that the positive class represents only the 2% of the total samples. In this ...
1 vote
3 answers
183 views

How to learn certain Maths to understand machine Learning papers?

I have done the deeplearning.ai course on deep learning. But I cannot Understand equations like minGmaxDV(D,G)=Ex∼pdata(x)[logD(x)]+Ez∼pz(z)[log(1−D(G(z)))] ...
1 vote
1 answer
627 views

Training neural network for regression with gaussian output layer

How does one train a neural network model that does regression over real values, using a gaussian output layer? ie estimating the mean and std parameters of the prediction. Since during training there ...
3 votes
2 answers
1k views

Neural network approach to the cocktail party effect

Imagine you have 2 people at 2 different microphones but in the same room. Each microphone is going to pick up some sound from the other person. Is there a good neural network based approach to ...
2 votes
1 answer
444 views

Cat2Vec implementation X = categorical and y = categorical

I am trying to convert categorical values (zipcodes) with Cat2Vec into a matrix which can be used as an input shape for categorical prediction of a target with binary values. After reading several ...
2 votes
1 answer
2k views

Backpropagation of convolutional neural network - confusion

I've already seen many articles about this topic and Backpropagation In Convolutional Neural Networks by Jefkine seems to be the best. Although, as author said, For the purposes of simplicity we ...
1 vote
1 answer
243 views

LSTM with input of actual time step

I'm working on an implementation of LSTM neural network to forecast energy consumption. I have a dataset with load, series of weather parameters and indicator of it's bank holiday or not. I first ...
0 votes
2 answers
294 views

Self-Attention Summation and Loss of Information

In self-attention, the attention for a word is calculated as: $$ A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>} $$ My question is why we sum over the ...
2 votes
2 answers
469 views

Estimating the uncertainty of regression models

Given a regression model, with n features, how can I measure the uncertainty or confidence of the model for each prediction? Suppose for a specific prediction the accuracy is amazing, but for another ...
1 vote
1 answer
40 views

Is vision transformer (ViT) always better than CNN?

The paper - AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE proposed vision transformer and outperformed CNN-based models in many cases. When it comes to sequential data, we ...
0 votes
1 answer
471 views

Validation loss not decreasing using dense layers altough training and validation data have the same distribution

I have a problem that I have great difficulties understanding the concept that leads to these results. I use a keras dense layer to map 13 input features to 3 output labels. During the training, the ...
3 votes
1 answer
318 views

Why does joint embedding of word and images work?

I often see some papers where the authors do point-wise multiplication of word and image embedding (e.g the image below). Why does this implementation works? I do not understand.
0 votes
1 answer
97 views

What can we learn from visualizing Feature Maps

I have the following classification model (dogs vs cats): ...
4 votes
1 answer
48 views

The end-to-end Training Process for Knowledge Distillation

I'm a bit confused on the complete training process for Knowledge Distillation. I was reading the Geoffrey Hinton "Distilling the Knowledge in a Neural Network" 2015 paper and some random ...
2 votes
2 answers
567 views

Neural network model for sparse multi-class classifier on Tensorflow

The problem I'm trying to solve is the following: the data is Movielens with N_users=6041 and N_movies=3953, ~1 million ratings. For each user, a vector of size N_movies is defined, and the values ...
0 votes
1 answer
160 views

Train and Validation Curve

I'm new in DeepLearning. I'm not good at understanding and commenting on graphics.Can you help me with these graphs
0 votes
0 answers
9 views

how to get similar traning data for a test sample?

I want to get the most similar samples for test sample on which the model choose it's - output. SHAP isn't useful because it show the contribution of each feature. I want to get the most similar ...
0 votes
1 answer
205 views

How many bounding boxes does the YOLOv6 model predict in total before thresholding?

I understand that the YOLOv5 model predicts 25200 bounding boxes between all 3 levels of output. How many does the YOLOv6 model predict, if the input resolution is 640x640?
0 votes
1 answer
476 views

Tensorflow model works for classification but not for regression (all predictions equal the output layer bias)

I'm trying to build a model for FX prediction. It's giving some promising results for classifying each period as buy/sell/neutral. When used as a classifier, actual returns are converted to 0, 1, or ...
0 votes
1 answer
96 views

A way to init sentence embedding for unsupervised text clustering, better than glove wordvec?

For unsupervised text clustering, the key thing is the init embedding for text. If we want to use deepcluster for text, the problem for text is how to get the init embedding from deep model. BERT can ...
0 votes
1 answer
193 views

Image classification with CNN Python

I'm working on image classification using CNN, my dataset contains more than 50 classes (50 folders) which represent the types of car parts, and in each folder we have vehicle brands, each vehicle ...
1 vote
1 answer
110 views

Working Behavior of BERT vs Transformers vs Self-Attention+LSTM vs Attention+LSTM on the scientific STEM data classification task?

So I just used BERT pre-trained with Focal Loss to classify Physics, Chemistry, Biology and Mathematics and got a good f-1 macro of 0.91. It is good given it only had to look for the tokens like ...
1 vote
1 answer
120 views

Train on multi-domains, then fine-tune on specific domain

Would it make sense to first train a model on images from multiple domains, and then do "fine-tuning" on one specific domain to improve its performance on it? For instance, one could train an object ...
0 votes
0 answers
25 views

Why Latent Space in Stable Diffusion has shape 64x64x3?

I am wondering why the dimensionality of Latent Space in Stable Diffusion is 64x64x3. Since ...
0 votes
1 answer
52 views

Questions about hidden states of bidirectional LSTMs

I read this in an article about bidirectional LSTM: In bidirectional LSTM, each word corresponds to two hidden states, one for each direction. Thus, we concatenate these two hidden states to ...
1 vote
0 answers
37 views

How can I use Time-GPT for pretraining my model

I am mentioning Time-GPT here as a placeholder example. It can be any pretrained model. Suppose I have a dataset that requires some time series prediction. How can I leverage a well-trained model and ...

1
2 3 4 5
98