Questions tagged [memory]
The memory tag has no usage guidance.
16
questions
2
votes
0
answers
15
views
Is batch size of 1 a valid choice for a very deep neural network with high memory requirement?
I am training a very deep neural network (Panoptic-DeepLab) with a ResNet34 backbone on Google Colab on CityScapes dataset for Panoptic Segmentation, and noticed that, with a big crop size, the batch ...
- 121
0
votes
0
answers
17
views
How to load several numpys in batch generator from directory
I have a deep learning model which has to be feed with a huge amount of data (200k of 256x256 images), so it runs out of memory. I have divided my data in several numpy arrays that are in an specified ...
1
vote
1
answer
49
views
What are different ways to reduce size of word2vec vectors file?
I am working on an application with memory constraints. We are getting vectors from python Gensim models but need to transmit copies of them to react native mobile app and potentially in-browser JS. ...
- 11
1
vote
0
answers
35
views
How to calculate the fully connected neural network footprint for each layer?
I'm using a MADDPG (RL algorithm) and trying to find out the memory footprint for each layer in it. Maddpg: https://github.com/openai/maddpg
The neural network is described here.
...
- 11
0
votes
0
answers
13
views
Can the Apple M1's iGPU access the entire RAM as "video memory" when training with typical deep learning frameworks?
Can the Apple M1's iGPU access the entire RAM as "video memory" when training with typical deep learning frameworks (e.g., tensorflow_macos)? If not, what memory do they use as video memory?
- 5,275
2
votes
1
answer
173
views
How can I measure time and memory complexity for a deep learning model?
How can I measure or find the time complexity and memory complexity for a model like VGG16 or Resnet50? Also, will it be different from one machine to another like using GTX GPU or RTX GPU?
- 45
0
votes
0
answers
52
views
Does using torch.cuda.empty_cache can decrease performance?
We're struggling with memory usage in a project that deploys multiple models to the same GPU (models are usually built with PyTorch and TensorFlow).
It was suggested that we could use ...
- 101
1
vote
1
answer
175
views
TF-IDF for 400,000+ unique words in corpus?
I have a corpus with over 400,000 unique words. I would like to build a TF-IDF matrix for this corpus.
I have tried doing this on my laptop (16GB RAM) and Google Colab, but am unable to do so due to ...
- 109
0
votes
1
answer
22
views
determining size of batch, time of sending and memory in to send from scala to ML section
I have a time series (sampling time: 66.66 micro second, number of samples/sampling time=151), I would like to determine some anomalies in them, the inputs are made by scala customer message bus.
...
- 1,526
1
vote
1
answer
47
views
Tips for clustering rows of a gigantic "distance" matrix
I've been assigned to the following task:
I was given 1,000,000 data points and was asked to create a sort of distance matrix and to cluster the rows.
So this matrix is 1,000,000 x 1,000,000 which is ...
- 197
1
vote
1
answer
124
views
PySpark for Big Data and RAM usage
I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this.
Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
- 11
1
vote
1
answer
321
views
Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?
I trained multiple models for my problem and most ensemble algorithms resulted in lengthy fit and train time and huge model size on disk (approx 10GB for RandomForest) but when I tried ...
- 35
0
votes
0
answers
13
views
Reducing dataset before computing similarity matrix
I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
2
votes
1
answer
106
views
Running out of memory when calculating loss using SigmoidFocalCrossEntropy
Code Versions:
python == 3.8
tensorflow == 2.2.0
tensorflow-addons == 0.11.2
Recently I've been using tensorflow addon's focal loss function for one of my models. ...
- 21
0
votes
1
answer
299
views
Memory issue when trying to initiate zero tensor with pytorch
I am facing a memory issue while trying to initialize a torch.zeros -
torch.zeros((2000,2000,3200), device=device)
Getting the following error:
...
0
votes
1
answer
340
views
Loading large numpy array (DAIC-WOZ) for LSTM model causes Out of memory errors
I am working on using the DAIC-WOZ Dataset of audio and video extracts for depression detection. While training the audio files, which has three input of eGeMAPS, MFCC, and DENSENET arrays. They are ...
- 133