Questions tagged [memory]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
0 answers
15 views

Is batch size of 1 a valid choice for a very deep neural network with high memory requirement?

I am training a very deep neural network (Panoptic-DeepLab) with a ResNet34 backbone on Google Colab on CityScapes dataset for Panoptic Segmentation, and noticed that, with a big crop size, the batch ...
user avatar
0 votes
0 answers
17 views

How to load several numpys in batch generator from directory

I have a deep learning model which has to be feed with a huge amount of data (200k of 256x256 images), so it runs out of memory. I have divided my data in several numpy arrays that are in an specified ...
user avatar
1 vote
1 answer
49 views

What are different ways to reduce size of word2vec vectors file?

I am working on an application with memory constraints. We are getting vectors from python Gensim models but need to transmit copies of them to react native mobile app and potentially in-browser JS. ...
user avatar
1 vote
0 answers
35 views

How to calculate the fully connected neural network footprint for each layer?

I'm using a MADDPG (RL algorithm) and trying to find out the memory footprint for each layer in it. Maddpg: https://github.com/openai/maddpg The neural network is described here. ...
user avatar
  • 11
0 votes
0 answers
13 views

Can the Apple M1's iGPU access the entire RAM as "video memory" when training with typical deep learning frameworks?

Can the Apple M1's iGPU access the entire RAM as "video memory" when training with typical deep learning frameworks (e.g., tensorflow_macos)? If not, what memory do they use as video memory?
user avatar
2 votes
1 answer
173 views

How can I measure time and memory complexity for a deep learning model?

How can I measure or find the time complexity and memory complexity for a model like VGG16 or Resnet50? Also, will it be different from one machine to another like using GTX GPU or RTX GPU?
user avatar
  • 45
0 votes
0 answers
52 views

Does using torch.cuda.empty_cache can decrease performance?

We're struggling with memory usage in a project that deploys multiple models to the same GPU (models are usually built with PyTorch and TensorFlow). It was suggested that we could use ...
user avatar
1 vote
1 answer
175 views

TF-IDF for 400,000+ unique words in corpus?

I have a corpus with over 400,000 unique words. I would like to build a TF-IDF matrix for this corpus. I have tried doing this on my laptop (16GB RAM) and Google Colab, but am unable to do so due to ...
user avatar
0 votes
1 answer
22 views

determining size of batch, time of sending and memory in to send from scala to ML section

I have a time series (sampling time: 66.66 micro second, number of samples/sampling time=151), I would like to determine some anomalies in them, the inputs are made by scala customer message bus. ...
user avatar
  • 1,526
1 vote
1 answer
47 views

Tips for clustering rows of a gigantic "distance" matrix

I've been assigned to the following task: I was given 1,000,000 data points and was asked to create a sort of distance matrix and to cluster the rows. So this matrix is 1,000,000 x 1,000,000 which is ...
user avatar
  • 197
1 vote
1 answer
124 views

PySpark for Big Data and RAM usage

I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this. Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
user avatar
  • 11
1 vote
1 answer
321 views

Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?

I trained multiple models for my problem and most ensemble algorithms resulted in lengthy fit and train time and huge model size on disk (approx 10GB for RandomForest) but when I tried ...
user avatar
  • 35
0 votes
0 answers
13 views

Reducing dataset before computing similarity matrix

I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
user avatar
2 votes
1 answer
106 views

Running out of memory when calculating loss using SigmoidFocalCrossEntropy

Code Versions: python == 3.8 tensorflow == 2.2.0 tensorflow-addons == 0.11.2 Recently I've been using tensorflow addon's focal loss function for one of my models. ...
user avatar
  • 21
0 votes
1 answer
299 views

Memory issue when trying to initiate zero tensor with pytorch

I am facing a memory issue while trying to initialize a torch.zeros - torch.zeros((2000,2000,3200), device=device) Getting the following error: ...
user avatar
0 votes
1 answer
340 views

Loading large numpy array (DAIC-WOZ) for LSTM model causes Out of memory errors

I am working on using the DAIC-WOZ Dataset of audio and video extracts for depression detection. While training the audio files, which has three input of eGeMAPS, MFCC, and DENSENET arrays. They are ...
user avatar