Questions tagged [memory]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
49 views

Holding batch size constant, will a bigger dataset consume more GPU memory?

If you hold (mini) batch size constant (as well as everything else) but increase the number of examples (and therefore the number of training iterations), should you expect a (significant) increase in ...
ubadub's user avatar
  • 101
0 votes
0 answers
115 views

Batch processing for a dictionary of dictionaries Python

I have a very large dictionary of the form vf = {key1:{...}, key2:{...}, key3:{...}, ...}, which is stored as an npz file. Trying to load it at once causes a memory error, so I tried to load it in ...
Andrew's user avatar
  • 103
0 votes
1 answer
46 views

Python version and performance in ML

I couldn't find any answer on that question, but does Python version really matter when you implement different ML models (like catboost or xgboost) or NN (like pytorch). I do know that these models ...
Master_Sniffer's user avatar
-1 votes
2 answers
216 views

Laptop for machine learning jobs

I am buying a new laptop for data science and web development jobs. Which combination is better: i9 (12900H) & NVIDIA T600 4GB or i7 (12800H) & NVIDIA RTX1000A 4GB? Both run on a DELL ...
Shad81's user avatar
  • 1
2 votes
2 answers
514 views

Which desktop hardware is best for DL?

I will be building my home Deep Learning workstation. Right now, I'm digging for some time about the best HW to use for home conditions. The workstation will be used for my work as a developer, but I ...
Jaroslav Štreit's user avatar
2 votes
0 answers
16 views

Can the Apple M1's iGPU access the entire RAM as "video memory" when training with typical deep learning frameworks?

Can the Apple M1's iGPU access the entire RAM as "video memory" when training with typical deep learning frameworks (e.g., tensorflow_macos)? If not, what memory do they use as video memory?
Franck Dernoncourt's user avatar
0 votes
0 answers
42 views

Memory Error when loading a txt file for an ML model

I am trying to run the Python code below: ...
lynx's user avatar
  • 113
6 votes
1 answer
6k views

What size language model can you train on a GPU with x GB of memory?

I'm trying to figure out what size language model I will be able to train on a GPU with a certain amount of memory. Let's for simplicity say that 1 GB = 109 bytes; that means that, for example, on a ...
HelloGoodbye's user avatar
2 votes
1 answer
116 views

Training Neural Network using TensorFlow on Large Video Dataset for Human Activity Recognition

I am working on a Human Activity Recognition on videos using deep learning and want to train my Neural Network (ConvLSTM & LRCN) on my custom dataset. The issue that I am facing is that my dataset ...
Sharjeel M. Rajput's user avatar
0 votes
1 answer
184 views

How to calculate the distance matrix when there is a lot of data And the memory does not support it

I am dealing with a data matrix in which most of the variables are binary or multilevel response. I would like to perform the MDS algorithm and for that, I need to calculate the distance matrix first. ...
PicaR's user avatar
  • 314
2 votes
1 answer
53 views

Is batch size of 1 a valid choice for a very deep neural network with high memory requirement?

I am training a very deep neural network (Panoptic-DeepLab) with a ResNet34 backbone on Google Colab on CityScapes dataset for Panoptic Segmentation, and noticed that, with a big crop size, the batch ...
A_C's user avatar
  • 121
1 vote
1 answer
462 views

What are different ways to reduce size of word2vec vectors file?

I am working on an application with memory constraints. We are getting vectors from python Gensim models but need to transmit copies of them to react native mobile app and potentially in-browser JS. ...
Aditya Jain's user avatar
1 vote
0 answers
91 views

How to calculate the fully connected neural network footprint for each layer?

I'm using a MADDPG (RL algorithm) and trying to find out the memory footprint for each layer in it. Maddpg: https://github.com/openai/maddpg The neural network is described here. ...
Student's user avatar
  • 11
3 votes
1 answer
2k views

How can I measure time and memory complexity for a deep learning model?

How can I measure or find the time complexity and memory complexity for a model like VGG16 or Resnet50? Also, will it be different from one machine to another like using GTX GPU or RTX GPU?
Lei's user avatar
  • 57
1 vote
1 answer
888 views

TF-IDF for 400,000+ unique words in corpus?

I have a corpus with over 400,000 unique words. I would like to build a TF-IDF matrix for this corpus. I have tried doing this on my laptop (16GB RAM) and Google Colab, but am unable to do so due to ...
user16584277's user avatar
0 votes
1 answer
28 views

determining size of batch, time of sending and memory in to send from scala to ML section

I have a time series (sampling time: 66.66 micro second, number of samples/sampling time=151), I would like to determine some anomalies in them, the inputs are made by scala customer message bus. ...
user10296606's user avatar
  • 1,834
1 vote
1 answer
536 views

Tips for clustering rows of a gigantic "distance" matrix

I've been assigned to the following task: I was given 1,000,000 data points and was asked to create a sort of distance matrix and to cluster the rows. So this matrix is 1,000,000 x 1,000,000 which is ...
crommy's user avatar
  • 197
1 vote
1 answer
345 views

PySpark for Big Data and RAM usage

I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this. Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
Shaun's user avatar
  • 11
1 vote
1 answer
2k views

Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?

I trained multiple models for my problem and most ensemble algorithms resulted in lengthy fit and train time and huge model size on disk (approx 10GB for RandomForest) but when I tried ...
ro23's user avatar
  • 35
0 votes
0 answers
19 views

Reducing dataset before computing similarity matrix

I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
Max van der Werf's user avatar
2 votes
1 answer
224 views

Running out of memory when calculating loss using SigmoidFocalCrossEntropy

Code Versions: python == 3.8 tensorflow == 2.2.0 tensorflow-addons == 0.11.2 Recently I've been using tensorflow addon's focal loss function for one of my models. ...
RLB's user avatar
  • 23
0 votes
1 answer
822 views

Memory issue when trying to initiate zero tensor with pytorch

I am facing a memory issue while trying to initialize a torch.zeros - torch.zeros((2000,2000,3200), device=device) Getting the following error: ...
Saikat Bhattacharya's user avatar
0 votes
1 answer
631 views

Loading large numpy array (DAIC-WOZ) for LSTM model causes Out of memory errors

I am working on using the DAIC-WOZ Dataset of audio and video extracts for depression detection. While training the audio files, which has three input of eGeMAPS, MFCC, and DENSENET arrays. They are ...
Saranya Prakash's user avatar