Questions tagged [memory]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
18 views

determining size of batch, time of sending and memory in to send from scala to ML section

I have a time series (sampling time: 66.66 micro second, number of samples/sampling time=151), I would like to determine some anomalies in them, the inputs are made by scala customer message bus. ...
1
vote
1answer
16 views

Tips for clustering rows of a gigantic “distance” matrix

I've been assigned to the following task: I was given 1,000,000 data points and was asked to create a sort of distance matrix and to cluster the rows. So this matrix is 1,000,000 x 1,000,000 which is ...
1
vote
1answer
14 views

PySpark for Big Data and RAM usage

I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this. Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
1
vote
1answer
27 views

Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?

I trained multiple models for my problem and most ensemble algorithms resulted in lengthy fit and train time and huge model size on disk (approx 10GB for RandomForest) but when I tried ...
0
votes
0answers
11 views

Reducing dataset before computing similarity matrix

I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
0
votes
0answers
16 views

Which memory usage is more accurate in nvidia-smi?

My nvidia-smi shows the following: ...
2
votes
0answers
31 views

Running out of memory when calculating loss using SigmoidFocalCrossEntropy

Code Versions: python == 3.8 tensorflow == 2.2.0 tensorflow-addons == 0.11.2 Recently I've been using tensorflow addon's focal loss function for one of my models. ...
0
votes
0answers
21 views

Best practices for memory management in image processing

I'm relatively new to data science, and I'm trying to create a CNN model for the Kaggle melanoma competition. I've created the following two functions to transform a folder of JPEG images to a Numpy ...
0
votes
1answer
55 views

Memory issue when trying to initiate zero tensor with pytorch

I am facing a memory issue while trying to initialize a torch.zeros - torch.zeros((2000,2000,3200), device=device) Getting the following error: ...
0
votes
1answer
74 views

Loading large numpy array (DAIC-WOZ) for LSTM model causes Out of memory errors

I am working on using the DAIC-WOZ Dataset of audio and video extracts for depression detection. While training the audio files, which has three input of eGeMAPS, MFCC, and DENSENET arrays. They are ...
0
votes
0answers
16 views

Can the Apple M1's iGPU access the entire RAM as “video memory” when training with typical deep learning frameworks?

Can the Apple M1's iGPU access the entire RAM as "video memory" when training with typical deep learning frameworks (e.g., tensorflow_macos)? If not, what memory do they use as video memory?