Questions tagged [memory]
The memory tag has no usage guidance.
12
questions
1
vote
1answer
34 views
TF-IDF for 400,000+ unique words in corpus?
I have a corpus with over 400,000 unique words. I would like to build a TF-IDF matrix for this corpus.
I have tried doing this on my laptop (16GB RAM) and Google Colab, but am unable to do so due to ...
0
votes
1answer
20 views
determining size of batch, time of sending and memory in to send from scala to ML section
I have a time series (sampling time: 66.66 micro second, number of samples/sampling time=151), I would like to determine some anomalies in them, the inputs are made by scala customer message bus.
...
1
vote
1answer
17 views
Tips for clustering rows of a gigantic "distance" matrix
I've been assigned to the following task:
I was given 1,000,000 data points and was asked to create a sort of distance matrix and to cluster the rows.
So this matrix is 1,000,000 x 1,000,000 which is ...
1
vote
1answer
33 views
PySpark for Big Data and RAM usage
I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this.
Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
1
vote
1answer
43 views
Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?
I trained multiple models for my problem and most ensemble algorithms resulted in lengthy fit and train time and huge model size on disk (approx 10GB for RandomForest) but when I tried ...
0
votes
0answers
11 views
Reducing dataset before computing similarity matrix
I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
0
votes
0answers
37 views
2
votes
0answers
45 views
Running out of memory when calculating loss using SigmoidFocalCrossEntropy
Code Versions:
python == 3.8
tensorflow == 2.2.0
tensorflow-addons == 0.11.2
Recently I've been using tensorflow addon's focal loss function for one of my models. ...
0
votes
0answers
28 views
Best practices for memory management in image processing
I'm relatively new to data science, and I'm trying to create a CNN model for the Kaggle melanoma competition. I've created the following two functions to transform a folder of JPEG images to a Numpy ...
0
votes
1answer
120 views
Memory issue when trying to initiate zero tensor with pytorch
I am facing a memory issue while trying to initialize a torch.zeros -
torch.zeros((2000,2000,3200), device=device)
Getting the following error:
...
0
votes
1answer
117 views
Loading large numpy array (DAIC-WOZ) for LSTM model causes Out of memory errors
I am working on using the DAIC-WOZ Dataset of audio and video extracts for depression detection. While training the audio files, which has three input of eGeMAPS, MFCC, and DENSENET arrays. They are ...
0
votes
0answers
16 views
Can the Apple M1's iGPU access the entire RAM as "video memory" when training with typical deep learning frameworks?
Can the Apple M1's iGPU access the entire RAM as "video memory" when training with typical deep learning frameworks (e.g., tensorflow_macos)? If not, what memory do they use as video memory?
