Questions tagged [parallel]
The parallel tag has no usage guidance.
39
questions
0
votes
0
answers
19
views
Interaction plot
I have some questions about the interaction plot. I tried to make it on my own but I am wrong in my approach and I would like to know how to construct this. I have made a log linear regression with ...
- 101
0
votes
0
answers
3
views
H20 AutoML Parallelism
Does H2O AutoML parallelize different models when launched via train ? (or you should specify it somehow?)
If so, can you show an example ?
2
votes
1
answer
108
views
Parallel Data preprocessing
I am looking for a suggestion. Is it possible to implement the data preprocessing steps like missing value imputation, outlier detection, normalization, label encoding in parallel? Can I implement ...
- 359
0
votes
1
answer
621
views
How to loop through multiple lists/dict?
I have the following code which finds the best value of k parameter in the KNNImputer. Basically it is looping through the list ...
- 1,576
1
vote
2
answers
216
views
How to load and run feature selection on a dataset with 5,000 samples and 500,000 features?
I have a dataset with 5000 samples and 500,000 features (all categorical with a cardinality of 3).
Two problems I'm trying to solve:
Loading the dataset - I can't load it in memory despite using a ...
0
votes
1
answer
63
views
How to run two different models in single frame?
I have mask_detector.model and yolov3 social distancing weights.
I want to run them simultaneously with a single webcam stream.
how can I run them both i.e. detecting mask and social distancing model ...
10
votes
3
answers
13k
views
What needs to be done to make n_jobs work properly on sklearn? in particular on ElasticNetCV?
The constructor of sklearn.linear_model.ElasticNetCV takesn_jobs as an argument.
Quoting the documentation here
n_jobs: int, ...
- 241
1
vote
0
answers
18
views
Pytorch Distributed Computing - Recomendations/Resources/Courses?
I would like to get into some distributed computing for processing Pytorch CNN models. I am completely fresh in this field and want to get some recommendations as to where I should start researching ...
- 11
1
vote
0
answers
56
views
Parallelization of a MIMO linear filter
I would like to implement a Multi Input Multi Output filtering operation, acting as fast as possible on batches of data. Here is my current implementation:
...
- 11
1
vote
1
answer
993
views
Would writing a decision tree algorithm in Pytorch or Tensorflow be faster than with Numpy?
Since these libraries can turn CPU arrays into GPU tensors, could you parallelize (and therefore accelerate) the calculations for a decision tree? I am considering making a decision tree class written ...
- 256
0
votes
3
answers
2k
views
Specifying number of threads using XGBoost.train
When using the xgboost.train() function, all the threads are used. I would like to use a specific amount. Unfortunately, this function does not accept the ...
- 11
1
vote
1
answer
37
views
Methodology for parallelising linked data?
If I have some form of data that can have inherent links to all other data in the set but I wish to parallelise out this data in order to increase computation time or to reduce the size of any ...
0
votes
1
answer
995
views
CUDA 8.0 is compatible with my GeForce GTX 670M Wikipedia says, but TensorFlow rises an error: GTX 670M's Compute Capability is < 3.0
According to Wikipedia, the GeForce GTX 670M has a Compute Capability of 2.1 (and a Fermi micro-architecture), which is confirmed by TensorFlow (I can read "2.1" in the error it rises).
...
1
vote
2
answers
107
views
Parallel hyperparameter optimization techniques?
Most hyperparameter optimization technique want to evaluate points one by one.
I have an expensive optimization problem, but i can run hundreds of evaluations in parallel.
The dimension of the problem ...
- 11
1
vote
0
answers
101
views
Updating Weight Using Updates on Related Data
Suppose
$$
x=Ay
$$
The $x$ is $M\times 1$, $y$ is $N \times 1$ and $A$ is $M\times N$
We have the data $x$ and would like to know what $y$ is.
However, the matrix $A$ is too large for pseudo-...
2
votes
1
answer
264
views
Can parallel computing be utilized for boosting?
Since boosting is sequential, does that mean we cannot use multi-processing or multi-threading to speed it up? If my computer has multiple CPU cores, is there anyway to utilized these extra resources ...
- 155
1
vote
1
answer
661
views
How can I parallelize GloVe reverse lookups in PyTorch?
I feel like I'm missing something obvious here because I can't find any discussion of this. I want to do a lot of reverse lookups (nearest neighbor distance searches) on the GloVe embeddings for a ...
- 133
1
vote
1
answer
4k
views
Multiple keras models parallel - time efficient
I am trying to load two different keras models in parallel. I tried to use the functional API model:
...
- 111
1
vote
1
answer
1k
views
Gunicorn workers timeout
I'm using Flask where i load some pre-trained machine learning models once. I'm also using Gunicorn usually with 2 or 4 workers to handle parallel requests.
Every request contains some texts that i ...
- 173
1
vote
1
answer
933
views
Model Parallelism not working in Inception v3 with Keras and TensorFlow
I have been stuck with a problem like this for a while now. I have an AWS setup with 500 GB of RAM and about 7 GPUs. Now the issue is that each time I try to run my Keras with TensorFlow as back-end ...
- 151
2
votes
0
answers
394
views
What should be the value of parallel iterations in tensorflow RNN implementations?
tf.nn.dynamic_rnn() and tf.nn.raw_rnn() take in an argument called parallel_iterations. The documentation says:
...
- 775
1
vote
0
answers
395
views
R studio, one multi core CPU vs dual single cores
So far I was using R on my home pc: i3 CPU, two cores, 4 threads. In order to run the code faster I was using the package "DoSnow", utilizing 3 out of the 4 cores in order not to choke my system ...
- 161
14
votes
1
answer
20k
views
Make Keras run on multi-machine multi-core cpu system
I'm working on Seq2Seq model using LSTM from Keras (using Theano background) and I would like to parallelize the processes, because even few MBs of data need several hours for training.
It is clear ...
- 1,954
11
votes
1
answer
3k
views
GPU Accelerated Data Processing for R in Windows
I'm currently taking a paper on Big Data which has us utilising R heavily for data analysis. I happen to have a GTX1070 in my pc for gaming reasons. Thus, I thought it would be really cool if I could ...
- 113
0
votes
1
answer
348
views
Do records with the same key in two RDDs repartitioned by key reside in the same node in spark?
I have two RDDs named "data" and "model", they are repartitioned by key described as below :
Does the tuple records with the same key reside in the same node in my cluster ?
Should it save IO cost ...
- 399
0
votes
1
answer
306
views
Efficiently Sending Two Series to a Function For Strings with an application to String Matching (Dice Coefficient)
I am using a Dice Coefficient based function to calculate the similarity of two strings:
...
1
vote
0
answers
75
views
Parallel processing for feature selection in microarray dataset
I want to apply feature selection on a dataset with some 30-40K columns and 100 rows ( total size: 400MB-800MB ). To decrease the time consumed for calculations involved (feature-feature), I want to ...
- 11
1
vote
0
answers
260
views
Scalable training/updating of many small LSTM models
My situation is that I have many thousands of devices which each have their own specific LSTM model for anomaly prediction. These devices behave wildly differently so I don't think there is any way to ...
- 33
0
votes
1
answer
39
views
How to optimize cohort sizes to reduce pair-wise comparisons?
I am making all pairwise comparisons in a dataset. The use-case is collapsing records into a unique ID based on fuzzy names and dates of birth. The size of the database is around 57,000 individuals. ...
- 263
0
votes
1
answer
373
views
How to reduce time R takes for model building
I am building machine learning algorithms in my laptop. It has i3 procesor and 16 GB RAM. Despite using multiple cores(3 out of 4), it takes 2 days to run all the techniques that i am trying to run an ...
- 865
0
votes
1
answer
899
views
GPU computing: how much VRAM do I need for mini batch gradient descent?
I want to do some GPU computing with an NVIDIA card, and am deciding between having a GTX 960 with a 2GB or 4GB ram. Which one should I take? How much difference would these make in terms of the batch ...
- 133
4
votes
2
answers
168
views
Parallel active optimization
I'm trying to optimize an expensive function for which I can choose sample points. The difficulty is that many function evaluations may be computed in parallel, taking varying amounts of time. I don't ...
- 213
4
votes
1
answer
1k
views
Parallel Q-learning
I'm looking for academic papers or other credible sources focusing on the topic of parralelized reinforcement learning, specifically Q-learning.
I'm mostly interested in methods of sharing Q-table ...
- 189
2
votes
1
answer
63
views
What makes a graph algorithm a good candidate for concurrency?
GraphX is the Apache Spark library for handling graph data. I was able to find a list of 'graph-parallel' algorithms on these slides (see slide 23). However, I am curious what characteristics of these ...
- 1,169
0
votes
1
answer
207
views
MPI, MapReduce, or Spark for complex datasets and processing
I have 2 data files: the first one is a database, potentially very large; the second one contains queries I want to answer. My program pipeline is processing the database to get some information first,...
- 101
12
votes
3
answers
2k
views
Instances vs. cores when using EC2
Working on what could often be called "medium data" projects, I've been able to parallelize my code (mostly for modeling and prediction in Python) on a single system across anywhere from 4 to 32 cores....
- 871
4
votes
1
answer
983
views
Open source solver for large mixed integer programming task?
I'm currently using General Algebraic Modeling System (GAMS), and more specifically CPLEX within GAMS, to solve a very large mixed integer programming problem. This allows me to parallelize the ...
- 203
27
votes
4
answers
19k
views
Is there a straightforward way to run pandas.DataFrame.isin in parallel?
I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...
- 871
16
votes
3
answers
1k
views
Parallel and distributed computing
What is(are) the difference(s) between parallel and distributed computing? When it comes to scalability and efficiency, it is very common to see solutions dealing with computations in clusters of ...
- 4,097