Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [parallel]

The tag has no usage guidance.

0
votes
1answer
24 views

Can parallel computing be utilized for boosting?

Since boosting is sequential, does that mean we cannot use multi-processing or multi-threading to speed it up? If my computer has multiple CPU cores, is there anyway to utilized these extra resources ...
1
vote
1answer
111 views

How can I parallelize GloVe reverse lookups in PyTorch?

I feel like I'm missing something obvious here because I can't find any discussion of this. I want to do a lot of reverse lookups (nearest neighbor distance searches) on the GloVe embeddings for a ...
0
votes
1answer
1k views

Multiple keras models parallel - time efficient

I am trying to load two different keras models in parallel. I tried to use the functional API model: ...
1
vote
1answer
275 views

Gunicorn workers timeout

I'm using Flask where i load some pre-trained machine learning models once. I'm also using Gunicorn usually with 2 or 4 workers to handle parallel requests. Every request contains some texts that i ...
1
vote
0answers
487 views

Model Parallelism not working? Inception v3 with keras and tensorflow

I have been stuck with a problem like this for a while now. I have an AWS setup with 500 GB of ram and about 7 GPUs. Now the issue is that each time I try to run my keras with tensorflow as back-end ...
1
vote
0answers
56 views

Multiple n_jobs in scikit-learn

How does scikit-learn handle it when there are multiple objects that can have the n_jobs argument? For example: ...
0
votes
0answers
186 views

What should be the value of parallel iterations in tensorflow RNN implementations?

tf.nn.dynamic_rnn() and tf.nn.raw_rnn() take in an argument called parallel_iterations. The documentation says: ...
1
vote
0answers
329 views

R studio, one multi core CPU vs dual single cores

So far I was using R on my home pc: i3 CPU, two cores, 4 threads. In order to run the code faster I was using the package "DoSnow", utilizing 3 out of the 4 cores in order not to choke my system ...
11
votes
1answer
10k views

Make Keras run on multi-machine multi-core cpu system

I'm working on Seq2Seq model using LSTM from Keras (using Theano background) and I would like to parallelize the processes, because even few MBs of data need several hours for training. It is clear ...
11
votes
1answer
2k views

GPU Accelerated Data Processing for R in Windows

I'm currently taking a paper on Big Data which has us utilising R heavily for data analysis. I happen to have a GTX1070 in my pc for gaming reasons. Thus, I thought it would be really cool if I could ...
0
votes
1answer
237 views

Do records with the same key in two RDDs repartitioned by key reside in the same node in spark?

I have two RDDs named "data" and "model", they are repartitioned by key described as below : Does the tuple records with the same key reside in the same node in my cluster ? Should it save IO cost ...
0
votes
1answer
131 views

Efficiently Sending Two Series to a Function For Strings with an application to String Matching (Dice Coefficient)

I am using a Dice Coefficient based function to calculate the similarity of two strings: ...
1
vote
0answers
62 views

Parallel processing for feature selection in microarray dataset

I want to apply feature selection on a dataset with some 30-40K columns and 100 rows ( total size: 400MB-800MB ). To decrease the time consumed for calculations involved (feature-feature), I want to ...
1
vote
0answers
214 views

Scalable training/updating of many small LSTM models

My situation is that I have many thousands of devices which each have their own specific LSTM model for anomaly prediction. These devices behave wildly differently so I don't think there is any way to ...
0
votes
1answer
25 views

How to optimize cohort sizes to reduce pair-wise comparisons?

I am making all pairwise comparisons in a dataset. The use-case is collapsing records into a unique ID based on fuzzy names and dates of birth. The size of the database is around 57,000 individuals. ...
0
votes
1answer
102 views

How to reduce time R takes for model building

I am building machine learning algorithms in my laptop. It has i3 procesor and 16 GB RAM. Despite using multiple cores(3 out of 4), it takes 2 days to run all the techniques that i am trying to run an ...
0
votes
1answer
468 views

GPU computing: how much VRAM do I need for mini batch gradient descent?

I want to do some GPU computing with an NVIDIA card, and am deciding between having a GTX 960 with a 2GB or 4GB ram. Which one should I take? How much difference would these make in terms of the batch ...
3
votes
2answers
146 views

Parallel active optimization

I'm trying to optimize an expensive function for which I can choose sample points. The difficulty is that many function evaluations may be computed in parallel, taking varying amounts of time. I don't ...
4
votes
1answer
403 views

Parallel Q-learning

I'm looking for academic papers or other credible sources focusing on the topic of parralelized reinforcement learning, specifically Q-learning. I'm mostly interested in methods of sharing Q-table ...
2
votes
1answer
56 views

What makes a graph algorithm a good candidate for concurrency?

GraphX is the Apache Spark library for handling graph data. I was able to find a list of 'graph-parallel' algorithms on these slides (see slide 23). However, I am curious what characteristics of these ...
0
votes
1answer
187 views

MPI, MapReduce, or Spark for complex datasets and processing

I have 2 data files: the first one is a database, potentially very large; the second one contains queries I want to answer. My program pipeline is processing the database to get some information first,...
11
votes
3answers
1k views

Instances vs. cores when using EC2

Working on what could often be called "medium data" projects, I've been able to parallelize my code (mostly for modeling and prediction in Python) on a single system across anywhere from 4 to 32 cores....
4
votes
1answer
910 views

Open source solver for large mixed integer programming task?

I'm currently using General Algebraic Modeling System (GAMS), and more specifically CPLEX within GAMS, to solve a very large mixed integer programming problem. This allows me to parallelize the ...
25
votes
4answers
11k views

Is there a straightforward way to run pandas.DataFrame.isin in parallel?

I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...
15
votes
3answers
785 views

Parallel and distributed computing

What is(are) the difference(s) between parallel and distributed computing? When it comes to scalability and efficiency, it is very common to see solutions dealing with computations in clusters of ...