Questions tagged [parallel]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
70 views

Would writing a decision tree algorithm in Pytorch or Tensorflow be faster than with Numpy?

Since these libraries can turn CPU arrays into GPU tensors, could you parallelize (and therefore accelerate) the calculations for a decision tree? I am considering making a decision tree class written ...
0
votes
1answer
64 views

Specifying number of threads using XGBoost.train

When using the xgboost.train() function, all the threads are used. I would like to use a specific amount. Unfortunately, this function does not accept the ...
1
vote
1answer
20 views

Methodology for parallelising linked data?

If I have some form of data that can have inherent links to all other data in the set but I wish to parallelise out this data in order to increase computation time or to reduce the size of any ...
0
votes
1answer
77 views

CUDA 8.0 is compatible with my GeForce GTX 670M Wikipedia says, but TensorFlow rises an error: GTX 670M's Compute Capability is < 3.0

According to Wikipedia, the GeForce GTX 670M has a Compute Capability of 2.1 (and a Fermi micro-architecture), which is confirmed by TensorFlow (I can read "2.1" in the error it rises). Wikipedia ...
0
votes
0answers
10 views

Not able to sentence encode a list of sentences using multiprocessing technique - pool.map() function in python

I am trying to embed a text data which is in the form of list, since its a huge data I wanted to embed it using the multiprocessing Pool map() function. The embedding technique I'm using is google's ...
1
vote
2answers
32 views

Parallel hyperparameter optimization techniques?

Most hyperparameter optimization technique want to evaluate points one by one. I have an expensive optimization problem, but i can run hundreds of evaluations in parallel. The dimension of the problem ...
1
vote
0answers
99 views

Updating Weight Using Updates on Related Data

Suppose $$ x=Ay $$ The $x$ is $M\times 1$, $y$ is $N \times 1$ and $A$ is $M\times N$ We have the data $x$ and would like to know what $y$ is. However, the matrix $A$ is too large for pseudo-...
0
votes
1answer
34 views

Can parallel computing be utilized for boosting?

Since boosting is sequential, does that mean we cannot use multi-processing or multi-threading to speed it up? If my computer has multiple CPU cores, is there anyway to utilized these extra resources ...
1
vote
1answer
264 views

How can I parallelize GloVe reverse lookups in PyTorch?

I feel like I'm missing something obvious here because I can't find any discussion of this. I want to do a lot of reverse lookups (nearest neighbor distance searches) on the GloVe embeddings for a ...
0
votes
1answer
2k views

Multiple keras models parallel - time efficient

I am trying to load two different keras models in parallel. I tried to use the functional API model: ...
1
vote
1answer
485 views

Gunicorn workers timeout

I'm using Flask where i load some pre-trained machine learning models once. I'm also using Gunicorn usually with 2 or 4 workers to handle parallel requests. Every request contains some texts that i ...
1
vote
0answers
719 views

Model Parallelism not working? Inception v3 with keras and tensorflow

I have been stuck with a problem like this for a while now. I have an AWS setup with 500 GB of ram and about 7 GPUs. Now the issue is that each time I try to run my keras with tensorflow as back-end ...
1
vote
0answers
114 views

Multiple n_jobs in scikit-learn

How does scikit-learn handle it when there are multiple objects that can have the n_jobs argument? For example: ...
0
votes
0answers
244 views

What should be the value of parallel iterations in tensorflow RNN implementations?

tf.nn.dynamic_rnn() and tf.nn.raw_rnn() take in an argument called parallel_iterations. The documentation says: ...
1
vote
0answers
350 views

R studio, one multi core CPU vs dual single cores

So far I was using R on my home pc: i3 CPU, two cores, 4 threads. In order to run the code faster I was using the package "DoSnow", utilizing 3 out of the 4 cores in order not to choke my system ...
10
votes
1answer
14k views

Make Keras run on multi-machine multi-core cpu system

I'm working on Seq2Seq model using LSTM from Keras (using Theano background) and I would like to parallelize the processes, because even few MBs of data need several hours for training. It is clear ...
11
votes
1answer
2k views

GPU Accelerated Data Processing for R in Windows

I'm currently taking a paper on Big Data which has us utilising R heavily for data analysis. I happen to have a GTX1070 in my pc for gaming reasons. Thus, I thought it would be really cool if I could ...
0
votes
1answer
257 views

Do records with the same key in two RDDs repartitioned by key reside in the same node in spark?

I have two RDDs named "data" and "model", they are repartitioned by key described as below : Does the tuple records with the same key reside in the same node in my cluster ? Should it save IO cost ...
0
votes
1answer
143 views

Efficiently Sending Two Series to a Function For Strings with an application to String Matching (Dice Coefficient)

I am using a Dice Coefficient based function to calculate the similarity of two strings: ...
1
vote
0answers
67 views

Parallel processing for feature selection in microarray dataset

I want to apply feature selection on a dataset with some 30-40K columns and 100 rows ( total size: 400MB-800MB ). To decrease the time consumed for calculations involved (feature-feature), I want to ...
1
vote
0answers
221 views

Scalable training/updating of many small LSTM models

My situation is that I have many thousands of devices which each have their own specific LSTM model for anomaly prediction. These devices behave wildly differently so I don't think there is any way to ...
0
votes
1answer
25 views

How to optimize cohort sizes to reduce pair-wise comparisons?

I am making all pairwise comparisons in a dataset. The use-case is collapsing records into a unique ID based on fuzzy names and dates of birth. The size of the database is around 57,000 individuals. ...
0
votes
1answer
116 views

How to reduce time R takes for model building

I am building machine learning algorithms in my laptop. It has i3 procesor and 16 GB RAM. Despite using multiple cores(3 out of 4), it takes 2 days to run all the techniques that i am trying to run an ...
0
votes
1answer
532 views

GPU computing: how much VRAM do I need for mini batch gradient descent?

I want to do some GPU computing with an NVIDIA card, and am deciding between having a GTX 960 with a 2GB or 4GB ram. Which one should I take? How much difference would these make in terms of the batch ...
4
votes
2answers
153 views

Parallel active optimization

I'm trying to optimize an expensive function for which I can choose sample points. The difficulty is that many function evaluations may be computed in parallel, taking varying amounts of time. I don't ...
4
votes
1answer
496 views

Parallel Q-learning

I'm looking for academic papers or other credible sources focusing on the topic of parralelized reinforcement learning, specifically Q-learning. I'm mostly interested in methods of sharing Q-table ...
2
votes
1answer
57 views

What makes a graph algorithm a good candidate for concurrency?

GraphX is the Apache Spark library for handling graph data. I was able to find a list of 'graph-parallel' algorithms on these slides (see slide 23). However, I am curious what characteristics of these ...
0
votes
1answer
188 views

MPI, MapReduce, or Spark for complex datasets and processing

I have 2 data files: the first one is a database, potentially very large; the second one contains queries I want to answer. My program pipeline is processing the database to get some information first,...
12
votes
3answers
1k views

Instances vs. cores when using EC2

Working on what could often be called "medium data" projects, I've been able to parallelize my code (mostly for modeling and prediction in Python) on a single system across anywhere from 4 to 32 cores....
4
votes
1answer
945 views

Open source solver for large mixed integer programming task?

I'm currently using General Algebraic Modeling System (GAMS), and more specifically CPLEX within GAMS, to solve a very large mixed integer programming problem. This allows me to parallelize the ...
25
votes
4answers
13k views

Is there a straightforward way to run pandas.DataFrame.isin in parallel?

I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...
13
votes
3answers
863 views

Parallel and distributed computing

What is(are) the difference(s) between parallel and distributed computing? When it comes to scalability and efficiency, it is very common to see solutions dealing with computations in clusters of ...