16

Simply set, 'parallel' means running concurrently on distinct resources (CPUs), while 'distributed' means running across distinct computers, involving issues related to networks. Parallel computing using for instance OpenMP is not distributed, while parallel computing with Message Passing is often distributed. Being in a 'distributed but not parallel' ...


11

When using IPython, you very nearly don't have to worry about it (at the expense of some loss of efficiency/greater communication overhead). The parallel IPython plugin in StarCluster will by default start one engine per physical core on each node (I believe this is configurable but not sure where). You just run whatever you want across all engines by ...


11

A general rule of thumb is to not distribute until you have to. It's usually more efficient to have N servers of a certain capacity than 2N servers of half that capacity. More of the data access will be local, and therefore fast in memory versus slow across the network. At a certain point, scaling up one machine becomes uneconomical because the cost of ...


8

Unfortunately, parallelization is not yet implemented in pandas. You can join this github issue if you want to participate in the development of this feature. I don't know any "magic unicorn package" for this purposes, so the best thing will be write your own solution. But if you still don't want to spend time on that and want to learn something new – you ...


7

In order to set the number of threads used in Theano (and, therefore, the number of CPU cores), you'll need to set a few parameters in the environment: import os os.environ['MKL_NUM_THREADS'] = '16' os.environ['GOTO_NUM_THREADS'] = '16' os.environ['OMP_NUM_THREADS'] = '16' os.eviron['openmp'] = 'True' This should allow you to use all cores of all CPUs. ...


6

All things considered equal (cost, CPU perf, etc.) you could choose the smallest instance that can hold all of my dataset in memory and scale out. That way you make sure not to induce unnecessary latencies due to network communications, and you tend to maximize the overall available memory bandwidth for your processes. Assuming you are running some sort ...


6

I think your best bet would be rosetta. I'm finding it extremely useful and easy. Check its pandas methods. You can get it by pip.


5

There is the useful dask library for parallel numpy/pandas jobs


4

From my experience setting up GPU processing for R is hard, setting it up on a Windows machine is even harder. Additionally, GPU processing can only be used for very specific types of calculations. If you just want to setup GPU processing for the sake of it, then my answer is quite useless. If you however care about general performance optimization of your ...


4

The terms "parallel computing" and "distributed computing" certainly have a large overlap, but can be differentiated further. Actually, you already did this in your question, by later asking about "parallel processing" and "distributed processing". One could consider "distributed computing" as the more general term that involves "distributed processing" as ...


3

The answers presented so far are very nice, but I was also expecting an emphasis on a particular difference between parallel and distributed processing: the code executed. Considering parallel processes, the code executed is the same, regardless of the level of parallelism (instruction, data, task). You write a single code, and it will be executed by ...


2

Never done stuff on that scale, but as no-one else has jumped in yet have you seen these two papers that discuss non-commercial solutions? Symphony and COIN-OR seem to be the dominant suggestions. Linderoth, Jeffrey T., and Andrea Lodi. "MILP software." Wiley encyclopedia of operations research and management science (2010). PDF version Linderoth, Jeffrey ...


2

There is a more common version of this question regarding parallelization on pandas apply function - so this is a refreshing question :) First, I want to mention swifter since you asked for a "packaged" solution, and it appears on most SO question regarding pandas parallelization. But.. I'd still like to share my personal gist code for it, since after ...


2

Would try to answer based on experience and understandings of parallel computing in production for DS/ML models: Answer to your questions as high level: Does the simple program above give you better performance with increasing n_jobs when you run it? answer: Yes and can be seen bellow in results. On what OS / setup? answer: OS:ubuntu, 2xCPUsx16Cores+...


2

When I ran your script, I got the same impression, that n_jobs was hurting you performance. However, you have to consider that parallelizing the cross-validation would only benefit if you have more data samples. With few data, the communication overhead indeed is more expensive than the processing cost involved on the task. I tried your script with more ...


2

I think you will like the following two papers: Available from: http://arxiv.org/abs/1507.04296 Nair A, Srinivasan P, Blackwell S, Alcicek C, Fearon R, De Maria A, et al. Massively Parallel Methods for Deep Reinforcement Learning. arXiv preprint arXiv:150704296 Available from: http://arxiv.org/abs/1602.01783 Mnih V, Badia AP, Mirza M, Graves A, Lillicrap ...


2

It's usually not very efficient to approach these types of problems in pythonic ways, with list comprehensions and such. This whole process can be done with some matrix math, which will be substantially faster (and able to be computed on the GPU using PyTorch). Using torch.dist with default p=2 will compute the Euclidean distance between two tensors, which ...


2

Having profiled and stepped through sklearn´s code, I´ve got some answers. The summary: Contrary to what has been suggested, sklearn's ElasticNetCV()'s poor scalability to n_jobs is not due to: the overhead of launching threads or processes. SequentialBackend always being used irrespective of n_jobs. (I cannot reproduce this problem as stated in n1tk's ...


2

5000 samples and 500,000 is not that big - it all depends how much memory you have. Also remember you can always and always optimize your data format. e.g. if they are float64 - do they need to be ? if they are categorical, how they are encoded ? (one character or a 20 character word?) and such. so Yes, if you can load the data into memory good for you if ...


1

As was suggested by Erik van de Ven, it sounds like running each model on a different process should provide the requested parallelism. I guess you could either run the fit function for each model in a different process Or you could even load them on different cpu cores: with K.device('cpu0'): input1 = Input(inputShapeOfModel1) output1 = model1(...


1

You can use the --timeout flag to increase the worker timeout. Run gunicorn --help for more information about the available flags


1

Using xgb.train function, you can set nthread in params. xgb.train({'nthread': 3}, dtrain) In the xgboost documentation, afaict, there is no clear way how to set the nthread parameter. Some global params are set by xgb.set_config and some are not, like nthread. But when I walk through their test script, I found nthread is set by params in xgb.train.


1

CUDNN and Tensorflow require a GPU which has a compute capacity of 3.0 at least: not only the CUDA version must take account of this CC, but also these both programs. Indeed: https://stackoverflow.com/questions/38542763/how-can-i-make-tensorflow-run-on-a-gpu-with-capability-2-0/38543201#38543201


1

The python hyperopt library will evaluate multiple trials in parallel, it's open source and there's a paper. Also I'm fairly sure AWS Sagemaker has a distributed Baysian algorithm, it doens't meet your criteria of open source though.


1

Bayesian optimisation is sequential in the sense that you need to know the value of the function for n point to decide through an acquisition criteria the next point to evaluate. Maybe you could customize it to your problem so that the acquisition returns not one point but a batch of them, which you distribute at the next step. You can also use an hybrid ...


1

You can estimate in parallel each of the weak learners. For example, searching for optimal splits in 'weak' decision trees can be streamlined by utilizing large number of cores.


1

The tuple of one partition is always on the same node because a partition itself is impartible. So if you do a groupBy or write your own partitioner which partitions by key, all records with the same key/partition number will be shuffled to the same node. Otherwise, transformations like mapPartition which pass an iterator to a user defined function wouldn't ...


1

There is a better way if I understand your question correctly. Here is the algorithm I propose: Initialize a 'window' list and a 'pairs' list Sort your data on birthday from old to young (or the other way around) Loop over your rows and keep track of all the rows that are still in the last three years since your current row. When you get to a new row, throw ...


1

Depends on the models you are trying to run. Your data isn't that big. For example using a support vector model from the kernlab package, you run into problems. Not every model is fast or has a fast implementation. Without more information on what you are doing it is difficult to say what causes the bottleneck. But if you just want a speed boost in running ...


Only top voted, non community-wiki answers of a minimum length are eligible