Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [distributed]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
19 views

Who is actually sharing physical RAM in a distributed sytem that has virtual shared memory? (Server and/or clients.)

There is a business with about 100 computers used by employees, and one high-powered server. It's called a "distributed system" by the system architect. It uses Distributed Shared Memory (DSM). ...
0
votes
0answers
6 views

Minimal Fragmentation in Distributed Database

In a distributed database, using selection we get a minimal fragmentation if If all the predicates of a set Pr are relevant, Pr is minimal. Given fragments generated by a set $M(P)$, we say a ...
0
votes
1answer
27 views

Speed of training decrease by adding more GPUs

I am using the distributed Tensorflow with Mirror Strategy. I am training the VGG16 based on custom Estimator. However, by increasing the number of GPUs time of training is increased. As I check, the ...
0
votes
0answers
12 views

Distributed optimization with a common variable among individual agents

I am new to optimization techniques and have a doubt in my approach. Consider I have an agent which tries to optimize on 3 variables C1,x11,x12 to minimize power. I have 60 such agents which ...
1
vote
0answers
99 views

Updating Weight Using Updates on Related Data

Suppose $$ x=Ay $$ The $x$ is $M\times 1$, $y$ is $N \times 1$ and $A$ is $M\times N$ We have the data $x$ and would like to know what $y$ is. However, the matrix $A$ is too large for pseudo-...
0
votes
0answers
16 views

Best way to optimize server runtime for multiple jobs?

I am not well versed in the data science field as I am working on devops now so I wanted to ask this community. If you can refer me to a more proper stackoverflow then let me know! Given servers and ...
0
votes
0answers
10 views

Should we invest more in exploration (as opposed to exploitation) when the distribution is fat-tailed in contrast to a bell-curve?

Not sure if this falls under data science but here it goes: In the exploration/exploitation trade-off (deliberation vs commitment to one choice), should we invest more in exploration when the ...
9
votes
1answer
126 views

What is meant by Distributed for a gradient boosting library?

I am checking out XGBoost documentation and it's stated that XGBoost is an optimized distributed gradient boosting library. What is meant by distributed? Have a nice day
0
votes
1answer
47 views

Why can distributed deep learning provide higher accuracy (lower error) than non-distributed one with the following cases?

Based on some papers which I read, distributed deep learning can provide faster training time. In addition, it also provides better accuracy or lower prediction error. What are the reasons? Question ...
1
vote
1answer
86 views

Implementation of a distributed data mining paper

I have a project about distributed data mining and I need to do some implementations, So I've searched and found this paper. The address of dataset is mentioned in the paper and I've downloaded it. ...
5
votes
2answers
336 views

Distributed PCA or an equivalent

We normally have fairly large datasets to model on, just to give you an idea: over 1M features (sparse, average population of features is around 12%); over 60M rows. A lot of modeling algorithms ...
0
votes
1answer
51 views

Which cloud platform to maximize my impact as a data scientist? [closed]

I am looking to pick up the knowledge/software skills to move towards becoming an end to end deep learning engineer. By this I mean handling the following on my own: preprocess big data at low ...
3
votes
2answers
7k views

What is the difference between Pytorch's DataParallel and DistributedDataParallel?

I am going through this imagenet example. And, in line 88, the module DistributedDataParallel is used. When I searched for the same in the docs, I haven’t found anything. However, I found the ...
7
votes
2answers
529 views

Understanding how distributed PCA works

As part of big data analysis project, I'm working on, I need to perform PCA on some data, using cloud computing system. In my case, I'm using Amazon EMR for the job and Spark in particular. Leaving ...
1
vote
1answer
174 views

How to Scaling Out Artifical Neural Networks?

I have started to study ANNs with Tensorflow and Keras. Now I want to find a solution to use ANNs over Hadoop. I have learnt that Spark 2.0 does have a Multilayer Perceptron Classifier, but as far as ...
1
vote
1answer
592 views

How to combine two CART decision trees learned in same type of data?

We have distributed data centers and we build decision trees in each data center. Our problem is to combine our CART decision trees into one CART decision tree. The data in each data center related to ...
1
vote
0answers
67 views

Parallel processing for feature selection in microarray dataset

I want to apply feature selection on a dataset with some 30-40K columns and 100 rows ( total size: 400MB-800MB ). To decrease the time consumed for calculations involved (feature-feature), I want to ...
6
votes
2answers
7k views

Large Graphs: NetworkX distributed alternative

I have built some implementations using NetworkX(graph Python module) native algorithms in which I output some attributes which I use them for classification ...
6
votes
1answer
3k views

How to decided which test of normality to use

Given a data set with features, that you want to check for normality, one feature at a time w/o a multivariate normal test, how do you decided which test of normality to use? For example, using the ...
0
votes
3answers
266 views

Predictive Analytics on distributed systems vs standalone system

I have somewhat philosophical question regarding performing predictive analytics on distributed systems (such as hadoop). I am no expert on this subject so maybe other folks who have more advance ...
1
vote
0answers
297 views

Multi-GPU, multi-machine computation with Torch

I know that Torch supports multi-GPU computation on the same machine (Example). Is it possible to perform multi-GPU, multi-machine computations with Torch?
5
votes
2answers
3k views

Distributed k-means in Spark

I want to implement K-means algorithm in Spark. I am looking for a starting point and I found Berkeley's naive implementation. However, is that distributed? I mean I see no mapreduce operations. Or ...
1
vote
1answer
369 views

Do something with the output of reducers

I am executing the job as: ...
5
votes
1answer
1k views

How to make k-means distributed?

After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code: ...
5
votes
2answers
1k views

Python distributed machine learning

I occasionally train neural nets for my research, and they usually take quite a long time to run (especially when I'm working on my laptop). I'm looking for a way to build the model on any computer ...
0
votes
2answers
596 views

Sampling from a multivariate von Mises-Fisher distribution in Python

I am looking for a simple way to sample from a multivariate von Mises-Fisher distribution in Python. I have looked in the stats module in scipy and the numpy module but only found the univariate von ...
0
votes
1answer
78 views

Possibility of working on KDDCup data in local system

I'm trying to apply classification algorithms to KDD Cup 2012 track2 data using R http://www.kddcup2012.org/c/kddcup2012-track2 It seems not possible to work with this 10GB training data on my local ...
4
votes
2answers
2k views

Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib

I want to implement Streaming Naive Bayes in a distributed system. What are the best approach to choose framework. Should I choose: Storm alone and implement streaming naive bayes on my own in storm ...
17
votes
3answers
6k views

Nearest neighbors search for very high dimensional data

I have a big sparse matrix of users and items they like (in the order of 1M users and 100K items, with a very low level of sparsity). I'm exploring ways in which I could perform kNN search on it. ...
5
votes
2answers
750 views

How to speedup message passing between computing nodes

I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much spread along the entire ...
31
votes
5answers
11k views

What are the use cases for Apache Spark vs Hadoop

With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've ...
14
votes
4answers
2k views

Looking for example infrastructure stacks/workflows/pipelines

I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
5
votes
2answers
507 views

How to measure execution time on distributed system

I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals. I have limited number of machines nearly ten ...
7
votes
3answers
119 views

How to compare experiments run over different infrastructures

I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
12
votes
2answers
7k views

Tradeoffs between Storm and Hadoop (MapReduce)

Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing ...
14
votes
3answers
840 views

Parallel and distributed computing

What is(are) the difference(s) between parallel and distributed computing? When it comes to scalability and efficiency, it is very common to see solutions dealing with computations in clusters of ...