Questions tagged [distributed]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
158 views

How to save hugging face fine tuned model using pytorch and distributed training

I am fine tuning masked language model from XLM Roberta large on google machine specs. When I copy the model using gsutil and subprocess from container to GCP ...
user avatar
  • 257
1 vote
2 answers
56 views

Combining CNNs for image classification

I would like to take the output of an intermediate layer of a CNN (layer G) and feed it to an intermediate layer of a wider CNN (layer H) to complete the inference. Challenge: The two layers G, H have ...
user avatar
  • 63
2 votes
1 answer
94 views

Distributed training with low level Tensorflow API

I am using low level Tensorflow API's for my model training. When I say low level it means I'm defining the tf.Session() object of the graph and evaluate graph with ...
user avatar
  • 1,204
1 vote
2 answers
100 views

Use distribution probability as a feature in ML model

I built an LSMT model to predict sick cows. I also have risk factors like cow size and height (static risk factor) that I want to combine into the ML model. I found that size is geometrically ...
user avatar
  • 11
1 vote
0 answers
70 views

CountVectorizer vs HashVectorizer for text

I'd like to tokenize a column of my training data (n-gram word-wise), but I'm working with a very large dataset distributed across a compute cluster. For this use case, Count Vectorizer doens't work ...
user avatar
  • 49
1 vote
0 answers
29 views

Federated learning - share of ROI

I am reading about federated learning and have a quick question 1) I know in federated learning, the model updates are shared to a central server 2) All the parties involved in FL can generate ...
user avatar
  • 2,215
1 vote
0 answers
17 views

Pytorch Distributed Computing - Recomendations/Resources/Courses?

I would like to get into some distributed computing for processing Pytorch CNN models. I am completely fresh in this field and want to get some recommendations as to where I should start researching ...
user avatar
0 votes
1 answer
72 views

Who is actually sharing physical RAM in a distributed sytem that has virtual shared memory? (Server and/or clients.)

There is a business with about 100 computers used by employees, and one high-powered server. It's called a "distributed system" by the system architect. It uses Distributed Shared Memory (DSM). ...
user avatar
0 votes
1 answer
194 views

Speed of training decrease by adding more GPUs

I am using the distributed Tensorflow with Mirror Strategy. I am training the VGG16 based on custom Estimator. However, by increasing the number of GPUs time of training is increased. As I check, the ...
user avatar
  • 1
1 vote
0 answers
101 views

Updating Weight Using Updates on Related Data

Suppose $$ x=Ay $$ The $x$ is $M\times 1$, $y$ is $N \times 1$ and $A$ is $M\times N$ We have the data $x$ and would like to know what $y$ is. However, the matrix $A$ is too large for pseudo-...
user avatar
9 votes
1 answer
176 views

What is meant by Distributed for a gradient boosting library?

I am checking out XGBoost documentation and it's stated that XGBoost is an optimized distributed gradient boosting library. What is meant by distributed? Have a nice day
user avatar
0 votes
1 answer
106 views

Why can distributed deep learning provide higher accuracy (lower error) than non-distributed one with the following cases?

Based on some papers which I read, distributed deep learning can provide faster training time. In addition, it also provides better accuracy or lower prediction error. What are the reasons? Question ...
user avatar
  • 103
0 votes
1 answer
96 views

Implementation of a distributed data mining paper

I have a project about distributed data mining and I need to do some implementations, So I've searched and found this paper. The address of dataset is mentioned in the paper and I've downloaded it. ...
user avatar
  • 3
5 votes
2 answers
475 views

Distributed PCA or an equivalent

We normally have fairly large datasets to model on, just to give you an idea: over 1M features (sparse, average population of features is around 12%); over 60M rows. A lot of modeling algorithms ...
user avatar
  • 198
1 vote
1 answer
57 views

Which cloud platform to maximize my impact as a data scientist? [closed]

I am looking to pick up the knowledge/software skills to move towards becoming an end to end deep learning engineer. By this I mean handling the following on my own: preprocess big data at low ...
user avatar
  • 113
9 votes
2 answers
13k views

What is the difference between Pytorch's DataParallel and DistributedDataParallel?

I am going through this imagenet example. And, in line 88, the module DistributedDataParallel is used. When I searched for the same in the docs, I haven’t found anything. However, I found the ...
user avatar
  • 8,016
8 votes
2 answers
871 views

Understanding how distributed PCA works

As part of big data analysis project, I'm working on, I need to perform PCA on some data, using cloud computing system. In my case, I'm using Amazon EMR for the job and Spark in particular. Leaving ...
user avatar
  • 183
1 vote
1 answer
179 views

How to Scaling Out Artifical Neural Networks?

I have started to study ANNs with Tensorflow and Keras. Now I want to find a solution to use ANNs over Hadoop. I have learnt that Spark 2.0 does have a Multilayer Perceptron Classifier, but as far as ...
user avatar
  • 7,927
1 vote
1 answer
843 views

How to combine two CART decision trees learned in same type of data?

We have distributed data centers and we build decision trees in each data center. Our problem is to combine our CART decision trees into one CART decision tree. The data in each data center related to ...
user avatar
  • 13
1 vote
0 answers
75 views

Parallel processing for feature selection in microarray dataset

I want to apply feature selection on a dataset with some 30-40K columns and 100 rows ( total size: 400MB-800MB ). To decrease the time consumed for calculations involved (feature-feature), I want to ...
user avatar
  • 11
10 votes
3 answers
13k views

Large Graphs: NetworkX distributed alternative

I have built some implementations using NetworkX(graph Python module) native algorithms in which I output some attributes which I use them for classification ...
user avatar
  • 610
5 votes
1 answer
4k views

How to decided which test of normality to use

Given a data set with features, that you want to check for normality, one feature at a time w/o a multivariate normal test, how do you decided which test of normality to use? For example, using the ...
user avatar
  • 109
0 votes
3 answers
275 views

Predictive Analytics on distributed systems vs standalone system

I have somewhat philosophical question regarding performing predictive analytics on distributed systems (such as hadoop). I am no expert on this subject so maybe other folks who have more advance ...
user avatar
1 vote
0 answers
301 views

Multi-GPU, multi-machine computation with Torch

I know that Torch supports multi-GPU computation on the same machine (Example). Is it possible to perform multi-GPU, multi-machine computations with Torch?
user avatar
5 votes
2 answers
3k views

Distributed k-means in Spark

I want to implement K-means algorithm in Spark. I am looking for a starting point and I found Berkeley's naive implementation. However, is that distributed? I mean I see no mapreduce operations. Or ...
user avatar
  • 291
6 votes
1 answer
2k views

How to make k-means distributed?

After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code: ...
user avatar
  • 291
7 votes
2 answers
1k views

Python distributed machine learning

I occasionally train neural nets for my research, and they usually take quite a long time to run (especially when I'm working on my laptop). I'm looking for a way to build the model on any computer ...
user avatar
  • 1,051
0 votes
2 answers
789 views

Sampling from a multivariate von Mises-Fisher distribution in Python

I am looking for a simple way to sample from a multivariate von Mises-Fisher distribution in Python. I have looked in the stats module in scipy and the numpy module but only found the univariate von ...
user avatar
  • 513
0 votes
1 answer
78 views

Possibility of working on KDDCup data in local system

I'm trying to apply classification algorithms to KDD Cup 2012 track2 data using R http://www.kddcup2012.org/c/kddcup2012-track2 It seems not possible to work with this 10GB training data on my local ...
user avatar
  • 69
4 votes
2 answers
2k views

Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib

I want to implement Streaming Naive Bayes in a distributed system. What are the best approach to choose framework. Should I choose: Storm alone and implement streaming naive bayes on my own in storm ...
user avatar
  • 141
21 votes
3 answers
7k views

Nearest neighbors search for very high dimensional data

I have a big sparse matrix of users and items they like (in the order of 1M users and 100K items, with a very low level of sparsity). I'm exploring ways in which I could perform kNN search on it. ...
user avatar
  • 431
5 votes
2 answers
886 views

How to speedup message passing between computing nodes

I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much spread along the entire ...
user avatar
  • 4,027
33 votes
5 answers
13k views

What are the use cases for Apache Spark vs Hadoop

With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've ...
user avatar
  • 511
14 votes
4 answers
2k views

Looking for example infrastructure stacks/workflows/pipelines

I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
user avatar
  • 143
4 votes
2 answers
747 views

How to measure execution time on distributed system

I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals. I have limited number of machines nearly ten ...
user avatar
  • 4,027
8 votes
3 answers
158 views

How to compare experiments run over different infrastructures

I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
user avatar
  • 4,027
12 votes
2 answers
7k views

Tradeoffs between Storm and Hadoop (MapReduce)

Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing ...
user avatar
  • 347
15 votes
3 answers
1k views

Parallel and distributed computing

What is(are) the difference(s) between parallel and distributed computing? When it comes to scalability and efficiency, it is very common to see solutions dealing with computations in clusters of ...
user avatar
  • 4,027