Questions tagged [distributed]
The distributed tag has no usage guidance.
38
questions
1
vote
0
answers
1k
views
How to save hugging face fine tuned model using pytorch and distributed training
I am fine tuning masked language model from XLM Roberta large on google machine specs.
When I copy the model using gsutil and subprocess from container to GCP ...
- 277
1
vote
2
answers
62
views
Combining CNNs for image classification
I would like to take the output of an intermediate layer of a CNN (layer G) and feed it to an intermediate layer of a wider CNN (layer H) to complete the inference.
Challenge: The two layers G, H have ...
- 63
2
votes
1
answer
131
views
Distributed training with low level Tensorflow API
I am using low level Tensorflow API's for my model training. When I say low level it means I'm defining the tf.Session() object of the graph and evaluate graph with ...
- 1,232
1
vote
2
answers
214
views
Use distribution probability as a feature in ML model
I built an LSMT model to predict sick cows. I also have risk factors like cow size and height (static risk factor) that I want to combine into the ML model. I found that size is geometrically ...
- 11
1
vote
0
answers
84
views
CountVectorizer vs HashVectorizer for text
I'd like to tokenize a column of my training data (n-gram word-wise), but I'm working with a very large dataset distributed across a compute cluster. For this use case, Count Vectorizer doens't work ...
- 49
1
vote
0
answers
32
views
Federated learning - share of ROI
I am reading about federated learning and have a quick question
1) I know in federated learning, the model updates are shared to a central server
2) All the parties involved in FL can generate ...
- 2,507
1
vote
0
answers
18
views
Pytorch Distributed Computing - Recomendations/Resources/Courses?
I would like to get into some distributed computing for processing Pytorch CNN models. I am completely fresh in this field and want to get some recommendations as to where I should start researching ...
- 11
0
votes
1
answer
97
views
Who is actually sharing physical RAM in a distributed sytem that has virtual shared memory? (Server and/or clients.)
There is a business with about 100 computers used by employees, and one high-powered server. It's called a "distributed system" by the system architect. It uses Distributed Shared Memory (DSM). ...
user79134
0
votes
1
answer
209
views
Speed of training decrease by adding more GPUs
I am using the distributed Tensorflow with Mirror Strategy. I am training the VGG16 based on custom Estimator. However, by increasing the number of GPUs time of training is increased. As I check, the ...
- 1
1
vote
0
answers
101
views
Updating Weight Using Updates on Related Data
Suppose
$$
x=Ay
$$
The $x$ is $M\times 1$, $y$ is $N \times 1$ and $A$ is $M\times N$
We have the data $x$ and would like to know what $y$ is.
However, the matrix $A$ is too large for pseudo-...
9
votes
1
answer
195
views
What is meant by Distributed for a gradient boosting library?
I am checking out XGBoost documentation and it's stated that XGBoost is an optimized distributed gradient boosting library.
What is meant by distributed?
Have a nice day
0
votes
1
answer
151
views
Why can distributed deep learning provide higher accuracy (lower error) than non-distributed one with the following cases?
Based on some papers which I read, distributed deep learning can provide faster training time. In addition, it also provides better accuracy or lower prediction error. What are the reasons?
Question ...
- 103
0
votes
1
answer
98
views
Implementation of a distributed data mining paper
I have a project about distributed data mining and I need to do some implementations, So I've searched and found this paper. The address of dataset is mentioned in the paper and I've downloaded it. ...
- 3
5
votes
2
answers
497
views
Distributed PCA or an equivalent
We normally have fairly large datasets to model on, just to give you an idea:
over 1M features (sparse, average population of features is around
12%);
over 60M rows.
A lot of modeling algorithms ...
- 198
1
vote
1
answer
63
views
Which cloud platform to maximize my impact as a data scientist? [closed]
I am looking to pick up the knowledge/software skills to move towards becoming an end to end deep learning engineer. By this I mean handling the following on my own:
preprocess big data at low ...
- 113
10
votes
2
answers
14k
views
What is the difference between Pytorch's DataParallel and DistributedDataParallel?
I am going through this imagenet example.
And, in line 88, the module DistributedDataParallel is used. When I searched for the same in the docs, I haven’t found anything. However, I found the ...
- 8,156
8
votes
2
answers
916
views
Understanding how distributed PCA works
As part of big data analysis project, I'm working on,
I need to perform PCA on some data, using cloud computing system.
In my case, I'm using Amazon EMR for the job and Spark in particular.
Leaving ...
- 183
1
vote
1
answer
185
views
How to Scaling Out Artifical Neural Networks?
I have started to study ANNs with Tensorflow and Keras. Now I want to find a solution to use ANNs over Hadoop. I have learnt that Spark 2.0 does have a Multilayer Perceptron Classifier, but as far as ...
- 8,297
1
vote
1
answer
896
views
How to combine two CART decision trees learned in same type of data?
We have distributed data centers and we build decision trees in each data center. Our problem is to combine our CART decision trees into one CART decision tree. The data in each data center related to ...
- 13
1
vote
0
answers
75
views
Parallel processing for feature selection in microarray dataset
I want to apply feature selection on a dataset with some 30-40K columns and 100 rows ( total size: 400MB-800MB ). To decrease the time consumed for calculations involved (feature-feature), I want to ...
- 11
12
votes
4
answers
14k
views
Large Graphs: NetworkX distributed alternative
I have built some implementations using NetworkX(graph Python module) native algorithms in which I output some attributes which I use them for classification ...
- 650
5
votes
1
answer
4k
views
How to decided which test of normality to use
Given a data set with features, that you want to check for normality, one feature at a time w/o a multivariate normal test, how do you decided which test of normality to use? For example, using the ...
- 109
0
votes
3
answers
277
views
Predictive Analytics on distributed systems vs standalone system
I have somewhat philosophical question regarding performing predictive analytics on distributed systems (such as hadoop). I am no expert on this subject so maybe other folks who have more advance ...
2
votes
0
answers
303
views
Multi-GPU, multi-machine computation with Torch
I know that Torch supports multi-GPU computation on the same machine (Example).
Is it possible to perform multi-GPU, multi-machine computations with Torch?
- 5,553
5
votes
2
answers
3k
views
Distributed k-means in Spark
I want to implement K-means algorithm in Spark. I am looking for a starting point and I found Berkeley's naive implementation. However, is that distributed?
I mean I see no mapreduce operations. Or ...
- 291
6
votes
1
answer
3k
views
How to make k-means distributed?
After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code:
...
- 291
7
votes
2
answers
1k
views
Python distributed machine learning
I occasionally train neural nets for my research, and they usually take quite a long time to run (especially when I'm working on my laptop).
I'm looking for a way to build the model on any computer ...
- 1,071
0
votes
2
answers
839
views
Sampling from a multivariate von Mises-Fisher distribution in Python
I am looking for a simple way to sample from a multivariate von Mises-Fisher distribution in Python. I have looked in the stats module in scipy and the numpy module but only found the univariate von ...
- 513
0
votes
1
answer
81
views
Possibility of working on KDDCup data in local system
I'm trying to apply classification algorithms to KDD Cup 2012 track2 data using R
http://www.kddcup2012.org/c/kddcup2012-track2
It seems not possible to work with this 10GB training data on my local ...
- 69
4
votes
2
answers
2k
views
Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib
I want to implement Streaming Naive Bayes in a distributed system. What are the best approach to choose framework. Should I choose:
Storm alone and implement streaming naive bayes on my own in storm ...
- 141
23
votes
3
answers
8k
views
Nearest neighbors search for very high dimensional data
I have a big sparse matrix of users and items they like (in the order of 1M users and 100K items, with a very low level of sparsity). I'm exploring ways in which I could perform kNN search on it. ...
- 451
5
votes
2
answers
910
views
How to speedup message passing between computing nodes
I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much spread along the entire ...
- 4,097
34
votes
5
answers
13k
views
What are the use cases for Apache Spark vs Hadoop
With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've ...
- 521
14
votes
4
answers
2k
views
Looking for example infrastructure stacks/workflows/pipelines
I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used ...
- 143
4
votes
2
answers
802
views
How to measure execution time on distributed system
I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals.
I have limited number of machines nearly ten ...
- 4,097
8
votes
3
answers
177
views
How to compare experiments run over different infrastructures
I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
- 4,097
12
votes
2
answers
7k
views
Tradeoffs between Storm and Hadoop (MapReduce)
Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing ...
- 347
16
votes
3
answers
1k
views
Parallel and distributed computing
What is(are) the difference(s) between parallel and distributed computing? When it comes to scalability and efficiency, it is very common to see solutions dealing with computations in clusters of ...
- 4,097