43
votes
Accepted
What are the use cases for Apache Spark vs Hadoop
Hadoop means HDFS, YARN, MapReduce, and a lot of other things. Do you mean Spark vs MapReduce? Because Spark runs on/with Hadoop, which is rather the point.
The primary reason to use Spark is for ...
- 6,415
23
votes
Accepted
Are there free cloud services to train machine learning models?
There are no unlimited free services*, but some have starting credit or free offers on initial signup. Here are some suggested to date:
AWS: If specifically deep learning on a large data set, then ...
22
votes
Are there free cloud services to train machine learning models?
I want to add one more resource, Google Colaboratory. It's a free cloud iPython notebook and gives you free usage of a GPU . I'm not sure of the exact limitations just yet, but it appears you get 12 ...
- 571
14
votes
Are there free cloud services to train machine learning models?
Check out Crestle.(Free one hour GPU compute time)
Google's colab
Floyd-hub
Paperspace (not free but it's easy to use referrals and earn 15$ compute time..)
Azure ($200)
deepcognition.ai (2 hours on ...
- 2,400
11
votes
Are there free cloud services to train machine learning models?
Yes, with limitations. Google Cloud Compute gives you 300 dollars worth of free credit signing up, and Microsoft Azure gives you 200 dollars (but their GPU time is a bit cheaper, so it's almost the ...
- 571
4
votes
Which cloud platform to maximize my impact as a data scientist?
It has been my experience that transitioning from local modeling to large-scale distributed programming is a lot more work than most data scientists realize, and leaves little room for anything BUT ...
- 151
4
votes
Accepted
How to download dynamic files created during work on Google Colab?
Based on what I've seen and experienced, the best way is to store and retrieve your data from your drive account. Actually your question is a bit unclear but first I say, try to use the following ...
- 13.3k
2
votes
Accepted
Different number of images in classes
Frankly, even 50 images will not be sufficient if you are going to create and use a CNN model. If you think you want more images for you model training, then go for data augmentation. It is a process ...
- 1,729
2
votes
Amazon Cloud Image istance most suited for R data mining
It very much depends on the calculations you are doing as well as the tools you are using to implement them and the size of data you are working with.
Some small rules of thumb:
R processes (...
- 161
1
vote
What is CI/CD Pipeline in DevOps?
CI = Continous Integration, CD = Continous Delivery.
What is CI/CD:
Continuous Integration (CI) means automatically testing each change made to your codebase as soon as possible. Continuous delivery (...
- 151
1
vote
Accepted
How to run unmodified Python program on GPU servers with scheduled GPUs?
A popular solution used for job management on GPU environments is SLURM.
SLURM allows specifying the resources needed by a job (e.g. 2 CPUs, 2Gb of RAM, 4 GPUs) and it will be scheduled for execution ...
- 15.4k
1
vote
Deep learning on cloud
Did you use your machine GPU?
You can use batch processing, and train you're data chunk by chunk, it will take some time to go through all of your 10GB, but it will eventually gives you the result ...
- 174
1
vote
How can I use my laptop to implement analysis in Python but use a different fast server for computation of analysis?
I have a similar set-up; I run both local and remote installations of Jupyter. On the server I have JupyterHub running, into which i can ssh. Locally I have Anaconda set up etc.
If you are meticulous ...
- 131
1
vote
Accepted
Train a deep reinforcement learning model using two computers
Distributed RL is very much a thing. Google have created a distributed setup called IMPALA for this, and there are multiple instances of A3C, PPO etc available if you search. I don't know much about ...
- 27.3k
1
vote
Different number of images in classes
You have a very low amount of images to create a CNN from scratch. You may be able to train your model via transfer learning from a pre-trained model, but you still may have too small of a dataset.
...
- 400
1
vote
Are there free cloud services to train machine learning models?
Microsoft's Azure Machine Learning Studio has an "always free" tier, subject to certain limitations, including
100 modules per experiment (a "module" in Azure-speak is any discrete operation such as "...
- 328
1
vote
What are the use cases for Apache Spark vs Hadoop
Not sure about the YARN, but I think that Spark makes a real difference compared to Hadoop (advertised as 100 times faster) if data can fit nicely in the memory of the computational nodes. Simply ...
- 599
Only top scored, non community-wiki answers of a minimum length are eligible