43 votes
Accepted

What are the use cases for Apache Spark vs Hadoop

Hadoop means HDFS, YARN, MapReduce, and a lot of other things. Do you mean Spark vs MapReduce? Because Spark runs on/with Hadoop, which is rather the point. The primary reason to use Spark is for ...
user avatar
  • 6,415
23 votes
Accepted

Are there free cloud services to train machine learning models?

There are no unlimited free services*, but some have starting credit or free offers on initial signup. Here are some suggested to date: AWS: If specifically deep learning on a large data set, then ...
22 votes

Are there free cloud services to train machine learning models?

I want to add one more resource, Google Colaboratory. It's a free cloud iPython notebook and gives you free usage of a GPU . I'm not sure of the exact limitations just yet, but it appears you get 12 ...
user avatar
14 votes

Are there free cloud services to train machine learning models?

Check out Crestle.(Free one hour GPU compute time) Google's colab Floyd-hub Paperspace (not free but it's easy to use referrals and earn 15$ compute time..) Azure ($200) deepcognition.ai (2 hours on ...
user avatar
  • 2,400
11 votes

Are there free cloud services to train machine learning models?

Yes, with limitations. Google Cloud Compute gives you 300 dollars worth of free credit signing up, and Microsoft Azure gives you 200 dollars (but their GPU time is a bit cheaper, so it's almost the ...
user avatar
4 votes

Which cloud platform to maximize my impact as a data scientist?

It has been my experience that transitioning from local modeling to large-scale distributed programming is a lot more work than most data scientists realize, and leaves little room for anything BUT ...
user avatar
4 votes
Accepted

How to download dynamic files created during work on Google Colab?

Based on what I've seen and experienced, the best way is to store and retrieve your data from your drive account. Actually your question is a bit unclear but first I say, try to use the following ...
user avatar
  • 13.3k
2 votes
Accepted

Different number of images in classes

Frankly, even 50 images will not be sufficient if you are going to create and use a CNN model. If you think you want more images for you model training, then go for data augmentation. It is a process ...
user avatar
  • 1,729
2 votes

Amazon Cloud Image istance most suited for R data mining

It very much depends on the calculations you are doing as well as the tools you are using to implement them and the size of data you are working with. Some small rules of thumb: R processes (...
user avatar
  • 161
1 vote

What is CI/CD Pipeline in DevOps?

CI = Continous Integration, CD = Continous Delivery. What is CI/CD: Continuous Integration (CI) means automatically testing each change made to your codebase as soon as possible. Continuous delivery (...
user avatar
  • 151
1 vote
Accepted

How to run unmodified Python program on GPU servers with scheduled GPUs?

A popular solution used for job management on GPU environments is SLURM. SLURM allows specifying the resources needed by a job (e.g. 2 CPUs, 2Gb of RAM, 4 GPUs) and it will be scheduled for execution ...
user avatar
  • 15.4k
1 vote

Deep learning on cloud

Did you use your machine GPU? You can use batch processing, and train you're data chunk by chunk, it will take some time to go through all of your 10GB, but it will eventually gives you the result ...
user avatar
  • 174
1 vote

How can I use my laptop to implement analysis in Python but use a different fast server for computation of analysis?

I have a similar set-up; I run both local and remote installations of Jupyter. On the server I have JupyterHub running, into which i can ssh. Locally I have Anaconda set up etc. If you are meticulous ...
user avatar
  • 131
1 vote
Accepted

Train a deep reinforcement learning model using two computers

Distributed RL is very much a thing. Google have created a distributed setup called IMPALA for this, and there are multiple instances of A3C, PPO etc available if you search. I don't know much about ...
user avatar
  • 27.3k
1 vote

Different number of images in classes

You have a very low amount of images to create a CNN from scratch. You may be able to train your model via transfer learning from a pre-trained model, but you still may have too small of a dataset. ...
user avatar
  • 400
1 vote

Are there free cloud services to train machine learning models?

Microsoft's Azure Machine Learning Studio has an "always free" tier, subject to certain limitations, including 100 modules per experiment (a "module" in Azure-speak is any discrete operation such as "...
user avatar
  • 328
1 vote

What are the use cases for Apache Spark vs Hadoop

Not sure about the YARN, but I think that Spark makes a real difference compared to Hadoop (advertised as 100 times faster) if data can fit nicely in the memory of the computational nodes. Simply ...
user avatar
  • 599

Only top scored, non community-wiki answers of a minimum length are eligible