0
$\begingroup$

It seems Hadoop, Spark, and different versions of Clouds offer facilities to store and analyze big data. There are some articles comparing Hadoop and Spark (for example, this article). There are also articles which discuss services on AWS for working with Hadoop/spark, such as this article.

My question is, if I am using cloud computing which provides storage and analytics services, will I still need hadoop/spark? Any references / readings which help in identifying different use cases and applications is highly appreciated!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Depends on what your 'storage and analytics services' are exactly.

For most cloud SaaS services they handle the technology under the hood. For example, Dropbox and Gmail probably have Hadoop and Spark underneath, which users never have to touch. However, read the price structure of the products carefully to see whether the service provider absorbs the cost.

Practically speaking, once you have an objective, just starting using the picked services until you hit a wall. This is the moment you know what you lack.

Improve this answer
$\endgroup$
4
  • $\begingroup$ thanks, by cloud, I meant something like AWS, Azure, or GCP, and not dropbox or gmail. Some planning is needed to reduce and justify project costs, so it would be good to have some sort of general understanding of how such technologies interact. So any reference to best practises or use cases of when to choose which technology (or combination of technogies) is helpful $\endgroup$
    – Tara
    Commented Apr 19, 2023 at 5:39
  • $\begingroup$ It's the same thing - the storage and analytics services are most likely built on top of AWS, GCP and Azure etc., just as Dropbox and Gmail do. I think you are confused by the terms. $\endgroup$
    – lpounng
    Commented Apr 19, 2023 at 7:16
  • $\begingroup$ Hadoop and Spark are software. They can be installed on your own computer, or you can rent a cluster from a cloud provider and install on it. Most cloud providers also offer self-served Hadoop/Spark services (e.g. Dataproc of GCP) so you do not have to deal with all the installation and maintenance hassles. This is roughly how Dropbox works - under the hood it rent a cluster from AWS (on which Hadoop is in place), and resell the storage space as service to public. $\endgroup$
    – lpounng
    Commented Apr 19, 2023 at 7:21
  • $\begingroup$ Move up a level, there are also self-served analytic services available, where you can feed your data in and get the result you want. Probably they also have Hadoop/Spark under the hood. However, I cannot say anything unless the service is named. $\endgroup$
    – lpounng
    Commented Apr 19, 2023 at 7:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.