3
$\begingroup$

Where exactly bigdata platforms fit in to a data science/ machine learning projects ?

Say I have a large dataset for a binary classification problem - cats and dogs.

Now I need to create a model for real time classification

Here is my question.

1 Since dataset is huge I can make use any distributed platform for faster computation and model creation right?

2 Once the model is ready, then there is no need of these distributed platform right ? or are they needed for feature extractions ?

Improve this question
$\endgroup$
3
  • $\begingroup$ I'd serve the model in something lighter, faster. Hadoop is not suited for serving models. I think you've got the idea. $\endgroup$
    – Emre
    Mar 9, 2017 at 17:38
  • $\begingroup$ so they can be used only for creating models, not for real time predictions right ? $\endgroup$
    – Sreejithc321
    Mar 13, 2017 at 5:46
  • $\begingroup$ Spark can be near real-time, but not Hadoop. oreilly.com/ideas/… $\endgroup$
    – Emre
    Mar 13, 2017 at 15:03

1 Answer 1

Reset to default
2
$\begingroup$

1 Since dataset is huge I can make use any disributed platform for faster computaion and model creation right?

Yes, that is what distributed platforms are for.

2 Once the model is ready, then there is no need of these disributed platform right ? or are they needed for feature extractions ?

Yes for data extraction, feature generation for scoring and scoring your dataset (the actual reason you trained at model for!), you would still need the platform as that data is also likely to be huge.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.