3
$\begingroup$

Where exactly bigdata platforms fit in to a data science/ machine learning projects ?

Say I have a large dataset for a binary classification problem - cats and dogs.

Now I need to create a model for real time classification

Here is my question.

1 Since dataset is huge I can make use any distributed platform for faster computation and model creation right?

2 Once the model is ready, then there is no need of these distributed platform right ? or are they needed for feature extractions ?

$\endgroup$
3
  • $\begingroup$ I'd serve the model in something lighter, faster. Hadoop is not suited for serving models. I think you've got the idea. $\endgroup$
    – Emre
    Commented Mar 9, 2017 at 17:38
  • $\begingroup$ so they can be used only for creating models, not for real time predictions right ? $\endgroup$ Commented Mar 13, 2017 at 5:46
  • $\begingroup$ Spark can be near real-time, but not Hadoop. oreilly.com/ideas/… $\endgroup$
    – Emre
    Commented Mar 13, 2017 at 15:03

1 Answer 1

2
$\begingroup$

1 Since dataset is huge I can make use any disributed platform for faster computaion and model creation right?

Yes, that is what distributed platforms are for.

2 Once the model is ready, then there is no need of these disributed platform right ? or are they needed for feature extractions ?

Yes for data extraction, feature generation for scoring and scoring your dataset (the actual reason you trained at model for!), you would still need the platform as that data is also likely to be huge.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.