3
$\begingroup$

I worked at a startup/medium sized company and I am concerned that we may be over-engineering one of our products.

In essence, we will be consuming real-time coordinates from vehicles and users and performing analytics and machine learning on this incoming data. This processing can be rather intensive as we try predict the ETAs of this entities matched to historical data and static paths.

The approach they want to take is using the latest and most powerful technology stack, that being Hadoop, Storm etc to process these coordinates. Problem is that no-one in the team has implemented such a system and only has had the last month or so to skill up on it.

My belief is that a safer approach would be to use NoSQL storage such as "Azure Table Storage" in an event based system to achieve the same result in less time. To me it's the agile approach, as this is a system that we are familiar with. Then if the demand warrants it, we can look at implementing Hadoop in the future.

I haven't done a significant amount of research in this field, so would appreciate your input.

Questions:

  • How many tracking entities (sending coordinates every 10 seconds) would warrant Hadoop?
  • Would it be easy to initially start off with a simpler approach such as "Azure Table Storage" then onto Hadoop at a later point?
  • If you had to estimate, how long would you say a team of 3 developers would take to implement a basic Hadoop/Storm system?
  • Is Hadoop necessary to invest from the get go as we will quickly incur major costs?

I know these are vague questions, but I want to make sure we aren't going to invest unnecessary resources with a deadline coming up.

|improve this question
$\endgroup$
  • $\begingroup$ How much data do you expect now, in 1 year and in 5 years? What kind of analysis do you plan to apply to your data? $\endgroup$ – ffriend Jul 28 '15 at 15:31
  • $\begingroup$ Hard to say as we are more building to cater for possible growth. Right now we have around 100 gigs of data for 30 entities over 3 years. So that is approximately 10 gigs per entity per year. We would like to be able to support thousands as we foresee major growth in the coming year or two. e.g. could break the 1T mark per year. The analysis currently is to predict the ETAs of these coordinates coming in. The final algorithm hasn't been decided on, but it will be a combination of machine learning from historical data and path matching. $\endgroup$ – Kyle Jul 29 '15 at 8:03
5
$\begingroup$

Yes, this is a how-long-is-a-piece-of-string question. I think it's good to beware of over-engineering, while also making sure you engineer for where you think you'll be in a year.

First I'd suggest you distinguish between processing and storage. Storm is a (stream) processing framework; NoSQL databases are a storage paradigm. These are not alternatives. The Hadoop ecosystem has HBase for NoSQL; I suspect Azure has some kind of stream processing story.

The bigger difference in your two alternatives is consuming a cloud provider's ecosystem vs Hadoop. The upside to Azure, or AWS, or GCE, is that these services optimize for integrating with each other, with billing, machine management, etc. The downside is being locked in to the cloud provider; you can't run Azure stuff anywhere but Azure. Hadoop takes more work to integrate since it's really a confederation of sometimes loosely-related projects. You're investing in both a distribution, and a place to run that distribution. But, you get a lot less lock-in, and probably more easy access to talent, and a broader choice of tools.

The Azure road is also a "big data" solution in that it has a lot of the scalability properties you want for big data, and the complexity as well. It does not strike me as an easier route. Do you need to invest in distributed/cloud anything at this scale? given your IoT-themed use case, I believe you will need to soon, if not now, so yes. You're not talking about gigabytes, but many terabytes in just the first year.

I'd give a fresh team 6-12 months to fully productionize something based on either of these platforms. That can certainly be staged as a POC, followed by more elaborate engineering.

|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for the response. Originally thought the "Big Data" eco-system was an all or nothing. Good to know that we can utilise some of the frameworks such as Storm without committing to the full stack. Want to incrementally implement this as oppose to building a full-fledged system straight away. The reason for this, is that initially we won't have any demand and only building this system to offer a service and scale it when the demand warrants it. Don't doubt that, should that service becomes popular, we will need Hadoop but wondering if the first implementation could do without it. $\endgroup$ – Kyle Jul 29 '15 at 8:29
  • $\begingroup$ I may very well be over-estimating the investment needed for Hadoop and the reason ATS would be an easier route is because we have implemented it before so it is a familiar technology. Also, being extra cautious as we do have an impending deadline on this service, so I want to make sure we aren't over-engineering our first release, while still ensuring that it is future proof. Last question, is it relatively easy to migrate from ATS to Hadoop? $\endgroup$ – Kyle Jul 29 '15 at 8:38
  • 1
    $\begingroup$ Yes the ecosystem is getting as broad as the umbrella term "Linux" -- hundreds of relevant projects. You'd never use more than a fraction. I don't think demand for your service determines how many things you put in your architecture; it determines how big your cluster is. You may add architecture as you add features. If by "Hadoop" you mean "more than one machine" I think you clearly need that from the start. There is no such thing as migrating from ATS to Hadoop; there are analogs (like HBase) but totally different API. Architecture is reusable; code is not. $\endgroup$ – Sean Owen Jul 29 '15 at 9:05
0
$\begingroup$

First, understand and solve the problem. On manageable data. Gather experience on how to organize the data, and where the difficulties are. Try to identify points where parallelism is possible.

Second, parrallelize and scale up as necessary.

Don't do it backwards, popular mistake. Solving the wrong problem with the wrong tools will fail big, with big data.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.