I worked at a startup/medium sized company and I am concerned that we may be over-engineering one of our products.
In essence, we will be consuming real-time coordinates from vehicles and users and performing analytics and machine learning on this incoming data. This processing can be rather intensive as we try predict the ETAs of this entities matched to historical data and static paths.
The approach they want to take is using the latest and most powerful technology stack, that being Hadoop, Storm etc to process these coordinates. Problem is that no-one in the team has implemented such a system and only has had the last month or so to skill up on it.
My belief is that a safer approach would be to use NoSQL storage such as "Azure Table Storage" in an event based system to achieve the same result in less time. To me it's the agile approach, as this is a system that we are familiar with. Then if the demand warrants it, we can look at implementing Hadoop in the future.
I haven't done a significant amount of research in this field, so would appreciate your input.
Questions:
- How many tracking entities (sending coordinates every 10 seconds) would warrant Hadoop?
- Would it be easy to initially start off with a simpler approach such as "Azure Table Storage" then onto Hadoop at a later point?
- If you had to estimate, how long would you say a team of 3 developers would take to implement a basic Hadoop/Storm system?
- Is Hadoop necessary to invest from the get go as we will quickly incur major costs?
I know these are vague questions, but I want to make sure we aren't going to invest unnecessary resources with a deadline coming up.