First of all, I am new in this field we call big data, so my questions may be naive.

In order to build an application, which deals with geolocation data, which could be : latitude and longitude coordinates and Geography SQL Server column types.

I need to have the following elements made easy:

  • Scalability : be prepared to receive huge amount of data, adding servers to the system have to be easy
  • Proximity requests : in example, how much points are in a circle (at meter scale).
  • Data must be accessible rapidly after being written.

I've been looking around for existing solutions, which are "Hadoop friendly" (Hortonworks, Cloudera) and available DBMS, like Cassandra. I have found some interesting information, but I still think it's hard to decide, which one to choose. It also need drivers for NodeJS & .NET (Hadoop with Cassandra seem to be OK with that). I've also looked around the MongoDB ecosystem, but, again, I feel that it is hard to know where to look at. By (little) experience with Mongoose, MongoDB can be disqualified by the third point because data writes are slow. But my model could certainly be improved.

Do any of you have some recent experiences, manipulating massive amount of geolocation data? I would appreciate sharing them here as well as any quality and recent literature on the subject.

  • 1
    $\begingroup$ What do you mean by "unstructured geolocalisation data"? A bunch of lat-long coordinates? A mix of coordinates, postal addresses and IP numbers? $\endgroup$ – Spacedman May 5 '15 at 20:23
  • $\begingroup$ Sorry I didn't mentioned it, it can be all of those examples. $\endgroup$ – Totem May 6 '15 at 6:15
  • 1
    $\begingroup$ I think you really need to spell out in more detail what you want. Even those four terms you have included could have a zillion possibilities. "Clustering" do you mean grouping location points together for display, for analysis, or is this database clustering (ie a group of database servers)? "Saving/restoring"? Of what? To where? And why? $\endgroup$ – Spacedman May 6 '15 at 14:16
  • $\begingroup$ You are right @Spacedman, I have edited my post. I hope it is now clearer. $\endgroup$ – Totem May 26 '15 at 9:03

One approach can be to use a no-SQL database running on top of a distributed store (like Cassandra or Hbase). Add an external index which support spatial indexing (say elastic) for fast search. This makes your solution scalable (due to distributed store) and fast enough for spatial search queries.

  • $\begingroup$ Thank you for sharing @Shagun, I will consider this option. $\endgroup$ – Totem May 27 '15 at 9:54
  • $\begingroup$ Sure. Do as share your experience. $\endgroup$ – Shagun Sodhani May 27 '15 at 10:28
  • $\begingroup$ What if, instead of using Elastic on top, I use geohashing as an index in Cassandra ? $\endgroup$ – Totem May 27 '15 at 13:31
  • $\begingroup$ Geohash is not a spatial index. I assume you were referring to some kind of spatial index. So yeah you can do that. My intent of mentioning Elastic (or any external index) was to take care of cases where the indexing offered by the database itself does not take care of spatial datatypes. $\endgroup$ – Shagun Sodhani May 27 '15 at 13:35
  • $\begingroup$ Ok I read too fast, thank you for your explanations @Shagun $\endgroup$ – Totem May 27 '15 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.