We created a social network application for eLearning purposes. It's an experimental project that we are researching on in our lab. It has been used in some case studies for a while and the data in our relational DBMS (SQL Server 2008) is getting big. It's a few gigabytes now and the tables are highly connected to each other. The performance is still fine, but when should we consider other options? Is it the matter of performance?
$\begingroup$
$\endgroup$
1
-
3$\begingroup$ For anything social networking, I would highly recommend a graph database like Neo4j or OrientDB $\endgroup$– ApolloCommented May 17, 2014 at 4:21
Add a comment
|
6 Answers
$\begingroup$
$\endgroup$
A few gigabytes is not very "big". It's more like the normal size of an enterprise DB. As long as you go over PK when joining tables it should work out really well, even in the future (as long as you don't get TB's of data a day).
Most professionals working in a big data environment consider > ~5TB as the beginning of the term big data. But even then it's not always the best way to just install the next best nosql database. You should always think about the task that you want to archive with the data (aggregate,read,search,mine,..) to find the best tools for you problem.
i.e. if you do alot of searches in you database it would probably be better to run a solr instance/cluster and denormalize your data from a DBMS like Postgres or your SQL Server from time to time and put it into solr instead of just moving the data from sql to nosql in term of persistence and performance.
$\begingroup$
$\endgroup$
To answer this question you have to answer which kind of compromise you can afford. RDBMs implements ACID. This is expensive in terms of resources. There are no NoSQL solutions which are ACID. See CAP theorem to dive deep into these ideas.
So you have to understand each compromise given by each solution and choose the one which is the most appropriate for your problem.
$\begingroup$
$\endgroup$
Big Data is actually not so about the "how big it is".
First, few gigabytes is not big at all, it's almost nothing. So don't bother yourself, your system will continu to work efficiently for some time I think.
Then you have to think of how do you use your data.
- SQL approach: Every data is precious, well collected and selected, and the focus is put on storing high valuable and well structured data. This can be costly, everything is interlink, and it's good for well stuctured system and functionnal data.
- Big Data approach: In big data you basically store almost everything, regardless of the value it has, and then do a active analytics process. Things are not linked, they are copied. For example let's say I have a blog entry. In Big Data there will not be a link to its author, but the author will be embedded inside the blog entry. Way more scalable, but require a different and more complex approach.
If your storing "functionnal" data use by your application, I will suggest you to remain on SQL. If your storing data in order to search on them later or to do reporting, and if this amount of data may increase quickly, I will suggest big data. In my opinion, big data is useful when you are dealing with real data that have to be collect and analyzed continuously.
$\begingroup$
$\endgroup$
I posted a pretty detailed answer on stackoverflow about when it is appropriate to use relational vs document (or NoSQL) database, here:
Motivations for using relational database / ORM or document database / ODM
Summary:
for small stuff, go with whatever tools you are familiar with
a few gigabytes is definitely small stuff: it doesn't get big until it is too big to fit in a single MySQL Cluster with a reasonable number of nodes (16-32), which means maybe 8-16TB data and a few million transactions per second (or a more conventional hard-drive-based database with up to 100's of TB data and a few thousand transactions per second).
if you're stuck with another database (not MySQL Cluster), get more mileage out of it by throwing in FusionIO hardware.
once you have data larger than a few TB and faster than thousands of transactions per second, it is a good time to look at moving to logical sharding in the application code first and then to NoSQL.
Cassandra :)
$\begingroup$
$\endgroup$
Is it the time to move to NoSQL will depends on 2 things:
- The nature/structure of your data
- Your current performance
SQL databases excel when the data is well structured (e.g. when it can be modeled as a table, an Excel spreadsheet, or a set of rows with a fixed number of columns). Also good when you need to do a lot of table joins (which it sounds like you do).
NoSQL databases excel when the data is un-structured beyond key-value pairs.
Performance wise, you gotta ask yourself one question: is your current SQL solution slow?
If not, go with the "IIABDFI" principle.
$\begingroup$
$\endgroup$
There are different ways for you to proceed.
1): Stay with SQL: If the performance stays ok and your data is structured, there is no need to change. Also, remember that if you change to NoSQL, you are losing the ACID properties meaning that your data is not consistent anymore.
Change to NoSQL: If your performance is getting worse, you should try to scale your database horizontally. That is not possible with SQL databases but with NoSQL or NewSQL databases it is. If your data is not structured then NoSQL is your solution.
Change to NewSQL: If performance is an issue, but you cannot compromise on ACID and have structured, relational data, then NewSQL is the perfect fit. It offers horizontal scalability combined with ACID and a relational schema.
