2
$\begingroup$

I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured data) as opposed to OLAP databases (Netezza)

$\endgroup$
4
$\begingroup$

'SQL' on Hadoop is very much a thing, though I use quotes since it's probably more accurate to say it's SQL-like. Some options for bringing SQL-like capabilities to Hadoop include Hue, Hive/bee (Heading towards Stinger? So punny Apache), Impala, SparkSQL (probably not a great solution for a bank given the possibility of concurrency issues), among others (Seems like everyone has their own version of it these days)

To be honest though, if you're asking if it could be helpful, you probably don't need Hadoop (sorry in advance of that comes off harshly, it's not intended to). Lots an lots of places think they need Hadoop, but very few actually need Hadoop. There are of business that are down on the tech because they transitioned to it when the need didn't really exist. If you truly do need Hadoop or another distributed system it'd be almost impossible to determine which setup would be beneficial to your org without an intimate understanding of your data, and your specific business means.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Ok, so the data needs to be moved from sql db to HDFS before Hadoop can work its magic? $\endgroup$ – Victor Aug 28 '15 at 13:54
  • 1
    $\begingroup$ Yes, if you were to switch to Hadoop you would be moving to a completely new architecture, with a SQL-like interface/wrapper running on top of Hadoop. It would not be an insignificant undertaking. Given the importance of your data it would be incredibly wise to bring in a vendor it consultant who specializes in creating new architectures from the ground up. Without intimate access to you current database structure, the data contained, and a look at the processes currently running on your system anything that me or anyone else could tell you online is merely speculation/best guess. $\endgroup$ – scribbles Aug 28 '15 at 14:05
1
$\begingroup$

I think you are referring to the HDFS part of Hadoop. If I'm correct, using Hadoop IMHO should not be considered as an alternative, but as a necessity. Hadoop answers the question : how can I take advantage advantage of the massive amount of data I have since I can't use it now as it is.

So yes, HDFS is a distributed file system (that's why Hadoop DFS), and if you are not experiencing the limits of your database tables for now (or planning to in the future), then there is no reason to consider switching. In addition I think that depending on the data, you might not be able to do whatever you want with a bank database.

I should add that in the big data ecosystem there are alternative to databases (NoSQL system for example), but my answer is entirely conditioned on the fact that you are not mentioning any specific task/goal so my answer is basically :

_ I have a computer which is sufficient for my work today and in the future, do I need to change it ?

_ If you believe it is sufficient for your future use, then no.

| improve this answer | |
$\endgroup$
1
$\begingroup$

It depends on what your specific situation is - I think you need to add more detail for you to get a good answer on this. e.g. what problems are you experiencing with data storage currently? What are you trying to change or achieve in the future?

Your data (you don't mention what exactly the data is, but I assume this is bank transactions, customer info etc?) will be in a traditional relational database management system (RDBMS), probably SQL based, for a very good reason - traditional RDBMSs are extremely good at transactional processing, i.e. being able to add new records, update and delete existing records, extremely quickly and without 'messing up' the data - e.g. if two updates on the same customer records occur almost simultaneously, the database system will order the updates correctly, 'lock' the relevant records to prevent conflicts, etc. so that you are never left with inconsistent data. That would be EXTREMELY bad for a bank - imagine if someone's bank account accidentally got overwritten with the wrong balance! This is generally referred to as OLTP (online transaction processing).

For this reason I imagine most banks will continue to use these systems for many many years into the future, which are extremely mature, well tested, well understood, and there are tons of people to hire with the necessary skills. Also, to change the underlying storage system for a banking system would be extremely risky, hence why most banks run very old software - the risk of something going wrong by updating software is just too big.

However, if you are talking about just wanting to produce analytical reports on this data, and keep the original source data as-is, then yes Hadoop may well be a useful system to investigate. The general idea will be to export your raw data from your banking system into HDFS for example, and use Hadoop to produce analytical reports. the benefit of this kind of system is that it may be much faster (and safer) to create a query such as 'what is the average bank balance of all my customers', without locking-up your source database system while the query runs (again, that would be really, really bad for a bank!). Hadoop and similar systems excel at this kind of task, generally called OLAP (offline analytical processing), but are no good at all for OLTP.

Sounds like you need to really think about what you're trying to do here and do lots of research into the differences in OLAP and OLTP as a starter, and go from there. Hope that helps.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.