92

To me (coming from a relational database background), "Big Data" is not primarily about the data size (which is the bulk of what the other answers are so far). "Big Data" and "Bad Data" are closely related. Relational Databases require 'pristine data'. If the data is in the database, it is accurate, clean, and 100% reliable. Relational Databases require "...


34

As you rightly note, these days "big data" is something everyone wants to say they've got, which entails a certain looseness in how people define the term. Generally, though, I'd say you're certainly dealing with big data if the scale is such that it's no longer feasible to manage with more traditional technologies such as RDBMS, at least without ...


22

Total amount of data in the world: 2.8 zetabytes in 2012, estimated to reach 8 zetabytes by 2015 (source) and with a doubling time of 40 months. Can't get bigger than that :) As an example of a single large organization, Facebook pulls in 500 terabytes per day, into a 100 petabyte warehouse, and runs 70k queries per day on it as of 2012 (source) Their ...


14

In order to understand the variety of ways machine learning can be integrated into production applications, I think it is useful to look at open source projects and papers/blog posts from companies describing their infrastructure. The common theme that these systems have is the separation of model training from model application. In production systems, ...


13

To me Big Data is primarily about the tools (after all, that's where it started); a "big" dataset is one that's too big to be handled with conventional tools - in particular, big enough to demand storage and processing on a cluster rather than a single machine. This rules out a conventional RDBMS, and demands new techniques for processing; in particular, ...


11

Data becomes "big" when a single commodity computer can no longer handle the amount of data you have. It denotes the point at which you need to start thinking about building supercomputers or using clusters to process your data.


11

Re: size of data The short answer Scala works for both small and large data, but its creation and development is motivated by needing something scalable. Scala is an acronym for “Scalable Language”. The long answer Scala is a functional programming language that runs on the jvm. The 'functional' part of this is a fundamental difference in the language ...


7

Big Data is defined by the volume of data, that's right, but not only. The particularity of big data is that you need to store a lots of various and sometimes unstructured stuffs all the times and from a tons of sensors, usually for years or decade. Furthermore you need something scalable, so that it doesn't take you half a year to find a data back. So ...


7

One of the most detailed and clear explanations of setting up a complex analytics pipeline is from the folks over at Twitch. They give detailed motivations of each of the architecture choices for collection, transportation, coordination, processing, storage, and querying their data. Compelling reading! Find it here and here.


7

Neo4j and Spark GraphX are meant for solving problem at different level and they are complimentary to each other. They can be connected by Neo4j's Mazerunner extension: Mazerunner is a Neo4j unmanaged extension and distributed graph processing platform that extends Neo4j to do big data graph processing jobs while persisting the results back to Neo4j. ...


6

Most of the efficient (and non trivial) statistic algorithms are iterative in nature so that the worst case analysis O() is irrelevant as the worst case is 'it fails to converge'. Nevertheless, when you have a lot of data, even the linear algorithms (O(n)) can be slow and you then need to focus on the constant 'hidden' behind the notation. For instance, ...


6

I'll share what Big Data is like in genomics, in particular de-novo assembly. When we sequence your genome (eg: detect novel genes), we take billions of next-generation short reads. Look at the image below, where we try to assemble some reads. This looks simple? But what if you have billion of those reads? What if those reads contain sequence errors? What ...


5

You can use map reduce algorithms in Hadoop without programming them in Java. It is called streaming and works like Linux piping. If you believe that you can port your functions to read and write to terminal, it should work nicely. Here is example blog post which shows how to use map reduce functions written in Python in Hadoop.


5

I gave a very limited partial answer for the confirmatory factor analysis package that I developed for Stata in this Stata Journal article based on timing the actual simulations. Confirmatory factor analysis was implemented as a maximum likelihood estimation technique, and I could see very easily how the computation time grew with each dimension (sample size ...


5

You mentioned regression and PCA in the title, and there is a definite answer for each of those. The asymptotic complexity of linear regression reduces to O(P^2 * N) if N > P, where P is the number of features and N is the number of observations. More detail in Computational complexity of least square regression operation. Vanilla PCA is O(P^2 * N + P ^ 3),...


5

ScalaNLP is a suite of machine learning and numerical computing libraries with support for common natural language processing tasks. http://www.scalanlp.org/ Here is a newly updated list of scala libraries for data science: https://www.datasciencecentral.com/profiles/blogs/top-15-scala-libraries-for-data-science-in-2018-1


4

I think that big data starts at the point where the size prevents you from doing what you want to. In most scenarios, there is a limit on the running time that is considered feasible. In some cases it is an hour, in some cases it might be few weeks. As long as the data is not big enough that only O(n) algorithms can run in the feasible time frame, you didn't ...


4

There will definitely be a translation task at the end if you prototype using just mongo. When you run a MapReduce task on mongodb, it has the data source and structure built in. When you eventually convert to hadoop, your data structures might not look the same. You could leverage the mongodb-hadoop connector to access mongo data directly from within ...


4

You also can create a MongoDB-Hadoop connection.


4

Data is "Big Data" if it is of such volume that it is less expensive to analyze it on two or more commodity computers, than on one high-end computer. This is essentially how Google's "BigFiles" file system originated. Page and Brin could not afford a fancy Sun server to store and search their web index, so hooked up several commodity computers


4

Can't you create a hash for each classes, and then merge rows by rows, field by field only the classes where the hash changed ? It should be faster if most of the classes don't change.. Or a hash of each rows or maybe columns.. depending on how the data normally change..


4

From listening to presentations by Martin Odersky, the creator of Scala, it is especially well suited for building highly scalable systems by leveraging functional programming constructs in conjuction with object orientation and flelxible syntax. It is also useful for development of small systems and rapid prototyping because it takes less lines of code than ...


3

This is a very good question and a common situation. In my opinion there are three different factors that must be controlled: Data: There exist already different benchmarks in order to evaluate algorithms and architectures. These data must be publicly available so that everybody can contrast their approaches. Architecture: My suggestion is to test ...


3

Sounds interesting. Could the solution be to dump the data out, build a fast custom processing thingie to run it through and then import it back to the database? I've seen some blazing fast Java-based text processing tools for topic modeling that handle millions of lines of text per second. If it's an option then you can build a shell script to first dump ...


3

I suggest the use of Hidden Markov Models, with two possible states: (1) high levels and (0) low levels. This technique might be helpful to decode your signal. Probably you would need a specific HMM for each codification. If noise is an issue an FIR filter with a Blackman-Harris window function would allow you to isolate the frequency you're concerned ...


3

Airbnb and Etsy both recently posted detailed information about their workflows.


3

Is there a specific reason beside the fact that you would like to contribute? I am asking because there is always pyspark that you can use, the Spark python API that exposes the Spark programming model to Python. For deep learning specifically, there are a lot of frameworks built on top of Theano -which is a python library for mathematical expressions ...


2

The following general answer is my uneducated guess, so take it with grain of salt. Hopefully, it makes sense. I think that the best way to describe or analyze experiments (as any other systems, in general) is to build their statistical (multivariate) models and evaluate them. Depending on whether environments for your set of experiments are represented by ...


Only top voted, non community-wiki answers of a minimum length are eligible