92

To me (coming from a relational database background), "Big Data" is not primarily about the data size (which is the bulk of what the other answers are so far). "Big Data" and "Bad Data" are closely related. Relational Databases require 'pristine data'. If the data is in the database, it is accurate, clean, and 100% reliable. Relational Databases require "...


48

A model underfits when it is too simple with regards to the data it is trying to model. One way to detect such situation is to use the bias–variance approach, which can represented like this: Your model is underfitted when you have a high bias. To know whether you have a too high bias or a too high variance, you view the phenomenon in terms of training ...


34

As you rightly note, these days "big data" is something everyone wants to say they've got, which entails a certain looseness in how people define the term. Generally, though, I'd say you're certainly dealing with big data if the scale is such that it's no longer feasible to manage with more traditional technologies such as RDBMS, at least without ...


25

xgboost.train will ignore parameter n_estimators, while xgboost.XGBRegressor accepts. In xgboost.train, boosting iterations (i.e. n_estimators) is controlled by num_boost_round(default: 10) In your case, the first code will do 10 iterations (by default), but the second one will do 1000 iterations. There won't be any big difference if you try to change clf = ...


22

Total amount of data in the world: 2.8 zetabytes in 2012, estimated to reach 8 zetabytes by 2015 (source) and with a doubling time of 40 months. Can't get bigger than that :) As an example of a single large organization, Facebook pulls in 500 terabytes per day, into a 100 petabyte warehouse, and runs 70k queries per day on it as of 2012 (source) Their ...


14

In order to understand the variety of ways machine learning can be integrated into production applications, I think it is useful to look at open source projects and papers/blog posts from companies describing their infrastructure. The common theme that these systems have is the separation of model training from model application. In production systems, ...


13

To me Big Data is primarily about the tools (after all, that's where it started); a "big" dataset is one that's too big to be handled with conventional tools - in particular, big enough to demand storage and processing on a cluster rather than a single machine. This rules out a conventional RDBMS, and demands new techniques for processing; in particular, ...


13

MapReduce: A fault tolerant distributed computational framework. MapReduce allows you to operate over huge amounts of data- with a lot of work put in to prevent failure due to hardware. MapReduce is a poor choice for computing results on the fly because it is slow. (A typical MapReduce job takes on the order of minutes or hours, not microseconds) A ...


13

The kernel trick is based on some concepts: you have a dataset, e.g. two classes of 2D data, represented on a cartesian plane. It is not linearly separable, so for example a SVM could not find a line that separates the two classes. Now, what you can do it project this data into an higher dimension space, for example 3D, where it could be divided linearly by ...


12

This is a pretty massive question, so this is not intended to be a full answer, but hopefully this can help to inform general practice around determining the best tool for the job when it comes to data science. Generally, I have a relatively short list of qualifications I look for when it comes to any tool in this space. In no particular order they are: ...


11

Data becomes "big" when a single commodity computer can no longer handle the amount of data you have. It denotes the point at which you need to start thinking about building supercomputers or using clusters to process your data.


10

To answer your question it is important to understand the frame of reference you are looking for, if you are looking for what philosophically you are trying to achieve in model fitting, check out Rubens answer he does a good job of explaining that context. However, in practice your question is almost entirely defined by business objectives. To give a ...


10

I think you messed up some things in your question. Lucene (I know nothing about Lucene,NET, but I suppose is the same) is a library used to analyze, split in tokens, and store documents in order to be able to query and retrieve them later. Lucene has a pretty old but effective model, it uses inverted trees to find and retrieve documents. Without further ...


10

For windows, use the task scheduler to set the task to run for example daily at 4:00 AM It gives you many other options regarding frequency etc. http://en.wikipedia.org/wiki/Windows_Task_Scheduler


10

State of the art as in: used in practise or worked on in theory? APRIORI is used everywhere, except in developing new frequent itemset algorithms. It's easy to implement, and easy to reuse in very different domains. You'll find hundreds of APRIORI implementations of varying quality. And it's easy to get APRIORI wrong, actually. FPgrowth is much harder to ...


9

How can I ask my computer to run this every night at 4 am so that I have an up to date report waiting for me in the morning? You can set up a cronjob on a Linux system. These are run at the set time, if the computer is on. To do so, open a terminal and type: crontab -e And add: 00 4 * * * r source(/home/FilePath/.../myRscript.R) Source: Stack Overflow


9

This is kind of like asking about the tradeoffs between frying pan and your drawer of silverware. They are not two things you compare, really. You might use them together as part of a larger project. Hadoop itself is not one thing, but a name for a federation of services, like HDFS, Hive, HBase, MapReduce, etc. Storm is something you use with some of these ...


7

I don't think that everyone reaches for C/C++ when performance is an issue. The advantage to writing low-level code is using fewer CPU cycles, or sometimes, less memory. But I'd note that higher-level languages can call down to lower-level languages, and do, to get some of this value. Python and JVM languages can do this. The data scientist using, for ...


7

Models are but abstractions of what is seen in real life. They are designed in order to abstract-away nitty-gritties of the real system in observation, while keeping sufficient information to support desired analysis. If a model is overfit, it takes into account too many details about what is being observed, and small changes on such object may cause the ...


7

Big Data is defined by the volume of data, that's right, but not only. The particularity of big data is that you need to store a lots of various and sometimes unstructured stuffs all the times and from a tons of sensors, usually for years or decade. Furthermore you need something scalable, so that it doesn't take you half a year to find a data back. So ...


7

One of the most detailed and clear explanations of setting up a complex analytics pipeline is from the folks over at Twitch. They give detailed motivations of each of the architecture choices for collection, transportation, coordination, processing, storage, and querying their data. Compelling reading! Find it here and here.


7

Firstly, I would generally agree with everything that AirThomas suggested. Caching things is generally good if you can, but I find it slightly brittle since that's very dependent on exactly what your application is. Data compression is another very solid suggestion, but my impression on both of these is that the speedups you're looking at are going to be ...


6

I'll share what Big Data is like in genomics, in particular de-novo assembly. When we sequence your genome (eg: detect novel genes), we take billions of next-generation short reads. Look at the image below, where we try to assemble some reads. This looks simple? But what if you have billion of those reads? What if those reads contain sequence errors? What ...


6

Most of the efficient (and non trivial) statistic algorithms are iterative in nature so that the worst case analysis O() is irrelevant as the worst case is 'it fails to converge'. Nevertheless, when you have a lot of data, even the linear algorithms (O(n)) can be slow and you then need to focus on the constant 'hidden' behind the notation. For instance, ...


5

Simply, one common approach is to increase the complexity of the model, making it simple, and most probably underfitting at first, and increasing the complexity of the model until early signs of overfitting are witnessed using a resampling technique such as cross validation, bootstrap, etc. You increase the complexity either by adding parameters (number of ...


5

Spam filtering, especially in email, has been revolutionized by neural networks, here are a couple papers that provide good reading on the subject: On Neural Networks And The Future Of Spam A. C. Cosoi, M. S. Vlad, V. Sgarciu http://ceai.srait.ro/index.php/ceai/article/viewFile/18/8 Intelligent Word-Based Spam Filter Detection Using Multi-Neural ...


5

The best language depends on what you want to do. First remark: don't limit yourself to one language. Learning a new language is always a good thing, but at some point you will need to choose. Facilities offered by the language itself are an obvious thing to keep into account but in my opinion the following are more important: available libraries: do you ...


5

You mentioned regression and PCA in the title, and there is a definite answer for each of those. The asymptotic complexity of linear regression reduces to O(P^2 * N) if N > P, where P is the number of features and N is the number of observations. More detail in Computational complexity of least square regression operation. Vanilla PCA is O(P^2 * N + P ^ 3),...


5

I gave a very limited partial answer for the confirmatory factor analysis package that I developed for Stata in this Stata Journal article based on timing the actual simulations. Confirmatory factor analysis was implemented as a maximum likelihood estimation technique, and I could see very easily how the computation time grew with each dimension (sample size ...


Only top voted, non community-wiki answers of a minimum length are eligible