Hot answers tagged

90

To me (coming from a relational database background), "Big Data" is not primarily about the data size (which is the bulk of what the other answers are so far). "Big Data" and "Bad Data" are closely related. Relational Databases require 'pristine data'. If the data is in the database, it is accurate, clean, and 100% reliable. Relational Databases require "...


46

Different people use different tools for different things. Terms like Data Science are generic for a reason. A data scientist could spend an entire career without having to learn a particular tool like hadoop. Hadoop is widely used, but it is not the only platform that is capable of managing and manipulating data, even large scale data. I would say that ...


42

First of all, dimensionality reduction is used when you have many covariated dimensions and want to reduce problem size by rotating data points into new orthogonal basis and taking only axes with largest variance. With 8 variables (columns) your space is already low-dimensional, reducing number of variables further is unlikely to solve technical issues with ...


41

Actually this is coming around. In the book R in a Nutshell there is even a section on using R with Hadoop for big data processing. There are some work arounds that need to be done because R does all it's work in memory, so you are basically limited to the amount of RAM you have available to you. A mature project for R and Hadoop is RHadoop RHadoop has ...


36

If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The pandas.read_csv method allows you to read a file in chunks like this: import pandas as pd for chunk in pd.read_csv(<filepath>, chunksize=<your_chunksize_here>) do_processing() train_algorithm() Here is ...


35

Or must I loose most of the efficiency gained by programming in C by calling on R scripts or other languages? Do the opposite: learn C/C++ to write R extensions. Use C/C++ only for the performance critical sections of your new algorithms, use R to build your analysis, import data, make plots etc. If you want to go beyond R, I'd recommend learning python. ...


34

As you rightly note, these days "big data" is something everyone wants to say they've got, which entails a certain looseness in how people define the term. Generally, though, I'd say you're certainly dealing with big data if the scale is such that it's no longer feasible to manage with more traditional technologies such as RDBMS, at least without ...


31

The main problem with using R for large data sets is the RAM constraint. The reason behind keeping all the data in RAM is that it provides much faster access and data manipulations than would storing on HDDs. If you are willing to take a hit on performance, then yes, it is quite practical to work with large datasets in R. RODBC Package: Allows connecting to ...


28

I would try to analyze and solve one or more of the problems published on Kaggle Competitions (https://www.kaggle.com/competitions). Note that the competitions are grouped by their expected complexity, from 101 (bottom of the list) to Research and Featured (top of the list). A color-coded vertical band is a visual guideline for grouping. You can assess time ...


23

To be perfectly honest, most NoSQL databases are not very well suited to applications in big data. For the vast majority of all big data applications, the performance of MongoDB compared to a relational database like MySQL is significantly is poor enough to warrant staying away from something like MongoDB entirely. With that said, there are a couple of ...


23

There are two possibilities: either you need to have all your data in memory for processing (e.g. your machine learning algorithm would want to consume all of it at once), or you can do without it (e.g. your algorithm only needs samples of rows or columns at once). In the first case, you'll need to solve a memory problem. Increase your memory size, rent a ...


22

Total amount of data in the world: 2.8 zetabytes in 2012, estimated to reach 8 zetabytes by 2015 (source) and with a doubling time of 40 months. Can't get bigger than that :) As an example of a single large organization, Facebook pulls in 500 terabytes per day, into a 100 petabyte warehouse, and runs 70k queries per day on it as of 2012 (source) Their ...


18

To clarify, I feel like the original question references by OP probably isn't be best for a SO-type format, but I will certainly represent python in this particular case. Let me just start by saying that regardless of your data size, python shouldn't be your limiting factor. In fact, there are just a couple main issues that you're going to run into dealing ...


17

Some good answers here. I would like to join the discussion by adding the following three notes: The question's emphasis on the volume of data while referring to Big Data is certainly understandable and valid, especially considering the problem of data volume growth outpacing technological capacities' exponential growth per Moore's Law (http://en.wikipedia....


14

News outlets tend to use "Big Data" pretty loosely. Vendors usually provide case studies surrounding their specific products. There aren't a lot out there for open source implementations, but they do get mentioned. For instance, Apache isn't going to spend a lot of time building a case study on hadoop, but vendors like Cloudera and Hortonworks probably ...


14

In order to understand the variety of ways machine learning can be integrated into production applications, I think it is useful to look at open source projects and papers/blog posts from companies describing their infrastructure. The common theme that these systems have is the separation of model training from model application. In production systems, ...


13

Well, I'm not sure if it is MapReduce that solves the problem, but it surely wouldn't be MapReduce alone to solve all these questions you raised. But here are important things to take into account, and that make it feasible to have such low latency on queries from all these TBs of data in different machines: distributed computing: by being distributed does ...


13

To me Big Data is primarily about the tools (after all, that's where it started); a "big" dataset is one that's too big to be handled with conventional tools - in particular, big enough to demand storage and processing on a cluster rather than a single machine. This rules out a conventional RDBMS, and demands new techniques for processing; in particular, ...


13

MapReduce: A fault tolerant distributed computational framework. MapReduce allows you to operate over huge amounts of data- with a lot of work put in to prevent failure due to hardware. MapReduce is a poor choice for computing results on the fly because it is slow. (A typical MapReduce job takes on the order of minutes or hours, not microseconds) A ...


13

There are couple of things you need to understand when dealing with Big data - What is Big data? You might be aware of famous V's of Big data - Volume, Velocity, Variety... So, Python may not be suitable for all. And it goes with all data science tools available. You need to know which tool is good for what purpose. If dealing with large Volume of data: ...


13

In reverse chronological order: data miner, statistician, (applied) mathematician.


13

I have used R, SAS Base and H2O. First, I do not think that H2O seeks to be either R or SAS. H2O provides data mining algorithms that are highly efficient. You can interface with H2O using several APIs such as their R API. The benefit of combining R and H2O is that H2O is very good at exploiting multi-cores or clusters with minimal effort of the user. It ...


13

You must look at this Multicore implementation of t-SNE. I actually tried it and can vouch for its superior performance.


12

R is great for "big data"! However, you need a workflow since R is limited (with some simplification) by the amount of RAM in the operating system. The approach I take is to interact with a relational database (see the RSQLite package for creating and interacting with a SQLite databse), run SQL-style queries to understand the structure of the data, and then ...


12

The one thing that you can say for sure is: Nobody can say this for sure. And it might indeed be opinion-based to some extent. The introduction of terms like "Big Data" that some people consider as "hypes" or "buzzwords" don't make it easier to flesh out an appropriate answer here. But I'll try. In general, interdisciplinary fields often seem to have the ...


12

What I am ending up using is a sort of hybrid solution: backup of the raw data git of the workflow manual snapshots of workflow + processed data, that are of relevance, e.g.: standard preprocessing really time-consuming for publication I believe it is seldom sensible to have a full revision history of large amount of binary data, because the time required ...


12

fillna fills the NaN values with a given number with which you want to substitute. It gives you an option to fill according to the index of rows of a pd.DataFrame or on the name of the columns in the form of a python dict. But interpolate is a god in filling. It gives you the flexibility to fill the missing values with many kinds of interpolations between ...


11

Note that there is an early version of LIBLINEAR ported to Apache Spark. See mailing list comments for some early details, and the project site.


11

There are tons of materials on financial (big) data analysis that you can read and peruse. I'm not an expert in finance, but am curious about the field, especially in the context of data science and R. Therefore, the following are selected relevant resource suggestions that I have for you. I hope that they will be useful. Books: Financial analysis (general /...


11

Howard Dresner, in 1989, is believed to have coined the term "business intelligence", to describe "concepts and methods to improve business decision making by using fact-based support systems.". When he was at Gartner Group. This is a common mantra, spread over the Web. I have not been able to trace the exact source for this origin yet. Many insist on he was ...


Only top voted, non community-wiki answers of a minimum length are eligible