Hot answers tagged

92 votes
Accepted

How big is big data?

To me (coming from a relational database background), "Big Data" is not primarily about the data size (which is the bulk of what the other answers are so far). "Big Data" and "Bad Data" are closely ...
user avatar
  • 1,102
47 votes
Accepted

Do I need to learn Hadoop to be a Data Scientist?

Different people use different tools for different things. Terms like Data Science are generic for a reason. A data scientist could spend an entire career without having to learn a particular tool ...
user avatar
45 votes
Accepted

How to do SVD and PCA with big data?

First of all, dimensionality reduction is used when you have many covariated dimensions and want to reduce problem size by rotating data points into new orthogonal basis and taking only axes with ...
user avatar
  • 2,771
42 votes
Accepted

Is the R language suitable for Big Data

Actually this is coming around. In the book R in a Nutshell there is even a section on using R with Hadoop for big data processing. There are some work arounds that need to be done because R does all ...
user avatar
41 votes
Accepted

Data Science in C (or C++)

Or must I loose most of the efficiency gained by programming in C by calling on R scripts or other languages? Do the opposite: learn C/C++ to write R extensions. Use C/C++ only for the performance ...
user avatar
40 votes

Opening a 20GB file for analysis with pandas

If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The pandas.read_csv method allows ...
user avatar
34 votes

How big is big data?

As you rightly note, these days "big data" is something everyone wants to say they've got, which entails a certain looseness in how people define the term. Generally, though, I'd say you're certainly ...
user avatar
  • 3,030
31 votes

Is the R language suitable for Big Data

The main problem with using R for large data sets is the RAM constraint. The reason behind keeping all the data in RAM is that it provides much faster access and data manipulations than would storing ...
user avatar
  • 654
29 votes
Accepted

Data Science Project Ideas

I would try to analyze and solve one or more of the problems published on Kaggle Competitions. Note that the competitions are grouped by their expected complexity, from ...
user avatar
28 votes

Opening a 20GB file for analysis with pandas

There are two possibilities: either you need to have all your data in memory for processing (e.g. your machine learning algorithm would want to consume all of it at once), or you can do without it (e....
user avatar
  • 2,101
26 votes
Accepted

Uses of NoSQL database in data science

To be perfectly honest, most NoSQL databases are not very well suited to applications in big data. For the vast majority of all big data applications, the performance of MongoDB compared to a ...
user avatar
  • 4,179
22 votes

How big is big data?

Total amount of data in the world: 2.8 zetabytes in 2012, estimated to reach 8 zetabytes by 2015 (source) and with a doubling time of 40 months. Can't get bigger than that :) As an example of a ...
user avatar
  • 3,062
18 votes
Accepted

Is Python suitable for big data

To clarify, I feel like the original question references by OP probably isn't be best for a SO-type format, but I will certainly represent python in this particular ...
user avatar
  • 4,179
17 votes

Is the R language suitable for Big Data

Some good answers here. I would like to join the discussion by adding the following three notes: The question's emphasis on the volume of data while referring to Big Data is certainly understandable ...
user avatar
15 votes

Big data case study or use case example

News outlets tend to use "Big Data" pretty loosely. Vendors usually provide case studies surrounding their specific products. There aren't a lot out there for open source implementations, but they ...
user avatar
15 votes
Accepted

Looking for example infrastructure stacks/workflows/pipelines

In order to understand the variety of ways machine learning can be integrated into production applications, I think it is useful to look at open source projects and papers/blog posts from companies ...
user avatar
  • 421
15 votes
Accepted

Difference between interpolate() and fillna() in pandas

fillna fills the NaN values with a given number with which you want to substitute. It gives you an option to fill according to ...
user avatar
  • 1,769
14 votes

Is Python suitable for big data

There are couple of things you need to understand when dealing with Big data - What is Big data? You might be aware of famous V's of Big data - Volume, Velocity, Variety... So, Python may not be ...
user avatar
  • 406
14 votes
Accepted

Improve the speed of t-sne implementation in python for huge data

You must look at this Multicore implementation of t-SNE. I actually tried it and can vouch for its superior performance.
user avatar
13 votes
Accepted

How does a query into a huge database return with negligible latency?

Well, I'm not sure if it is MapReduce that solves the problem, but it surely wouldn't be MapReduce alone to solve all these questions you raised. But here are important things to take into account, ...
user avatar
  • 4,027
13 votes

How big is big data?

To me Big Data is primarily about the tools (after all, that's where it started); a "big" dataset is one that's too big to be handled with conventional tools - in particular, big enough to demand ...
user avatar
  • 231
13 votes
Accepted

Tradeoffs between Storm and Hadoop (MapReduce)

MapReduce: A fault tolerant distributed computational framework. MapReduce allows you to operate over huge amounts of data- with a lot of work put in to prevent failure due to hardware. MapReduce is a ...
user avatar
  • 421
13 votes
Accepted

What is an 'old name' of data scientist?

In reverse chronological order: data miner, statistician, (applied) mathematician.
user avatar
  • 10.4k
13 votes
Accepted

How is H2O faster than R or SAS?

I have used R, SAS Base and H2O. First, I do not think that H2O seeks to be either R or SAS. H2O provides data mining algorithms that are highly efficient. You can interface with H2O using several ...
user avatar
  • 1,383
12 votes

Is the R language suitable for Big Data

R is great for "big data"! However, you need a workflow since R is limited (with some simplification) by the amount of RAM in the operating system. The approach I take is to interact with a relational ...
user avatar
  • 325
12 votes
Accepted

Is Data Science just a trend or is a long term concept?

The one thing that you can say for sure is: Nobody can say this for sure. And it might indeed be opinion-based to some extent. The introduction of terms like "Big Data" that some people consider as "...
user avatar
  • 400
12 votes
Accepted

How to deal with version control of large amounts of (binary) data

What I am ending up using is a sort of hybrid solution: backup of the raw data git of the workflow manual snapshots of workflow + processed data, that are of relevance, e.g.: standard preprocessing ...
user avatar
  • 246
12 votes

Data Science in C (or C++)

I agree that the current trend is to use Python/R and to bind it to some C/C++ extensions for computationally expensive tasks. However, if you want to stay in C/C++, you might want to have a look at ...
user avatar
12 votes
Accepted

Machine Learning Best Practices for Big Dataset

I'll list some practices I've found useful, hope this helps: Irrespective of whether the data is huge or not, cross validation is a must when building any model. If this takes more time than an end ...
user avatar
11 votes

How big is big data?

Data becomes "big" when a single commodity computer can no longer handle the amount of data you have. It denotes the point at which you need to start thinking about building supercomputers or using ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible