Hot answers tagged

41 votes

Opening a 20GB file for analysis with pandas

If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The pandas.read_csv method allows ...
Olel Daniel's user avatar
29 votes

Opening a 20GB file for analysis with pandas

There are two possibilities: either you need to have all your data in memory for processing (e.g. your machine learning algorithm would want to consume all of it at once), or you can do without it (e....
KT.'s user avatar
  • 2,121
15 votes
Accepted

Difference between interpolate() and fillna() in pandas

fillna fills the NaN values with a given number with which you want to substitute. It gives you an option to fill according to ...
Kiritee Gak's user avatar
  • 1,789
14 votes
Accepted

Improve the speed of t-sne implementation in python for huge data

You must look at this Multicore implementation of t-SNE. I actually tried it and can vouch for its superior performance.
Nilav Baran Ghosh's user avatar
12 votes
Accepted

Machine Learning Best Practices for Big Dataset

I'll list some practices I've found useful, hope this helps: Irrespective of whether the data is huge or not, cross validation is a must when building any model. If this takes more time than an end ...
Sandeep S. Sandhu's user avatar
12 votes

Opening a 20GB file for analysis with pandas

I just had this issue a few days ago! Not sure if this helps in your specific case since you aren't providing so many details, but my situation was to work offline on a 'large' dataset. The data was ...
Marcus Jones's user avatar
11 votes

Can we take of benefit of using transfer learning while training a word2vec models?

Yes, you can take benefit of pre-trained models. Most famous one being the GoogleNewsData trained model which you can find here. Pre-trained word and phrase vectors https://drive.google.com/file/d/...
Guru's user avatar
  • 316
9 votes

How to deal with version control of large amounts of (binary) data

This is a pretty common problem. I had this pain when I did research projects for a university and now - in industrial data science projects. I've created and recently released an open source tool to ...
Dmitry Petrov's user avatar
8 votes

Improve the speed of t-sne implementation in python for huge data

Check out FFT-accelerated Interpolation-based t-SNE (paper, code, and Python package). From the abstract: We present Fast Fourier Transform-accelerated Interpolation-based t-SNE (FIt-SNE), ...
The_Anomaly's user avatar
8 votes

Improve the speed of t-sne implementation in python for huge data

Try UMAP. It's significantly faster than t-SNE.
patel ashutosh's user avatar
8 votes

How to deal with large training data?

I do something similar with keras and GPU training, where i also have only a small memory amount available. The idea would be split the numpy files into smaller ones, let's say 64 samples per file and ...
hh32's user avatar
  • 2,722
7 votes
Accepted

Machine Learning in Spark

Probability can be found for the test dataset once you trained the model and transformed for the test dataset e.g: if your trained Naive Bayes model is model then ...
krishna Prasad's user avatar
7 votes
Accepted

Is Java or Python a better choice for an application involving data intensive algorithms employing natural language processing?

I work with python and java in big-data settings every day. python is definitely my ...
Uri Goren's user avatar
  • 438
6 votes

What is the best Big-Data framework for stream processing?

You might also want to take a look at Apache Flink. It is a new contender to Apache Spark as it can do both, realtime and batch analysis. It also claims to be faster than Apache Spark, but I think ...
chAlexey's user avatar
  • 161
6 votes

Improve the speed of t-sne implementation in python for huge data

Since, there are no answers in SO, I have asked myself in github page and the issue has been closed by stating the following reply by GaelVaroquaux.. If you only want to parallelise vector ...
chmodsss's user avatar
  • 1,964
6 votes

What is most likely the bare minimum knowledge one has to have to become data scientist?

Just a few thoughts which aren't covered in the link I pasted above ... Big data != data science. If you are a data scientist you may or may not be using big data tools. Your question wasn't clear ...
Marcus D's user avatar
  • 571
6 votes
Accepted

Instead of one-hot encoding, can I store the same information in one column using a single value?

I suppose you could do this, but if your goal is simply to store 15 boolean values in a single column you are complicating things unnecessarily. Instead of going to all the trouble to compute the ...
timleathart's user avatar
  • 3,930
6 votes

Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set?

If "variables" refers to training examples: You can use Stochastic Gradient Descent (SGD) where each iteration uses one training example. Or you could use Mini-Batch Gradient Descent where ...
Ben's user avatar
  • 2,562
5 votes

Simple Explanation of Apache Kafka

Can anybody explain Apache Kafka for me in a plain language? As the official documentation of Kafka says: Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. It ...
Dawny33's user avatar
  • 8,266
5 votes

Simple Explanation of Apache Flume

What is it used for? Data ingestion into a distributed datastore (e.g. HDFS). See image (I did not make the image and am only including the image for visual aid). There are other tools that will help ...
Kevin Vasko's user avatar
5 votes

Machine Learning Best Practices for Big Dataset

The question is, how much data does it take to saturate your model? To determine this you can plot learning curves with varying amounts of data, perhaps scaling up/down in size by a constant factor. ...
Emre's user avatar
  • 10.5k
5 votes
Accepted

Understanding how distributed PCA works

The question is more related to Apache Spark architecture and map reduce; there are more than one questions here, however, the central piece of your question perhaps is For example, one of the means ...
Ironluca's user avatar
  • 187
5 votes

Opening a 20GB file for analysis with pandas

In my experience, initializing read_csv() with parameter low_memory=False tends to help when reading in large files. I don't ...
chainD's user avatar
  • 166
4 votes

How big is big data?

Data is "Big Data" if it is of such volume that it is less expensive to analyze it on two or more commodity computers, than on one high-end computer. This is essentially how Google's "BigFiles" file ...
Neil McGuigan's user avatar
4 votes

A Simple Explanation of ZooKeeper in Hadoop

I noticed you are asking several questions related to the Hadoop ecosystem. It is hard to judge how much you know while explaining the things you are asking in a very simplistic manner without knowing ...
Kevin Vasko's user avatar
4 votes

Simple Explanation of Apache Flume

What is Apache Flume? Apache Flume is a tool for designed for streaming data ingestion in HDFS. Objective: The main objective of Flume is to capture streaming data from various web servers to HDFS. ...
kavya's user avatar
  • 41
4 votes
Accepted

What is most likely the bare minimum knowledge one has to have to become data scientist?

From personal experience (so take into consider that I might not be representative although I'm probably not that far away too) the people that approached me with a job offer for Data Scientist ...
armatita's user avatar
  • 349
4 votes
Accepted

is there big difference between data Science , big Data and database?

Well, they are absolutely different things but that are somehow linked. I gonna go through each of them. Data Base Think of a data base (DB from now) like a computer which only purpose is to store ...
sergiOrtiz's user avatar
4 votes

Instead of one-hot encoding, can I store the same information in one column using a single value?

"Premature optimization is the root of all evil". --Knuth You could do this, but why? You definitely don't want to do this, if you're going to feed the result into a classifier: most classifiers ...
D.W.'s user avatar
  • 3,341

Only top scored, non community-wiki answers of a minimum length are eligible