Skip to main content
41 votes

Opening a 20GB file for analysis with pandas

If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The pandas.read_csv method allows ...
Olel Daniel's user avatar
29 votes

Opening a 20GB file for analysis with pandas

There are two possibilities: either you need to have all your data in memory for processing (e.g. your machine learning algorithm would want to consume all of it at once), or you can do without it (e....
KT.'s user avatar
  • 2,121
15 votes
Accepted

Difference between interpolate() and fillna() in pandas

fillna fills the NaN values with a given number with which you want to substitute. It gives you an option to fill according to ...
Kiritee Gak's user avatar
  • 1,799
14 votes
Accepted

Improve the speed of t-sne implementation in python for huge data

You must look at this Multicore implementation of t-SNE. I actually tried it and can vouch for its superior performance.
Nilav Baran Ghosh's user avatar
12 votes
Accepted

Machine Learning Best Practices for Big Dataset

I'll list some practices I've found useful, hope this helps: Irrespective of whether the data is huge or not, cross validation is a must when building any model. If this takes more time than an end ...
Sandeep S. Sandhu's user avatar
12 votes

Opening a 20GB file for analysis with pandas

I just had this issue a few days ago! Not sure if this helps in your specific case since you aren't providing so many details, but my situation was to work offline on a 'large' dataset. The data was ...
Marcus Jones's user avatar
9 votes

How to deal with version control of large amounts of (binary) data

This is a pretty common problem. I had this pain when I did research projects for a university and now - in industrial data science projects. I've created and recently released an open source tool to ...
Dmitry Petrov's user avatar
8 votes

Improve the speed of t-sne implementation in python for huge data

Check out FFT-accelerated Interpolation-based t-SNE (paper, code, and Python package). From the abstract: We present Fast Fourier Transform-accelerated Interpolation-based t-SNE (FIt-SNE), ...
The_Anomaly's user avatar
8 votes

Improve the speed of t-sne implementation in python for huge data

Try UMAP. It's significantly faster than t-SNE.
patel ashutosh's user avatar
8 votes

How to deal with large training data?

I do something similar with keras and GPU training, where i also have only a small memory amount available. The idea would be split the numpy files into smaller ones, let's say 64 samples per file and ...
hh32's user avatar
  • 2,762
7 votes
Accepted

Machine Learning in Spark

Probability can be found for the test dataset once you trained the model and transformed for the test dataset e.g: if your trained Naive Bayes model is model then ...
krishna Prasad's user avatar
7 votes
Accepted

Is Java or Python a better choice for an application involving data intensive algorithms employing natural language processing?

I work with python and java in big-data settings every day. python is definitely my ...
Uri Goren's user avatar
  • 438
6 votes

What is most likely the bare minimum knowledge one has to have to become data scientist?

Just a few thoughts which aren't covered in the link I pasted above ... Big data != data science. If you are a data scientist you may or may not be using big data tools. Your question wasn't clear ...
Marcus D's user avatar
  • 571
6 votes
Accepted

Instead of one-hot encoding, can I store the same information in one column using a single value?

I suppose you could do this, but if your goal is simply to store 15 boolean values in a single column you are complicating things unnecessarily. Instead of going to all the trouble to compute the ...
timleathart's user avatar
  • 3,940
6 votes

Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set?

If "variables" refers to training examples: You can use Stochastic Gradient Descent (SGD) where each iteration uses one training example. Or you could use Mini-Batch Gradient Descent where ...
Ben's user avatar
  • 2,572
5 votes

Machine Learning Best Practices for Big Dataset

The question is, how much data does it take to saturate your model? To determine this you can plot learning curves with varying amounts of data, perhaps scaling up/down in size by a constant factor. ...
Emre's user avatar
  • 10.5k
5 votes
Accepted

Understanding how distributed PCA works

The question is more related to Apache Spark architecture and map reduce; there are more than one questions here, however, the central piece of your question perhaps is For example, one of the means ...
Ironluca's user avatar
  • 187
5 votes

Opening a 20GB file for analysis with pandas

In my experience, initializing read_csv() with parameter low_memory=False tends to help when reading in large files. I don't ...
chainD's user avatar
  • 166
4 votes

How big is big data?

Data is "Big Data" if it is of such volume that it is less expensive to analyze it on two or more commodity computers, than on one high-end computer. This is essentially how Google's "BigFiles" file ...
Neil McGuigan's user avatar
4 votes

Simple Explanation of Apache Flume

What is Apache Flume? Apache Flume is a tool for designed for streaming data ingestion in HDFS. Objective: The main objective of Flume is to capture streaming data from various web servers to HDFS. ...
kavya's user avatar
  • 41
4 votes
Accepted

What is most likely the bare minimum knowledge one has to have to become data scientist?

From personal experience (so take into consider that I might not be representative although I'm probably not that far away too) the people that approached me with a job offer for Data Scientist ...
armatita's user avatar
  • 349
4 votes
Accepted

is there big difference between data Science , big Data and database?

Well, they are absolutely different things but that are somehow linked. I gonna go through each of them. Data Base Think of a data base (DB from now) like a computer which only purpose is to store ...
sergiOrtiz's user avatar
4 votes

Instead of one-hot encoding, can I store the same information in one column using a single value?

"Premature optimization is the root of all evil". --Knuth You could do this, but why? You definitely don't want to do this, if you're going to feed the result into a classifier: most classifiers ...
D.W.'s user avatar
  • 3,381
4 votes
Accepted

Avoid reloading DataFrame between different python kernels

If it's important for your use cases, you could try switching to Apache Zeppelin. As all Spark notebooks there share the same Spark context, same Python running environment. https://zeppelin.apache....
Tagar's user avatar
  • 198
4 votes
Accepted

Handling large imbalanced data set

There are multiple options, depending on your problem and the algorithms you want to use. The most promising (or closest to your original plan) is to use a generator to prepare batches of training ...
Jan van der Vegt's user avatar
4 votes
Accepted

Training a Convnet on 300GB data

You don't need to load the whole dataset into memory at once. The only data you need in memory are the samples in a single training batch. Use the fit_generator ...
David Marx's user avatar
  • 3,258
4 votes

SGDClassifier fit and partial_fit functions

fit(), always initializes the parameters like a new object, and trains the model with the dataset passed in fit() method. ...
vipin bansal's user avatar
  • 1,272
4 votes
Accepted

Euclidean vs. cosine similarity

On L2 normalized data it is an easy and good exercise to prove that they are equivalent. So you should try to solve the math yourself. Hint: use squared Euclidean. Note that it is common with tfidf ...
Has QUIT--Anony-Mousse's user avatar
4 votes

Why imbalanced data-set will bias the prediction model towards the more common class?

Let us say you have a skewed class (which is linearly inseparable, otherwise the prediction is bound to be correct always) like this: Image Credit: Learning from Imbalanced Classes Now you can ...
DuttaA's user avatar
  • 793

Only top scored, non community-wiki answers of a minimum length are eligible