41
votes
Accepted
Data Science in C (or C++)
Or must I lose most of the efficiency gained by programming in C by calling on R scripts or other languages?
Do the opposite: learn C/C++ to write R extensions. Use C/C++ only for the performance-...
- 536
41
votes
Opening a 20GB file for analysis with pandas
If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The pandas.read_csv method allows ...
- 511
29
votes
Opening a 20GB file for analysis with pandas
There are two possibilities: either you need to have all your data in memory for processing (e.g. your machine learning algorithm would want to consume all of it at once), or you can do without it (e....
- 2,101
15
votes
Accepted
Difference between interpolate() and fillna() in pandas
fillna fills the NaN values with a given number with which you want to substitute. It gives you an option to fill according to ...
- 1,779
14
votes
Accepted
Improve the speed of t-sne implementation in python for huge data
You must look at this Multicore implementation of t-SNE.
I actually tried it and can vouch for its superior performance.
13
votes
Accepted
What is an 'old name' of data scientist?
In reverse chronological order: data miner, statistician, (applied) mathematician.
- 10.5k
13
votes
Accepted
How is H2O faster than R or SAS?
I have used R, SAS Base and H2O. First, I do not think that H2O seeks to be either R or SAS.
H2O provides data mining algorithms that are highly efficient. You can interface with H2O using several ...
- 1,403
12
votes
Accepted
How to deal with version control of large amounts of (binary) data
What I am ending up using is a sort of hybrid solution:
backup of the raw data
git of the workflow
manual snapshots of workflow + processed data, that are of relevance, e.g.:
standard preprocessing
...
- 246
12
votes
Data Science in C (or C++)
I agree that the current trend is to use Python/R and to bind it to some C/C++ extensions for computationally expensive tasks.
However, if you want to stay in C/C++, you might want to have a look at
...
- 5,478
12
votes
Which is faster: PostgreSQL vs MongoDB on large JSON datasets?
For data load, Postgre outperforms MongoDB.
MongoDB is almost always faster when returning query counts.
PostgreSQL is almost always faster for queries using indexes.
Check out this website and this ...
12
votes
Accepted
Machine Learning Best Practices for Big Dataset
I'll list some practices I've found useful, hope this helps:
Irrespective of whether the data is huge or not, cross validation is a must when building any model. If this takes more time than an end ...
- 2,427
12
votes
Opening a 20GB file for analysis with pandas
I just had this issue a few days ago! Not sure if this helps in your specific case since you aren't providing so many details, but my situation was to work offline on a 'large' dataset. The data was ...
- 221
11
votes
Accepted
Machine Learning on financial big data
There are tons of materials on financial (big) data analysis that you can read and peruse. I'm not an expert in finance, but am curious about the field, especially in the context of data science and R....
- 6,518
11
votes
Data Science in C (or C++)
As Andre Holzner has said, extending R with C/C++ extension is a very good way to take advantage of the best of both sides. Also you can try the inverse , working with C++ and ocasionally calling ...
- 211
11
votes
Accepted
Original Meaning of "Intelligence" in "Business Intelligence"
Howard Dresner, in 1989, is believed to have coined the term "business intelligence", to describe "concepts and methods to improve business decision making by using fact-based support systems.". When ...
- 1,447
11
votes
Can we take of benefit of using transfer learning while training a word2vec models?
Yes, you can take benefit of pre-trained models. Most famous one being the GoogleNewsData trained model which you can find here.
Pre-trained word and phrase vectors https://drive.google.com/file/d/...
- 196
10
votes
How to deal with version control of large amounts of (binary) data
Try looking at Git Large File Storage (LFS). It is new, but might be the thing worth looking at.
As I see, a discussion on Hacker News mentions a few other ways to deal with large files:
git-annex (...
- 746
10
votes
Accepted
Classifying transactions as malicious
This problem is popularly called the "Credit Card Fraud Detection"
There are several classification algorithms, which aim to tackle this problem.
With the knowledge of the dataset you ...
- 8,136
9
votes
How to deal with version control of large amounts of (binary) data
I have used Versioning on Amazon S3 buckets to manage 10-100GB in 10-100 files. Transfer can be slow, so it has helped to compress and transfer in parallel, or just run computations on EC2. The boto ...
- 236
9
votes
How to deal with version control of large amounts of (binary) data
This is a pretty common problem. I had this pain when I did research projects for a university and now - in industrial data science projects.
I've created and recently released an open source tool to ...
- 261
9
votes
Accepted
How to detect overfitting of a stock screener
Learning curves or bias-variance decomposition are the gold standard for detecting high variance, aka: overfitting. Separate your data (in your case the "back data") into 60% training data and 40% ...
- 6,778
8
votes
Is Python suitable for big data
Python has some very good tools for working with big data:
numpy
Numpy's memmory-mapped arrays let you access a file saved on disk as though it were an array. Only the parts of the array you are ...
- 226
8
votes
Improve the speed of t-sne implementation in python for huge data
Check out FFT-accelerated Interpolation-based t-SNE (paper, code, and Python package).
From the abstract:
We present Fast
Fourier Transform-accelerated Interpolation-based t-SNE (FIt-SNE),
...
- 180
8
votes
Improve the speed of t-sne implementation in python for huge data
Try UMAP.
It's significantly faster than t-SNE.
8
votes
How to deal with large training data?
I do something similar with keras and GPU training, where i also have only a small memory amount available. The idea would be split the numpy files into smaller ones, let's say 64 samples per file and ...
- 2,522
7
votes
Is the R language suitable for Big Data
R is great for a lot of analysis. As mentioned about, there are newer adaptations for big data like MapR, RHadoop, and scalable versions of RStudio.
However, if your concern is libraries, keep your ...
7
votes
How much of a background in programming is necessary to become a data scientist?
First of all, the fact that you have known some Java, even ten years ago, already means that you don't "know nothing about programming" (I suggest you update the title of your question to reflect that ...
- 6,518
7
votes
Accepted
What's an efficient way to compare and group millions of store names?
This is an entity resolution aka record linkage aka data matching problem.
I would solve this by removing all of the non-alphabetical characters including numbers, casting into all uppercase and then ...
- 6,778
7
votes
Accepted
Machine Learning in Spark
Probability can be found for the test dataset once you trained the model and transformed for the test dataset e.g: if your trained Naive Bayes model is model then ...
- 1,147
Only top scored, non community-wiki answers of a minimum length are eligible