AN6U5
  • Member for 6 years, 8 months
  • Last seen more than a week ago
When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?
Accepted answer
198 votes

There are some cases where LabelEncoder or DictVectorizor are useful, but these are quite limited in my opinion due to ordinality. LabelEncoder can turn [dog,cat,dog,mouse,cat] into [1,2,1,3,2], but ...

View answer
What is a good way to transform Cyclic Ordinal attributes?
Accepted answer
42 votes

The most logical way to transform hour is into two variables that swing back and forth out of sync. Imagine the position of the end of the hour hand of a 24-hour clock. The x position swings back and ...

View answer
Time series prediction using ARIMA vs LSTM
31 votes

Statement 1 is correct, statement 2 is correct, but requires elaboration, and statement 3 is incorrect for seasonal ARIMA: The following might point you in the right direction but hopefully you'll ...

View answer
Calculation and Visualization of Correlation Matrix with Pandas
Accepted answer
28 votes

I suggest some sort of play on the following: Using the UCI Abalone data for this example... import matplotlib import numpy as np import matplotlib.pyplot as plt %matplotlib inline # Read file into ...

View answer
Can you explain the difference between SVC and LinearSVC in scikit-learn?
28 votes

A regular SVM with default values uses a radial basis function as the SVM kernel. This is basically a Gaussian kernel aka bell-curve. Meaning that the no man's land between different classes is ...

View answer
When do I have to use aucPR instead of auROC? (and vice versa)
Accepted answer
16 votes

Yes, you are correct that the dominant difference between the area under the curve of a receiver operator characteristic curve (ROC-AUC) and the area under the curve of a Precision-Recall curve (PR-...

View answer
Training Deep Nets on an Ordinary Laptop
Accepted answer
11 votes

Yes, a laptop will work just fine for getting acquainted with some deep learning projects: You can pick a smallish deep learning problem and gain some tractable insight using a laptop so give it a ...

View answer
With unbalanced class, do I have to use under sampling on my validation/testing datasets?
Accepted answer
11 votes

Great question... Here are some specific answers to your numbered questions: 1) You should cross validate on B not B`. Otherwise, you won't know how well your class balancing is working. It couldn'...

View answer
How to create US state choropleth map
10 votes

Plotly in Python provides a nice solution: !pip install chart_studio from chart_studio import plotly as py import pandas as pd df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/...

View answer
Logistic regression on biased data
Accepted answer
9 votes

Background I'll start with some background to help you research the solution yourself and then will add some specifics. What you refer to as "biased data" is more commonly known as unbalanced ...

View answer
How to evaluate clustering success in a completely unsupervised system?
9 votes

First, PCA is not a clustering method. It is a dimensionality reduction scheme. You can assess the performance of PCA through analysis of the percent of variance in the dataset that is retained as ...

View answer
For which real world data sets does DBSCAN surpass K-means.?
8 votes

I play around quite a bit with location data and have found examples both where k-means works fine and where k-means is a poor representation and DBSCAN is a great fit. If you've ever gone hiking or ...

View answer
How to detect overfitting of a stock screener
Accepted answer
8 votes

Learning curves or bias-variance decomposition are the gold standard for detecting high variance, aka: overfitting. Separate your data (in your case the "back data") into 60% training data and 40% ...

View answer
Creating new columns by iterating over rows in pandas dataframe
7 votes

There's a much more pythonic solution in pandas... This takes less than a second on 10 Million rows on my laptop: for x in X11.E.unique(): X11[x]=(X11.E==x).astype(int) X11 Here are the details ...

View answer
Boundary conditions for clustering
Accepted answer
7 votes

Since you accepted another answer, which says this can't be done, I am editing this to include an example of it being done. Hope this helps! Original Answer: The most logical way to transform hour ...

View answer
What's an efficient way to compare and group millions of store names?
Accepted answer
7 votes

This is an entity resolution aka record linkage aka data matching problem. I would solve this by removing all of the non-alphabetical characters including numbers, casting into all uppercase and then ...

View answer
Clustering users based on buying behaviour
6 votes

Big Picture: First of all, the feature set in your data is pretty sparse and uninteresting, so you should not expect to gain much traction from this problem. Use your human mind to think about the ...

View answer
Simple example of genetic alg minimization
Accepted answer
6 votes

Here is a trivial example, which captures the essence of genetic algorithms more meaningfully than the polynomial you provided. The polynomial you provided is solvable via stochastic gradient descent,...

View answer
How to evaluate distance in k-means clusters?
Accepted answer
5 votes

There are several important points to keep in mind in considering your questions: You should always normalize or standardize your data before applying k-means clustering. This is true of most other ...

View answer
Why use sum and not average for sentiment analysis?
5 votes

Consider the following two sentences: My awesome girlfriend bought me a delicious popsicle at the store. 0--1-------0----------0------0--0-1---------0--------0--0---0-----:2:11 My awesome girlfriend,...

View answer
Are there any interesting application of linear regression
Accepted answer
5 votes

The way that you have phrased this question makes it tough for people to answer without first offering you some background on linear regression (LR). Its great that you are interested in learning ...

View answer
how to generate sample dataset for classification problem
Accepted answer
4 votes

I know this isn't answering the question that you actually asked, but I suggest that you NOT generate data for your 'short text' categorization problem. Generated data can work for certain cases when ...

View answer
How data representation affects neural networks?
Accepted answer
4 votes

It will have very little effect The answer most will give is that it will have no effect, but adding one more feature will decrease the ratio of records to features so will slightly increase the bias ...

View answer
Research in random forest algorithms able to switch data sets
Accepted answer
4 votes

Semi-Supervised Learning The combination of unsupervised learning and supervised learning is referred to as semi-supervised learning, which is the concept that I believe you are searching for. Label ...

View answer
Why does applying PCA on targets causes underfitting?
4 votes

First, thanks for the edits to your original question since we now know that you are applying the same transformation to all of your data. Q: Why do perceptrons perform so much better than ...

View answer
Vector space model cosine tf-idf for finding similar documents
4 votes

Unfortunately, the math simplifies to show that you can't rigorously justify restricting the cosine similarity comparison of the vectors based on their lengths. The key point is that the cosine ...

View answer
How to split train/test in recommender systems
3 votes

Leave-one-out cross validation is probably the most straight-forward way to address this. If you happen to be using a model that requires a lot of time to train, then leave n% out might be more ...

View answer
How would you teach a machine how to count?
3 votes

I suggest using supervised learning and employing a linear model: linear regression. This is a perfectly linear system (y=x+1), so linear regression will work just fine i.e. perfectly. Further, you ...

View answer
Deep learning facial recognition research project
3 votes

First, tensorflow would be a great resource as it is gaining a userbase with exponential growth. Tensorflow has gpu support for single node calculations, and has recently added a distributed version ...

View answer
Should I use regularization every time?
3 votes

Adding a few more specifics to the previous two responses which both contain useful insight and perspective: Regularization is used to control overfitting (more formally high variance) scenarios. You ...

View answer