TwinPenguins
  • Member for 3 years, 11 months
  • Last seen this week
4 answers
21 votes
20k views
Different Test Set and Training Set Distribution
22 votes

Great question, this is what is known in Machine Learning paradigm as either "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on. One of the critical assumption one would make to ...

View answer
2 answers
18 votes
11k views
Why do we have to divide by 2 in the ML squared error cost function?
Accepted answer
16 votes

It is simple. It is because when you take the derivative of the cost function, that is used in updating the parameters during gradient descent, that $2$ in the power get cancelled with the $\frac{1}{2}...

View answer
2 answers
23 votes
23k views
Why do we need to discard one dummy variable?
Accepted answer
15 votes

Simply put because one level of your categorical feature (here location) become the reference group during dummy encoding for regression and is redundant. I am quoting form here "A categorical ...

View answer
1 answers
10 votes
7k views
Confusion about Entity Embeddings of Categorical Variables - Working Example!
Accepted answer
14 votes

For those who are interested, I've spent some time, finally figured out that the problem was the way one has to prepare the categorical encoding for the Entity Embedding suitable for a neural network ...

View answer
1 answers
9 votes
823 views
Assumptions of linear regression
14 votes

There are three major assumptions (statistically strictly speaking): There is a linear relationship between the dependent variables and the regressors (right figure below), meaning the model you are ...

View answer
2 answers
14 votes
3k views
Efficient dimensionality reduction for large dataset
Accepted answer
13 votes

Have you heard of Uniform Manifold Approximation and Projection (UMAP)? UMAP (Uniform Manifold Approximation and Projection) is a novel manifold learning technique for non-linear dimension ...

View answer
3 answers
12 votes
10k views
How can I do classification with categorical data which is not fixed?
13 votes

It is very good question; in fact this problem has been around for a while and I have not yet found the perfect solution. Yet more than happy to share my experience: Avoid one-hot-encode as much as ...

View answer
1 answers
5 votes
29k views
Confusion Matrix three classes python
12 votes

Multi-class Confusion Matrix is very well established in literature; you could find it easily on your own. Anyhow, Scikit-learn can do it easily like: from sklearn.metrics import confusion_matrix ...

View answer
3 answers
1 votes
615 views
Is there any python framework which take a dataframe and give all important relation?
8 votes

There is a one-liner solution using a Python package called pandas-profiling that gives you a quick way into most crucial statistical explanatory analysis including various correlations and many more. ...

View answer
3 answers
5 votes
14k views
How to get spike values from a value sequence?
Accepted answer
8 votes

This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply: data_df[data_df > 15000] If this data ...

View answer
1 answers
4 votes
7k views
interpreting multi linear regression results
7 votes

There are a lot going on in a simple OLS model. I strongly encourage you to learn more about them from textbooks. One of the best place to start is the free online book An Introduction to Statistical ...

View answer
2 answers
7 votes
12k views
Display Images (url) Inside Pandas Dataframe
Accepted answer
7 votes

Actually the solution2 worked; I just had to be a bit more patient. I am posting it here in case someone have difficulties, like me, getting this to work: import pandas as pd from IPython.display ...

View answer
3 answers
4 votes
27k views
Which parameters are hyper parameters in a linear regression?
Accepted answer
7 votes

I like the way Wikipedia generally defines it: In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. By contrast, the values of other ...

View answer
3 answers
11 votes
3k views
How to encode a class with 24,000 categories?
6 votes

Entity Embedding for Categorical Variables (original pager) would be a very suitable approach here. Read on here, or here. I have actually put pieces of codes from here and there and made an complete ...

View answer
3 answers
3 votes
11k views
Mean Average Precision python code
5 votes

This library called Metrics provides most of metrics for Machine Learning including MAP for Recommendation systems. If you only interested in metrics for recommendation systems, perhaps you can see ...

View answer
2 answers
4 votes
599 views
Logistic regression cost function
Accepted answer
5 votes

The cost function of the Logistic Regression derived via Maximum Likelihood Estimation: If y = 1 (positive): i) cost = 0 if prediction is correct (i.e. h=1), ii) cost $\rightarrow \infty $ if $...

View answer
1 answers
3 votes
3k views
Data scaling before PCA: how to deal with categorical values?
5 votes

You can not use PCA, or at least it is not recommended, for mixed data. It is best to use Factor analysis of mixed data. You are lucky that Prince is a Python package that covers all data scenarios, ...

View answer
1 answers
3 votes
19k views
Python: Unable to install pandas_profiling
Accepted answer
5 votes

Try: pip install pandas-profiling or: conda install -c anaconda pandas-profiling

View answer
1 answers
4 votes
9k views
Choosing a model for dataset with categorical variables
5 votes

I know you ask about the model choice here, but it is worth to discuss about your input data first. Data with many categorical features is still an active research; so it is not that straightforward. ...

View answer
2 answers
6 votes
20k views
Always drop the first column after performing One Hot Encoding?
4 votes

This question, in a slightly different form, was discussed herein earlier. You are kind of right, but the best and safest way is to do One-Hot-Encoding and drop at the end because which column you ...

View answer
1 answers
2 votes
9k views
Find Cluster Diameter and Associated Cluster Points with KMeans Clustering (scikit learn)
4 votes

For the time being, I've prepared a workaround solution: #iris example iris = datasets.load_iris() x = iris.data y = iris.target estimator = KMeans(n_clusters=3) y_kmeans = estimator.fit_predict(x) ...

View answer
2 answers
3 votes
497 views
Logistic regression in python
4 votes

Logistic Regression is rather a hard algorithm to digest immediately as details often are abstracted away for the sake of simplicity for practitioners. To explain the idea behind logistic regression ...

View answer
3 answers
7 votes
3k views
How to estimate the variance of regressors in scikit-learn?
Accepted answer
4 votes

I believe it is the probabilistic nature of a model that allows you to get the variance of predictions, or more generally defined as the uncertainty of predictions, like the Gaussian process you ...

View answer
3 answers
5 votes
19k views
Feature Selection in Linear Regression
4 votes

Why don't you consider Gradient Boosting Decision Trees (GBDT) for Regression which you will find many Python implementation for (XGboost, LightGBM and CatBoost). The good things about GBDTs (more ...

View answer
1 answers
6 votes
2k views
Instead of one-hot encoding a categorical variable, could I profile the data and use the percentile value from it's cumulative density distribution?
Accepted answer
4 votes

(Edited after @D.W. suggestion). To the best of my knowledge, there is nothing wrong with what you have in mind; thus it is certainly valid.. As you said, you have to try out all possible ways you ...

View answer
1 answers
4 votes
3k views
Cleaning input data with pd.get_dummies()
Accepted answer
4 votes

Categorical features need to be converted to numerical values. They are various ways to do that. I would recommend reading this blog and this one to learn what are the advantages and disadvantages of ...

View answer
1 answers
3 votes
53 views
NLP: Information extraction
3 votes

What you need is perhaps Named Entity Recognition with custom entity dictionary. See this example: Many packages like NLTK or Spacy have a large dictionary of such entities that enable models to ...

View answer
1 answers
-1 votes
727 views
Recommend another product only on purchase history of users available
3 votes

This is a typical recommendation system. Just to recap, the three most popular ones are: Collaborative models use only collaborative information – implicit or explicit interactions of users with ...

View answer
3 answers
2 votes
3k views
Add noise to an example set
3 votes

In addition to what Lupacante conceptually and nicely showed such that the added feature(s) has(have) to be informative for the model otherwise it can get ignored by majority of models (perhaps easily ...

View answer
1 answers
4 votes
4k views
changing cost function in xgboost
3 votes

This is discussed in stackoverflow, just to recap: Define your your customized cost function, e.g.: def new_cost(y_pred, y_true): # perform calculation for new cost return 'new_cost', ...

View answer