20k views

Great question, this is what is known in Machine Learning paradigm as either "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on. One of the critical assumption one would make to ...

11k views

It is simple. It is because when you take the derivative of the cost function, that is used in updating the parameters during gradient descent, that $2$ in the power get cancelled with the $\frac{1}{2}... View answer 2 answers 23 votes 23k views Accepted answer 15 votes Simply put because one level of your categorical feature (here location) become the reference group during dummy encoding for regression and is redundant. I am quoting form here "A categorical ... View answer 1 answers 10 votes 7k views Accepted answer 14 votes For those who are interested, I've spent some time, finally figured out that the problem was the way one has to prepare the categorical encoding for the Entity Embedding suitable for a neural network ... View answer 1 answers 9 votes 823 views 14 votes There are three major assumptions (statistically strictly speaking): There is a linear relationship between the dependent variables and the regressors (right figure below), meaning the model you are ... View answer 2 answers 14 votes 3k views Accepted answer 13 votes Have you heard of Uniform Manifold Approximation and Projection (UMAP)? UMAP (Uniform Manifold Approximation and Projection) is a novel manifold learning technique for non-linear dimension ... View answer 3 answers 12 votes 10k views 13 votes It is very good question; in fact this problem has been around for a while and I have not yet found the perfect solution. Yet more than happy to share my experience: Avoid one-hot-encode as much as ... View answer 1 answers 5 votes 29k views 12 votes Multi-class Confusion Matrix is very well established in literature; you could find it easily on your own. Anyhow, Scikit-learn can do it easily like: from sklearn.metrics import confusion_matrix ... View answer 3 answers 1 votes 615 views 8 votes There is a one-liner solution using a Python package called pandas-profiling that gives you a quick way into most crucial statistical explanatory analysis including various correlations and many more. ... View answer 3 answers 5 votes 14k views Accepted answer 8 votes This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply: data_df[data_df &gt; 15000] If this data ... View answer 1 answers 4 votes 7k views 7 votes There are a lot going on in a simple OLS model. I strongly encourage you to learn more about them from textbooks. One of the best place to start is the free online book An Introduction to Statistical ... View answer 2 answers 7 votes 12k views Accepted answer 7 votes Actually the solution2 worked; I just had to be a bit more patient. I am posting it here in case someone have difficulties, like me, getting this to work: import pandas as pd from IPython.display ... View answer 3 answers 4 votes 27k views Accepted answer 7 votes I like the way Wikipedia generally defines it: In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. By contrast, the values of other ... View answer 3 answers 11 votes 3k views 6 votes Entity Embedding for Categorical Variables (original pager) would be a very suitable approach here. Read on here, or here. I have actually put pieces of codes from here and there and made an complete ... View answer 3 answers 3 votes 11k views 5 votes This library called Metrics provides most of metrics for Machine Learning including MAP for Recommendation systems. If you only interested in metrics for recommendation systems, perhaps you can see ... View answer 2 answers 4 votes 599 views Accepted answer 5 votes The cost function of the Logistic Regression derived via Maximum Likelihood Estimation: If y = 1 (positive): i) cost = 0 if prediction is correct (i.e. h=1), ii) cost$\rightarrow \infty $if$...

3k views

You can not use PCA, or at least it is not recommended, for mixed data. It is best to use Factor analysis of mixed data. You are lucky that Prince is a Python package that covers all data scenarios, ...

19k views

Try: pip install pandas-profiling or: conda install -c anaconda pandas-profiling

9k views

I know you ask about the model choice here, but it is worth to discuss about your input data first. Data with many categorical features is still an active research; so it is not that straightforward. ...

20k views

This question, in a slightly different form, was discussed herein earlier. You are kind of right, but the best and safest way is to do One-Hot-Encoding and drop at the end because which column you ...

9k views

For the time being, I've prepared a workaround solution: #iris example iris = datasets.load_iris() x = iris.data y = iris.target estimator = KMeans(n_clusters=3) y_kmeans = estimator.fit_predict(x) ...

497 views

Logistic Regression is rather a hard algorithm to digest immediately as details often are abstracted away for the sake of simplicity for practitioners. To explain the idea behind logistic regression ...

3k views

I believe it is the probabilistic nature of a model that allows you to get the variance of predictions, or more generally defined as the uncertainty of predictions, like the Gaussian process you ...

19k views

Why don't you consider Gradient Boosting Decision Trees (GBDT) for Regression which you will find many Python implementation for (XGboost, LightGBM and CatBoost). The good things about GBDTs (more ...

2k views

(Edited after @D.W. suggestion). To the best of my knowledge, there is nothing wrong with what you have in mind; thus it is certainly valid.. As you said, you have to try out all possible ways you ...

3k views

Categorical features need to be converted to numerical values. They are various ways to do that. I would recommend reading this blog and this one to learn what are the advantages and disadvantages of ...

53 views

What you need is perhaps Named Entity Recognition with custom entity dictionary. See this example: Many packages like NLTK or Spacy have a large dictionary of such entities that enable models to ...

727 views

This is a typical recommendation system. Just to recap, the three most popular ones are: Collaborative models use only collaborative information – implicit or explicit interactions of users with ...

3k views