TwinPenguins
  • Member for 3 years, 11 months
  • Last seen this week
3 answers
1 votes
24 views
What to do about the failed portion of trained dataset?
0 votes

This is where so-called term Baseline comes into play. One needs to have a baseline either a simple model prediction performance (accuracy, precision or recall whatever) set, and try to improve upon ...

View answer
2 answers
6 votes
20k views
Always drop the first column after performing One Hot Encoding?
4 votes

This question, in a slightly different form, was discussed herein earlier. You are kind of right, but the best and safest way is to do One-Hot-Encoding and drop at the end because which column you ...

View answer
2 answers
4 votes
125 views
What is the best model for a recommendation system using implicit ratings?
Accepted answer
0 votes

If you are after a baseline model, which I also strongly recommend, matrix-factorization (MF) approach (aka collaborative models) you mentioned is the most basic one and super fast and easy to ...

View answer
1 answers
3 votes
53 views
NLP: Information extraction
3 votes

What you need is perhaps Named Entity Recognition with custom entity dictionary. See this example: Many packages like NLTK or Spacy have a large dictionary of such entities that enable models to ...

View answer
3 answers
1 votes
5k views
Remove all columns where the entire column is null
Accepted answer
1 votes

[Updated]: Just realized it is about pyspark! It is still simple! A concrete example (idea heavily borrowed from this answer): Creating a dummy dataset import pandas as pd import numpy as np import ...

View answer
2 answers
7 votes
314 views
How important is advanced SQL for data science?
0 votes

It is a very good point, and IMHO it is often overlooked and underestimated by Data Scientists. I have come to believe it strongly depends on the following variables that are largely intertwined (...

View answer
1 answers
2 votes
317 views
Distance of a people in an image
2 votes

Updated: Measuring distance from camera to object Perhaps you could search and find tons of materials. This is a well-established task, for example this tutorial, of more advanced methods combining ...

View answer
3 answers
1 votes
85 views
Classifying boat images
Accepted answer
2 votes

By looking at your code snippet, I realize you are training your CNN from scratch. Use Transfer Learning Instead. Training a new model (choice of model architecture i.e. how deep your model should ...

View answer
1 answers
0 votes
52 views
How to split pieces of dataframe and create new dataframes based on it?
1 votes

One way of doing this is explained here, for example: import pandas as pd import numpy as np df = pd.DataFrame([('bird', 'Falconiformes', 389.0), ('bird', 'Psittaciformes', 24.0),...

View answer
1 answers
1 votes
366 views
Ignore unseen columns with OneHotEncoder
1 votes

It is not possible. And this has nothing to do with OHE, if you choose any other encoding methods, still you are facing the same problem! Not having similar feature sets between train and test ...

View answer
1 answers
0 votes
2k views
Surface Pro 6 vs Macbook Pro for Professional Data Science Practice
Accepted answer
0 votes

Although once asked I knew this was a very subjective question, and I was not looking for specific yes or no, but rather hearing the community's experience, here now I would like the share the ...

View answer
2 answers
1 votes
94 views
Meaning of axes in Linear Discrimination Analysis
Accepted answer
0 votes

You could be asking yourself the same questions using PCA! Usually these linear transformations (or even non-linear for that matter, PCA, LDA, ICA, UMAP etc.) are used to reduce the high dimensional ...

View answer
1 answers
-1 votes
729 views
Recommend another product only on purchase history of users available
3 votes

This is a typical recommendation system. Just to recap, the three most popular ones are: Collaborative models use only collaborative information – implicit or explicit interactions of users with ...

View answer
1 answers
1 votes
160 views
Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?
0 votes

Short answer: NO. The moment you convert the spark dataframe into a pandas dataframe, all of the subsequent operations (pandas, ml etc.) will be run on a single-core as those algorithms and ...

View answer
1 answers
1 votes
352 views
redundancy or functional dependency of two columns
Accepted answer
2 votes

Well, it always depends, for example, on what model you might be training (i.e. some are robust to multicollinearity). I am pretty sure you are aware, but to have it said as a rule of thumb it is ...

View answer
2 answers
2 votes
1k views
Should we use one hot encoder class in data having 2 as maximum numeric representation of categorical feature in each column?
2 votes

To answer the question straightaway: Should I use OneHotEncoder() class afterwards? No. After LabelEncoder you do not need to do OneHotEncoder again. I do not know what you meant by: Or 2 isn'...

View answer
1 answers
0 votes
52 views
Basic sympy problem in anaconda
Accepted answer
0 votes

Just change the ’ to " it works just fine, see: from sympy import * x,y=symbols("x,y"); solution=solve((4*x-3*y-17,7*x+5*y-11),x,y); P=(solution[x],solution[y]); print(P) (118/41, -75/41)

View answer
3 answers
11 votes
3k views
How to encode a class with 24,000 categories?
6 votes

Entity Embedding for Categorical Variables (original pager) would be a very suitable approach here. Read on here, or here. I have actually put pieces of codes from here and there and made an complete ...

View answer
1 answers
5 votes
29k views
Confusion Matrix three classes python
12 votes

Multi-class Confusion Matrix is very well established in literature; you could find it easily on your own. Anyhow, Scikit-learn can do it easily like: from sklearn.metrics import confusion_matrix ...

View answer
1 answers
3 votes
365 views
Incorrect correlation results
2 votes

Pearson Correlation is a technique for investigating the relationship between two quantitative, continuous variables, for example, age and blood pressure. Pearson's correlation coefficient (r) is a ...

View answer
1 answers
1 votes
356 views
Decision tree classifier prediction changes from one run of the model to the next
Accepted answer
2 votes

While this is a duplicate and the suggested link answers your question, for learning purposes I would like to suggest that you plot your DecisionTree every time you have a new run to see by yourself ...

View answer
1 answers
3 votes
2k views
How to automate ANOVA in Python
Accepted answer
1 votes

I am not sure ANOVA is the best and easiest way to find correlation between these categorical features and your target. You may see this great post where they propose many other methods along with ...

View answer
1 answers
2 votes
9k views
Find Cluster Diameter and Associated Cluster Points with KMeans Clustering (scikit learn)
4 votes

For the time being, I've prepared a workaround solution: #iris example iris = datasets.load_iris() x = iris.data y = iris.target estimator = KMeans(n_clusters=3) y_kmeans = estimator.fit_predict(x) ...

View answer
2 answers
0 votes
188 views
LinearRegression with multiple binary features sometimes performs poorly
0 votes

Prior to jumping to any conclusions, some questions that immediately comes to my mind: What is the correlation between your features and target? Do you have any numerical features too? How large is ...

View answer
1 answers
0 votes
224 views
How to predict specific user from session logs?
Accepted answer
0 votes

There are certainly different ways to approach the problem, I am giving you what I think. The main key is to create as many as features as you can, well it depends on what algorithm you may end up ...

View answer
2 answers
0 votes
73 views
What's the name of data points that trained model gets on input in production?
1 votes

It is subjective what you are going to call your "Unseen Data", and it would not matter IMHO. You may call it production data as it is suggested above, or even unseen data or test data. In the Machine ...

View answer
2 answers
18 votes
11k views
Why do we have to divide by 2 in the ML squared error cost function?
Accepted answer
16 votes

It is simple. It is because when you take the derivative of the cost function, that is used in updating the parameters during gradient descent, that $2$ in the power get cancelled with the $\frac{1}{2}...

View answer
3 answers
3 votes
11k views
Mean Average Precision python code
5 votes

This library called Metrics provides most of metrics for Machine Learning including MAP for Recommendation systems. If you only interested in metrics for recommendation systems, perhaps you can see ...

View answer
1 answers
10 votes
7k views
Confusion about Entity Embeddings of Categorical Variables - Working Example!
Accepted answer
14 votes

For those who are interested, I've spent some time, finally figured out that the problem was the way one has to prepare the categorical encoding for the Entity Embedding suitable for a neural network ...

View answer
2 answers
3 votes
497 views
Logistic regression in python
4 votes

Logistic Regression is rather a hard algorithm to digest immediately as details often are abstracted away for the sake of simplicity for practitioners. To explain the idea behind logistic regression ...

View answer