Julio Jesus
  • Member for 1 year, 8 months
  • Last seen this week
1 answers
0 votes
24 views
Clustering method for 2-D data that self-detects number of clusters and takes care of outliers
2 votes

DBSCAN is the algorithm of choice for this task. This a density based algorithm which will look for clusters according to two main parameters, epsilon and min samples. It will also identify those ...

View answer
1 answers
0 votes
1k views
neg_mean_squared_error in cross_val_score
2 votes

You are right, neg_mean_squared_error is simple -1 * mean_squared_error. This is because a convention in the Scikit-learn api that all the scorers follow. According to scikit-learn documentation (some ...

View answer
3 answers
1 votes
494 views
How to create a big data frame in Python
2 votes

I have had to deal with huge data frames as you mention, in mi case the problem was "solved" by storing the data frame as pickle pd.to_pickle() and not as csv. The memory usage reduced by 60%...

View answer
1 answers
1 votes
82 views
Are there any tree-based models that use a genetic algorithm to generate the trees?
Accepted answer
2 votes

Very new release: genetic-tree The main objective of the package is to allow creating decision trees that are better in some aspects than trees made by greedy algorithms. The creation of trees is ...

View answer
2 answers
0 votes
122 views
How to group every data point with HDBSCAN to some group to have no noise?
2 votes

DBSCAN will always mark noisy points according to epsilon and min_samples parameters, so there is no way to avoid that unless you have very compact and "well defined" clusters, what seems ...

View answer
2 answers
1 votes
761 views
When should we choose agglomerative clustering over K-means clustering?
Accepted answer
2 votes

To add to WBM great citation, you should use K-means over Agglomerative when your final objetive is to use the trained algorithm to make inference over new unseen observations. I will try to ...

View answer
2 answers
0 votes
131 views
Why is the idf important in tf-idf when it seems to just re-scale your features?
2 votes

To complement my comment I'm taking those paragraphs from data camp tutorial in which they explain this in a very clear way

View answer
1 answers
0 votes
40 views
Which data science model is best for explainability for prediction problems?
2 votes

Personally I think linear (through model's coefficients/weights) and tree-based models (gain importance) are the best for explainability But this is not restricted to those models since you can use ...

View answer
1 answers
2 votes
72 views
Does RandomForest convergence imply I can solve a problem with a NN too?
2 votes

To me those are separate things since both models have a different cost function to be optimized. On the other hand you could combine those models by constructing embeddings based on random forest ...

View answer
1 answers
0 votes
1k views
How to replace values in a numpy array?
Accepted answer
2 votes

import numpy as np a = np.array(['PAIDOFF', 'COLLECTION', 'COLLECTION', 'PAIDOFF']) f = lambda x: 1 if x == "COLLECTION" else 0 np.fromiter(map(f,a),dtype=np.int) Alternative: np.where(a ...

View answer
1 answers
1 votes
113 views
How to interpret results of a Clustering Heart Failure Dataset?
Accepted answer
2 votes

I would use all the features and see how the separateness of my clusters behave according to some metric, for example, silhouette score Additionally, it is very important to scale your data prior to ...

View answer
2 answers
0 votes
54 views
Building a summary string in a Pandas groupby (Possibly cross-tab or pivot-table question)
2 votes

Does that work for you? df.groupby(["fruit","name"]).num.sum().unstack().assign(suma = lambda x: x.sum(axis = 1)) Returns: name Bob Tim suma fruit Apples 10 20 30 ...

View answer
2 answers
0 votes
561 views
Replace value of a column if the value of another column is a duplicate
Accepted answer
2 votes

A cleaner way would be: df["date"] = df.groupby("name").date.transform("min") Outputs:

View answer
1 answers
0 votes
17 views
Searchable list of Kaggle challenges
Accepted answer
2 votes

Have you checked the competition's stage? In there you will see all the active and concluded contests https://www.kaggle.com/competitions

View answer
14 answers
153 votes
259k views
Train/Test/Validation Set Splitting in Sklearn
2 votes

How about using numpy random choice import numpy as np from sklearn.datasets import load_iris def ttv_split(X, y = None, train_size = .6, test_size = .2, validation_size = .2, random_state = 42): ...

View answer
1 answers
1 votes
1k views
Plotting multiple precision-recall curves in one plot
Accepted answer
2 votes

Try using Matplotlib gca() method in this way you can indicate what axis you want to plot in from sklearn.metrics import precision_recall_curve from sklearn.metrics import plot_precision_recall_curve ...

View answer
1 answers
0 votes
83 views
High probabilities of success for wrong predictions
2 votes

It is difficult to diagnose without having more info. But based upon what you mention I point: Since you have a class imbalance, you MUST use regularizarided models that penalize the cost function ...

View answer
4 answers
13 votes
6k views
How do I make an interactive PCA scatterplot in Python?
2 votes

I highly recommend using PlotlyExpress instead This code is plotting the first 3 components on the iris dataset import plotly.express as px from sklearn.datasets import load_iris from ...

View answer
1 answers
0 votes
27 views
roc_auc_score from sk-learn gives error when test label vector with classes has only a subset of the whole set
Accepted answer
1 votes

predict_proba method will return a numpy array of shape (n_samples,2) with the probability of Y == 1 and Y == 0 but you need to pass only the probability of Y == 1 for roc calculation so: from sklearn....

View answer
1 answers
0 votes
36 views
KNN error: could not find function "train"
1 votes

As someone that is more used to use Python's structure, I highly recommend to use the package/class name before the method. So if you are using the method train, you want to specify that this method ...

View answer
1 answers
0 votes
29 views
Select Random Value from Pandas list column for each row ensuring that value don't get picked again
Accepted answer
1 votes

This is a first approach, and even though this is not the best in terms of performance it makes the work: def urandom(frame): ls = list() for idx, row in frame.iterrows(): val = np....

View answer
2 answers
0 votes
75 views
calibrated classifier ValueError: could not convert string to float
Accepted answer
1 votes

Once I assume you are using text data as your input matrix X. The first point is that you have to include your preprocessing step as you would do when not using a calibrated classifier, so as you ...

View answer
2 answers
0 votes
42 views
Text mining match in Python
Accepted answer
1 votes

As far as I understand from your question, you are trying to compare sentences on word level, but it seems like you are interested in finding the number of words in sentence A that are contained in ...

View answer
1 answers
0 votes
49 views
Turning regression problem into "classification + regression"
1 votes

As you well noticed there is no way to know the bin in wich an unseen data's target value will be. So what you can do is to train a model that splits/clusters your data and then run a model on each ...

View answer
4 answers
1 votes
134 views
What do "Under fitting" and "Over fitting" really mean? They have never been clearly defined
1 votes

Personally I find Victor Lavrenko's explanation of underfitting and overfitting the most intuitive and concise definition: This definition is very useful for at least these two points: This is not ...

View answer
1 answers
0 votes
39 views
Standardizing giving worse results
Accepted answer
1 votes

As you well mentioned, tree-based models are not sensitive to feature scaling, but on the contrary it might help with the convergency of finding the minimum in the optimization on boosted models I ...

View answer
1 answers
0 votes
26 views
need an explanation of the For Loop in the DBSCAN algorithm Demo
1 votes

First I'm going to use a simplier way (gives the same plot just without changing dots size according to its distance to core samples) of visualizing the cluster results: plt.scatter(X[:,0], X[:,1], c =...

View answer
2 answers
0 votes
29 views
How to create classification decision trees on a dataset that has both numerical and categorical variables?
1 votes

What you mentioned is true, for 99% of Scikit-learn's estimators, the data X must be numeric (I think only HistGradientBoosting works with no numerical categorical data) So when working with mixed ...

View answer
3 answers
1 votes
43 views
What data visualization for N elements switched from x to y
1 votes

You could go for a heat map, whose rows and columns would be the software from and software to, respectively and the values are the number of users that switched from one to another. This might look ...

View answer
1 answers
0 votes
238 views
OneHotEncoder Showing error while encoding two columns
1 votes

Try: from sklearn.compose import ColumnTransformer from sklearn.preprocessing import OneHotEncoder ct = ColumnTransformer(transformers = [('encoder', OneHotEncoder(),[1,2])], remainder ='passthrough') ...

View answer