Julio Jesus
  • Member for 1 year, 8 months
  • Last seen this week
2 answers
0 votes
29 views
What is the best way to limit number of features in TF-IDF?
3 votes

Generally speaking the correct representation on td-idf encoding is a hyperparameter to be optimized. As suggested in the above's answers, you can go for the regularization parameter i.e min_df which ...

View answer
1 answers
0 votes
27 views
roc_auc_score from sk-learn gives error when test label vector with classes has only a subset of the whole set
Accepted answer
1 votes

predict_proba method will return a numpy array of shape (n_samples,2) with the probability of Y == 1 and Y == 0 but you need to pass only the probability of Y == 1 for roc calculation so: from sklearn....

View answer
1 answers
3 votes
464 views
What Shape Does Naive Bayes make?
Accepted answer
6 votes

Specifically talking about Gaussian Naive Bayes, the decision boundary are ellipsoids characterized by the mean and standard deviation of the Gaussian distribution. Image: https://scikit-learn.org/...

View answer
1 answers
0 votes
36 views
KNN error: could not find function "train"
1 votes

As someone that is more used to use Python's structure, I highly recommend to use the package/class name before the method. So if you are using the method train, you want to specify that this method ...

View answer
1 answers
0 votes
29 views
Select Random Value from Pandas list column for each row ensuring that value don't get picked again
Accepted answer
1 votes

This is a first approach, and even though this is not the best in terms of performance it makes the work: def urandom(frame): ls = list() for idx, row in frame.iterrows(): val = np....

View answer
1 answers
0 votes
959 views
Is there a way to force a transformer to return a pandas dataframe?
2 votes

Might be late but for anyone with the same question the answers (as almost everything with Scikit-learn) is the usage of Pipelines from sklearn.impute import SimpleImputer from sklearn.preprocessing ...

View answer
2 answers
1 votes
36 views
Multiple Merges make the data frame in pandas to explode and causing Memory Issue in jupyter notebook
2 votes

First things first: Check that you are joining by primary keys only i.e, yo do not have any duplicate value at any of the columns you are joining, otherwise you will end up with a huge and ...

View answer
2 answers
4 votes
191 views
Remove outliers from a noisy curve
2 votes

You can think of this problem as basically trying to find dense areas inside a cloud with noise. This is not the only possible solution but you could use a clustering algorithm, and specifically one ...

View answer
1 answers
2 votes
25 views
ggplot2 for Cluster analysis (non-readible row names)
Accepted answer
2 votes

Try adding theme to your plot layout So: library("reshape2") library("purrr") library("dplyr") library("dendextend") dendro <- as.dendrogram(aggl.clust.c) ...

View answer
1 answers
2 votes
36 views
How can I get the dataframe after scikit pipeline?
2 votes

You can try applying your preprocessor to your X_train and X_test: preprocessor = ColumnTransformer( transformers=[ ('num', numeric_transformer, numericas_all) ,('cat', categorical_transformer, ...

View answer
2 answers
0 votes
65 views
Which machine learning model is best for a combination of numerical and categorical data?
2 votes

Even though virtually any supervised classification algorithm can be used when having categorical features by applying some encoding technique, my first thought is using Catboost, an algorithm ...

View answer
2 answers
0 votes
52 views
improve LinearSVC
2 votes

Additionally to the previous answer, I would go for POS tagging features (features that count the number of verbs, adverbs, nouns, etc contained in your review), since you are trying to distinguish ...

View answer
2 answers
0 votes
78 views
calibrated classifier ValueError: could not convert string to float
Accepted answer
1 votes

Once I assume you are using text data as your input matrix X. The first point is that you have to include your preprocessing step as you would do when not using a calibrated classifier, so as you ...

View answer
1 answers
2 votes
139 views
sklearn models Parameter tuning GridSearchCV
Accepted answer
3 votes

The correct way of calling the parameters inside Pipeline is using double underscore like named_step__parameter_name .So the first thing I noticed is in this line: parameters = {'vect__ngram_range': [(...

View answer
3 answers
1 votes
26 views
Work with large number of features for machine learning with pandas and sklearn
2 votes

I'm assuming you already have made feature selection so all your ~200 features are the ones that describe your target So particularly for models that use SGD you can train your model in batches i.e ...

View answer
2 answers
0 votes
42 views
Text mining match in Python
Accepted answer
1 votes

As far as I understand from your question, you are trying to compare sentences on word level, but it seems like you are interested in finding the number of words in sentence A that are contained in ...

View answer
2 answers
1 votes
46 views
How do I determine the top "reason" for anomaly when using Isolation Forests
2 votes

A naive approach would be to use a supervised model to predict the target anomaly vs no anomaly that your IsolationForest model outputs, then if and only if this supervised binary classification model ...

View answer
1 answers
0 votes
49 views
Turning regression problem into "classification + regression"
1 votes

As you well noticed there is no way to know the bin in wich an unseen data's target value will be. So what you can do is to train a model that splits/clusters your data and then run a model on each ...

View answer
1 answers
1 votes
46 views
Can the feature importance scores of multiple ML algorithms be combined?
3 votes

In recent years I have read different approaches on what you mention, with the argument of applying an ensamble on feature selection, the same way it is applied to model prediction (stacking models) ...

View answer
4 answers
1 votes
134 views
What do "Under fitting" and "Over fitting" really mean? They have never been clearly defined
1 votes

Personally I find Victor Lavrenko's explanation of underfitting and overfitting the most intuitive and concise definition: This definition is very useful for at least these two points: This is not ...

View answer
2 answers
2 votes
179 views
How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline
Accepted answer
2 votes

Might be late but I found this question interesting: Try: import pandas as pd from sklearn.datasets import load_iris from sklearn.pipeline import Pipeline from sklearn.compose import ...

View answer
1 answers
0 votes
39 views
Standardizing giving worse results
Accepted answer
1 votes

As you well mentioned, tree-based models are not sensitive to feature scaling, but on the contrary it might help with the convergency of finding the minimum in the optimization on boosted models I ...

View answer
1 answers
0 votes
26 views
need an explanation of the For Loop in the DBSCAN algorithm Demo
1 votes

First I'm going to use a simplier way (gives the same plot just without changing dots size according to its distance to core samples) of visualizing the cluster results: plt.scatter(X[:,0], X[:,1], c =...

View answer
4 answers
0 votes
3k views
Remove columns with a certain number of consecutive zeros
0 votes

An alternative to archived what you are looking for is: threshold = 12 drops = [l[0] for l in list(filter(lambda x: x[1] > threshold,[(col, (df.groupby((df[col] != 0).cumsum()).cumcount()).max()) ...

View answer
3 answers
4 votes
783 views
Is it always possible to get well-defined clusters from the data?
Accepted answer
8 votes

First of all a picture should not be taken to define if there are or not groups on your data, since no matter what projection you use (linear with PCA or manifold with tSNE) you are reducing a 64-...

View answer
6 answers
7 votes
2k views
Why do we need data scientists if there are websites like this?
13 votes

Without having a complete knowledge of the features on that website I would say: Data visualization is only one part on data scientist (ds) pipeline from data understanding thought model validation ...

View answer
2 answers
0 votes
29 views
How to create classification decision trees on a dataset that has both numerical and categorical variables?
1 votes

What you mentioned is true, for 99% of Scikit-learn's estimators, the data X must be numeric (I think only HistGradientBoosting works with no numerical categorical data) So when working with mixed ...

View answer
1 answers
1 votes
38 views
Check if distribution per week is the same
0 votes

You can create n matrices where n is the number of different products {Alpha, Beta, ...,} then for each different product you group your daily value into weekly so you have something like Product1 : {...

View answer
3 answers
1 votes
43 views
What data visualization for N elements switched from x to y
1 votes

You could go for a heat map, whose rows and columns would be the software from and software to, respectively and the values are the number of users that switched from one to another. This might look ...

View answer
1 answers
0 votes
24 views
Run linear regression fit on 2 1D array
Accepted answer
0 votes

By using np.reshape you can transform your input x to Scikit's necessary input shape from sklearn.model_selection import train_test_split x_train, x_test, y_train, y_test = train_test_split(x,y, ...

View answer