New answers tagged scikit-learn
0
import spacy
nlp = spacy.load('en_core_web_trf')
def identify_futuristic(sentence):
sentence_doc = nlp(sentence)
if any((token.morph.get('Tense') == [] and
token.morph.get('VerbForm') == ['Fin'] and
token.morph.get('Mood') == [])
or
(token.morph.get('Tense') == ['Pres'] and
token.morph.get('...
3
The term learning curve can mean different things in different context, which is confusing.
When talking about neural networks (and other iteratively trained models) the learning curve describes the model's training progress. It is often used to determine when it's time to stop training.
In scikit-learn, the learning curve is interpreted differently. It ...
2
You can try applying your preprocessor to your X_train and X_test:
preprocessor = ColumnTransformer(
transformers=[
('num', numeric_transformer, numericas_all)
,('cat', categorical_transformer, categoricas_all)
])
X_train_pipe = preprocessor.transform(X_train)
X_test_pipe = preprocessor.transform(X_test)
Edit:
Since you did not use any transformer that ...
1
It is correct that calling learning_curve will refit your model multiple times for different training dataset sizes. You can simply pass specific hyperparameters when initializing the model you want to use, which you can then pass to learning_curve for the estimator argument. The actual loss funtion that is used depends on the type of estimator you are ...
2
A million observations of 20 features should be very manageable on a laptop, if a little slow. Cloud computing for very large datasets is staggeringly expensive and offers little or no benefit unless and until you have good parallelization in place. I would recommend keeping that option as your last resort.
For the initial data exploration and ...
1
Parallelize your analyses on a single (multi-cpu) machine with e.g. pandarallel or the like or go for broke with scala if the problem wont fit on a single machine.
4
There are 2 things you can do here:
1.) Use libraries like Dask to speed up your data preprocessing. Here is the link
2.) Use cloud computing services like Azure, AWS or GCP. I am not aware of other two but I have worked on Azure and it provides a lot of options for implementing a data science solution. You get options like Auto-ML, Azure Designer, Python ML ...
1
For "why", I think it's just a new transformer that maybe didn't get thoroughly thought-out. The default of "half the features" in particular seems very odd to me.
A middle ground, that I think is more useful, is to select features until there is no (or little) further improvement. That's being implemented in PR20145. If they would ...
0
Imputer
fit() : provides statistics for the imputer i.e., fits data to imputer
transform() : imputes and fills the missing values
fit_transform() : Fit to data, then transform it.
Pipeline ideally apply a list of transforms so the final estimator only needs to implement fit
So to answer your question in the pipeline, the transform is already in place, so ...
0
Dask can help you out. Basically it uses sparse data to load your dataset so that even datasets much larger than your compute memory can be loaded.
0
I believe you need to add () where you add scaler to the pipeline: ('std_scaler',StandardScaler) --> ('std_scaler',StandardScaler())
1
Following the link in zachdj's answer, they spell it with a lowercase "s" and lowercase "l" in both papers appearing in the citation guide provided. Here is an example found in the paper Scikit-learn: Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825-2830, 2011:
While scikit-learn focuses on ease of use, and is mostly ...
0
There are many ways to do this. For example, you could use pandas to cross-tabulate the label values. Note that, judging by your output, the true labels are actually the second column in your table.
import pandas as pd
df = pd.read_csv('labels.csv', header=None)
df.columns = ['predicted', 'actual']
print(pd.crosstab(df.actual, df.predicted))
predicted 0 ...
0
I hope I'm phrasing this correctly.
You can call it as a attribute from the best estimator, i.e.
grid_model.best_estimator_.oob_score_
so you can't really use grid_model.oob_score_ because gridsearchcv does not have such an attribute; but you can call the best instance of the random forest model, then call the attribute.
1
Both are within one-vs-all scheme when there is a classification task.
LabelBinarizer it turn every variable into binary within a matrix where that variable is indicated as a column. In other words, it will turn a list into a matrix, where the number of columns in the target matrix is exactly as many as unique value in the input set. If your input labels ...
0
Scikit-learn's LabelBinarizer converts input labels into binary labels, each example belongs to a single class or not.
Scikit-learn's MultiLabelBinarizer
converts input labels into multilabel labels, each example can belong to multiple classes.
1
Some difference is expected as you set random_state = None and shuffle=True in your model. This results in weights being initialized randomly and training data to be used in different orders. For reproducible results, you should set it to an integer. See Scikit documentation for random_state variable.
0
The Jaccard index or score is often used for bounding boxes or semantic segmentation in machine learning, i.e. in computer vision problems. Your problem is a classification problem using tabular data, and therefore this metric is not really applicable for this type of problem. Accuracy (and maybe even more so precision and recall) are more valuable metrics ...
1
Why do I get a tuple as output and not vectors of 1 and 0?
You get this because by default OneHotEncoder() uses sparse matrix representation. Hence, it transforms the elements of y into elements of type -
<1x3 sparse matrix of type '<class 'numpy.float64'>'
with 1 stored elements in Compressed Sparse Row format>
If you want the output as ...
0
You have to remember your training data has to have all the possible values you know you will have in the test set. The data set should be shuffled before splitting so your case should not append. Remember a model cannot predict correctly on unknown category value never seen during training. So always shuffle and/or get more data so every category values are ...
1
Do not split the test set into half for the second train_test_split. Instead first split your whole data into train and test set. Then split the train set into train and validation sets as shown below.
X_train, X_test, y_train, y_test
= train_test_split(X, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val
= train_test_split(X_train,...
1
It would be helpful if you could post the full stack trace, so that we can see which line your error occurs at. In general, the more information you can provide in a question, the better.
In this case, it looks like your full_model_pipeline may somehow become a numpy array. Since you have a one-element pipeline, you could try changing
full_model_pipeline = ...
1
Additionally to the previous answer, I would go for POS tagging features (features that count the number of verbs, adverbs, nouns, etc contained in your review), since you are trying to distinguish between two kind of reviews, it sounds reasonable to think that something that talks about the function of the product has for example more adjetives.
Of course ...
0
The primary issue is that fowlkes_mallows_score is designed to evaluate clustering and you are trying to apply it to evaluate binary classification.
0
There are several ways the data distribution can change after training.
If there is a new category, that will require retraining the model to handle the new category.
If the relative frequency of the categories change, that will not effect most models. Most models learn a decision boundary based on features. If the model is based on prior priority of class ...
1
Once I assume you are using text data as your input matrix X. The first point is that you have to include your preprocessing step as you would do when not using a calibrated classifier, so as you already know you can use a Pipeline like so:
calibrated_svc = CalibratedClassifierCV(linear_svc,
method='sigmoid',
...
1
Assuming you mean sklearn.cluster.KMeans, you are able to pass in the initialization points using the init argument:
init : {‘k-means++’, ‘random’}, callable or array-like of shape
(n_clusters, n_features), default=’k-means++’ Method for
initialization:
‘k-means++’ : selects initial cluster centers for k-mean clustering in
a smart way to speed up ...
1
There are many ways to increase the accuracy.
1.) Try to get more data. More data usually helps in getting better results. (usually, not always!)
2.) Although you mention you have tried different models and I'm not sure how many, but there are still more models you can try.
3.) Try hyperparameter tuning for all the models you have tried, not only for linear ...
2
For any kind of Machine Learning task or a NLP task (which is what you are doing), you need to convert string/text values to numeric values. The machine cannot uderstand or work with string values. It only understands numeric values.
So for example if you are doing a machine learning task, you would use libraries like OneHotEncoder, LabelEncoder etc to ...
0
I am assuming, your focus here is the prediction accuracy and not interpretability?
So, as there is a class imbalance, you can do two things:
As suggested by the other user, you can use SMOTE or any technique.
Use a non-parametric method that is more robust in handling the class imbalance.
I tried to use Random Forest on your data, and the classification ...
0
You can look into SMOTE & ADAYSN techniques. This will help you in reducing the imbalance in the dataset by creating synthetic data
https://medium.com/coinmonks/smote-and-adasyn-handling-imbalanced-data-set-34f5223e167
0
Building model is an iterative process, we cannot guess which algorithm and which parameters give good results to your data in the beginning itself. So it is always good to start with the model with default parameters and then go for tuning the parameters.
Also, the default parameters are not same as the one you are providing in GridSearchCV. For example, In ...
3
The correct way of calling the parameters inside Pipeline is using double underscore like named_step__parameter_name .So the first thing I noticed is in this line:
parameters = {'vect__ngram_range': [(1, 1), (1, 2)],'tfidf__use_idf': (True, False),'clf__alpha': (1e-2, 1e-3) }
You are calling vect__ngram_range but this should be tfidf__ngram_range
Now this ...
2
If your datasets are random (with no real connection between the class and predictive variables), then "the right" model is a constant one: in (A), the predicted probabilities should be roughly $0.3, 0.2, 0.5$, whereas in (B) they should be $0.33, 0.33, 0.33$. When making the hard classifier then, in (A) the maximum probability will nearly always ...
0
The best approach is remove all the data that is not labeled 4 or 5.
Then rerun all the steps. Redo the train/test split and retrain the entire pipeline from scratch, including the CountVectorizer.
2
Regarding the case when you have 'bad' category present in the entire dataset, I would recommend using sklearn.model_selection.train_test_split function, with stratify option set to a corresponding variable. If stratify option is set to a list of all categories, it will be guaranteed that every single category will be included in both training and testing ...
1
You can reserve a special ordinal value to indicate "unknown/unseen during training." You would use this special value for any and all values of x that you encounter in the test set and in production.
In fact, scikit-learn's OrdinalEncoder does this for you via the handle_unknown parameter.
2
I'm assuming you already have made feature selection so all your ~200 features are the ones that describe your target
So particularly for models that use SGD you can train your model in batches i.e adding new observations each time
In your case if using python you can make usage of SGDClassfiier with loss = log to optimize logistic regression cost function, ...
0
Run a PCA on or LDA your data set. Here is some sample code to start with.
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
cancer = load_breast_cancer()
df = pd.DataFrame(data=cancer.data, columns=cancer.feature_names)
df.head()
X = df.values
X.shape
from sklearn....
0
Running models on all your available variables may not be the deal, since you'd have too much variables not directly linked to your result, and you might lose performance (you flood your model with unnecessary info, so it's lost and can't find direct and necessary one). You should start trying to produce the model starting with only a few variables you think ...
1
The y_pred vector should hold all predictions for your observations present in the X_test dataset, which should be 756 observations. If you want to use your model on the whole dataset you can simply use the .predict() method on your X_train dataset:
# predict on your training dataset
ann.predict(X_train)
# predict on your test dataset
ann.predict(X_test)
0
There are a couple scikit learn implementations of CatBoost and LGBM(not sure about this one) that are robust to nan values. I am sure catboost can handle nan values.
Top 50 recent answers are included