New answers tagged scikit-learn
0
The way you are imputing your feature can't be replicated in the test set, because it needs knowledge of the target classes!
You need to select a different imputation strategy, that doesn't rely on your target feature.
Assuming that you are using another feature, the same way you were using your target, you need to store the value(s) you are imputing each ...
0
Something's wrong with your feature selection tool: p-value is NaN, confidence interval includes $0$. Confusion matrix shows that all observations are predicted as Class 1. How many explanatory variables do you have? Try using all of them instead of just one. Are you sure
logit_model = sm.Logit(y_train, X_train).fit()
is correct? Shouldn't it be the other ...
0
It depends on the approach you take for label encoding. If you want to use .cat.codes then you need to convert it into category datatype.
You can also use sklearn labelencoder which does this inherently bcoz it can be used to transform non-numerical labels (as long as they are hashable and comparable) to numerical labels.
0
It is not the number of features that is the problem with Gaussian Naive Bayes (GaussianNB). It is the decision boundary that GaussianNB is learning. Naive Bayes is constrained to the learn the marginal distribution of the data because "naive" assumption. Often times the conditional distribution is useful to make predictions. Given the performance ...
3
The intersection of the precision and recall curves is certainly a good choice, but it's not the only one possible.
The choice depends primarily on the application: in some applications having very high recall is crucial (e.g. a fire alarm system), whereas in some other applications precision is more important (e.g. deciding if somebody needs a risky medical ...
0
The other answers make sense but I would be more categorically negative about the idea:
Is this approach a correct approach, or logical with respect to machine learning principles ?
No, it's not. The parameters of a ML model (whether supervised or unsupervised) are estimated using a particular set of features designed as the input for the problem. Changing ...
3
Is this approach a correct approach, or logical with respect to machine learning principles?
It will affect the performance of the model in the sense that your algorithm learned to separate the clusters based upon distance according to all the features. I have read discussions about how to calculate feature importance on unsupervised problems like yours, so ...
1
Interesting question. The answer is: It depends.
The best way to find out how it would affect your model is with the shap package. You can use it to uncover the importance of features and reveal interaction effects in the model.
There could be a very different effect depending on how „important“ the excluded features are.
Let‘s assume a very simple decision ...
0
By using .values in your definition of X, you've converted to a numpy array and lost the column names. Just removing that, you'll provide a frame to SRS, and mlxtend will use the column names in the k_feature_names_ attribute; so that's probably the best approach.
There are two other approaches: one is to add the custom_feature_names parameter of mlxtend....
0
Based on this scikit-learn documentation, you can get a boolean mask (in the same order) of the input features, via the get_support method:
2
you can try below approach
centroids is a matrix with all cluster centers
centroids=[[20,40,60,80],[60,120,180,240],[100,200,300,400]]
TestData_vector=[130,170,250,300] #you new test data as a vector
import numpy as np
from sklearn.metrics.pairwise import euclidean_distances
euc_res=euclidean_distances(np.array(centroids), np.array([TestData_vector]))
# ...
1
News sentences will have more unique tokens than normal conversations.
Conversations have more stop words than news articles.
I think you can use bert or normal wordvect classification to train a baseline model here.
I would play aroud the pipeline of fake news classifier and news-conversation classifier. like passing the text to news classifier first and ...
0
It is really hard to figure out the logic behind what you are doing, it look odd
But assuming you are trying to apply a preprocessing step to a data frame I would go as follows:
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import OrdinalEncoder
ordinal_features=['LotShape','ExterQual','ExterCond','BsmtQual','BsmtCond',
'...
0
In the documentation categories parameter is explained as: categories[i] holds the categories expected in the ith column. The passed categories should not mix strings and numeric values, and should be sorted in case of numeric values.
You can pass a two dimensional array to the categories parameter in which each element of the array is an another array holds ...
1
Sklearn provides a predict function for the KMeans object. So something like this should work:
model = KMeans(clusters=2, random_state=42)
model.fit(X_train)
# get centroids
centroids = model.cluster_centers_
test_data_point = pass
model.predict([test_data_point])
KMeans assigns data points to clusters is by calculating the Euclidean distance between ...
1
If you want to apply the result of fit_transform, you must assign to your columns.
columns = ['S_LENGTH', 'S_WIDTH', 'P_LENGTH', 'P_WIDTH']
min_max = preprocessing.MinMaxScaler()
df[columns] = min_max.fit_transform(df[columns])
df.head()
Output
ID S_LENGTH S_WIDTH P_LENGTH P_WIDTH SPECIES
0 1 0.0 0.0 1.0 0.0 VIRGINICA
1 ...
0
You can use df.select_dtypes(exclude='object') to exclude categorical columns.
Also while importing dataset, set the index_col='ID' to use ID as index instead of column.
0
GridSearchCV does allow param_grid to be a list of grid-dicts, which sometimes is sufficient. In this case, separate grids are generated and their union is searched.
There isn't quite a convenient implementation by which you provide your own list of hyperparameter points. But looking at the source code for GridSearchCV, you'll notice that it's amazingly ...
2
You can absolutely get this information, but not from the confusion matrices. You want to be comparing the prediction vectors themselves, not the confusion matrices, because as you've rightly identified, the confusion matrix dumps all the false postives / negatives into the same buckets (so that we can see how full each bucket is, also useful information).
...
2
Despite the fact that I'm not sure why it's that, it's seems that's normal. Please see the example from sklearn documentation: https://scikit-learn.org/stable/modules/impute.html#multivariate-feature-imputation
import numpy as np
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
imp = IterativeImputer(...
1
You are using the right method but in a wrong way :)
nan_to_num is a method of numpy module, not numpy.ndarray. So instead of calling nan_to_num on you data, call it on numpy module giving your data as a paramter:
import numpy as np
data = np.array([1,2,3,np.nan,np.nan,5])
data_without_nan = np.nan_to_num(data)
prints:
array([1., 2., 3., 0., 0., 5.])
In ...
1
You can also use frequency encoding in which you map values to their frequencies
Example taken from How to Win a Data Science Competition from Coursera,
eg. for titanic dataset :
encoding = titanic.groupby('Embarked').size()
encoding = encoding/len(titanic) // calculates frequency
titanic['enc'] = titanic.embarked.map('Encoding')
This will preserve ...
1
That is not a regression problem. Thus those results can not be interpreted.
It should be framed as probabilistic binary classification. The target is binary because there are two outcomes - disease or not. Probabilistic because it on a scale from 0 (not possible) to 1 (completely certain).
1
You can pass in a list of file paths into scikit-learn's train_test_split.
Here is an example:
from pathlib import Path
from sklearn.model_selection import train_test_split
file_locations = [Path("images1.png"), Path("images2.png")]
X_train, X_test = train_test_split(file_locations)
The choice between loading data all-at-once vs batch-...
1
The point of sample_weights is to give weights to specific sample (e.g. by their importance or certainty); not to specific classes.
Apparently, the "balanced accuracy" is (from the user guide):
the macro-average of recall scores per class
So, since the score is averaged across classes - only the weights within class matters, not between classes.....
0
This depends largely on the software. sklearn classifiers will know not to treat label-encoded data as ordered; that said, most/all of them will take the raw string data just fine (and in many cases will use a LabelEncoder internally, for computational efficiency). If you one-hot encode multiclass data, sklearn will generally think your problem is ...
0
What you want to do is teach a model how to predict something using your train set, and test it, like in real conditions, with a test set.
For that you have to provide the model some train data associated with the known result, so the model can learn which patern is usually labeled 1 and which one is usually labaled 0. So you have to fit your model giving ...
0
When you are fitting a supervised learning ML model (such as linear regression) you need to feed it both the features and labels for training. The features are your X_train, and the labels are your y_train. In your case:
from sklearn.linear_model import LinearRegression
LinReg = LinearRegression()
LinReg.fit(X_train, y_train)
If you are performing some ...
0
As of version 0.24, it does!
Announcement, documentation
2
1. what are the rows before the first red row? I thought it may be the combinations of parameters but that doesn't make a lot of sense because those are not enough
Parameters, which are the candidates of the CV are printed
2. What is the meaning of the row between the red rows? Why those parameters are there? is this after one CV?
It is printed after the ...
0
MLPerf is a set of benchmarks designed just for that purpose. It tests a variety of common machine learning tasks across a variety of possible hardware.
1
My two cents: a general way to think about this process is in terms of learning and transformations. Scaling (standardization) is a transformation that you apply to every sample both in your training and test/validation/production set. These transformations are done using parameters that are learned using the training set.
The aim of up/down sampling is to ...
1
Having 2 uncorrelated normal variables $x_1 \sim N(0, 1)$ and $x_2 \sim N(0, 1)$, one can correlate them through a linear transformation:
$$A = \begin{bmatrix}a & b \\ c & d\end{bmatrix}$$
$$X' = AX$$
The covariance matrix of the transformed correlated variables $X'$ is given by:
$$\Sigma' = A A^T$$
For a rotation by an angle $\theta$, and scaling ...
0
It has to be saved somewhere i.e. Database after the training is done.
The saved values should be used on new data and all these steps should work in a loop i.e. when you re-train again the values will be updated and saved again.
e.g. If we see the Keras pre-trained models, it provides the necessary pre-processing function. We can directly use that
from ...
0
I tried to fix it like this, not sure if it is correct.
from sklearn.utils.multiclass import unique_labels
target_names_filtered = [twenty_test.target_names[i] for i in unique_labels(twenty_test.target[:n_samples], predicted)]
then i used target_names_filtered as the target_names param.
2
Yes, a GBM with Huber loss initializes with the median. The relevant bit of code is the method init_estimator of the loss class, in the file _gb_losses.py. For HuberLossFunction:
def init_estimator(self):
return DummyRegressor(strategy='quantile', quantile=.5)
(source)
1
Demonstrating and visualizing cluster centroid updates by using the first four iterations in Python?
You can use the plt.scatter() and plt.subplots() to achieve this as follows:
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
data = make_blobs(n_samples=200, n_features=8,
centers=6, cluster_std=1.8,random_state=101)
fig, ax = plt.subplots(nrows=2, ncols=2,figsize=(10,10))
from sklearn.cluster import ...
1
For documentation, check the Scikit-Learn code at Github i.e. line#535 [Link]
value : array of double, shape [node_count, n_outputs, max_n_classes]
Contains the constant prediction value of each node.
You can check the same using this sample code.
Identify the leaf nodes
Slice the value attributes for leaf nodes
from sklearn.datasets import load_diabetes
...
0
Demonstrating and visualizing cluster centroid updates by using the first four iterations in Python?
You could either write a KMeans algorithm yourself such that you can perform each update step-by-step and easily incorporate the plotting of the points into your code, but could probably also use the KMeans implementation from scikit-learn. To stop the algorithm to fully converge you could limit max_iter to [1, 2, 3, 4] since you want to plot just the first ...
0
The decision depends on the loss function used. If you are using the Mean Square Error you use the within-leaf mean. The fastest way is to use the .apply() method and estimate the average of observations that fall in the same leaf.
1
You can perform a statistical test to confirm your data is normally distributed
Try:
from scipy import stats
np.random.seed(42)
x = np.random.normal(2, 1, size=1000)
k2, p = stats.normaltest(x)
alpha = 0.001
print("p = {:g}".format(p))
if p < alpha: # null hypothesis: x comes from a normal distribution
print("The null hypothesis can ...
0
The split actually happens with mini gini or entropy. See if you have numeric values first arrange the values in ascending order, let's say we have values from 60 to 220.
Arrange in ascending order, first it calculates >60 and less than < 65 , so on. And calculates gini for every split and does the minimal split. It's not random.
1
There are two estimators i.e. The initial predictor and the sub-estimators
init_estimator
The estimator that provides the initial predictions. Set via the init argument or loss.init_estimator.
estimators_
ndarray of DecisionTreeRegressor of shape (n_estimators, 1)
The collection of fitted sub-estimators.
Prediction after the first (i.e. init) estimator is ...
1
It's hard to say without your data. I'd comment this, but it's a bit long and better formatted as an answer.
The part of the source code that's relevant is quite short, so you can go through step by step to see what's wrong:
bins = np.linspace(0., 1. + 1e-8, n_bins + 1)
binids = np.digitize(y_prob, bins) - 1
bin_sums = np.bincount(binids, weights=y_prob, ...
Top 50 recent answers are included
