New answers tagged scikit-learn
1
My guess would be that in the following piece of code:
for i,j in enumerate(indices):
temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
matches.append(temp)
The variable (array element):
distances[i][0]
contains the top element match for the $i^{th}$ line. Replace the second array index so it becomes:
distances[i][1]
...
-1
To Pieter21's point, it may not a good idea to split the data by year since the year itself could an important feature.
Without more context, you can use sklearn.model_selection.train_test_split twice as explained in this post->Train/Test/Validation Set Splitting in Sklearn
If you do have a valid reason to split by year, you can use wmh's code.
0
Seems to me the best (or at least quickest) way to do this would be have all the data in a Pandas dataframe, then create masks based on year and create new dataframes for each group. Ex:
train_df = data[data['year'].isin(['2015', '2016', '2017'])
validate_df = data[data['year'] == '2018']
test_df = data[data['year'] == '2019']
Hope this is what you're ...
2
Problem is the way you're onehot encoding.
Best practice for any type of encoding :
You should train an estimator for Onehot encoding on the training data only, and when encoding test data, you should use the same estimator used on training data.
Eg : sklearn.preprocessing.OneHotEncoder does this, and it has a parameter called : handle_unknown.
...
1
At a high level you need to find historical data matching what you need to forecast (average temperature) and any factors which influence it. So you need to be a domain expert, have access to one, or find data and do exploratory analysis to find for yourself what factors are important.
If you're using 30% to train the model, you're holding 70% back to test ...
1
Do you have context where this problem originates from? You seem to be mixing up a few things.
Q1: depends on what dataset do you have available, or can you find? It can be just year and average temperature of the year, and then you'd apply a regression or ARIMA model.
But maybe you have average temperatures for various countries and regions, and then you ...
1
The 'StratifiedShuffleSplit' function takes parameters on how the split needs to take place and returns a function to do the split.
The 'split' variable in the first line is used to store this function. In Python, functions/procedures can be stored as variables.
'n_splits' indicates the number of folds.
'test_size' indicates the proportion of the test ...
0
Decision trees usually base feature importance on the impurity reduction achieved by splitting on the features. In classification a usual choice is gini impurity, while regression trees typically use the mean squared error or node sample variance. This is also the case in scikit learn.
For a given (binary) node $m$ with left and right child nodes the ...
0
The log data is forward propagation loss,still have a backward modify.
If use model train one time,you will get the data you wanted.
3
If the "cost" for experimenting is not really that big I suggest you take the time to experiment and take this as a learning opportunity and just try if it could actually work.
There are many approaches to address class imbalance and setting class weight is one of them and the easiest to implement.
Change loss function (for example to focal loss for ...
0
Alternatively, just implement a simple Grid Search algorithm yourself. The book "Introduction to Machine Learning with Python" by Mueller and Guido includes an example using an SVC:
# naive grid search implementation
from sklearn.svm import SVC
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, random_state=0)
print("Size of ...
0
I am not sure about pixelwise distance but what I could help is on applying KMeans on this picture. Let's say I give you this picture (I cannot get your original image so I'll just use mine).
Implementing KMeans on image is actually quite straightforward. What you might want to pay attention to is to the size of the image since big image like I what I am ...
1
If speed is the only only issue then i have few suggestions that will definitely improve the algorithm run time by 5-10times(which i experienced), without compromising on any other input:
1) Increase the number of jobs submitted in parallel, use (n_jobs = -1) in the algorithm parameters. This will run the algo in parallel instead of series(and will cut down ...
2
By passing a callable for parameter scoring, that uses the model's oob score directly and completely ignores the passed data, you should be able to make the GridSearchCV act the way you want it to. Just pass a single split for the cv parameter, as @jncranton suggests; you can even go further and make that single split use all the data for the training ...
1
There are a few ways of making this faster:
Decrease the CV value, as mentioned by @jncraton
Decrease the search space for the hyperparameters (test only a few parameters or decrease the ranges for parameters)
Additionally, you might consider using a more efficient way of hyperparameter searching by using hyperopt or nevergrad.
6
GridSearchCV is built around cross validation, but if speed is your main concern, you may be able to get better performance using a smaller number of folds.
From the docs:
class sklearn.model_selection.GridSearchCV(estimator, param_grid, scoring=None, n_jobs=None, iid='deprecated', refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', error_score=nan, ...
4
Sure, here is a skeleton in TF 2.0.
import tensorflow as tf
weights = tf.Variable(tf.random.normal(shape=(784, 10), dtype=tf.float64))
biases = tf.Variable(tf.random.normal(shape=(10,), dtype=tf.float64))
def logistic_regression(x):
lr = tf.add(tf.matmul(x, weights), biases)
return lr
def cross_entropy(y_true, y_pred):
y_true = tf.one_hot(...
0
As the new version of Pandas 0.24.0 you can change the code from .as_matrix() to .to_numpy()
Examples:
input: >>> pd.DataFrame({"A": [1, 2], "B": [3, 4]}).to_numpy()
output: array([[1, 3], [2, 4]])
1
As promised, here you can find an example of how you could apply kfold cross validation for a defined convolutional neural network model, applied to an augmented dataset. You can find the code as a simple gist here
It is done as follows:
for a subset of the CIFAR10 images dataset, generate 3 augmented images (by applying horizontal_flip) per original image, ...
0
Update:
One thing that worked the best with my data was converting the words into tf-idf vectors per document and applying Naive bayes on it to predict the probability per document or word.
0
It looks like you need to add parentheses to your calls to .as_matrix(). Only calling as_matrix will return a bound method instead, not the data.
1
A direct way to find the words which are the most representative of a class is to calculate the probability of the class given a word:
$$p(c|w)=\frac{\#\{\ d\ |\ label(d)=c\ \land w\in d\}}{\#\{\ d\ |\ w\in d\ \}}$$
Ranking the words according to their probability $p(c|w)$ gives:
highest values: the most correlated words for the class
lowest values: the ...
0
See the section 6.1.4 in the documentation.
As per the documentation, whenever the transformer expects a 1D array as input, the columns were specified as a string ("title"). For the transformers which expects 2D data, we need to specify the column as a list of strings (["title"]).
1
I think here you must maintain the actual tf-idf and create corpus over it.. Assuming you already have lables for documents available. You can rum classification over it.
Best classification I am anticipating for this problem would be naive bayes..
1
As Sergey has pointed out on the original question, going through the following link one finds the reference to the original article. So, we're going to proced by unraveling the components of each formula (explaining each of them) and then doing a proper interpretation of the splitting criterion that Jerome Friedmann proposed.
1. Probabilities $p_{k}$
...
4
Searching the source code of Sklearn for SimpleImputer (with strategy= "most_frequent"), the most frequent value is calculated within a loop in python, therefore that is the part of code that is so slow. In the source code of SimpleImputer there is also the comment that explains why they do not use the scipy.stats.mstats.mode, which is mush faster:
scipy....
1
In R you can run a linear regression. Consider this "academic" minimal example:
df = data.frame(c(3,5,2,7,5,3), c(1,0,1,0,1,0), c(0,1,1,0,1,0))
colnames(df) = c("A", "B", "C")
df
Take this data as an example:
A B C
1 3 1 0
2 5 0 1
3 2 1 1
4 7 0 0
5 5 1 1
6 3 0 0
Now we can see how B and C describe A in the best way.
reg = lm(A~B+C, data=df)
summary(...
1
You can read the file line by line (or block by block), assuming it is in a format where that can be done, and keep track of the unique values of the category.
Can handle multiple fields in 1 pass of the data instead of the 1 field like the below code.
Of course the easiest way is to get more memory or move to a cluster. The line-by-line solution will not ...
0
The issue is your data violates the requirements of StratifiedShuffleSplit, specifically it's not possible to do a 70:30 split of the data and maintain the same number of distinct y values in the test and train set - most likely because you have a y value which only occurs 1 time?
Perhaps just use ShuffleSplit?
1
Lambda is a tuning parameter („how much regularisation“, I think called alpha in sklearn) and you would choose lambda so that you optimise fit (e.g. by MSE). You can do this by running cross validation.
This page (for the GLMnet package in R) explains how to apply Lasso in a very instructive way (alpha is the elastic-net mixing parameter here, Lambda is ...
0
There is nothing that guarantee the six features will be a subset of the 55. Intuitively, if you parametrize you problem with number of features instead of lambda, it is easier to see how two variables could better explain the output than a single different one.
Another problem that can arise from these regularisation methods come from correlated variables....
1
In multiclass classification each class is mutually exclusive, but in multilabel classification each class basically represents a different binary classification task.
An example.
Multiclass: Images that could contain a dog, a cat or a frog. Each image contains only one of the animals.
vs
Multilabel: Movie Genre Classification based on poster images. You ...
0
Create a new column based on your formula and train your decision tree.
Then, based on the outcome class of the tree, pass the data to different regression.
This would be a sort of Stacking.
Please treat this just an engineering approach, it might not be a good solution based on your data and need.
0
The F1 measure is a type of class-balanced accuracy measure - when there are only two classes, it's very straightforward, as there's only one possible way to compute it. With 3 classes, however, you could compute the F1 measure for classes A and B, or B and C, or C and A, or between all three of A, B and C.
It seems that the "labels" parameter is telling ...
0
The metrics differ in the macro case because macro in scikit-learn ignores class imbalances. micro in scikit-learn looks at global rates. The data has class imbalances, thus the macro argument will return different values compared to micro.
1
This method was researched in the 90s and I'm afraid that it wasn't very successful. Do a search on treed-regression.
1
Mutually exclusive classes are usually integers; in your case, win=0, draw=1, loss=2 (for instance).
It is mostly a convention and is convenient for implementation.
1
Yes, you do not need to fit or train explicitly when using cross_validate(). Also see this example from the SKLearn documentation:
from sklearn import datasets, linear_model
from sklearn.model_selection import cross_validate
from sklearn.metrics import make_scorer
from sklearn.metrics import confusion_matrix
from sklearn.svm import LinearSVC
diabetes = ...
0
This has already been answered in stackoverflow. This is a summary of the answers:
R^2 is bounded above by 1.0, but it is not bounded below, so it's Ok that you get negative values.
From your code, it seems you are invoking sklearn.metrics.r2_score correctly, i.e. r2_score(y_true, y_pred).
The cause may be in the data, e.g. if the mean of your test data is ...
3
You dont use self.k in your closest_n method (for starters).
U dont use it at all, you just initialise the property.
I would check sklearn implementation, just look under the hood here
0
The way the scikit-learn MinMaxScaler works is:
fit operation: finds the minimum and maximum values of your feature column (mind this scaling is applied separately for each one of your dataframe attributes/columns)
transform: applies the min max scaling operation, with the values found in the 'fit' operation
Worked example:
let's assumme we have one ...
7
I think the easiest way to do this would be to have a decision tree where the final decision results in a linear formula.
Setting aside whether this is actually easiest/best, this type of model does exist, usually called "model-based recursive partitioning". See e.g. https://stats.stackexchange.com/q/78563/232706
There are several packages in R for this: ...
2
Even after your update, I think Noah's hint to spline regression is the best way to approach the problem. Here is a brief example in R:
# Generate data
x <- -50:100
y <- 0.001*x^3
plot(x,y)
df = data.frame(y,x)
# Linear regression
reg_ols=lm(y~.,data=df)
pred_ols = predict(reg_ols, newdata=df)
# GAM with regression splines
library(gam)
reg_gam = gam(...
5
I would suggest using spline regression. or some polynomial regression.
Why? what you are basically approximating is the (almost) step-wise function. See here
More info in here and a great background introduction in here.
0
Yes, it is possible. You don't really need a new column, you could just convert your existing columns to their representative weights.
I would suggest for you to:
Convert your [0,1] to [-1,1];
Multiply by a float weight in the range [0.0, 1.0].
I suggest you change the range from [0, 1] to [-1,1] because of the following example where it is not possible ...
1
In addition to the comment made by Oxbowerce, you can reason about it as follows:
in a real case, you would expect the distribution of your X_train data to be similiar to the X_test data, so applying the MinMaxScaler transformer to the X_test data fitted "only" on X_train means (or should mean) no actual difference compared to fitting it also with X_test; ...
1
If you would fit the scaler using x_test you would be using information from your test set and would be leaking data. This is information that you would not have if your model was in production and can therefore not use when fitting your model.
1
#Taking care of missing data
from sklearn.impute import SimpleImputer
imputer=SimpleImputer(missing_values=np.nan,strategy='mean')
imputer=imputer.fit(X[:,1:3])
X[:,1:3]=imputer.transform(X[:,1:3])
Result
array([['France', 44.0, 72000.0],
['Spain', 27.0, 48000.0],
['Germany', 30.0, 54000.0],
['Spain', 38.0, 61000.0],
['Germany'...
3
A few things going on here:
Your matrix is 100x100. So you have no degrees of freedom left in a linear model, which will cause $R^2=1$. See this post.
You use random numbers. Thus, they should make little sense in terms of explaining your dependent variable (it's basically noise). Since Lasso "shrinks" parameters which are not useful (and none is useful ...
2
From sklearn's documentation for the score function
Returns the coefficient of determination R^2 of the prediction.
R^2 is a measure of how well the variability of the data is explained by the model. So, at 0.97, the model is able to explain that really well. And yes, it is a measure of accuracy for regression models. However, you cannot construct ...
Top 50 recent answers are included
