New answers tagged

0

Assuming you mean sklearn.cluster.KMeans, you are able to pass in the initialization points using the init argument: init : {‘k-means++’, ‘random’}, callable or array-like of shape (n_clusters, n_features), default=’k-means++’ Method for initialization: ‘k-means++’ : selects initial cluster centers for k-mean clustering in a smart way to speed up ...


0

Looks the model is good, If training accuracy is 100% then try increasing the dropout percentage in the initial layers or reduce some hidden layers. If training accuracy is still less than 100% then try decreasing the dropout percentage in the last few layers of nodes with count as 32. Refer this page https://stats.stackexchange.com/questions/417055/dropping-...


0

If you are just dropping the columns like X_train = train1.drop(['Sales', 'Date', 'Customers'], axis = 1) will give a dataframe, when you use .values at the end then the result type will be numpy array. So please remove the .values also the reshape line won't be required. Thanks


0

I am assuming, your focus here is the prediction accuracy and not interpretability? So, as there is a class imbalance, you can do two things: As suggested by the other user, you can use SMOTE or any technique. Use a non-parametric method that is more robust in handling the class imbalance. I tried to use Random Forest on your data, and the classification ...


0

You can look into SMOTE & ADAYSN techniques. This will help you in reducing the imbalance in the dataset by creating synthetic data https://medium.com/coinmonks/smote-and-adasyn-handling-imbalanced-data-set-34f5223e167


2

If your datasets are random (with no real connection between the class and predictive variables), then "the right" model is a constant one: in (A), the predicted probabilities should be roughly $0.3, 0.2, 0.5$, whereas in (B) they should be $0.33, 0.33, 0.33$. When making the hard classifier then, in (A) the maximum probability will nearly always ...


2

Regarding the case when you have 'bad' category present in the entire dataset, I would recommend using sklearn.model_selection.train_test_split function, with stratify option set to a corresponding variable. If stratify option is set to a list of all categories, it will be guaranteed that every single category will be included in both training and testing ...


1

You can reserve a special ordinal value to indicate "unknown/unseen during training." You would use this special value for any and all values of x that you encounter in the test set and in production. In fact, scikit-learn's OrdinalEncoder does this for you via the handle_unknown parameter.


0

Actually, the answer was already provided by Oxbowerce, if the data is in a form of numpy array, use this code: import numpy as np arr = np.array(['A', 'B', 'C', 1, 2, 3, 4, 5, 6]) arr = arr.reshape([-1,3]) if the data is in a pandas DataFrame, the things get messy so I recommend you just converting it into the numpy array and reshaping it: import pandas ...


1

Use itertools doc for this purpose. Without knowing your exact codes I just made some lists up: import itertools as it nums = [x for x in range(37)] single = ["_"] abc = list('abcdefgh') codes = [f"123{x}" for x in range(20)] len(abc) * len(nums) * len(codes) # 5920 list(it.product(abc, single, nums, codes)) # len(...) -> 5920 This ...


0

While Tim Earhart has already provided the answer, I would like to add here there are cases when rather than using choosing df.mean() to substitute your NA values, it is better to choose df.median() - which calculates your median value. Mean is notorious for taking into consideration even the outliers. Since you are a beginner, you might want to try both.


0

BorutaShap is looking for input that has the columns attribute. Try converting your data to a Pandas dataframe. If you try that, you'll likely also discover that BorutaShap wants your data's columns attribute elements to be in string format. import pandas as pd data = pd.DataFrame(data) data.columns = [str(i) for i in data.columns] # or use the actual ...


1

The problem is closer to bin packing - fix capacity container and the goal is find the minimum number of containers. You might want to looking into inverse bin packing problem - both the number of bins and their sizes are fixed, but the item sizes can be changed. First-fit-decreasing bin packing algorithm might work.


1

It seems that the reason why you're getting the better result has nothing to do with cross-validation, but rather with weight adjustment happening during calculation of cross-validation result in train_decode function. To check this, I have used the following simulation, results = {"beta": [], "folds": []} noises = np.arange(0,1,0.1) ...


1

The model learns the significant patterns that it finds in the data. If the model is fed with noisy/inconsistent data, it might learn the noise. The solution is simply to clean up the data, in order to avoid feeding noise to the model. If I understand your scenario correctly, it's possible to detect whether an instance contradicts the business rules. ...


0

Normalizing the entire dataset for a single new observation may not be practical. If normalization gives a value outside [0, 1], consider using 0 or 1 (as the case may be) as an approximation. Usually, it is sufficient. Do remember to flag the event with appropriate markers and alarms so that the risks are known to the users of the prediction. If these ...


0

Normalization is a transformation of the data. The parameters of that transformation should be found on the training dataset. Then the same parameters should be applied during prediction. You should not re-find the normalization parameters during prediction. A machine learning model maps feature values to target labels. If you should not change feature ...


1

A few comments: I don't know this dataset but it seems to be a difficult one to classify since the performance is not much better than a random baseline (the random baseline in binary classification gives 50% accuracy, since it guesses right half the time). If I'm not mistaken the majority class (class 1) has 141 instances out of 252, i.e. 56% (btw the ...


2

In multiclass classification, the assumption is that every instance has exactly one class. Example: a poll asks people their favourite colour among blue (B), yellow (Y) or red (R). Each instance represents a person's answer, either B, Y or R. The "one vs. rest" method means that 3 binary classifiers are trained: "B" vs "not B", ...


1

The normalization is straightforward division by the sum of the probabilities: source. As to why, it's obviously desirable to have a sum of 1, but beyond that it's perhaps subjective. There's a discussion here suggesting that simple normalization outperformed other aggregations, but if I'm reading the linked paper correctly, it was a marginal gain and ...


0

There are a couple scikit learn implementations of CatBoost and LGBM(not sure about this one) that are robust to nan values. I am sure catboost can handle nan values.


2

I faced a similar situation regarding the imbalance among all possible classes when building a topics classifier in my current company, in this case for a callcenter conversations dataset. As @Erwan also suggests, I would try to: focus first on a model for the most frequent classes (as analogy, in my case it was about modeling first only the main topics); ...


3

With 200k instances, a class which has less than 10 instances represents less than 0.005% of the data. It's very unlikely that a model can learn to distinguish all these classes, especially the smallest ones. Data augmentation is really not recommended with text. Text is very diverse so there's no way to generate a good representative sample from a few ...


2

The sklearn implementation of AdaBoost takes the base learner as an input parameter, with a decision tree as the default, so it cannot modify the tree-learning algorithm to short-circuit at a "good-enough" split; it will search all possible splits. It manages to be fast at that because the tree learning is done in Cython. Another option for ...


3

I fully agree with @Erwan that this is a very bad idea. All performance estimates based on a finite sample of data will have some amount of variation caused by sampling from an underlying population, but this is just "noise". In terms of test-training splits, if one test training split gives you better results than another, it probably means that ...


2

It would be a very bad idea to select the "optimal split". The goal of cross-validation is to evaluate the model more accurately by minimizing the effect of chance due to the splitting. Selecting the "optimal split" goes against the idea of reliably estimating the performance, in fact this would purposefully overestimate the model. It's ...


0

As I found on stackoverflow something similar that did the job for me, here is the link [StackOverflow Link][1]. Which I'm citing " I think I found the mistake. You are rescaling your train and test data with the ImageDataGenerator. But you are not doing that when testing a single image. Try this: #Making new Predictions import numpy as np ...


1

The problem seems to be with the discrete_features flag inside mutual_info_regression. If you remove it completely (or set it to 'auto') it will work fine!


0

According to the glossery entry for partial_fit: Generally, estimator parameters should not be modified between calls to partial_fit, although partial_fit should validate them as well as the new mini-batch of data. In contrast, warm_start is used to repeatedly fit the same estimator with the same data but varying parameters. So the practical implication of ...


0

Maybe you could use Matrix Profile (see Matrix Profile Foundation) to solve your problem. A Matrix Profile is a new time series that measures the similarity of one time series to another, or a "self similarity" measure of parts of a time series to other parts of the same series. There is a parameter, usually called sub_len, which could be adjusted ...


0

is it the precision= 56% or 25% and also for recall and f1-score ? No, because precision, recall and f1-score are defined only for binary classification, and this report is about a multi-class classification problem (with 8 classes). Note: in order to understand this kind of classification report one needs to first understand how things work in a confusion ...


0

No - There is no way to get character level embeddings for SpaCy. One option for character level embeddings is the flair package which implements Contextual String Embeddings for Sequence Labeling.


0

Here is a solution by using bisect Python standard library from bisect import bisect from random import sample data = sample(range(10_000), 1_000) breakpoints = [1, 5, 25, 50, 150, 250, 1_000, 5_000, 10_000] buckets = {} for i in data: buckets.setdefault(breakpoints[bisect(breakpoints, i)], []).append(i) this will result in a dictionary with ...


0

You can approach the problem in following 2 ways: You may convert the bs4.ResultSet to a list object (if you are more familiar with Python lists) and then by calling the pop function of Python lists to store the value in the new variable. OR You may simply call the pop function of bs4.ResultSet and store the value in a new variable. Either way once you ...


0

Preprocessing.LabeledEncoder() is to encode the target y only accordinf to scikit doc. To encode Nominal Feature you should use sklearn.preprocessing.OneHotEncoder. But beware if unique values are too numerous, this may be a problem. doc here: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing....


3

This specific format to me looks like graphviz. So if you can extract the tree edges from your original object, then you can render it, example below (some roundabout to convert between different objects): import networkx as nx import pydot import graphviz # Just a part of your graph G = nx.Graph() ed = [('n3','n0'), ('n0','MusicHendrixA'), ('n0'...


4

Disclaimer: This is actually a tentative explanation, it provides a possible answer, but it does not contain proof. First of all, contrary to added comments, cosine similarity is not always in the range $[0,1]$. This range is valid if the vectors contain positive values, but if negative values are allowed, negative cosine similarity is possible. Take for ...


1

I suppose you want to "stack" multiple models? If so, you can use sklearn.ensemble.StackingClassifier. Example from the docs: from sklearn.datasets import load_iris from sklearn.ensemble import RandomForestClassifier from sklearn.svm import LinearSVC from sklearn.linear_model import LogisticRegression from sklearn.preprocessing import ...


2

As I understand this, what I've described so far is stochastic gradient descent. Without any way to generate a gradient or any mention of using gradients, this is not gradient descent. Your choice of the word "mutate" plus terms like "parents" and "breed" would lead me to believe that you initially want to train your neural ...


0

Try using Erode and Dilation Operations to remove the white area.


0

You pretend the out-of-sample data set does not exist. Any feature manipulations should be based on the in-sample (training) data. If your in-sample data set is $\{1,2,3,4,5,6\}$ and your test set is $\{1,3,7\}$, you would do your normalization based on the in-sample set and accept that the $7$ in the test set will be normalized to a value exceeding $1$. ...


0

No - using a recommender system to predict watch duration is not going to work well. Watch duration is a continuous target so it would be more useful to model your scenario as a regression problem. Try to the find the user and video features that are associated with differences in watch duration time.


0

This would need some data preprocessing: Get the different main categories (ex: bikes and car) If there are several mix of categories, get the quantities of each configuration in order to get the right proportions of the samples (see 4). Get random sample within each category (10% bikes and 10% cars) Ensure that those samples have the right quantity in ...


1

I had the same question, and according to this link: Grey represents the categorical values which cannot be scaled in high or low. Concerning the other questions, I found this link: https://github.com/slundberg/shap/issues/960 Where slundberg states: In the linear model SHAP does indeed give high importance to outlier feature values. For a linear (or ...


1

The data description says: Note: Each of these 10 feature variables have been mean centered and scaled by the standard deviation times n_samples (i.e. the sum of squares of each column totals 1). https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html For more information see: Bradley Efron, Trevor Hastie, Iain Johnstone and Robert Tibshirani (2004) "...


0

I found the solution, using the built-in breakpoint() function as FLAK-ZOSO mentioned here https://stackoverflow.com/questions/69062229/how-to-set-a-breakpoint-inside-a-custom-metric-function-in-keras?noredirect=1#comment122059117_69062229


0

There are many options. Here a couple: Values that very high or very low. Values that are many standard deviations from current distributions. Multivariate anomalies - the combination of values across features are anomalous


0

The original problem might be too complex for neuroevolution to learn. You can train in easier version then progressive make the examples more difficult. This is commonly called curriculum learning.


1

You don't need to make preprocessing as I understand, and the reason for this is that the Transformer makes an internal "dynamic" embedding of words that are not the same for every word; instead, the coordinates change depending on the sentence being tokenized due to the positional encoding it makes. Note the difference with Word2Vec, GloVe or ...


0

Took a bit of time, but I understand what I was asking. The process of kNN is to find the nearest "majority" major (determined by param k). My misunderstanding came from interpreting what the training set was being used for. The training set and the test set both are mapped in the same feature space. To classify the test set images, a distance (...


Top 50 recent answers are included