New answers tagged

0

I'm not very knowledgeable about this but here a some ideas: Statistical testing: in order to know whether there is any statistical difference between the values in two categories, one can use the Student t-test (for normal distributions) or the Wilcoxon test (for any distribution). In the case where there is a significant difference there are methods to ...


2

It is great you are thinking like this. Often when we build a model, we do not think how we will get the data for the next model build or the biases in this building model from previous models. In many models, like fraud or credit, the current model and business policy biases future data. I have built fraud models with event rates < 1%. Personally I do ...


2

There are a few things you can do. Keep in mind there are some people who would disagree with one method while others agree on the same. 1.) Since it an imbalanced dataset, you can apply any of the sampling techniques (either oversampling or undersampling). I would suggest oversampling (SMOTE) since undersampling leads to information loss. Now there are ...


1

Check this paper. Its introduction gives a very good definition of both: The classic approach towards the assessment of any machine learning model revolves around the evaluation of its generalizability i.e. its performance on unseen test scenarios. Evaluating such models on an available non-overlapping test set is popular, yet significantly limited in its ...


0

Your problem has 3 main sections as below: Text data (function description) as input. Looks like a multi-label & multi-class classification problem There is hierarchy/dependency between the two classifiers (Parent and sub category) Based on this information, I would suggest you have a look at BERT/transformers based multi-label & multi-class ...


0

Here a are a few things I'd look into: Are the categories balanced in training-2.json? Class imbalance is a well-known issue in ML development, particularly whenever the class distribution on the training set does not match the distribution on the test set. More interestingly, even if the classes are balanced (which, again, is a strong assumption that I ...


0

It is not the number of layers that matter, but the number of trainable parameters. This number should definitely be greater compared to the size of training data you have, and by size I mean: (number of samples)X(number of features). This number should at least be an order of magnitude greater compared to trainable parameters or lower: if your data set has ...


0

There are many ways to do this. For example, you could use pandas to cross-tabulate the label values. Note that, judging by your output, the true labels are actually the second column in your table. import pandas as pd df = pd.read_csv('labels.csv', header=None) df.columns = ['predicted', 'actual'] print(pd.crosstab(df.actual, df.predicted)) predicted 0 ...


2

A machine learning model (simple or neural net) is agnostic of the image type or the object depicted. That is because in both cases images consist of pixels (i.e a matrix of numbers with a certain range) and a CNN model, for instance, will identify a dog or a drawing based on the image's pixels pattern (in a very high level). Therefore, both tasks can be ...


1

Both are within one-vs-all scheme when there is a classification task. LabelBinarizer it turn every variable into binary within a matrix where that variable is indicated as a column. In other words, it will turn a list into a matrix, where the number of columns in the target matrix is exactly as many as unique value in the input set. If your input labels ...


0

Scikit-learn's LabelBinarizer converts input labels into binary labels, each example belongs to a single class or not. Scikit-learn's MultiLabelBinarizer converts input labels into multilabel labels, each example can belong to multiple classes.


0

You can't say you are not getting better performance after just checking 3 models. There are a whole lot of models that you can use with your dataset to get the best performing one. Also the data cleaning part can be done using different libraries (depending on the data). I don't know what your dataset looks like but I am sure you can try much more technique ...


1

At some time, I'll also require the assistance of programmers. He initially approached a private individual, but he worked on the project for a long time with no results. I needed to go a step further and work with a single global company https://www.avenga.com/industries/pharma-life-sciences/. The guys worked quickly and efficiently to complete the job! The ...


1

Here is a very simple way to do label encoding. import pandas as pd # Intitialise data of lists data = [{'Year': 2020, 'Airport':2000, 'Casino':5000, 'Stadium':9000, 'Size':'Small'}, {'Year': 2019, 'Airport':3000, 'Casino':4000, 'Stadium':12000, 'Size':'Medium'}, {'Year': 2018, 'Airport':5000, 'Casino':9000, 'Stadium':10000, 'Size':'Medium'},...


1

Since you have less data, please provide more data for training as try split like 80% 20%. If training accuracy is 100% then try increasing the dropout percentage. If training accuracy is still less than 100% then try decreasing the dropout percentage and add more convolution layer. Thanks


0

The Jaccard index or score is often used for bounding boxes or semantic segmentation in machine learning, i.e. in computer vision problems. Your problem is a classification problem using tabular data, and therefore this metric is not really applicable for this type of problem. Accuracy (and maybe even more so precision and recall) are more valuable metrics ...


0

your use case isn't entirely clear but if i may make some assumptions the company ID in both tables refer to the same company (so ID 0 is the same company in both table) you already have a good idea of feature engineering and know which algo to use for your final classification BUT i am going to treat it like i am modeling it if i were you, i would use 2 ...


2

The issue that you are running into is because you are using the fit_transform method on both your training and test dataset. The correct way of using a transformer is to use fit_transform only on the training dataset so that it learns the parameters and applies the transformation, and then use transform on your test set to apply those learned parameters to ...


1

Normalization is done only for numerical variables and One Hot Encoding only for categorical variables. I would advise split you data into 2 dataframes. One for numerical features and other contains only categorical features. Then perform Normalization and One Hot Encoding for respective datasets. That way you don't get confused about the order! Feature ...


1

One hot encoding is only for categorical features Normalization is only for numerical features Thus these two steps can be done in any order, they are independent. Feature selection should be done with the final set of features as used by the model, so it must be the last step.


1

Try this, col_names = list(tranformer.named_transformers_['one_hot'].get_feature_names())+numerical_features df1 = pd.DataFrame.sparse.from_spmatrix(transformed_X) df1.columns = col_names df1.head()


2

Functional API allows you to design more complicated models, including multi-output models. Check the documentation to see how you can connect specific neurons to others of your choice. You should be able to make custom layers from scratch. Once you build distinct output layers, probabilities within each can be set just as usual by using softmax activation.


1

Notice the terminology that precision and recall both depend on "positive" predictions and actual "positives". Both of the classes in binary classification can be considered as "positive". In the classification report that you shared, there are two classes: 0 and 1. Case 1: We consider 1 as the positive class. Here, predicted ...


2

Your confusion matrix does not correspond to your classification report. Also the matrix that you show is not standard: the labels "True Positive" and "True negative" are confusing because these terms should only be used for the classification status (see below). They mean "true class is positive" and "true class is ...


0

The usual approach with unbalanced classes is just to make the train and test sets as homogenous as possible. So make sure that proportions of the classes in both sets are the same. There are many factors that can be taken into account when splitting data, but I'm gonna guess that you just need the basic approach. In sklearn that would be any stratified ...


0

What percentages are you using for buy, hold and sell classes? From the data you share in the question, I am guessing it is a stock that has been going up rather than down for the most of the days. So, if you increase percentage cutoffs you have for the stock, you will have a balanced data. As you don't share the details in your question, let's assume you ...


2

First I will try to explain it in words - for me it always help to grasp the idea. So class precision supposed to measure how precise is your prediction given the class you predicted. For example lets say you want to predict 'rainy' or 'not-rainy' for tomorrow. It might be that when your model predicts 'rainy' the probability of being correct is higher than ...


1

Why do I get a tuple as output and not vectors of 1 and 0? You get this because by default OneHotEncoder() uses sparse matrix representation. Hence, it transforms the elements of y into elements of type - <1x3 sparse matrix of type '<class 'numpy.float64'>' with 1 stored elements in Compressed Sparse Row format> If you want the output as ...


0

You are correct. Sanity check: The final (incorrect) formulation of the optimization problem would not make sense because when you maximize over a certain variable, you essentially take that variable out of the expression. It's fixed. In that vein, I think Ng's notation would be more informative in equation (1) if they wrote the lagrangian as $L(\alpha)$ ...


0

The primary issue is that fowlkes_mallows_score is designed to evaluate clustering and you are trying to apply it to evaluate binary classification.


0

Q1. The equation is still valid if $\|w\|\neq1$, but the interpretation of $w_1$ as the (signed) distance from the origin is not. Q2. You haven't specified a learning algorithm, but for example with SVM, the popular libsvm formulates the problem(s) with $w$ not a unit vector, instead scaling so that $\|w\|$ gives the margin width. But also, quite often under ...


2

There is relatively little data for a deep learning solution - 220 total data points and 20 data points for each of the 11 labels. Increasing the amount of data would probably have the greatest impact on model performance. The best option would be to collect more data. Another option would be data augmentation.


0

The sign is just a matter of convention. If you use plus instead of minus, it simply flips the sign of the multiplier itself. The method of finding them is the same. I am not sure if I understand the second part of your question but the first equation is for the general case where the number of Lagrange multipliers can be more than one - if you have more ...


0

I am assuming, your focus here is the prediction accuracy and not interpretability? So, as there is a class imbalance, you can do two things: As suggested by the other user, you can use SMOTE or any technique. Use a non-parametric method that is more robust in handling the class imbalance. I tried to use Random Forest on your data, and the classification ...


0

You can look into SMOTE & ADAYSN techniques. This will help you in reducing the imbalance in the dataset by creating synthetic data https://medium.com/coinmonks/smote-and-adasyn-handling-imbalanced-data-set-34f5223e167


2

First, you can notice that the points which are satisfying the constraint are the surface of a norm-ball. Hence they don't form a convex set. Also, consider ∥x∥=1 and ∥-x∥=1. You can easily observe that (1/2)(x+(−x)) has 0 norm. So, it is not closed under convex combination.


2

If your datasets are random (with no real connection between the class and predictive variables), then "the right" model is a constant one: in (A), the predicted probabilities should be roughly $0.3, 0.2, 0.5$, whereas in (B) they should be $0.33, 0.33, 0.33$. When making the hard classifier then, in (A) the maximum probability will nearly always ...


1

I don't think there is a built-in loss function for what you want - I had the same issue a few years back and I found a custom loss function for this purpose. It is called Ordinal Categorical Classification problem. I have not checked this in a while now but I believe it is still not implemented in Keras. You can also check this cross-validated question and ...


0

Compute class weights for the labels in the dataset and then pass these weights to the loss function so that it takes care of the class imbalance. In PyTorch it can be done as shown below: from sklearn.utils.class_weight import compute_class_weight #compute the class weights class_weights = compute_class_weight('balanced', np.unique(train_labels), ...


0

This problem is called text classification (it belongs to the more general case of document classification). There are plenty of resources online about this, e.g. here, here or here. There are also a lot of research papers on the topic. General text classification consists in two steps: Represent the text as features Train a classification model The first ...


0

Your "adviser" can use the correlation between the explanatory variables and the explained variable. You can also use information provided by the p-value More details here : https://towardsdatascience.com/feature-selection-correlation-and-p-value-da8921bfb3cf


0

Try using stratify to see if the problem is solved. When splitting add the stratify parameter as train_test_split(x, y, test_size = 0.2, random_state = 69, startify = y).


2

In multiclass classification, the assumption is that every instance has exactly one class. Example: a poll asks people their favourite colour among blue (B), yellow (Y) or red (R). Each instance represents a person's answer, either B, Y or R. The "one vs. rest" method means that 3 binary classifiers are trained: "B" vs "not B", ...


1

There is nothing wrong with an imbalanced training dataset. It's possible that no changes are required. When your training set is highly imbalanced like this, models in early training stages will predict everything to be the most prevalent class (positive in this case). After a longer training period, usually the model moves out of this local minima and ...


Top 50 recent answers are included