New answers tagged

0

There are many options to spend this up: Get a better CPU. Distribute the process across a cluster since each document is independent. Reduce the size of the vocabulary. If only the top-n most popular words are used, it greats reduces the size of the data. Reduce the size of the embedding space. Switch to doc2vec so the document themselves are a learned ...


0

Your model is overfitting to the training data. You are adding more data to training data but the model is overfitting to that additional data. To reduce overfitting, you need to increase regularization. Common options: Keep adding data to the training dataset until you cover all possible scenarios. Add data augmentation. Increase dropout.


0

I'd say yes : if you have mislabeled data in train, the model won't learn correctly, and if you have them in test/val, your test results won't represent the actual model results. Since that's directly the label, that's ok to correct/remove them, since you'd never have the label in real new cases. Moreover, if you encounter the same problem with a variable, ...


1

There is no fixed rule while selecting the size of the training set and testing set. Its all about trial and error, so try out different ratios 80-20, 70-30, 65-35 and pick one that gives the best performance result. Its suggested in several machine learning research articles to generally opt for Training dataset to be 70% (for setting model parameters) ...


3

Epoch One epoch leads to underfitting of the curve in the graph (below). Increasing number of epochs helps to increase number of times the weight are changed in the neural network and the curve goes from underfitting to optimal to overfitting curve. Number of epochs is related to how diverse your data is. Read this article for better understanding Batch ...


0

Primarily, go for CV for the training and test set. If you still get the same type of result, then choose the second model. The first model has a very large difference in accuracy between the training and test set. It is a very specific model. There is a chance that the high accuracy on the test set appeared due to data leakage. The second model is a more ...


0

In theory, the very first thing to do should be fixing noises and seasonality. There are various ways of approaching the noises. If possible, try to understand the reason for noisy samples and extract the noise part from them manually. Or, use the smoothing algorithms to minimize the noises. In addition, you can create a dummy predictor variable that will be ...


0

It seems it currently has an issue with data leakage in the cross validation function: https://github.com/microsoft/LightGBM/issues/4319


2

There is no fixed rule while selecting the size of the training set and testing set. Its all about trial and error, so try out different ratios 80-20, 70-30, 65-35 and pick one that gives the best performance result. Its suggested in several machine learning research articles to generally opt for Training dataset to be 70% (for setting model parameters) ...


0

This might help, Size of the training set and test test should not be in similar size why ? because what ever the model your are testing it will fist applicable on the training set equal size of data will create noise. In the machine learning world, data scientists are often told to train a supervised model on a large training dataset and test it on a ...


0

The loss is the sum of errors based on your train and validation datasets, whereas the accuracy is the percentage of good results obtained with the validation dataset. So, if your loss is decreasing and your accuracy is increasing, it means that it has worse results in your train data sets but better ones in your validation dataset. There is probably a way ...


0

The train_test_split function returns four arrays with the first two arrays being your input arrays and last two being the values you're trying to predict. You should therefore reorder your variable assignment on your second line as follows: X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)


0

Your options are: Do nothing with the model. (Then why did you build it?) Use the model, knowing that it works well enough, even though it makes mistakes. (Evaluating if it works well enough is a separate discussion.) Speech recognition software like Siri make mistakes, yet the software remains useful. The alternative is not to have any speech recognition ...


0

In short I would say: stick with the old probably manual process if it suits you better or try to understand why the result is only 85% and improve the model/data quality try some other models, parameters, engineer other features get more and cleaner data or hire a top DataScientist who could help with that issue When deciding if 85% is enough or not, you ...


0

This is where so-called term Baseline comes into play. One needs to have a baseline either a simple model prediction performance (accuracy, precision or recall whatever) set, and try to improve upon it. Or in a more natural way, when available, it is best to have a human baseline. The latter is quite common in industries, but it is more costly to obtain ...


0

It is based on the output classes if they are mutually exclusive or not. For example in a multi-label classification problem, we use multiple sigmoid functions for each output because it is considered as multiple binary classification problems. But if the output classes are mutually exclusive. In this case, the best choice is to use softmax, because it will ...


Top 50 recent answers are included