Questions tagged [training]

Training is the part of machine learning whereby a model is "trained" on a define portion of a dataset to learn attributes and statistical features of the data. It's counterparts are called Testing and Validation. After training a model is tested and validated on another portion of the dataset.

Filter by
Sorted by
Tagged with
3 votes
1 answer
2k views

Does validation data has any effect on training or it acts solely without affecting the training?

When using Keras library of Python, we use validation data with training data while training our model. In every epoch, we get a validation accuracy. Does this validation accuracy have any effect on ...
Rawnak Yazdani's user avatar
1 vote
0 answers
68 views

How to do online retraining of model on a single new data point/observation?

I am trying to investigate the effect on performance on old data and new data when a classifier is retrained on only the new observation when it is encountered. The aim is to retrain the classifier on ...
Abanoub Ghobrial's user avatar
2 votes
0 answers
90 views

When to stop the final model training?

Let's say I'm participating in a Kaggle image recognition competition. Firstly, I create a train/validation split and find the good hyperparameters for my model. Here the stopping criterion is when ...
SpaceCossack's user avatar
0 votes
2 answers
455 views

how to correct mislabeled data in training, validation and test set

In an image classification task, I know there are mislabeled data. should I remove/correct them in all training / validation / test set ? I saw this article https://arxiv.org/pdf/2103.14749.pdf but I ...
user702846's user avatar
0 votes
1 answer
320 views

Best choice for splitting data given a quantity and a expected accuracy

I have a dataset with at least 1,000,000 images (from IDs) which I am using to detect the presence of sealed IDs. The legacy algorithm got nearly 60% accuracy, but my current algorithm yielded almost ...
Manu's user avatar
  • 103
1 vote
0 answers
221 views

YOLO: how to deal with object partially in training image

I am trying to train a YOLO (v3) network on a set of images, but I am faced with a problem when preparing the training set. Sometimes, an image contains my object of interest, but only half of the ...
S.I.'s user avatar
  • 11
0 votes
1 answer
692 views

How can I choose num of epochs and batch size?

I have the coco 2014 dataset and need to train it as training is around 82700 and testing is 40500. However, I got the same sentence with different values every time with ...
user5520049's user avatar
0 votes
2 answers
88 views

Is it reasonable to do train/test splitting based upon information/entropy?

I want to divide my time series dataset into training and test sets. The data is seasonal and very noisy. When I randomly split, the test and train samples do not resemble in their ...
Ather Cheema's user avatar
0 votes
1 answer
1k views

Why does Light GBM model produce different results while testing?

Using the Light GBM regressor, I have trained my data and, using Grid Search, I got the best parameters, but while testing with the best parameters I am getting different results each time, which ...
HEMANTHKUMAR GADI's user avatar
1 vote
2 answers
4k views

Training set and test set size

How to correctly approach the generation of a training/test set? I am doing several experiments testing the generalization ability of my neural network model, so my test set is different from my ...
lmtr339's user avatar
  • 21
1 vote
1 answer
67 views

How to train a deep neural network to return the input as it is?

The task is to train a neural network to return the input as it is, like X -> X or Y -> Y. The network should contain at ...
Kaiyakha's user avatar
  • 111
0 votes
1 answer
1k views

ValueError: Layer model expects 2 input(s), but it received 3 input tensors using generator

I am trying to fit a model using generator function and I get the following error: ...
manix velu's user avatar
1 vote
3 answers
32 views

What to do about the failed portion of trained dataset?

I've seen lots of tutorials and papers about this or that model getting some great accuracy score. In this case, let's say 85%. But what I never see is what you are supposed to do with the remaining ...
Malik A. Rumi's user avatar
1 vote
0 answers
57 views

Dataset format for Transformer text-generation

I'm trying to find some tutorials on training Transformer for generating comments on articles. So far, I found an article showing how to train GPT2 as a chat-bot. Input files in that example are given ...
Bojan Vukasovic's user avatar
3 votes
1 answer
314 views

Difference in performance Sigmoid vs. Softmax

For the same Binary Image Classification task, if in the final layer I use 1 node with Sigmoid activation function and ...
Eric Cartman's user avatar
1 vote
0 answers
33 views

How can I prepare my discrete batches of data for training?

I'm trying to calculate effect of parameters of an operation on the thickness of a wall. Each operation is thinning the wall thickness and at some point the wall is replaced and operation starts again....
Oncel Umut TURER's user avatar
1 vote
1 answer
47 views

Quantitative measure of the smoothness of learning curves

$\DeclareMathOperator{\loss}{loss}$ $\DeclareMathOperator{\AvgVar}{AvgVar}$ Lat's say we have some deep learning task. We have our model and two sets of hyperparameters $A$ and $B$. We train both ...
Fallen Apart's user avatar
0 votes
2 answers
356 views

How should I implement machine learning for multi-tenant website?

The company I work for has a website for personal use to track leads and opportunities. I implemented a linear regression algorithm to predict a score for opportunities which is trained on the ...
Ishaan's user avatar
  • 111
1 vote
0 answers
23 views

Training data with one class

I have a real-time scenario, finding out whether a transaction is fraudulent or not. I have a dataset that contains only fraudulent transactions. For any binary classification algorithm, we may need ...
Yasin's user avatar
  • 11
0 votes
0 answers
62 views

Is it right method to remove instances that are hard to predict before train test split?

In a binary classification problem, I have a slightly unbalanced medical dataset with class distribution: 0:5600, 1:1500 0 without a problem and 1 with a problem. I tried many pipelines, automls, and ...
DOT's user avatar
  • 103
2 votes
1 answer
1k views

Should I train from scratch or use pre-trained weights?

With yolov4, I am training an 80k images dataset that is used to classify different species of fish. Currently, I am using the following pre-trained weights: ...
Dark Apostle's user avatar
1 vote
1 answer
234 views

Average loss is 0 when training dataset with darknet yolov4

I am currently training a dataset using yolov4 darknet from AlexeyAB Github found here: https://github.com/AlexeyAB/darknet The dataset I am training is called FishNet Open Images. The dataset has 86,...
Dark Apostle's user avatar
1 vote
1 answer
30 views

Rule based prediction for known data

Lets say we have trained our model on 900 records (training data) . During prediction on test data of 100 records, assume model produces 95% accuracy. The question here is, can a mechanism be built, ...
Sandeep Bhutani's user avatar
1 vote
1 answer
269 views

Applying the same changes to the test set

I'm busy working through Aurélien Géron's book. (Hands-On Machine Learning with Scikit-Learn, Keras, and Tensorflow) The idea is to split the data into train and test set as early as possible in order ...
Neal Liddle's user avatar
1 vote
0 answers
27 views

Iteratively improving ML model on a small dataset

I have a spam classification model which I created using a very small dataset.I have exported it as shown ...
AK10's user avatar
  • 11
1 vote
2 answers
198 views

How to train and evaluate machine learning models with growing/changing datasets over time

Assume that you have a classification machine learning model, and you start with an initial dataset that contains 3 classes. You split the initial dataset into training/testing spits, you train the ...
hajem_badger's user avatar
2 votes
2 answers
96 views

How do you effectively predict the top 20% most likely customers to churn from a dataset?

I am looking to work out that if I have a dataset with 100,000 existing customers who didn't churn and 20,000 previous customers that churned in the past and the business objective is to target the 20%...
Dean F's user avatar
  • 21
2 votes
1 answer
26 views

Machine learning problem with only original data without test and validation data

I am new to machine learning and I am trying to solve a problem where I have to predict if a customer will buy a home insurance product or not. I have got a dataset which tells me that which of the ...
Django0602's user avatar
1 vote
1 answer
25 views

Lower performace with same script on google cloud vs laptop

So I want to test a lot of hyperparameters for an xgboost classification model and also do cross validation for all of these. To do this I use a gridsearch. To speed up the process I want to use as ...
JohnDoe's user avatar
  • 11
1 vote
1 answer
94 views

Why do we need cross validation set? [closed]

I know we need to test our model on onseen data, but isn't that test set are for? Also what will happen if we increase K value in kfold?
Alan's user avatar
  • 11
0 votes
1 answer
137 views

Training the network with some batch size - code

There is my "training" code below, I wrote it based on one youtube tutorial. I don't understand actually one part: batch_X = train_X[i:i+BATCH_SIZE], batch_y = train_y[i:i+BATCH_SIZE]. How ...
Adolf Miszka's user avatar
2 votes
0 answers
445 views

Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?

I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in ...
bkaankuguoglu's user avatar
0 votes
1 answer
33 views

Which dataset I should use when I am retraining my model?

I trained my deep learning model using x dataset and now I got new dataset and I give it name as y. I want to retrain my model on this new dataset which is y. Do I need to use x+y dataset or just y? ...
Yash Choksi's user avatar
0 votes
0 answers
238 views

Train/ Test split on small dataset along with SMOTE

I have a binary classification imbalanced dataset with 1000 samples ( 15% of class 1, 85% of the rest). My main goal is to build a robust classifier using the following approach. Wanted to know if ...
Vardaan Khanted's user avatar
4 votes
3 answers
7k views

Alternatives with better GPU than Google Colab Pro

I am currently running/training MAchine learning models that are very GPU expensive, Google Colab Pro is not giving me enough GPU/RAM Is there any alternatives with better GPU and more RAM than ...
The Dan's user avatar
  • 183
2 votes
0 answers
1k views

XGBoost incremental training for big datasets

I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred). I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
Mattia Surricchio's user avatar
0 votes
1 answer
42 views

What if Training and testing dataset comes from the same source?

I am working on a classification problem in which I have to distinguish between healthy and damaged plates. when I use the combination of k-means clustering and SVM algorithm together with 10-fold ...
Syed Haider's user avatar
0 votes
1 answer
148 views

Train and Validation Curve

I'm new in DeepLearning. I'm not good at understanding and commenting on graphics.Can you help me with these graphs
Sdeveci's user avatar
0 votes
1 answer
65 views

Lower Variance vs. Higher Validation Scores

So I'm trying to compare between two models, say model(1) has training accuracy of 90% and validation accuracy of 86%, while model(2) has training accuracy of 87% and validation accuracy of 85%. Now, ...
Mourad Askar's user avatar
0 votes
1 answer
43 views

Adding Validation PyTorch

First of all, I'm new in this field and it's my first this kind of work. I'm trying to train EfficientNet (CNN), the code below is working fine, but I can't succeed to add also validation set to the ...
Adolf Miszka's user avatar
0 votes
0 answers
61 views

How many epochs should all the data be trained on after training with validation finds when validation and training diverge?

One uses and train/test split to use their training data to get an idea of how many epochs to train with. If the validation accuracy starts going down while the training accuracy is still going up, ...
user14094230's user avatar
0 votes
1 answer
7k views

Plot a training/validation curve in Pytorch Training [closed]

I have the following training method and I'm confused how may I modify the code to plot a training and validation curve history graph with matplotlib ...
Charith Jayasanka's user avatar
2 votes
1 answer
262 views

Training a neural network with TWO possible correct outputs for one input

I have a system as a black box that has two correct outputs for a single input sample. now I want to train a neural network to generate at least one of the correct outputs for that input sample. what ...
Abolfazl Sajady's user avatar
0 votes
1 answer
114 views

Need for LIME explainer

Is it possible to train a LIME explainer for a binary classfier on a dataset without labels? I need to understand what is the value of storing a LIME explainer object trained on the same data used to ...
user3076574's user avatar
0 votes
1 answer
245 views

Help needed in interpreting the loss, val_loss vs epoch plots for an autoencoder training?

I am training a variational autoencoder and I am getting a loss-plot as follows: Rigt after epoch 224, val-loss overtakes train-loss and sort of getting bigger but at an extremely slow pace as you ...
user62198's user avatar
  • 1,091
2 votes
0 answers
275 views

How to calculate MAE and threshold in a multivariate time series

I'm trying to understand how to calculate the MAE in my time series and then the thresholds to understand which of my data in the test set are anomalies. I'm following this tutorial, which is based on ...
Fabio's user avatar
  • 53
2 votes
1 answer
663 views

Improve Convolutional Autoencoder

I just built a Convolutional Autoencoder to try to reconstruct a time series with shape (4000, 10, 30). This is the code, I used a batch size of 32, but I think it ...
Fabio's user avatar
  • 53
2 votes
0 answers
19 views

Best Approach and Classifier for Binary Classification Problem

I am trying to build a binary classifier, and I am wondering what is the best approach for data segmentation, training/testing, performance evaluation, selecting classifier type, and overall approach ...
xyztg's user avatar
  • 21
1 vote
1 answer
392 views

What is the proper proportion for train and test set for classification system?

I have recently googled the best proportion for training and test set for classifying physiological data between normal and abnormal. Much of the source tells that the proportion should be 70:30 or 80:...
Naufal's user avatar
  • 21
0 votes
1 answer
28 views

How do Classification Algorithms such as Catboost and Random Forest parse test data?

I would like to know how classification works with the algorithms listed above. My specific question is this, say I have a high signal continuous feature which has a certain distribution and I train a ...
Nathan's user avatar
  • 1

1
3 4
5
6 7
14