Questions tagged [training]

Training is the part of machine learning whereby a model is "trained" on a define portion of a dataset to learn attributes and statistical features of the data. It's counterparts are called Testing and Validation. After training a model is tested and validated on another portion of the dataset.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

How to deal with errors or inconsistencies in the training data?

There are inconsistant wrong labels and consistant errors in training data. For the former I tried MC-dropout and data Shapley. For the later I wonder if manual data curation is a requisite?
0
votes
0answers
18 views

Generating unique points with an auto-encoder

I have been working on some research using a type of auto-encoder to generate new points with specific desirable properties. I trained my network and successfully generated some points, but when I ...
0
votes
0answers
31 views

pytorch lightning produces no checkpoint when learning rate fine tuning ison

My problem is concerning with using the automatic learning rate finder of pytorch lightning. In case I use this feature there isn't any checkpoint output produced at any time during the training of ...
0
votes
1answer
1k views

1D target tensor expected, multi-target not supported

I am trying to train my model. My model outputs a [4,2] tensor where 4 is the batch size and 2 because of binary classification. After receiving the outputs I found the index of the maximum element ...
0
votes
0answers
42 views

Custom stratification for train and test

The data contains half-year amounts of sales. The train dataset would be fit on the second half of 2020 and the test dataset would be fit on the first half of 2021. There are 2 columns containing this,...
0
votes
0answers
12 views

Validation output in a custom training loop not working - Tensorflow

I am new to Deep Learning and I am trying to learn more about implementation in Tensorflow and Keras. I am basing my work on this link : https://www.tensorflow.org/guide/keras/...
0
votes
1answer
19 views

Aren't balanced data sets important in regression?

Why is it that the necessity for balanced data sets is (almost) always exclusively mentioned in the context of classification but not of regression?
1
vote
2answers
51 views

Spliting Training Test and Validation for Image Dataset

I have 600 images in the training folder, 200 images in the validation folder, and 200 images in the test folder. Suppose if I fit the training data generator and validation data generator for some ...
0
votes
2answers
25 views

No target response variable in my testing data

I have two datasets which are the training and testing set. The training data has a target variable, but the testing set does not. What should I do to fix the issue with the testing set?
0
votes
0answers
34 views

How can I check a confusion_matrix after fine-tuning with custom datasets?

Background I would like to check a confusion_matrix, including precision, recall, and f1-score like below after fine-tuning with custom datasets. Fine tuning process and the task are Sequence ...
1
vote
1answer
177 views

How train - test split works for Graph Neural Networks

I have recently started studying GNN's. I have covered GCN and GraphSage so far. But I am confused regarding the process when testing occurs. Now suppose in the graph above I am using the nodes as ...
0
votes
1answer
39 views

Does validation data has any effect on training or it acts solely without affecting the training?

When using Keras library of Python, we use validation data with training data while training our model. In every epoch, we get a validation accuracy. Does this validation accuracy have any effect on ...
0
votes
0answers
16 views

Weight for Samples on SVM

there is a option sample_weight in fit(X[, y, sample_weight]) function (OneClassSVM, sklearn library). If I use the option ...
1
vote
0answers
48 views

How to do online retraining of model on a single new data point/observation?

I am trying to investigate the effect on performance on old data and new data when a classifier is retrained on only the new observation when it is encountered. The aim is to retrain the classifier on ...
2
votes
0answers
26 views

When to stop the final model training?

Let's say I'm participating in a Kaggle image recognition competition. Firstly, I create a train/validation split and find the good hyperparameters for my model. Here the stopping criterion is when ...
0
votes
1answer
29 views

how to correct mislabeled data in training, validation and test set

In an image classification task, I know there are mislabeled data. should I remove/correct them in all training / validation / test set ? I saw this article https://arxiv.org/pdf/2103.14749.pdf but I ...
0
votes
1answer
36 views

Best choice for splitting data given a quantity and a expected accuracy

I have a dataset with at least 1,000,000 images (from IDs) which I am using to detect the presence of sealed IDs. The legacy algorithm got nearly 60% accuracy, but my current algorithm yielded almost ...
0
votes
0answers
12 views

Training a 3D reconstruction from a single image model

I want to know if training the model only on images of cars will give better results in term of the final shape details, instead of using a pre-trained model (trained on different images of objects) ...
1
vote
0answers
42 views

YOLO: how to deal with object partially in training image

I am trying to train a YOLO (v3) network on a set of images, but I am faced with a problem when preparing the training set. Sometimes, an image contains my object of interest, but only half of the ...
0
votes
1answer
88 views

How can I choose num of epochs and batch size?

I have the coco 2014 dataset and need to train it as training is around 82700 and testing is 40500. However, I got the same sentence with different values every time with ...
0
votes
0answers
27 views

Best image recognition API to implement for eCommerce Lifestyle/Sculpture site

I'm planning an eCommerce site currently. We are likely running WooCommerce and looking to implement Algolia for our search features. We feel that for our particular purposes, a visual search would be ...
0
votes
1answer
31 views

Is it reasonable to do train/test splitting based upon information/entropy?

I want to divide my time series dataset into training and test sets. The data is seasonal and very noisy. When I randomly split, the test and train samples do not resemble in their ...
0
votes
1answer
121 views

Why does Light GBM model produce different results while testing?

Using the Light GBM regressor, I have trained my data and, using Grid Search, I got the best parameters, but while testing with the best parameters I am getting different results each time, which ...
0
votes
2answers
64 views

Training set and test set size

How to correctly approach the generation of a training/test set? I am doing several experiments testing the generalization ability of my neural network model, so my test set is different from my ...
0
votes
0answers
10 views

Volatile training loss chart on transfer learning mobile_netV1

Above is a graph of my latest attempt at transfer learning from a mobile_netV1 model already trained to 1 million steps, doing transfer learning with 50000 additional steps to my new dataset. The ...
1
vote
1answer
40 views

How to train a deep neural network to return the input as it is?

The task is to train a neural network to return the input as it is, like X -> X or Y -> Y. The network should contain at ...
0
votes
0answers
389 views

ValueError: Layer model expects 2 input(s), but it received 3 input tensors using generator

I am trying to fit a model using generator function and I get the following error: ...
1
vote
3answers
25 views

What to do about the failed portion of trained dataset?

I've seen lots of tutorials and papers about this or that model getting some great accuracy score. In this case, let's say 85%. But what I never see is what you are supposed to do with the remaining ...
1
vote
0answers
29 views

Dataset format for Transformer text-generation

I'm trying to find some tutorials on training Transformer for generating comments on articles. So far, I found an article showing how to train GPT2 as a chat-bot. Input files in that example are given ...
1
vote
1answer
120 views

Difference in performance Sigmoid vs. Softmax

For the same Binary Image Classification task, if in the final layer I use 1 node with Sigmoid activation function and ...
1
vote
0answers
28 views

How can I prepare my discrete batches of data for training?

I'm trying to calculate effect of parameters of an operation on the thickness of a wall. Each operation is thinning the wall thickness and at some point the wall is replaced and operation starts again....
1
vote
1answer
31 views

Quantitative measure of the smoothness of learning curves

$\DeclareMathOperator{\loss}{loss}$ $\DeclareMathOperator{\AvgVar}{AvgVar}$ Lat's say we have some deep learning task. We have our model and two sets of hyperparameters $A$ and $B$. We train both ...
0
votes
0answers
17 views

Error due to Colab RAM depletion when implementing Multi-label classification with BERT and Pytorch

Background: I'm implementing multi-label classification for tones (7 types of tones). Dataset shape: train_df=(5392, 8); val_df = (1348, 8) The modelling approach remains the same as this multi-label ...
0
votes
2answers
108 views

How should I implement machine learning for multi-tenant website?

The company I work for has a website for personal use to track leads and opportunities. I implemented a linear regression algorithm to predict a score for opportunities which is trained on the ...
1
vote
0answers
22 views

Training data with one class

I have a real-time scenario, finding out whether a transaction is fraudulent or not. I have a dataset that contains only fraudulent transactions. For any binary classification algorithm, we may need ...
0
votes
0answers
34 views

Is it right method to remove instances that are hard to predict before train test split?

In a binary classification problem, I have a slightly unbalanced medical dataset with class distribution: 0:5600, 1:1500 0 without a problem and 1 with a problem. I tried many pipelines, automls, and ...
2
votes
1answer
218 views

Should I train from scratch or use pre-trained weights?

With yolov4, I am training an 80k images dataset that is used to classify different species of fish. Currently, I am using the following pre-trained weights: ...
0
votes
0answers
21 views

Will training with yolov4 backup weights early cause weak prediction or is a true backup?

I am training a dataset in yolov4 using the repo from AlexeyAB darknet. In his repo, backup weights are created every so iterations but you originally train with a pretrained weights file. I was ...
1
vote
0answers
104 views

Average loss is 0 when training dataset with darknet yolov4

I am currently training a dataset using yolov4 darknet from AlexeyAB Github found here: https://github.com/AlexeyAB/darknet The dataset I am training is called FishNet Open Images. The dataset has 86,...
1
vote
1answer
20 views

Rule based prediction for known data

Lets say we have trained our model on 900 records (training data) . During prediction on test data of 100 records, assume model produces 95% accuracy. The question here is, can a mechanism be built, ...
0
votes
0answers
31 views

What should be the input shape for convLSTM if ResNet-50 is applied?

I have a dataset 12 videos. Each video is comprised of 179 frames. On these frames, I have applied ResNet-50 to extract features, and I received (179,7,7,2048) features. As far I know, 179=Total ...
1
vote
1answer
41 views

Applying the same changes to the test set

I'm busy working through Aurélien Géron's book. (Hands-On Machine Learning with Scikit-Learn, Keras, and Tensorflow) The idea is to split the data into train and test set as early as possible in order ...
1
vote
0answers
20 views

Iteratively improving ML model on a small dataset

I have a spam classification model which I created using a very small dataset.I have exported it as shown ...
1
vote
2answers
37 views

How to train and evaluate machine learning models with growing/changing datasets over time

Assume that you have a classification machine learning model, and you start with an initial dataset that contains 3 classes. You split the initial dataset into training/testing spits, you train the ...
0
votes
0answers
23 views

How to retrain a model without bias

Let's say I have a model in production that predicts a certain customer behavior (for example, propensity to buy), and let's say that there is a business process underneath that takes some actions ...
1
vote
1answer
43 views

How do you effectively predict the top 20% most likely customers to churn from a dataset?

I am looking to work out that if I have a dataset with 100,000 existing customers who didn't churn and 20,000 previous customers that churned in the past and the business objective is to target the 20%...
2
votes
1answer
21 views

Machine learning problem with only original data without test and validation data

I am new to machine learning and I am trying to solve a problem where I have to predict if a customer will buy a home insurance product or not. I have got a dataset which tells me that which of the ...
1
vote
1answer
17 views

Lower performace with same script on google cloud vs laptop

So I want to test a lot of hyperparameters for an xgboost classification model and also do cross validation for all of these. To do this I use a gridsearch. To speed up the process I want to use as ...
1
vote
1answer
43 views

Why do we need cross validation set? [closed]

I know we need to test our model on onseen data, but isn't that test set are for? Also what will happen if we increase K value in kfold?
0
votes
0answers
8 views

What techniques are there to train custom sentence classification models with reasonable memory footprint?

We are currently working on tasks that involve user-inputted data (e.g., question-answers, short-answer-grading), with a framework that will allow them to be improved through active learning. However, ...

1
2
3 4 5
12