Questions tagged [training]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
2answers
32 views

How many trees does a random forest need?

At first, I did a GridsearchCV and the best parameter is a random forest with just 100 trees. My trainset has 80.000 rows and 669 columns. My test set has 20.000 rows and 669 columns. How is it ...
1
vote
1answer
84 views

How to keep the test data from leaking into the training process of a machine learning algorithm?

I read in many different sources that I need to split my data into a training set and a test set. Then I have to make sure that the algorithm is trained only on the training data, and do my best to ...
0
votes
0answers
14 views

Is it normal that CUDA will sporadically run out of memory?

Sometimes I will train a model successfully, othertimes CUDA will run out of memory in the middle of training with the exact same network, batch size, etc. Similar behaviour like models not loading ...
0
votes
1answer
34 views

What is the main concept of using lexical,linguistic, semantic or syntactic approach in NLP for cyberbullying

I am really in need of some explanation, I am working on an NLP cyber-bullying detection tool which I will deploy to the web using Django framework, however, am stuck on some idea, can someone explain ...
0
votes
2answers
23 views

Present results of the best or the last iteration on dev set?

Which is the correct way - presenting the results of the best or the last iteration on the dev set in a paper? In research papers I usually see only one value, is it the best iteration of all? I'm ...
1
vote
1answer
78 views

Incremental training and Auto Machine Learning for big datasets

I built a NLP sentence classifier, which uses vectors from word embedding as features. Training dataset is big (100k sentences). Every sentence has 930 features. I found the best model using an auto ...
31
votes
8answers
7k views

What would I prefer - an over-fitted model or a less accurate model?

Let's say we have two models trained. And let's say we are looking for good accuracy. The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted. The second has an ...
1
vote
0answers
18 views

Vector Autoregression forecasting with large dataset

I am trying to use VAR to forecast electricity price for a whole day and I have a dataset with over 20000 observations (price for every hour) from 2015-2017. My first intuition was to select 19975 ...
0
votes
0answers
161 views

InvalidArgumentError: Incompatible shapes while training

InvalidArgumentError: Incompatible shapes: [15,31744] vs. [15,31680][[{{node loss_4/output_loss/logistic_loss/mul}}]] Has anyone of you ever got this kind of ...
1
vote
1answer
24 views

Including the validation file in the training process after tunning

Should I include the validation file in the training process after finishing the tuning process (e.g. searching for params using the validation file)?
1
vote
2answers
47 views

How to train the machine so that it can give 'out of bound/classes' as an output for neural network

I know I was not able to word the title of the question properly. So I am trying to explain the problem here: Suppose, I built and trained a CNN to identify numbers from 0 to 9. However, when I ...
0
votes
1answer
181 views

splitting into train test by train_test_split of float values?

How to split into train test by train_test_split of float values ? I used LabelEncoder but I have about 300K lines and when I used the cross_val I saw ...
0
votes
0answers
25 views

How to convert non image data to lmdb format for caffe

I have data where the input is of size [1,1,625] ( width and height is 1 and number of channels is 625). The input size cannot be changed since I need to use the weights of a pretrained model which ...
1
vote
1answer
168 views

Is it reasonable that a problem “solved” by a traditional ANN can also be solved by a CNN?

There is a data science numerical problem, which me and my team were able to get an ANN model that predicts down to a 1% MAPE error (with roughly 70000+ trainable parameters). Given the nature of the ...
0
votes
0answers
20 views

Loss is decreasing but the predictions are not getting well

I am trying to implement a Dependency Parsing model using the transformer model in here with a few changes. On the training, my loss has decreasing trend; but the predictions at the end of 20 epochs ...
1
vote
1answer
390 views

What's the best way to train a NER model?

I am trying to do a project using NLP. My goal is to process Cyber Threat Intelligence articles like this to extract information such as actor’s name, malwares and tools used… To do that I want to ...
0
votes
0answers
19 views

Best model for removing a consistent background image

I'm working on an ML driven background replacement from photos, where the background will always be consistent, but the objects in front of that background (in this case humans) will be different - ...
0
votes
0answers
29 views

What will cause high accuracy but a big loss?

In the question of What is the relationship between the accuracy and the loss in deep learning?, @Jérémy Blain gave a fantastic interpretation of 'relationship' between accuracy and loss: 1 - low ...
0
votes
0answers
22 views

Find Correspondence between Image Pair

I have a problem with the following provided: (for some reason I can't disclose the dataset) Dataset My dataset is a bunch of raw images with a lot of strip-like objects. Each image has a lot of ...
0
votes
1answer
43 views

How to treat time based ticket prices for train/test split

I have a dataset of airfare price tickets that were scraped throughout a 6 month period where each observation represents a particular price for a specific flight on a specific date that it was ...
1
vote
2answers
96 views

Accuracy of KFold Cross Validation for Neural Network

I have a neural network that Im evaluating using 10 -Fold cross validation. The validation accuracy for a fold changes alot during training in the range of -+10% So for example the validation ...
4
votes
1answer
44 views

Overfitting due to features correlating with training set generation rules

As background, I am using a Deep Neural Network built using Keras to classify inputs into 5 categories. The current structure of the network is: Input layer (~450 nodes) Dense layer (750 nodes) ...
0
votes
0answers
9 views

should I re-initialize my optimizer and my scheduler before I try to fine tune my neural network on the different dataset?

I am doing NLP, and I have this block of Transformer body that was already trained on dataset A. Now I am interested in fine tuning this same Transformer on a new dataset B. In my Python code, should ...
0
votes
2answers
32 views

Can I use more features for my training data than my test data will supply?

I am pretty new to the data science game so pardon me, if the answer to my question should be a no-brainer. We are looking at manufacturing / quality data where products are labeled 'okay' or 'not ...
1
vote
1answer
468 views

Error: An operation has `None` for gradient with categorical_crossentropy

I am trying to train my discriminator network using Keras with TensorFlow backend. The network is meant to classify the input into one of the 9 output labels. I am passing a 2D input (height, width, ...
0
votes
0answers
15 views

How to best use Large images in training set for deep learning

I would like to ask you about how I should deal with the images I have. They are really large. They have this shape: (3000, 4000, 3). I'm working on a multilabel classification model. And I want to ...
0
votes
0answers
107 views

saving a model during training of an RL agent

I am training an RL agent using PPO2 algorithm. Iam using stable-baselines library. During the training process, my rewards are slowly increasing and stabilizing, but are falling down suddenly. I ...
0
votes
1answer
106 views

Target encoding before split on skewed data

Hi My data is distributed like follow: And I only have categorical variables on many many levels. As I need to make a regression task I thought about doing leave one out encoding on my categories. ...
0
votes
0answers
24 views

CNN models comparison

I coded a 38 layer CNN and 8 layer CNN but there's something wrong in my 38 layer CNN, which doesn't learn anything. Not able to fugure out what's wrong. They were trained on CIFAR.
4
votes
2answers
286 views

Training a model sample by sample

I'm training a Scikit model but it seems that in all examples, they call the fit method on the entire training set. What I want to do however is call it per sample (...
0
votes
0answers
6 views

Training Algorithm for pointcloud data (3d point data)

I have a "PointNet" neural network which theoretically can work with any number of points as input. I have trained the model using an equal number of points from each object. That is fairly simple and ...
0
votes
0answers
16 views

Accuracy to big when train yolov3

I am trainning yolov3. My result is not as my expected. I think accuracy must be less then 1.0 But I got accuracy and avg too large, in this case is 1577.76 average. Do I still working well?
0
votes
2answers
25 views

Suggestions on how to explain 'models' & 'algorithms'

I guess other members of this Stack have ran in to this before, but I may be wrong: Have you ever been approached and asked to explain the difference between models and algorithms? This happened to be ...
1
vote
0answers
134 views

Grape disease detection

i'm trying to realize a detector for diseased grape leaves, for this par of the project i'm just interested in detecting lets say, the percentage of diseased to healty leaves and/or place a flag where ...
0
votes
0answers
30 views

What should be ideal ratio for number of unique target label vs number of training set samples

For a multiclass classification problem in Machine Learning, is there any rule for ratio of number of target class values vs number of training samples? For example, I have 2000 records to train on ...
0
votes
0answers
6 views

Is there any relation between binary/ternary quantization using in deep learning and fuzzy?

I am new with binary/ternary quantization but its structure seems to have some relation with fuzzy. Am I in right way? Is there any relation between binary/ternary quantization using in deep learning ...
2
votes
1answer
48 views

How to deal with annotation errors?

I know my annotators are not perfect, sometimes making mistakes. What would be the best way to deal with the annotation errors for my training data? Thanks!
0
votes
1answer
22 views

Transfer learning VGG16 does not work as expected. (Detect tacos as hamburgers)

I am new in this field of machine learning, to test I wanted to do a simple project. Create a cnn capable of recognizing hamburger images. As I do not have the ability to collect more than 10,000 ...
1
vote
2answers
659 views

Oversampling/Undersampling only train set only or both train and validation set

I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...
0
votes
0answers
24 views

How do I compare more than 20 deep learning models?

I have to compare several deep learning models (CNNs) based on the same dataset. For estimating the model skill's I use the train_test_split instead of ...
0
votes
1answer
1k views

clarification on train, test and val and how to use/implement it

So far I think I understood the differences between the training, test and validation set. Basically it is like in this image: Training set: The data where the model is trained on Validation set: ...
1
vote
1answer
28 views

How to construct validation set for time series for NN?

I would like to train my model with a validation set. As the data is a time series I have to use timeseriessplit: ...
1
vote
0answers
21 views

How to teach algorithm to mimic paths in a certain enviroment

I have a set of scenarios which represent the movement of a car in a certain environment containing some obstacles. So for each scenario I have the position of the car (x,y,t) and a description of the ...
0
votes
0answers
14 views

Training data for Tesseract

I have to train my tesseract to detect different variations of a letter for example, u,û,ü,ù should all mean u. Is this possible and if yes how should I train it and how should the dataset be made ...
0
votes
0answers
18 views

How to avoid different accuracies when training with subsets?

when trying to train a CNN with randomly selected small subsets (each same size) of the training data set, I get different results in accuracy (the accuracy varies from 0.75 to 0.85). I determine the ...
3
votes
2answers
157 views

Why can we not split train test data with 0.01 as parameter or 99% training data

Most of the blogs mention about a good thumb rule to be 80-20 split for the train and test respectively. My special case is a time series dataset and it is for the stock prices, which IMO is very ...
1
vote
3answers
136 views

How to deal with a feature that has lot of categorical values?

I know this question has been asked before and I have tried a few things but those things are not working as expected for my usecase. I have a 500 length feature vector. One of these features is a ...
1
vote
1answer
33 views

Accuracy noise patterns during model training

I'm training a logistic regression model on a small dataset. I have about 1300 samples that I split into a training and a testing set (70% and 30% respectively). The training seems ok, however when I ...
2
votes
2answers
2k views

What can be the cause of a sudden explosion in the loss when training a CNN (Deeplab)

I am training the following deeplab CNN: https://github.com/tensorflow/models/tree/master/research/deeplab During training I see the following loss: The first 50k steps of the training the loss is ...
3
votes
2answers
174 views

Given a machine learning algorithm, what is the minimum size of the training set for it?

I understand that the more data we have, the more reliable is our model trained on that data. I also understand that the more parameters a machine learning model has, the more training data it ...

1 2
3
4 5
8