New answers tagged machine-learning
0
votes
Accepted
How does oversampling or undersampling approch is going to help during the testing on real time data?
The key here is how you define "help" regarding the measurement of performance. Oversampling/undersampling may not help increasing accuracy. However, it may help increase other performance ...
- 23.8k
0
votes
Test set won't have all the features present in the training set
Running a customer's job may require time consuming
equipment setup, and time is money. Predict the dollars for setup.
Your existing production data captures number of seconds to set up
each of {...
- 343
0
votes
How can Time Series Analysis be done with Categorical Variables
I have splitted the periods not in fixed time windows, but in dynamic ones. Eg. If we take @Ricardo's Alerts, I would take the values of all the variables from that "period". The table ...
- 101
0
votes
What exactly is Gradient norm?
The concept of norm comes from functional analysis (and is found both in linear algebra and in optimization methods and other fields). There are several types of norm. The one in question is the norm ...
- 1
1
vote
Out-of-Range Target Variable in Sequence-based Machine Learning Model
"Interpolation is easy. Extrapolation is hard."
Extrapolating might be the right thing to do.
But always be suspect of a model that is leading
you to big unexplored regions of the state ...
- 343
9
votes
Accepted
Why images, audio/video clips, text are regarded as unstructured data?
Tell me the schema for a relation.
Maybe students.csv has column attributes of
name, gender, ...
- 343
0
votes
0
votes
What are the "EMA" weights included with certain pre-trained models?
It may be a late answer, but I couldn't let this question go unanswered.
The difference between the two files is likely related to the use of Exponential Moving Average (EMA) in training.
The file <...
1
vote
Employee Attrition - Binary Classification - Predicting leavers
I don't have a ready answer, but I come up with some ideas that may help you approach this problem.
As I understood, the main difficulty in treating it as a binary classification of this data is that ...
- 106
0
votes
How to use External Data Sets in test set
When dealing with time series prediction, incorporating external datasets into both the training and testing phases can provide valuable information to improve the model's performance. Here are some ...
- 71
0
votes
How to create a bot for a real-time PvP game using machine learning?
Creating a bot for real-time PvP games raises ethical concerns and is often against the terms of service of the game. Using bots to automate gameplay can provide unfair advantages and is generally ...
- 71
0
votes
How do I get my Neural network to ignore certain values?
My experience - I struggled with a similar issue some time ago in a different problem, where I wanted my network to ignore part of the input data. I tried a few approaches: setting a constant small ...
- 106
1
vote
Semantic Search on numeric data
You could feed the LLM a description of the file format and then request it to generate a piece of code to extract the information you want, for instance, in Python. Then, you would run the generated ...
- 23.8k
2
votes
What methods do top Kagglers employ for score gain?
There are many methods that can be employed to increase your score on a Kaggle competition. Here are a few examples:
Advanced classification techniques: Using advanced classification techniques such ...
- 2,879
2
votes
What methods do top Kagglers employ for score gain?
Top Kagglers often employ a combination of traditional machine learning techniques and creative, out-of-the-box strategies to gain an edge in competitions. Here are some techniques:
1. Ensemble ...
- 71
1
vote
Accepted
How do I use ML models to estimate current stress level based on past data?
Yes, both these scenarios are different.
Estimating Current Stress level - Your target variable here is stress level and features are heart rate and blood pressure. In order to estimate current ...
- 188
1
vote
Should I apply the same data transformations in production for my classification model's inference steps
Should I use the same scaler() I used in training during my inference?
Yes, the data that is coming into your model in production should go through the same transformations as the data you used to ...
- 273
0
votes
Why use both validation set and test set?
In theory (an ideal world)
Training set: The model's many (e.g. 10 million) parameters are directly tuned using the training set, typically via gradient descent.
Validation set: The model's few (e.g. ...
- 258
3
votes
Beginner basic clustering model and one-hot encoding?
Once you are done fitting the model, you can label each of your records based on the cluster.
df['cluster_labels'] = kmeans.labels_
For ease of analysis, you can ...
- 188
0
votes
Accepted
Beginner clustering project, what are the input features and how do I analyze the data?
Let me provide some guidance on the various aspects you've mentioned:
1. Encoding Categorical Variables
Yes, you should encode categorical variables like REGION, <...
- 71
0
votes
Is it possible to run ML models on SBC boards?
Yes, you may be able to run some sizes of Whisper, depending on the available amount of RAM.
Specifically, you may be able to run Whisper tiny or base through whisper.cpp. Here you can find benchmarks ...
- 23.8k
1
vote
Accepted
Which machine learning models are rational to use on NP-hard and NP-complete "theoretical" problems?
Is there any point in using machine learning algorithms on these "well-defined" NP-hard problems (instead of using dynamical programming)?
In short, yes. Consider that for whatever problem ...
- 556
2
votes
Accepted
activation=tf.keras.activations.relu vs activation='relu'
Your results were probably result of the randomness in the training. Both activation=tf.keras.activations.relu and ...
- 23.8k
2
votes
Accepted
Confusion with tensorflow's Sequential Dense Layers
Keras expects the inputs $X$ to be batched, i.e., of shape $B\times D$ in your case (for a feed-forward NN), where $B$ is the batch size (by default is 32, which can be assigned in ...
- 556
1
vote
randomness in lightgbm model training
What can be the other factors adding randomness to the training process that I did not notice?
By "adding randomness to the training process", I'm assuming that you mean things that could ...
- 121
1
vote
How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?
Fine-tune a pretrained model like BERT on your open-source data first. This will provide a strong baseline model. Then continue fine-tuning on the small real dataset. The pretrained weights will help ...
- 2,879
2
votes
How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?
Given that your finetuning dataset is very small, I suggest using multitask finetuning (multitask learning but in a finetuning situation):
Download other legal datasets from the internet. If possible,...
- 23.8k
1
vote
How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?
You may consider the following options:
Artificially enhance the real data through data augmentation. You can utilize libraries specifically designed for this purpose. This technique allows you to ...
11
votes
Accepted
How high of a correlation coefficient of a feature with a target variable is considered too high?
You should not remove features just because their correlation to the target is high. Such high correlation is a sign of potential target leakage. You should understand why their correlation is high.
...
- 23.8k
1
vote
One class classification using Exclusively Positive Examples
Sure, you can use a OneClassSVM model or an IsolationForest (read also this). Basically, you feed them the data you have (regardless of the class label) and the model scores your data (according to an ...
- 556
1
vote
How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?
Given your question details, it seems you need to try transfer learning, where you reuse an existing pre-trained model on new data to fine tune the model generalization capabilities according to the ...
- 156
1
vote
Why SMOTE is not used in prize-winning Kaggle solutions?
SMOTE or data augmentation techniques introduce bias.
In academic settings, researchers tend to overlook this, but in real world applications the issue is serious.
For example, some well-known ...
- 156
0
votes
Why the one validation score is lower than the other sections of cross validation
That is the code:
...
0
votes
Dense layer: how to transform hgih dim vector to one dim? (sigmoid function)
Don't think about each layer performing a dimension transformation on your data. If your input is 20-dimensional and your network structure is composed of a input layer, a 100-dimensional hidden layer ...
- 326
1
vote
Why the one validation score is lower than the other sections of cross validation
This problem also might be related to the seeding of the random number generator (RNG). Typically, your RNG spits out a sequence of pseudo-random numbers, meaning, that they look random but there ...
- 191
0
votes
Why the one validation score is lower than the other sections of cross validation
Maybe data is not shuffled properly. Try to shuffle it explicitly before cross_val_score() is called. What is your dataset?
0
votes
What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering?
From a data management, data engineering, and data analytics perspective, basic EDA will force you to slice and group your data with liked data types. Creating a situation where you will be forced to ...
- 156
0
votes
Why not train the final model on the entire data after doing hyper-paramaeter tuning basis test data and model selection basis validation data?
Never do anything with the test dataset. I am surprised this question has many positive votes.
Overall. The point of having the test set on the sidelines is to evaluate ML model generalization ...
- 156
1
vote
how to build model using two input dataset in which there is no common column to merge or combine
It looks like your problem is more suitable for optimization (operational research) techniques than machine learning algorithms. If I understood it right, you could build a model that can decide how ...
- 326
0
votes
Decision boundary in a classification task
First of all, the plot in the question does not look correct to me. This is what I would get when I perform the same calculation:
There, on the left the small markers show the plane in "3D" ...
- 191
0
votes
How to encode & scale IP addresses as input for ML models
The domain of all possible random IP addresses is astronomically large, and they have no relation to each other. So any given IP address only provides information about that specific IP address, ...
- 1
0
votes
In Gradient descent, Why the gradient of cost function do not have to be normalized into unit vector
Your intuition is partly correct, and in fact the family of adaptive learning rate methods (adagrad, adadelta, adam, etc) takes into consideration the magnitude of the gradient, quoting from ADADELTA: ...
- 1
1
vote
Very basic but how to understand data statistically for machine learning?
That's actually a good question - this is nothing to be embarrassed about IMO :)
I'll attempt to answer your question as a practitioner (but hopefully people who are more knowledgeable can provide ...
- 36
1
vote
Accepted
How can I approach this transactions data problem?
There are so many things to take into consideration but my answer will focus on some divergent thoughts to help you with your modeling.
1 - I would start by understanding the underlying distribution ...
- 36
0
votes
causal inference vs sensitivity analysis to create acurate ML predictions?
Your post has 2 questions, so I will try to answer both:
Difference between Causal Inference (CI) and Sensitivity Analysis (SA).
Sensitivity Analysis (SA) can be described as the following (Naik & ...
- 273
1
vote
Accepted
How does the stacking works?
You are partially correct:
Yes, the meta-model is trained to predict the label $y_{true}$.
But $y_{true}$ is not used as input feature for the meta model. This would not work in any application / ...
- 1,352
0
votes
How to do Feature clustering?
It depends on what you are doing. I would consider:
K-means clustering
Analysis of covariance
PCA loadings
- 101
2
votes
Accepted
Supervised or Unsupervised Learning Classification: Facebook Prophet vs. ARIMA
Supervised Learning is learning from labeled data, while in Unsupervised Learning, you learn without labels. Now to your questions:
Could someone clarify whether Facebook's Prophet and ARIMA are ...
- 271
0
votes
Supervised or Unsupervised Learning Classification: Facebook Prophet vs. ARIMA
Supervised Learning methods are characterized for using a target value to drive the learning process. I uderstand that your question comes from the impression that a time series doesn't have a "...
- 326
Top 50 recent answers are included