New answers tagged

0 votes
Accepted

How does oversampling or undersampling approch is going to help during the testing on real time data?

The key here is how you define "help" regarding the measurement of performance. Oversampling/undersampling may not help increasing accuracy. However, it may help increase other performance ...
noe's user avatar
  • 23.8k
0 votes

Test set won't have all the features present in the training set

Running a customer's job may require time consuming equipment setup, and time is money. Predict the dollars for setup. Your existing production data captures number of seconds to set up each of {...
J_H's user avatar
  • 343
0 votes

How can Time Series Analysis be done with Categorical Variables

I have splitted the periods not in fixed time windows, but in dynamic ones. Eg. If we take @Ricardo's Alerts, I would take the values of all the variables from that "period". The table ...
M. Chris's user avatar
  • 101
0 votes

What exactly is Gradient norm?

The concept of norm comes from functional analysis (and is found both in linear algebra and in optimization methods and other fields). There are several types of norm. The one in question is the norm ...
Karina's user avatar
  • 1
1 vote

Out-of-Range Target Variable in Sequence-based Machine Learning Model

"Interpolation is easy. Extrapolation is hard." Extrapolating might be the right thing to do. But always be suspect of a model that is leading you to big unexplored regions of the state ...
J_H's user avatar
  • 343
9 votes
Accepted

Why images, audio/video clips, text are regarded as unstructured data?

Tell me the schema for a relation. Maybe students.csv has column attributes of name, gender, ...
J_H's user avatar
  • 343
0 votes

Proper way to build time series dataset

You should be good to aggregate the data.
Kriti's user avatar
  • 188
0 votes

What are the "EMA" weights included with certain pre-trained models?

It may be a late answer, but I couldn't let this question go unanswered. The difference between the two files is likely related to the use of Exponential Moving Average (EMA) in training. The file <...
Luke Wardford's user avatar
1 vote

Employee Attrition - Binary Classification - Predicting leavers

I don't have a ready answer, but I come up with some ideas that may help you approach this problem. As I understood, the main difficulty in treating it as a binary classification of this data is that ...
Tomasz Witkowski's user avatar
0 votes

How to use External Data Sets in test set

When dealing with time series prediction, incorporating external datasets into both the training and testing phases can provide valuable information to improve the model's performance. Here are some ...
lvvittor's user avatar
0 votes

How to create a bot for a real-time PvP game using machine learning?

Creating a bot for real-time PvP games raises ethical concerns and is often against the terms of service of the game. Using bots to automate gameplay can provide unfair advantages and is generally ...
lvvittor's user avatar
0 votes

How do I get my Neural network to ignore certain values?

My experience - I struggled with a similar issue some time ago in a different problem, where I wanted my network to ignore part of the input data. I tried a few approaches: setting a constant small ...
Tomasz Witkowski's user avatar
1 vote

Semantic Search on numeric data

You could feed the LLM a description of the file format and then request it to generate a piece of code to extract the information you want, for instance, in Python. Then, you would run the generated ...
noe's user avatar
  • 23.8k
2 votes

What methods do top Kagglers employ for score gain?

There are many methods that can be employed to increase your score on a Kaggle competition. Here are a few examples: Advanced classification techniques: Using advanced classification techniques such ...
Multivac's user avatar
  • 2,879
2 votes

What methods do top Kagglers employ for score gain?

Top Kagglers often employ a combination of traditional machine learning techniques and creative, out-of-the-box strategies to gain an edge in competitions. Here are some techniques: 1. Ensemble ...
lvvittor's user avatar
1 vote
Accepted

How do I use ML models to estimate current stress level based on past data?

Yes, both these scenarios are different. Estimating Current Stress level - Your target variable here is stress level and features are heart rate and blood pressure. In order to estimate current ...
Kriti's user avatar
  • 188
1 vote

Should I apply the same data transformations in production for my classification model's inference steps

Should I use the same scaler() I used in training during my inference? Yes, the data that is coming into your model in production should go through the same transformations as the data you used to ...
Adam Jaamour's user avatar
0 votes

Why use both validation set and test set?

In theory (an ideal world) Training set: The model's many (e.g. 10 million) parameters are directly tuned using the training set, typically via gradient descent. Validation set: The model's few (e.g. ...
Mateen Ulhaq's user avatar
3 votes

Beginner basic clustering model and one-hot encoding?

Once you are done fitting the model, you can label each of your records based on the cluster. df['cluster_labels'] = kmeans.labels_ For ease of analysis, you can ...
Kriti's user avatar
  • 188
0 votes
Accepted

Beginner clustering project, what are the input features and how do I analyze the data?

Let me provide some guidance on the various aspects you've mentioned: 1. Encoding Categorical Variables Yes, you should encode categorical variables like REGION, <...
lvvittor's user avatar
0 votes

Is it possible to run ML models on SBC boards?

Yes, you may be able to run some sizes of Whisper, depending on the available amount of RAM. Specifically, you may be able to run Whisper tiny or base through whisper.cpp. Here you can find benchmarks ...
noe's user avatar
  • 23.8k
1 vote
Accepted

Which machine learning models are rational to use on NP-hard and NP-complete "theoretical" problems?

Is there any point in using machine learning algorithms on these "well-defined" NP-hard problems (instead of using dynamical programming)? In short, yes. Consider that for whatever problem ...
Luca Anzalone's user avatar
2 votes
Accepted

activation=tf.keras.activations.relu vs activation='relu'

Your results were probably result of the randomness in the training. Both activation=tf.keras.activations.relu and ...
noe's user avatar
  • 23.8k
2 votes
Accepted

Confusion with tensorflow's Sequential Dense Layers

Keras expects the inputs $X$ to be batched, i.e., of shape $B\times D$ in your case (for a feed-forward NN), where $B$ is the batch size (by default is 32, which can be assigned in ...
Luca Anzalone's user avatar
1 vote

randomness in lightgbm model training

What can be the other factors adding randomness to the training process that I did not notice? By "adding randomness to the training process", I'm assuming that you mean things that could ...
James Lamb's user avatar
1 vote

How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?

Fine-tune a pretrained model like BERT on your open-source data first. This will provide a strong baseline model. Then continue fine-tuning on the small real dataset. The pretrained weights will help ...
Multivac's user avatar
  • 2,879
2 votes

How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?

Given that your finetuning dataset is very small, I suggest using multitask finetuning (multitask learning but in a finetuning situation): Download other legal datasets from the internet. If possible,...
noe's user avatar
  • 23.8k
1 vote

How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?

You may consider the following options: Artificially enhance the real data through data augmentation. You can utilize libraries specifically designed for this purpose. This technique allows you to ...
Kübra Nilgün Karaca's user avatar
11 votes
Accepted

How high of a correlation coefficient of a feature with a target variable is considered too high?

You should not remove features just because their correlation to the target is high. Such high correlation is a sign of potential target leakage. You should understand why their correlation is high. ...
noe's user avatar
  • 23.8k
1 vote

One class classification using Exclusively Positive Examples

Sure, you can use a OneClassSVM model or an IsolationForest (read also this). Basically, you feed them the data you have (regardless of the class label) and the model scores your data (according to an ...
Luca Anzalone's user avatar
1 vote

How Can I Train a Real-World-Ready Classifier with Limited Real Data and Abundant Open-Source Data?

Given your question details, it seems you need to try transfer learning, where you reuse an existing pre-trained model on new data to fine tune the model generalization capabilities according to the ...
Full Array's user avatar
1 vote

Why SMOTE is not used in prize-winning Kaggle solutions?

SMOTE or data augmentation techniques introduce bias. In academic settings, researchers tend to overlook this, but in real world applications the issue is serious. For example, some well-known ...
Full Array's user avatar
0 votes

Why the one validation score is lower than the other sections of cross validation

That is the code: ...
Emir Kutsal's user avatar
0 votes

Dense layer: how to transform hgih dim vector to one dim? (sigmoid function)

Don't think about each layer performing a dimension transformation on your data. If your input is 20-dimensional and your network structure is composed of a input layer, a 100-dimensional hidden layer ...
Giovanni Amorim's user avatar
1 vote

Why the one validation score is lower than the other sections of cross validation

This problem also might be related to the seeding of the random number generator (RNG). Typically, your RNG spits out a sequence of pseudo-random numbers, meaning, that they look random but there ...
BanDoP's user avatar
  • 191
0 votes

Why the one validation score is lower than the other sections of cross validation

Maybe data is not shuffled properly. Try to shuffle it explicitly before cross_val_score() is called. What is your dataset?
citara3996's user avatar
0 votes

What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering?

From a data management, data engineering, and data analytics perspective, basic EDA will force you to slice and group your data with liked data types. Creating a situation where you will be forced to ...
Full Array's user avatar
0 votes

Why not train the final model on the entire data after doing hyper-paramaeter tuning basis test data and model selection basis validation data?

Never do anything with the test dataset. I am surprised this question has many positive votes. Overall. The point of having the test set on the sidelines is to evaluate ML model generalization ...
Full Array's user avatar
1 vote

how to build model using two input dataset in which there is no common column to merge or combine

It looks like your problem is more suitable for optimization (operational research) techniques than machine learning algorithms. If I understood it right, you could build a model that can decide how ...
Giovanni Amorim's user avatar
0 votes

Decision boundary in a classification task

First of all, the plot in the question does not look correct to me. This is what I would get when I perform the same calculation: There, on the left the small markers show the plane in "3D" ...
BanDoP's user avatar
  • 191
0 votes

How to encode & scale IP addresses as input for ML models

The domain of all possible random IP addresses is astronomically large, and they have no relation to each other. So any given IP address only provides information about that specific IP address, ...
ekohrt's user avatar
  • 1
0 votes

In Gradient descent, Why the gradient of cost function do not have to be normalized into unit vector

Your intuition is partly correct, and in fact the family of adaptive learning rate methods (adagrad, adadelta, adam, etc) takes into consideration the magnitude of the gradient, quoting from ADADELTA: ...
Toonia's user avatar
  • 1
1 vote

Very basic but how to understand data statistically for machine learning?

That's actually a good question - this is nothing to be embarrassed about IMO :) I'll attempt to answer your question as a practitioner (but hopefully people who are more knowledgeable can provide ...
NNZ's user avatar
  • 36
1 vote
Accepted

How can I approach this transactions data problem?

There are so many things to take into consideration but my answer will focus on some divergent thoughts to help you with your modeling. 1 - I would start by understanding the underlying distribution ...
NNZ's user avatar
  • 36
0 votes

causal inference vs sensitivity analysis to create acurate ML predictions?

Your post has 2 questions, so I will try to answer both: Difference between Causal Inference (CI) and Sensitivity Analysis (SA). Sensitivity Analysis (SA) can be described as the following (Naik & ...
Adam Jaamour's user avatar
1 vote
Accepted

How does the stacking works?

You are partially correct: Yes, the meta-model is trained to predict the label $y_{true}$. But $y_{true}$ is not used as input feature for the meta model. This would not work in any application / ...
Broele's user avatar
  • 1,352
0 votes

How to do Feature clustering?

It depends on what you are doing. I would consider: K-means clustering Analysis of covariance PCA loadings
BigMistake's user avatar
2 votes
Accepted

Supervised or Unsupervised Learning Classification: Facebook Prophet vs. ARIMA

Supervised Learning is learning from labeled data, while in Unsupervised Learning, you learn without labels. Now to your questions: Could someone clarify whether Facebook's Prophet and ARIMA are ...
Dr. Snoopy's user avatar
0 votes

Supervised or Unsupervised Learning Classification: Facebook Prophet vs. ARIMA

Supervised Learning methods are characterized for using a target value to drive the learning process. I uderstand that your question comes from the impression that a time series doesn't have a "...
Giovanni Amorim's user avatar

Top 50 recent answers are included