Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

New answers tagged

1 vote

How to explain missing dates to a model?

I am not sure how you could define such a predictive model using Nixtla (I am unfamiliar with this software). However, tensorflow has a way you can mask specific timestamps. Perhaps this is what you'...
healthydata's user avatar
0 votes

Why do we need to concatenate in a U-Net?

The information at the first layers of the network can get "muddy" as it passes through to the final layers of the network and skip connections are a great way to remind the network what it ...
Frobot's user avatar
  • 99
2 votes

Can anyone help me understand this problem in my data?

First, your absolute values, for the time-series are extremely large. Quite often these packages would then try to compute variance, and you get numerical overflows. Consider normalizing this. Second, ...
Cryo's user avatar
  • 553
1 vote

Improving GPU Utilization in LLM Inference System

The system you have designed is not capable of processing multiple requests concurrently. GPUs cannot process separate workloads in parallel (at least not without exerting control at the SM layer and ...
Karl's user avatar
  • 746
1 vote

How to measure different models' feature importance using a generic and common standard?

Permutation importance is a relatively simply model-agnostic approach. You train and score a model the usual way to get a reference score. Then you take each feature in turn, and score the model after ...
MuhammedYunus's user avatar
0 votes

Test Error is extremely higher than Training error after gridsearch and crossvalidation

A few comments. I might be missing something, but I can't find your normalize() function, it might have a bug. For random forest there is no need to normalize anything anyway. If you don't have a ...
Yair Beer's user avatar
1 vote

How good are LSTMs in generalizing when learning curves?

LSTMs in an encoder-decoder arrangement would ingest $f$, render an encoding, and then decode that into a new sequence $g$. LSTMs are prone to overfitting, especially for small datasets, so I don't ...
MuhammedYunus's user avatar
2 votes
Accepted

How do I interpret probability results in conjunction with my known precision/accuracy/recall scores?

So for example, let's say I give the model a new piece of data to classify and it returns: (0.3, 0.7) Is there only a 90% chance that the 0.7 is correct? The 90% precision means that when the model ...
MuhammedYunus's user avatar
1 vote

Bagging vs pasting in ensemble learning

In the context of ensembling, the aim of both bagging and pasting is to get a diverse set of estimators despite each estimator using the same algorithm. The diversity comes from how you set up the ...
MuhammedYunus's user avatar
1 vote
Accepted

Determining the threshold value for the neural network

There are two steps and I am in line with other answers and comments that you should reconsider step 1: 1. Find a distance / similarity measure If you want to identify similar names, you need to ...
Broele's user avatar
  • 1,450
4 votes
Accepted

Using training data that requires manual interpretation

There are many cases where creating labels is expensive and time consuming. For that reason, there exist multiple approaches for that case. In the following, I give a brief overview of some techniques....
Broele's user avatar
  • 1,450
0 votes

Determining the threshold value for the neural network

I'm not sure if you need a neural network for your problem. A classical tool for this is Levenshtein distance. It will allow you to calculate the similarity between surnames. As it's a distance, you ...
Tomasz Witkowski's user avatar
1 vote

Optimized input data structure for ML model training

For the date columns I would suggest Feature on given date being a bank holiday in a given country Feature on weekday, do an ordinal encoding Add seasonality indicators, which month it is, how many ...
dmayilyan's user avatar
0 votes

I need suggestion for a project

You can just use a single date like the beginning or end of the month to represent each month value. An example of this is done here: Catfish Monthly Sales Prediction I'm not sure how to to model ...
avb0101's user avatar
  • 11
2 votes

CS undergrad query about DS

“Data science” as a distinct term is fairly new, especially compared to related fields like statistics, computer science, and economics. Consequently, it is not totally clear what constitutes data ...
Dave's user avatar
  • 3,903
1 vote

Fitting users' reports with joint time-semantic model

What about using a Bayesian Classifier? https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Examples Define as many features as you need. I mean, inner definitions of the data you want to study: ...
Alfred's user avatar
  • 11
1 vote

How to weight loss in regression

To handle imbalanced data in regression tasks, use the DenseWeights package. It assigns weights to data points using kernel density estimation (KDE) based on the ...
Kevin Patel's user avatar
2 votes

Autoencoders are fitting anomalies too good

In principle it's a feasible way to use CAE's (Convolutional AutoEncoders) for this. It's a bit hard to really tell you what to do better without knowing the code. But I can give you some points I ...
burn4science's user avatar
1 vote

Should I standardise time series data for deep learning classification?

Most time series classification algorithms do not take the time index as input, so there's no point standardising the time index. The description for some algorithms will mention a time index, but ...
Lynn's user avatar
  • 1,307
2 votes
Accepted

What type of machine learning am I looking for with these column types?

Before struggling on 'what machine learning model to use', let's take a step back and ask 'does my features have a causal connection with the target?' Machine learning is no magic. It cannot predict ...
lpounng's user avatar
  • 1,075
0 votes

What method can I use to interpolate a low frequency time series to a high frequency one, using the pattern from a higher resolution time series?

We deal with two types of data: high frequency (HF) and low frequency (LF). When we want to increase the frequency of LF data to match HF data, we have two main strategies: 1.Regression of LF on HF (...
far had's user avatar
  • 31
1 vote

How many ways are there to check model overfitting?

Evaluating training scores—such as accuracy for classification and (adjusted) R-squared for regression—against test scores can indicate potential overfitting. However, this comparison alone doesn't ...
Seyit Hocuk's user avatar
2 votes

What do the terms in this equation $$θ1x1+θ2x2+θ0=0$$ represent?

Nice work on self-studying! To understand the equation you posted, let's go through the most basic concepts and eventually build up the linear separator mentioned in the video. A lot of these concepts ...
F-said's user avatar
  • 39
1 vote

Model evaluation approach allowing manual experimentation without data leakage

Instead of splitting the data in two parts, train and test, you could split the data into more parts. Basically, every time you want to evaluate something you need data that is completely unseen.
alepfu's user avatar
  • 31
1 vote

Custom loss function in python

worried that the evaluation of the loss function won't capture the "anomaly-part" of the loss function as this would be lost in the aggregation? You're talking about weighting the minority ...
J_H's user avatar
  • 1,110
0 votes
Accepted

Data binning for interval data

regression This is a regression problem, not a classification problem. So model it with a regressor. Your loss function can choose to discretize each prediction before scoring it, if that's what you ...
J_H's user avatar
  • 1,110
0 votes

Why is each successive tree in GBM fit on the negative gradient of the loss function?

The concept is by fitting the sequential tree by the "residual" (Practically the gradient relative to the current model) the overall process will be equivalent to Gradient Descent over the ...
Royi's user avatar
  • 167
1 vote

How to train a recommender based on user_features, item_features and likes?

The simplest method would be to generate a feature vector of [[user_features], [item_features]] and send it to a random forest or logistic regression model. If you ...
Karl's user avatar
  • 746
-2 votes

What are the disadvantages of Azure's ML vs a pure code approach (R/SKlearn)

Seems to be a misapplication of the algorithms. Tuning and tweeking is the meat and potatoes of Data science. If your looking for performance and dimensions of truths previously hidden then why would ...
Frederick Duff's user avatar
1 vote

Text Classification with unlimited labels, Text Extraction?

Unfortunately, it is not possible to do a classifier with “unlimited labels”. In general, your best bet is to use a transformer model adapted to certain tasks: Text generation: GPT, Llama Named ...
AndrewJaeyoung's user avatar

Top 50 recent answers are included