New answers tagged

0 votes

Trimming "unused" neurons from the bottleneck of an autoencoder

For anyone stumbling on this question, I've been digging and found some info. This paper does basically what I was looking for on VAEs, and this one is related but not directly about autoencoders.
  • 101
3 votes
Accepted

Predict actual result after model trained with MinMaxScaler LinearRegression

First, you can't use anything from the test set before training. This means that the scaling should be done using only the test set, otherwise there's a risk of data leakage. Then remember that ...
  • 22.8k
0 votes

How to learn steep functions using neural network?

Your problems seem like solving a stock exchange problem with respect to some given values. Take it as a time series problem where the curve of your graph changes according to some low peak and high ...
0 votes

Clustering by using Locality sensitive hashing *after* Random projection

It makes sense to reduce the dimensionality with Random Projection (RP) and then cluster with Locality Sensitive Hashing (LSH). One of the primary ways of improving LSH is running it multiple times ...
1 vote

adding conditional variables

If you are asking, to identify people who will become all star players based on their current stats using a model of past players, both all star and not, yes, you will likely need to do some feature ...
  • 251
2 votes
Accepted

Increasing/Decreasing importance of feature/thing in ML/DL

I would not say these models "misunderstand" anything. They simply learn from the data provided based on their inductive biases. I hypothesize that all three cases might be caused by the ...
  • 5,065
4 votes
Accepted

Can you do the math for this simple treeSHAP example (decisionTree)?

You seem to have entirely the right idea, you just miscalculated the second and fourth contributions you listed. Below are the corrected calculations, bolded to indicate where a change has been made: ...
  • 10.2k
0 votes

Custom loss function for regression

A bit late to the party, but I had a similar challenge. I wanted my model to favour low prediction errors for high values of the target variable. I can't tell if this is what you are trying to achieve,...
0 votes

How do transformers differ from feature selection and regular machine learning?

Your question does not necessarily apply to transformers but to machine learning in general. A question I can answer is: What is the difference between feature selection and machine learning? The ...
0 votes

Should a model be re-trained if new observations are available?

I don't understand the third question Is it over-fitting if the parameters are re-optimised for the aggregated data?, but I get the answers to these three: I have not been able to find any literature ...
0 votes

Unsupervised learning for anomaly detection

You can consider this package, pyod. It has various anomaly/outlier detection algorithms all in one package. Approaches : Use a dimension reduction technique like tsne, which can collapse high D into ...
  • 13
0 votes
Accepted

Given an image how to find height of an object?

Put the apple next to a penny and capture a top level view. Then by using the known ground truth (since you know dimensions of penny), you can estimate the distance to the ground and distance to the ...
1 vote

How to Predict future temperatures based on past data with years

Predicting temperature is often framed as a time series problem. Given the seasonal effects, a simplifying assumption is that each month can be predicted separately. You can start with a baseline ...
1 vote

How to learn steep functions using neural network?

The problem isn't that the loss function is steep, the problem is that it is not convex/differentiable / there are local minima. Read this answer.
0 votes

Why does my mAP metric value start so high in the first epoch?

I would make sure that the the classifier does not expect bounding boxes in the centroid format of (xmid, ymid, width, height). Dig more into the IoU calculation code to find which format the ...
2 votes

Why Deep Learning / Neural Networs don't achieve state of the art results in tabular data problems?

My intuition is that it is because tabular data does not necessarily form a manifold. There is some limited and indirect support in the literature for this hypothesis: According to [1], the manifold ...
  • 5,065
1 vote

Stacking: Use predictions of train or test to create features for level 1 classifier

Since my dataset was not very big and I didn't want to split it (as @Erwan suggested), I ended up doing the same thing sklearn does: Train level 0 classifier on ...
  • 143
4 votes
Accepted

Stacking: Use predictions of train or test to create features for level 1 classifier

I'm not sure if there's any standard about this, but I usually proceed by splitting the training set into two parts A and B: A is used as training set for level 0 models B is used as test set for the ...
  • 22.8k
0 votes

How to learn steep functions using neural network?

My understanding of your question is that you have a multivariate regression problem (i.e., 19 different target variables). Given it's a multivariate regression problem and you have specific ...
0 votes
Accepted

same random_state or mean of the different random_state?

Yes, ideally, you should run experiments with different random seeds. Explanation The reason why it is recommended to use a fixed random seed is reproducibility, i.e. you don`t want to get different ...
  • 5,065
0 votes

What are available Python libraries for Interpretable ML?

I will suggest Dalex as it has a very easy workflow and has both Python and R APIs. Also Interpret-ML from Microsoft has very good features.
2 votes

Sklearn vs Pytorch vs Tensorflow vs Keras

The scikit-learn is a library that is used most often when working with the more traditional non neural network models, whereas the other three are more focused on ...
  • 6,169
1 vote

Predicting products to be sold in a store - problem formulation

Typically in a sales forecast task you will use data structured like the following: ...
  • 5,065
0 votes

Plotting a no-skill model in a precision-recall curve

Assuming the dummy model yielding constant probability for all samples, there are two possible thresholds: either everything is classified as positive or negative. So starting point will be ...
  • 446
3 votes
Accepted

How can I say if I have a class imbalance issue in my data?

'Imbalance problem' is a mix up of several loosely related issues, mainly these two: It's hard to generalize when there's too few of a certain class' samples, especially with lots of dimensions. ...
  • 446
1 vote

How do I match the number of the features of new text data to the data used in the training of the model

If you obtained only 19 features it's likely because you used fit_transform instead of transform for your instance. It's ...
  • 22.8k
0 votes

What is degenerate feedback loop and how to detect and prevent it?

A degenerate feedback loop can happen when the predictions themselves influence the feedback, which, in turn, influences the next iteration of the model. More formally, a degenerate feedback loop is ...
0 votes

How to save API data into CSV format?

You can also do this in bash. For example, if your CSV file is a list of queries for the API eg query1 query2 ...
  • 101
0 votes

Evaluation of a model of imbalanced data

First, as far as I know a ROC curve and AUC can perfectly be used with imbalanced (binary classification) data. However I think there is a problem with the ROC curve you show at the end: it seems to ...
  • 22.8k
1 vote
Accepted

I am struggling to understand the point of supervised ML models in real world scenarios

Supervised means that the training stage is supervised and requires labels. It does not mean that you need labels during inference. Here is small example using a Random Forest classifier with scikit ...
  • 5,065
0 votes

Xgboost Multiclass evaluation Metrics

In this snippet: ...
  • 10.2k
2 votes

Evaluation of a model of imbalanced data

As discussed on the stats.SE meta, class imbalance leads to a lot of misconceptions, and it is important to have a strong understanding of the underlying statistics in order to overcome those ...
  • 3,278
2 votes
Accepted

How to calculate a trend to use as a feature in a machine learning model?

You have several choices: The trend can be calculated as $\frac{last - first}{first}$, e.g. $\frac{3.9 - 3.6}{3.6}$ You can perform a linear regression including the 4 points and use the slope as ...
  • 22.8k
0 votes

Libraries for Online Machine Learning

River - Online Machine Learning in Python (website) previously creme and scikit-multiflow.
1 vote

Enocding of months for machine learning project

There is a column with months, but it is categorical. Was wondering what kind of encoding I should use. If you really want to feed the month as a feature into your model, then the best approach is ...
1 vote

Enocding of months for machine learning project

Trees can performs splits efficiently with ordinal features and handle OHE somewhat worse, so label encoding for months looks like a good start. As for more complex cases: There's a meaningful order: ...
  • 446
1 vote

Fuzzy Classification in NLP

I think the first question you may need to ask is what does a "fuzzy decision" in language mean? The objective of language at its base requirement is to convey a clear message that on one ...
0 votes

Aggregating multiple encoded categorical values

What I do in this case is to use the min_frequency parameter of SK-learn's OneHotEncoder, check it out : https://scikit-learn.org/stable/modules/generated/sklearn....
  • 161
0 votes

Currency Normalization for Salary Prediction

The same skill-set wouldn't give the same salary in different countries because of difference in the living standard. So what I would do is convert the salaries into a common currency, but using the ...
  • 161
1 vote

Is it possible to attribute reasons for poor (anomalous) target performance?

I suggest Random Forest for a start because it classifies many cases automatically so that you can see what is going on with your data with good accuracy. This is a relatively robust algorithm ...
0 votes

How to handle undefined or null data in a neural network

There is not one absolutely correct method to handle this issue. Depending on dataset and algorithm used (eg NNs, SVMs) it is more appropriate in certain cases to simply add default values, but in ...
  • 2,206
0 votes

Max pooling has no parameters and therefore doesn't affect the backpropagation?

This is a trick question. On the one hand yes, you don't have trainable parameters in max pooling - but this doesn't mean it has no affect on the backpropagation process. BUT - when thinking on the ...
  • 63
4 votes
Accepted

Is Pearson correlation a good loss function?

I think Dave's answer points out the most pressing issues: translational invariance Absolute scale invariance In Tensorflow we can define our correlation function: ...
  • 162
0 votes

Loss function for age classification

What you're doing is not quite classification but ordinal regression. There is no issue with this when you work with an ordinal logistic regression such as rms::orm in R. Your CNN is just an extension ...
  • 3,278
5 votes

Is Pearson correlation a good loss function?

Correlation does not make a useful loss function for many reasons. One reason is that correlation only measures how linearly related two variables are. A model can have a strong linear relationship ...
0 votes

How to use hierarchical variable in a ML model

Of course you could use only granular level variables but this would throw away a lot of information. The are different ways to leverage hierarchies. One way would be target encoding as described here ...
8 votes

Is Pearson correlation a good loss function?

UPDATE (I WAS WRONG) Maximizing correlation misses a lot and makes for a terrible function to optimize. For instance, correlation will not detect if you consistently predict too high or too low. For ...
  • 3,278
0 votes

What is the difference between ensemble methods and hybrid methods, or is there none?

Hybrid usually implies - symbolic + subsymbolic learning (KG/Rule-Based + ML/DL) Ensemble Learning implies - similar/dissimilar methods competing against each other to reduce error where the models ...
0 votes

how to set threshold for anomaly detection

The way to tune the anomaly detection threshold is as follows: Construct a train set using a large sample of observations without anomalies. Take a smaller sample of observations containing anomalies ...
2 votes

Classification on severe Class Imbalance high dimensional data

I have implemented something at work on the dataset which has so many classes as above and it worked well for me, though generalization might be an issue but its worth trying, what I would do in your ...

Top 50 recent answers are included