Skip to main content
Share Your Experience: Take the 2024 Developer Survey
7 votes

Why does data science see class imbalance as a problem for supervised learning when statistics does not?

It's generally not related to Data Science but what goes around; typically all sort of bad practice relating to laziness / looking for short term rewards. I wouldn't say DS is pushing for it but ...
Lucas Morin's user avatar
  • 2,244
4 votes

Parameter estimation in linear regression

The typical way to encode a category is with a 0/1 indicator about absence or presence of that category. If you do this, then model $\#1$ results in your model matrix having three columns (the ...
Dave's user avatar
  • 3,903
4 votes

Parameter estimation in linear regression

This might be a trick question, if so then the answer could be the first option. The reason being that the first model estimates a coefficient for males and females separately, but the effect of one ...
user2974951's user avatar
3 votes
Accepted

Linear Regression and Logistic Regression

Yes, you are right. Both are unified under the term Generalized Linear Model.
noe's user avatar
  • 26.9k
2 votes

Is probabilities mean of predicted class (RandomForest) a consistent estimator of class recall?

Unfortunately this doesn't work. To understand why, let's think about the perfect classifier and the maximally wrong classifier. The perfect classifier has 100% recall on any given class. If there are ...
Nicholas James Bailey's user avatar
2 votes
Accepted

how to evaluate a model on our data when the model is imported from a library and thus not trained by us?

If you want to avoid annotate the data yourself, you can try to evaluate the model on one or more benchmark datasets that are available for sentiment analysis. This may work unless the model is very ...
Luca Anzalone's user avatar
2 votes

Using scipy poisson, is it possible to calculate lambda if given random variable (k) and cumulative density function (p) when k is a large value?

There is no direct way to do this. But the cdf for a fixed number k is a monotonic function of mu. This means that we can use ...
Broele's user avatar
  • 1,460
2 votes

Multiple Hypothesis Testing in feature selection process

In a basic statistical model like a logistic regression on the features, this kind of univariate feature selection is problematic. Briefly, a candidate feature can be fairly unrelated to the outcome ...
Dave's user avatar
  • 3,903
2 votes

Why does data science see class imbalance as a problem for supervised learning when statistics does not?

Statisticians typically focus on probabilistic classification. In the way they build their models they are interested in predicting $P(Y|X) = P(X|Y)P(Y)/P(X)$. Now we find : The predicted ...
Ggjj11's user avatar
  • 216
1 vote

Time Series Analysis and Price Elasticity

Can I ask what the logic is for dividing the log-differenced number of sales by the log-differenced moving average of price? Not saying the logic is wrong, I just haven't really seen this before. ...
aranglol's user avatar
  • 2,196
1 vote

Negative Log Likelihood of a Gaussian Model can be negative?

You aren't missing anything. The negative log-likelihood of any distribution can be negative. In the case of the negative log-likelihood for a Gaussian random variable, this occurs when the function ...
aranglol's user avatar
  • 2,196
1 vote

Simple rebasing formula - beginner quesitons!

On your diagram it appears that Risk and Impact have different units and different numeric ranges. The usual way to put disparate continuous variables on the same footing is to turn them into Z-scores....
J_H's user avatar
  • 1,110
1 vote

Very basic but how to understand data statistically for machine learning?

That's actually a good question - this is nothing to be embarrassed about IMO :) I'll attempt to answer your question as a practitioner (but hopefully people who are more knowledgeable can provide ...
NNZ's user avatar
  • 36
1 vote

Workflow when making a machine learning model

There is always a workflow for doing anything, even for ML. The question you are trying to ask is somewhat related to MLOp. Please do your research on this as I am not the right person to explain ...
Pushpak Ruhil's user avatar
1 vote
Accepted

Suggestions to learn the Machine Learning models in greater depth?

I guess I can answer this question well as I have been studying the intensive mathematics required for ML. People often tend to ignore the maths required for Machine Learning and directly jump to the ...
Pushpak Ruhil's user avatar
1 vote

Approaches to dataset, whose elements have different size

Do you expect the size of the dataset to be highly correlated with the classification into clusters? If so, then it could be a straightforward approach to simply count the number of columns and use ...
brewmaster321's user avatar
1 vote

Is probabilities mean of predicted class (RandomForest) a consistent estimator of class recall?

This does not make sense. When you predict the probability, there is not the sense of being right or wrong like there is when you predict a category. Therefore, there is not a notion of recall, let ...
Dave's user avatar
  • 3,903
1 vote

Bibliographic references on statistical analysis of classification results in machine learning

Benavoli, Alessio, et al. "Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis." The Journal of Machine Learning Research 18.1 (2017): 2653-2688. This ...
Dave's user avatar
  • 3,903
1 vote

Which statistical procedure to use?

You may apply paired t-test to test your hypothesis: the students overall course grades are the same in both tests.
Subhash C. Davar's user avatar
1 vote

What is a good metric for comparing a single value to a distribution?

The z-score could help you understand how far away a data point in one distribution is from the mean of that distribution, in terms of standard deviations. This can be useful for comparing individual ...
Saurabh Anand's user avatar
1 vote

Creating well-separated bins

Here are a few hints: Your task might not be well-defined: to "separate" data point can be understood as to find the (four) different clusters that they belong to -- a problem of ...
BanDoP's user avatar
  • 231
1 vote
Accepted

Pearson coefficients different for same data?

Pearson coefficient is the ratio between the covariance of two variables and the product of their standard deviations. If the data is same, their standard deviation, mean, covariance and hence Pearson ...
shivani's user avatar
  • 140
1 vote

Choice of ML approach

This is a classic problem of causal inference, where you are trying to estimate the effect of a treatment (in this case, the type of food) on an outcome (in this case, the amount of sleep), given that ...
Harshad Patil's user avatar
1 vote

Survival analysis on time series data for predictive maintenance

@rvdinter You could try time-varying survival analysis if you have long formatted data. One package in python that would allow you to do this is Lifelines. If instead your data can be broken up into ...
healthydata's user avatar
1 vote

Why are normal distributions so important in deep learning?

You seem to have mixed up two similar names. The normal (Gaussian) distribution, which is a “bell curve” A normalized distribution that has been centered and scaled by subtracting the mean and the ...
Dave's user avatar
  • 3,903
1 vote

Simplex method for equality optimization

I would recommend starting to explore Pyomo on Python or JuMP on Julia for solving linear programming problems. Those modeling languages provide a user-friendly interface and offer various solver ...
Josa Ferreira's user avatar
1 vote

Simplex method for equality optimization

I think the term you are looking for is "linear programming". The constraints you provided as an example are too broad, and they lead to an infinite number of solutions. However, you can ...
noe's user avatar
  • 26.9k
1 vote

What is the difference between conformal prediction and uncertainty estimation

The main difference is that Conformal Prediction is a much better framework because it is: distribution free works with any model without ... provided mathematical guarantees of validity for final ...
valeman's user avatar
  • 11
1 vote

When to use mean vs median

I was trying to find a center where the absolute difference with all the other values are minimized. It turned out to be median. $$ f(center) = \sum | center - x | $$ After we sort all the values, ...
KRoy's user avatar
  • 111

Only top scored, non community-wiki answers of a minimum length are eligible