Nuclear Hoagie
  • Member for 4 years, 4 months
  • Last seen this week
13 answers
42 votes
23k views
Data science related funny quotes
42 votes

Q: How many machine learning specialists does it take to change a light bulb? A: Just one, but they require a million light bulbs to train properly. Q: How many machine learning specialists does it ...

View answer
2 answers
3 votes
797 views
How to handle missing data for machine learning
Accepted answer
6 votes

There are three main approaches to handling missing data. Impute - use some method to fill in the missing values with reasonable guesses. You could interpolate between two time points, take the ...

View answer
1 answers
5 votes
4k views
Are neural networks able to deal with non-normalised inputs?
Accepted answer
5 votes

Normalization helps to eliminate scale factors that might exist between variables in your data. Take, for example, the classic problem of predicting home prices. If you represent the square footage of ...

View answer
2 answers
2 votes
3k views
What could explain a much higher F1 score in comparision to accuracy score?
Accepted answer
4 votes

You have imbalanced classes. Notice that your accuracy is very close to your precision, and quite dissimilar to your recall. This means that your precision (accuracy of positive predictions) is ...

View answer
6 answers
27 votes
15k views
What is the reason behind taking log transformation of few continuous variables?
4 votes

Yet another reason why logarithmic transformations are useful comes into play for ratio data, due to the fact that log(A/B) = -log(B/A). If you plot a distribution of ratios on the raw scale, your ...

View answer
1 answers
3 votes
7k views
The Area Under an ROC Curve (AUC) vs Confusion Matrix for classifier evaluation?
Accepted answer
4 votes

A confusion matrix can be used to measure the performance of a particular classifier with a fixed threshold. Given a set of input cases, the classifier scores each one, and score above the threshold ...

View answer
1 answers
2 votes
23 views
P-value mining on large number of combinations of variables
3 votes

You need to investigate multiple hypothesis correction methods, like Bonferroni correction or Benjamini-Hochberg false discovery rate. The problem with this sort of analysis is that your associations ...

View answer
2 answers
1 votes
32 views
Dealing with zeros when plotting log-scaled data
2 votes

For this type of issue, I typically add the reciprocal of the log base. For data that's being log10-scaled, this results in adding 0.1 to all values. For data that's being log2-scaled, this results in ...

View answer
2 answers
1 votes
607 views
Is it correct to say that a high variance feature will be important to my model?
2 votes

High feature variance does not imply any sort of relationship to the target variable you're interested in modeling. Suppose you are looking at a population and have two variables for each person, ...

View answer
3 answers
2 votes
140 views
Interpreting the new dimensions after PCA
2 votes

One way to explore the mapping between the original dimensions and and PCA dimensions is to look at something called the factor loadings. These are essentially projections of your original dimensions ...

View answer
2 answers
1 votes
66 views
Interpretation of PCA visualisation
2 votes

Do not apply PCA to categorical data PCA attempts to find the dimensions which contains the most variance in a dataset. When you have categorical variables, the distance between points and the ...

View answer
3 answers
3 votes
4k views
Purpose of converting continuous data to categorical data
Accepted answer
2 votes

Your intuition is generally correct - in many cases, premature discretization of continuous variables is undesirable. Doing so throws away potentially meaningful data, and the result can be highly ...

View answer
2 answers
0 votes
151 views
Why does prediction by a consensus of classifier work better than prediction by a single classifiers?
Accepted answer
2 votes

For a very simple example, imagine you have three independent classifiers that each have 60% accuracy. If you use any one of them to classify a random sample, you have a 60% chance of getting it right....

View answer
1 answers
0 votes
32 views
Machine learning accuracy for not a class-imbalanced problem
Accepted answer
1 votes

Accuracy treats all misclassifications as the same - we only care whether we got the answer right or not, but don't care about what kind of error was made. Even for class-balanced problems, this may ...

View answer
1 answers
0 votes
33 views
Replace NAs with random string from a list
Accepted answer
1 votes

This should do the trick. You set up your vector of possible fruits to select from, and then everywhere there is an NA in df$fruit, pick a random element from the possible fruit vector to overwrite it ...

View answer
1 answers
0 votes
50 views
Determining extreme outliers from boxplot (by eye)
1 votes

These images aren't very useful to make decisions by eye, because the scale is far too compressed. In the second and third images the mean, entire box and both whiskers are all represented by a single ...

View answer
1 answers
0 votes
39 views
Using Gini index, how to calculate the probability of correctly classifying a new randomly selected case to the highest probability class?
1 votes

While you can calculate the underlying class probabilities from the Gini index (for binary classification), it'll be more straightforward to calculate it from the "value" line in each box. ...

View answer
3 answers
1 votes
1k views
Does shuffling data for time series forecasting help?
1 votes

Shuffling data would not seem to make sense here, since your model has "memory". You're not predicting $y_i$ from only $x_i$, but also $x_{i-1}$ and $x_{i-2}$. If you shuffle the data and perform ...

View answer
1 answers
0 votes
21 views
prediction() returning mistakenly false positives
Accepted answer
1 votes

The slot "fp" counts how many false positives there are at each choice of classification cutoff (which can be found in the "cutoff" slot). The cutoff represents at what value you set the threshold to ...

View answer
2 answers
2 votes
111 views
What is a manifold for Unsupervised Learning?
1 votes

A manifold is some kind of low-dimensions structure that exists in a higher-dimensional space. The classic example of this is the Swiss Roll dataset, which simply looks like a spiral with values that ...

View answer
4 answers
0 votes
1k views
How to know if my Decision tree model is good or bad?
1 votes

It appears that you are training your model and generating predictions on the same dataset (X_new). You should not attempt to evaluate your model's performance using this output - because you are ...

View answer
1 answers
0 votes
25 views
Statistical comparison of model performance when training and validation data is always the same
Accepted answer
1 votes

If your model is deterministic (no randomness), then repeating the training/testing on the exact same set of data is pointless - you will get the exact same answer every time. The benefit of cross-...

View answer
2 answers
1 votes
3k views
difference betwen predicting seen and unseen data
1 votes

For data with lots of features, it's generally the case that many of those features will be unrelated or weakly related to your target variable of interest. It's possible to build a model using even ...

View answer
2 answers
0 votes
4k views
Smaller test data set than training data set in machine learning
1 votes

There is no reason to sub-sample your test data. The test data serves to give an unbiased estimation of your model learner's performance on unseen data, and more test data only gives you a more ...

View answer
3 answers
0 votes
74 views
How to deal with new features values in my classification model?
1 votes

You could include a category of Other into which you can bin all non-valid responses. This person's response means that they are not single, married, or divorced - it does not represent Unknown, and ...

View answer
2 answers
1 votes
26 views
Sampling n data points from high dimensional data
0 votes

One approach will be to run an unsupervised clustering algorithm of your choosing, parameterized to return a 100-cluster solution. If there are indeed groups of images, you should see that similar ...

View answer
2 answers
1 votes
64 views
What's my target variable?
Accepted answer
0 votes

Your target variable is whatever you want to predict. For this particular dataset, logical choices are probably "death", "event", or "AZT". You'd typically want to use some kind of patient data to ...

View answer
2 answers
2 votes
362 views
Unable to understand the usage of labels argument in sklearn.metrics.f1_score
0 votes

The F1 measure is a type of class-balanced accuracy measure - when there are only two classes, it's very straightforward, as there's only one possible way to compute it. With 3 classes, however, you ...

View answer
3 answers
0 votes
59 views
Bias and variance in the model o in the predictions?
0 votes

In some cases, you may have a model that's a black box - you feed input features and get output predictions, without knowing or caring what happens in the middle. In those situations, the model is, in ...

View answer
2 answers
2 votes
74 views
Sequence prediction with unlimited predictions
0 votes

I don't have a proof for this, but I expect the best you can do is estimate the normal distribution of interval times from your training data, and then just predict points at the mean interval time. ...

View answer