1
$\begingroup$

I'm just starting out with using sklearn for my own Machine Learning project and I'm using sklearn's built-in "Diabetes" dataset.

While performing data exploration on the features, I noticed something a bit confusing to me about the sex feature. Here's the hist plot:

enter image description here

Now there are 2 things I do understand here:

  1. The binary histogram makes sense, there are in this dataset 2 distinct 'sexes' of male and female.
  2. Them being numerical also makes sense, as it appears all features in this dataset have already been 'normalized'.

What I don't understand is why the values are the way they are? (See below for what the values are)


>>> from sklearn import datasets
>>> diab_df = datasets.load_diabetes(as_frame=True)
>>> features = diab_df['data']
>>> features.sex.unique()

array([ 0.05068012, -0.04464164])

How are these numbers derived? At first, I thought it could be some sort of stratified sampling, where if the true population distribution is say, 53% male, 47% female, then I'd maybe expect to see the values in this hist to be -0.47 & 0.53 or something?

$\endgroup$
1
  • 3
    $\begingroup$ Have a look at the standardization procedure: "Note: Each of these 10 feature variables have been mean centered and scaled by the standard deviation times n_samples (i.e. the sum of squares of each column totals 1)." When you apply this to the un-standardized dataset you should get the standardized values as in given in the sklearn dataset. $\endgroup$
    – Jonathan
    Sep 5, 2021 at 12:10

1 Answer 1

1
$\begingroup$

The data description says:

Note: Each of these 10 feature variables have been mean centered and scaled by the standard deviation times n_samples (i.e. the sum of squares of each column totals 1). https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html

For more information see: Bradley Efron, Trevor Hastie, Iain Johnstone and Robert Tibshirani (2004) "Least Angle Regression," Annals of Statistics (with discussion), 407-499. http://web.stanford.edu/~hastie/Papers/LARS/LeastAngle_2002.pdf

from sklearn import datasets
print(datasets.load_diabetes())
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.