2
$\begingroup$

I'm studying the following code, which cross_val_score_ was used as well as .mean() and .std(). I read many documentation of the meanings, but didn't get what each of the above does.

import pandas as pd 
import numpy as np
from sklearn import tree 
import graphviz
from sklearn.model_selection import cross_val_score

#importing the dataset
d = pd.read_csv('student-por.csv', sep= ';')

d['pass'] = d.apply(lambda row: 1 if (row['G1']+ row['G2']+ row ['G3']) >= 35 else 0 , axis=1)
d = d.drop(['G1', 'G2','G3'], axis=1 )

#Doing one-hot encoding
d=pd.get_dummies(d, columns =['sex','activities','school', 'address', 'famsize','Pstatus','Mjob','Fjob','reason','guardian','schoolsup','famsup','paid','nursery','higher','internet','romantic'])

#shuffle rows
d = d.sample(frac=1)

#split traning and test
d_train = d[:500]
d_test = d[500:]

d_train_att = d_train.drop(['pass'], axis=1)
d_train_pass= d_train['pass']

d_test_att = d_test.drop(['pass'], axis=1)
d_test_pass= d_test['pass']

d_att = d.drop(['pass'], axis=1)
d_pass = d['pass']

t = tree.DecisionTreeClassifier(criterion ='entropy', max_depth = 5)
t= t.fit (d_train_att, d_train_pass)

#to export the tree
dot_data = tree.export_graphviz(t,out_file = 'students-tree.png', label ='all', impurity=False, proportion= True, feature_names=list(d_train_att), class_names=['fail', 'pass'], filled = True, rounded=True)

t.score (d_test_att, d_test_pass)
scores = cross_val_score(t, d_att,d_pass, cv=5)

print ('Acuracy %0.2f (+/- %0.2f)' % (scores.mean(), scores.std() *2))

in short this is what I need to know:

scores = cross_val_score(t, d_att,d_pass, cv=5)

print ('Acuracy %0.2f (+/- %0.2f)' % (scores.mean(), scores.std() *2))

one more thing, am I suppose to get the same score as in the original code publisher? because I didn't.

|improve this question
$\endgroup$
  • $\begingroup$ In general, model training depends to some extent on randomness, in which case you would expect not to get the same answer twice. Sometimes you can set the random "seed" to mitigate this behavior, if you so choose. $\endgroup$ – Scott Jul 9 '18 at 4:08
1
$\begingroup$

The source, around line 274 is where the default scoring for cross_validation_score gets set, if you pass in None for the scorer argument. For classifiers, the usual default score is accuracy. For regression, it's rmse, IIRC. So, since you're applying a decision tree classifier, cross_val_score splits the data into 5 equalish sized pieces, trains on each combination of 4 and gives back the accuracy of the estimator on the 5th. The mean and std of these accuracies presumably tells one something about the performance of the family of decision tree classifiers on your dataset, but I would take it with a grain of salt.

|improve this answer
$\endgroup$
  • 1
    $\begingroup$ So, what is the important of cross_val_score ? will it override the .fit to get the maximum score? $\endgroup$ – Mohamed Abduljawad Jul 7 '18 at 6:52
  • 1
    $\begingroup$ No. The goal of cross validation is to do a thing that can ordinarily only be done with an out-of-training sample, using only training data. For example, accuracy on the training set is pretty meaningless for many model families (especially decision trees, which are notorious for overfitting). The score of interest is accuracy on out-of-training data. One way to measure this is to set aside a test set. Another way that does not involve setting aside perfectly good training data is to do cross validation. $\endgroup$ – Scott Jul 7 '18 at 20:43
  • 1
    $\begingroup$ So cross_val_score estimates the expected accuracy of your model on out-of-training data (pulled from the same underlying process as the training data, of course). The benefit is that one need not set aside any data to obtain this metric, and you can still train your model on all of the available data. $\endgroup$ – Scott Jul 7 '18 at 20:47
  • 1
    $\begingroup$ On a lower level, the importance of scoring in general is that, once you have .fit your model, it is completely unclear as to whether the model actually learned anything meaningful whatsoever. Thus, it is important to have metrics by which one can measure how well your model is doing. A common way to do this for classification problems is with accuracy on an out-of-training sample. However, this is not fool-proof, and it is necessary to use your headbrain to consider the implications and non-implications of any metric. $\endgroup$ – Scott Jul 7 '18 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.