2
$\begingroup$

The short version:
I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after using PCA in terms of accuracy and runtime. For some reason the runtime of the classifiers (XGBoost and AdaBoost to take 2 as an example) after the use of PCA is 3 times (approximately) the runtime of the classifiers before the use of PCA. My question is: why? am I doing something wrong or is it possible?

The long version:
my understanding of how to use PCA:

  • have normalized and clean datasets split into training and testing sets (using train_test_split).
  • PCA fit and transform the X_train and save it to a new df
  • Using the fitted PCA, transform (without fitting) the X_test
  • run the classifier with the transformed X_train and X_test

    PS: I have checked that the number of dimentions is decreasing (from 21 to the number specified: 17 in case of 90% of the variance). The dataset size is around 130000 entries, taken from kaggle. The code written to achieve this:
pca = PCA(n_components=0.9)
X_train_Reduced = pca.fit_transform(X_train)
X_test_Reduced = pca.transform(X_test)

Classifier (XGBoost) before the use of PCA:

start_timeXGBoost = time.time()
warnings.filterwarnings('ignore')
modelXGBoost = XGBClassifier(learning_rate = 0.2, n_estimators = 200, verbosity = 0, use_label_encoder = False, n_jobs = -1)
modelXGBoost.fit(X_train, y_train)
predictionsXGBoost = modelXGBoost.predict(X_test)
accuracyXGBoost = metrics.accuracy_score(y_test, predictionsXGBoost)
print("Accuracy (XGBoost): ", accuracyXGBoost)
timeXGBoost = time.time() - start_timeXGBoost
print("Time taken to achive result: %s seconds" % (timeXGBoost))

Output of code:

Accuracy (XGBoost): 0.9655066214967662
Time taken to achive result: 3.33561372756958 seconds


Classifier (XGBoost) After PCA:

start_timeXGBoost = time.time()
warnings.filterwarnings('ignore')
modelXGBoost = XGBClassifier(learning_rate = 0.2, n_estimators = 200, verbosity = 0, use_label_encoder = False,
                             n_jobs = -1)
modelXGBoost.fit(X_train_Reduced, y_train)
predictionsXGBoost = modelXGBoost.predict(X_test_Reduced)
accuracyXGBoost = metrics.accuracy_score(y_test, predictionsXGBoost)
print("Accuracy (XGBoost): ", accuracyXGBoost)
timeXGBoost = time.time() - start_timeXGBoost
print("Time taken to achive result: %s seconds" % (timeXGBoost))

Output of Code:

Accuracy (XGBoost): 0.93032029565753
Time taken to achive result: 10.376214981079102 seconds

Another example (AdaBoost)
Classifier (AdaBoost) before PCA:

start_timeAdaBoost = time.time()
modelDecTree = DecisionTreeClassifier(random_state=0, max_depth=2)
modelAdaBoost = AdaBoostClassifier(modelDecTree, n_estimators = 1000, random_state = 0, learning_rate = 1)
modelAdaBoost.fit(X_train, y_train)
predictionsAdaBoost = modelAdaBoost.predict(X_test)
accuracyAdaBoost = metrics.accuracy_score(y_test, predictionsAdaBoost)
print("Accuracy (AdaBoost): ", accuracyAdaBoost)
timeAdaBoost = time.time() - start_timeAdaBoost
print("Time taken to achive result: %s seconds" % (timeAdaBoost))

Output of code:

Accuracy (AdaBoost): 0.9575762242069603
Time taken to achive result: 103.38761949539185 seconds

Classifier (AdaBoost) after PCA:

start_timeAdaBoost = time.time()
modelDecTree = DecisionTreeClassifier(random_state=0, max_depth=2)
modelAdaBoost = AdaBoostClassifier(modelDecTree, n_estimators = 1000, random_state = 0, learning_rate = 1)
modelAdaBoost.fit(X_train_Reduced, y_train)
predictionsAdaBoost = modelAdaBoost.predict(X_test_Reduced)
accuracyAdaBoost = metrics.accuracy_score(y_test, predictionsAdaBoost)
print("Accuracy (AdaBoost): ", accuracyAdaBoost)
timeAdaBoost = time.time() - start_timeAdaBoost
print("Time taken to achive result: %s seconds" % (timeAdaBoost))

Output of code:

Accuracy (AdaBoost): 0.9141515244841392
Time taken to achive result: 295.6763050556183 seconds


I would very much appreciate any help in the matter of understanding what I have done wrong (or right).
Thank you all in advance
$\endgroup$

1 Answer 1

3
$\begingroup$

This is likely behavior when several of your original features are discrete.

Each tree, when splitting, considers a split for each unique value of each feature (in the current node). For discrete features this is often significantly smaller than the number of rows, while for continuous features it is often very nearly the same as the number of rows. When some features are discrete then, the count of candidate splits is generally much smaller than when all the features are continuous.

Since PCA amounts to essentially a rotation of the feature space, the resulting feature space will generally be purely-continuous, making the resulting tree decisions more expensive even when removing a few of the last principal components.

You could try using histogram splitting in xgboost, which discretizes the features before splitting to reduce computation time. I don't think this is currently available in sklearn's implementation of adaboost.

$\endgroup$
4
  • $\begingroup$ Thanks for the answer! Yes, almost all features of the data are discrete as a matter of fact. So basically using PCA for data which is mostly discrete is not advised? ever? Or are there instances when it is a good idea to do so? $\endgroup$ Aug 3, 2021 at 12:45
  • $\begingroup$ Also, why does PCA work on images (grayscale for example to make it easier). The data can also be counted as discrete (255 unique values). Or is it that there have to be enough unique values for the algorithm to work? if so, do you have an approximate number? $\endgroup$ Aug 3, 2021 at 13:09
  • $\begingroup$ None of my answer should suggest that it affects the performance with PCA, only the computation time needed for a tree algorithm to train. $\endgroup$
    – Ben Reiniger
    Aug 3, 2021 at 13:58
  • $\begingroup$ Of course, that is what I meant. $\endgroup$ Aug 3, 2021 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.