0
$\begingroup$

I am using a simple autoencoder to extract the informative features and I have multiple Q:

  1. I know that the features extracted will be a linear combination of the original features so I consider that the feature that has a larger mean weight (has the highest percentage in the formation of new features) will be important so I will take that features but I don't know if this is true or not

  2. the second things is that I want to apply the grid search to find the optimal hyperparameters for the model but I can't do that please if anyone can help me in this and save my life

$\endgroup$

2 Answers 2

0
$\begingroup$
  1. Autoencoders normally aren't linear models. If you make them linear (i.e. you create a shallow Autoencoder with linear activations) then you get exactly a PCA result. The power of Neural Networks is their non-linearity, if you want to stick with linearity go for PCA imho.

  2. Keep a Train-Validation-Test set split, and try different configurations of hyperparams checking their performance on Validation data. Alternatively there are many libraries, such as hyperopt, that let you implement more sophisticated Bayesian hyperparameter searches, but unless you want to be published at a conference or win some competition it's a bit overkill. If you're still interested, the internet is plenty of tutorials like this one.

$\endgroup$
5
  • $\begingroup$ if you want to stick with linearity go for PCA imho linear autoencoders are as powerful as PCA, so there is actually no "imho" $\endgroup$
    – anon
    Commented Nov 9, 2022 at 11:50
  • $\begingroup$ A shallow Autoencoder with linear activations returns exactly the same results as PCA $\endgroup$
    – Leevo
    Commented Nov 9, 2022 at 19:21
  • $\begingroup$ no, PCA has more constraints that AE, for example the principal components are orthogonal and ordered by importance, but the PCA matrix and the AE matrix span the same space $\endgroup$
    – anon
    Commented Nov 10, 2022 at 17:28
  • $\begingroup$ "If the autoencoder uses only linear activations and the cost function is the mean squared error (MSE), then it ends up performing Principal Component Analysis". Cit. from Aurelien Geron, "Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow", Chapter 17, p. 570 $\endgroup$
    – Leevo
    Commented Nov 11, 2022 at 15:24
  • $\begingroup$ Performing PCA and having the same result is not the same. You will have the same result, but it's going to be obtained with different matrices, but with the same spans... see _Baldi and Hornik, 1989: _.. this is due to the fact that even if the problem is convex, it's not strictly convex, ie can have infinite local minima, just like $f(x,y)=x^2$ has infinite solution, as y can be whatever $\endgroup$
    – anon
    Commented Nov 11, 2022 at 17:03
0
$\begingroup$

I am a beginner for AE. I have similar question with Q1. I know AE could compress information and extract new features which represent the input data. I found a paper which used AE to evaluate the importance of every feature in the origin matrix. In the subsequent analysis, the research leveraged these selected features to establish machine learning model like randomforest, XGBoost, and logistic regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.