Feature Extraction using deep learning but classification Using boosting or other ML algorithms

Question

Is it logical to perform feature extraction using deep learning but classification using traditional machine learning or boosting techniques at the same time?

Is it okay to use ML algorithms for classification rather than fully connected layers where the feature extraction is already been done using deep learning I am aware that if feature engineering is successful, the model will automatically perform well.

For example: If I want to do a text classification problem, can I build such a model as ( Stack of RNN blocks + Adaboost), where RNN blocks perform feature extraction and Adaboost does classification? Does it make sense?

I have proposed an architecture using an integrated deep learning framework for feature extraction, however, can I incorporate an integrated framework with traditional ML or boosting algorithm for classification? Will it be a redundant thing?

Answer 1

Yes, this is a thing. Maybe looking into the paper "ConvXGB: A new deep learning model for classification problems based on CNN and XGBoost" will give you an intuition - although it does take advantage of CNN instead of RNN [...].

The contributions of this paper are:

– A new deep learning model for classification problems called “ConvXGB” based on combine between CNN and XGBoost.

– The ConvXGB architecture consists of a net with several stacked convolutional layers and with XGBoost as the last layer of the model. It differs from the traditional CNN, because there is neither a pooling layer nor a Fully Connected (FC) layer. This introduces simplicity and reduces the number of calculation parameters, since it is not necessary to bring weights from the FC layers back to re-adjust weights in the previous layers.

– ConvXGB uses auto feature learning effectively and predicts class labels, with higher accuracy than the two individual models, which are the current prototypes for modeling, and other extant models, e.g. Decision Tree Classification, Multilayer Perceptron and Support Vector Classification.

Answer 2

In my opinion, it does not make sense from a hypothetical perspective. The Boosting models and Recurrent Neural Network models are designed to serve different purposes. If we consider natural language processing-based tasks like the one you mentioned regarding text classification, we were previously solving text classification problems using the bagging models, then we brought boosting models, then we brought Recurrent Neural Networks, and now we are utilizing Transformer based architectures using an attention mechanism. If you already have used an advanced model in the previous stage which did great work in converting textual features into representative numeric features while preserving the context of the data, it is not a costly thing to use a Sigmoid or Softmax layer to classify such text. Bringing another boosting or all-alone machine learning architecture does not make sense in this case. If that would make sense Google Research, Microsoft Research, NeurIPS papers, and ICML papers would not bring novel architectures based on new concepts rather they would have randomly stacked several machine learning methods over one another to solve real-world problems.