Is pre-processing always neccessary?

Question

I'm working on classification of two classes of Raman spectra. And while I was working on finding the optimal steps for pre-processing, I started to wonder if it is really necessary.

I have a lot of Raman spectra data and and most of them differ between both classes even to the naked eye (characteristic peaks or the lack of). While I do have many datasets I know were not recorded under exact same conditions, I also have a few sets for which exact same conditions apply for both classes. They still look obviously different from each other. Since my primary goal is to develop a robust algorithm with an unsupervised classificator, I now start to wonder:

Assuming the setup and thus noise and any sort of distortion is the same in both classes, even if dominant, do I need to bother with pre-processing when various classifiation models already return satisfying results with the raw data?

Most (if not all) papers I read about Raman spectra pre-processing (there's a lot on that topic) gave me the impression, the authors are always interested in separating the Raman spectrum of interest from all the unwanted other signals (fluorescent background, white noise, cosmic spikes etc). While this certainly would be interesting, I actually only want to sort them.

I'm asking this question from a rather hypothetical standpoint. I know that in reality you can hardly guarantee exact same conditions (especially with random occurrences like cosmic spikes).

Answer 1

[Not specific for Raman spectroscopy]

There is a trend in some areas of machine learning to aim at end-to-end learning, that is, devise machine learning algorithms that take raw input data and gives the desired output (e.g. classification label, regression values). This way, end-to-end learning avoids introducing expert knowledge or preprocessing, usually relying only on huge amounts of data. This has been specially true in the last years in the field of deep learning, where images are taken with no preprocessing, audio is not filter-banked, and games are played by receiving only raw pixels.

So it is certainly possible to skip data preprocessing completely.

We should nevertheless take into account that most of the successful end-to-end machine learning approaches impose constraints that enable the learning to exploit the very aspects of the input data that were previously defined by experts. An example for this are Convolutional Networks, that impose a feature locality constraint that enables the learning to identify features, which could have been pre-defined as gabor filters instead.

BUT, in more traditional data science, feature engineering is a key part of the process of devising ML systems and data exploration and expert knowledge are normally used to identify the kind of preprocessing to be applied to data.