34

The idea with Neural Networks is that they need little pre-processing since the heavy lifting is done by the algorithm which is the one in charge of learning the features. The winners of the Data Science Bowl 2015 have a great write-up regarding their approach, so most of this answer's content was taken from: Classifying plankton with deep neural networks. ...


29

In the interest of preventing information about the distribution of the test set leaking into your model, you should go for option #2 and fit the scaler on your training data only, then standardise both training and test sets with that scaler. By fitting the scaler on the full dataset prior to splitting (option #1), information about the test set is used to ...


17

Great question, this is what is known in Machine Learning paradigm as either "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on. One of the critical assumption one would make to build a machine learning model for future prediction is that unseen data (test) comes from the same distribution as training data! However, in reality this rather ...


15

Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while LabelEncoder is for converting target variable. That's why OrdinalEncoder can fit data that has the shape of (n_samples, n_features) while LabelEncoder can only fit data that has the shape of (n_samples,) (though in the past one ...


13

One of the benefits of decision trees is that ordinal (continuous or discrete) input data does not require any significant preprocessing. In fact, the results should be consistent regardless of any scaling or translational normalization, since the trees can choose equivalent splitting points. The best preprocessing for decision trees is typically whatever is ...


13

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the split at 3. in your flow chart. A common step I think you have missed is imputation of missing values. I would put that before feature engineering. Overall I ...


10

This question on stackoverflow might help you. To sum up, some deep learning researchers think that padding a big part of the image is not a good practice, since the neural network has to learn that the padded area is not relevant for classification, and it does not have to learn that if you use interpolation, for instance.


8

Benchmarking is the process of comparing your result to existing methods. You may compare to published results using another paper, for example. If there is no other obvious methodology against which you can benchmark, you might compare to the best naive solution (guessing the mean, guessing the majority class, etc) or a very simple model (a simple ...


8

there are multiple issues with the code: You force the values in the image to be uint8 (8-bit integer). Since the values are floats they will be casted/rounded to either 0 or 1. This will later be interpreted as image in black and the darkest form of gray (1 out of 255). Once you have proper floats as values PIL or pillow can't handle the array (they only ...


8

This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply: data_df[data_df > 15000] If this data is sitting in a particular column, you can use this instead: data_df[data_df['column_name'] > 15000] These will return the peak values. Updated Answer: If ...


7

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each new category. However, that speed and simplicity also leads to the "curse of dimensionality" by creating a new dimension for each category. Embedding is a ...


7

It would be better if you could provide some code which allows us to reproduce at least part of your DataFrame, such as this: import pandas as pd df = pd.DataFrame({'code': ['6254', '5854, 5676, 7265, 6051', '5815']}) At the start, your dataframe looks like this: code 0 6254 1 5854, 5676, 7265, 6051 2 ...


7

Yes, the encoding would be lost. You should instead use sklearn OneHotEncoder and save the corresponding encoder instance so that you can re-load it on unseen data. One can do something along these lines: import pandas as pd import pickle from sklearn.preprocessing import OneHotEncoder def get_encoder_inst(feature_col): """ returns: an instance of ...


6

Welcome to the real world of data science. Here, the data sets are not as clean as you thought while doing those courses/tutorials online. Those are super polished and refined. But, the real world data is not so. The step where one does the cleaning and scrubbing is called the data pre-processing step. So, some nice data cleaning techniques, in addition ...


6

We first center our data by subtracting the mean of the batch. We also divide by the standard deviation, so our formula becomes: $ z = \frac{x - \mu}{\sigma} $ where: $ x $ is the pixel value $ \mu $ is the arithmetic mean of the channel distribution $ \sigma $ is the standard deviation of the channel, calculated as such: $ \sigma = \sqrt{\frac{\sum^...


6

It could be your sampling strategy If you are oversampling by just duplicating data from class 0, then it is likely that you are overfitting. The same datapoint will be seen over and over. You could try another oversampling strategy, for example, SMOTE or ADASYN. These techniques create data points that are closets to decision boundaries so you are less ...


6

Good question. Your interpretation is adequate. Using a logarithmic function reduces the skewness of the target variable. Why does that matter? Transforming your target via a logarithmic function linearizes your target. Which is useful for many models which expect linear targets. Scikit-Learn has a page describing this phenomenon: https://scikit-learn.org/...


5

If I understood them correctly, both Jeremy and Edmund's (first) solutions are the same, namely, plain euclidean distance in a 4-dimensional space of IP addresses.BTW, I think a very fast alternative to euclidean distance would be to calculate a hamming distance bit-wise. Edmund's first update would be better than his second. The reason is simple to state: ...


5

I want to subsample observations from training set which closely resembles test set. I am not sure you'd want to do that. The whole purpose is rather to train your algorithm so that it generalises well to unseen data. Usually, one should adapt its test data to its train data (e.g. standardising test data according to train data) and not the other way ...


5

You have a few options: For Small Images: upsample through interpolation pad the image using zeros If you are unable to maintain the aspect ratio via upsampling, you can upsample and also crop the excess pixels in the largest dimension. Of course this would result in losing data, but you can repeatedly shift the center of your crop. This would help your ...


5

As for differences in OrdinalEncoder and LabelEncoder implementation, the accepted answer mentions the shape of the data: (OrdinalEncoder for 2D data; shape (n_samples, n_features), LabelEncoder is for 1D data: for shape (n_samples,)) That's why a OrdinalEncoder would get an error: ValueError: Expected 2D array, got 1D array instead: ...if trying to fit ...


5

No, it does not make sense to do this. You model has learned how to map one input space to another, that is to say it is itself function approximation, and will likely not know what to for the unseen data. By not performing the same scaling on the test data, you are introducing systematic errors in the model. This was pointed out in the comments by nanoman ...


4

That's a very interesting question. Similarity here should be computed component-wise, but the thing is from a "business logic" perspective, the similarity of the last 3 numbers doesn't matter if the other 3 sets of numbers are not the same. Keeping that in mind, I would probably do something like the following (there is probably a more elegant way of doing ...


4

Recurrent Neural Networks can deal with variable length data. You might want to have a look at: Andrej Karpathy: The Unreasonable Effectiveness of Recurrent Neural Networks. Christopher Olah: Understanding LSTM Networks. Hochreiter, Schmidthuber: Long short-term memory. Another idea (which I have not tested so far and just came to my mind) is using a ...


4

There is an easy way to use one hot encoding in pandas and you can read about it in the following link: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html The get_dummies method is easy and optimized for use in pandas Data Frames. Good luck!


4

It may be a late answer, but I got the same problem and below is the solution # Don't use categorical_features=[10] in encoder init from sklearn.preprocessing import OneHotEncoder onehotencoder=OneHotEncoder() Y= onehotencoder.fit_transform(X[:,[10]]).toarray()


4

It seems that Embedding vector is the best solution here. However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick". In this variant, when the number of unique words is too large to be assigned a unique index in a dictionary, one may hash words of into vector of fixed size. One advantage in your use case is that you may ...


4

Why are you convinced that the issue is in pre-processing? It could be in many other spots or a combination of steps in your process. If I were you, I would revisit WHY you're using a decision tree. There are many other approaches to image processing and neural networks are far superior - in most cases - to decision trees.


4

Assuming both of x_data and labels are lists or numpy arrays, train_data = [] for i in range(len(x_data)): train_data.append([x_data[i], labels[i]]) trainloader = torch.utils.data.DataLoader(train_data, shuffle=True, batch_size=100) i1, l1 = next(iter(trainloader)) print(i1.shape)


Only top voted, non community-wiki answers of a minimum length are eligible