39

In the interest of preventing information about the distribution of the test set leaking into your model, you should go for option #2 and fit the scaler on your training data only, then standardise both training and test sets with that scaler. By fitting the scaler on the full dataset prior to splitting (option #1), information about the test set is used to ...


35

The idea with Neural Networks is that they need little pre-processing since the heavy lifting is done by the algorithm which is the one in charge of learning the features. The winners of the Data Science Bowl 2015 have a great write-up regarding their approach, so most of this answer's content was taken from: Classifying plankton with deep neural networks. ...


35

Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while LabelEncoder is for converting target variable. That's why OrdinalEncoder can fit data that has the shape of (n_samples, n_features) while LabelEncoder can only fit data that has the shape of (n_samples,) (though in the past one ...


22

Great question, this is what is known in Machine Learning paradigm as either "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on. One of the critical assumption one would make to build a machine learning model for future prediction is that unseen data (test) comes from the same distribution as training data! However, in reality this rather ...


16

As for differences in OrdinalEncoder and LabelEncoder implementation, the accepted answer mentions the shape of the data: OrdinalEncoder is for 2D data with the shape (n_samples, n_features) LabelEncoder is for 1D data with the shape (n_samples,)) Maybe that's why the top-voted answer suggests OrdinalEncoder is for the "features" (often a 2D ...


15

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each new category. However, that speed and simplicity also leads to the "curse of dimensionality" by creating a new dimension for each category. Embedding ...


13

One of the benefits of decision trees is that ordinal (continuous or discrete) input data does not require any significant preprocessing. In fact, the results should be consistent regardless of any scaling or translational normalization, since the trees can choose equivalent splitting points. The best preprocessing for decision trees is typically whatever is ...


13

This question on stackoverflow might help you. To sum up, some deep learning researchers think that padding a big part of the image is not a good practice, since the neural network has to learn that the padded area is not relevant for classification, and it does not have to learn that if you use interpolation, for instance.


13

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the split at 3. in your flow chart. A common step I think you have missed is imputation of missing values. I would put that before feature engineering. Overall I ...


11

Assuming both of x_data and labels are lists or numpy arrays, train_data = [] for i in range(len(x_data)): train_data.append([x_data[i], labels[i]]) trainloader = torch.utils.data.DataLoader(train_data, shuffle=True, batch_size=100) i1, l1 = next(iter(trainloader)) print(i1.shape)


9

Benchmarking is the process of comparing your result to existing methods. You may compare to published results using another paper, for example. If there is no other obvious methodology against which you can benchmark, you might compare to the best naive solution (guessing the mean, guessing the majority class, etc) or a very simple model (a simple ...


8

there are multiple issues with the code: You force the values in the image to be uint8 (8-bit integer). Since the values are floats they will be casted/rounded to either 0 or 1. This will later be interpreted as image in black and the darkest form of gray (1 out of 255). Once you have proper floats as values PIL or pillow can't handle the array (they only ...


8

This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply: data_df[data_df > 15000] If this data is sitting in a particular column, you can use this instead: data_df[data_df['column_name'] > 15000] These will return the peak values. Updated Answer: If ...


7

We first center our data by subtracting the mean of the batch. We also divide by the standard deviation, so our formula becomes: $ z = \frac{x - \mu}{\sigma} $ where: $ x $ is the pixel value $ \mu $ is the arithmetic mean of the channel distribution $ \sigma $ is the standard deviation of the channel, calculated as such: $ \sigma = \sqrt{\frac{\sum^...


7

You have a few options: For Small Images: upsample through interpolation pad the image using zeros If you are unable to maintain the aspect ratio via upsampling, you can upsample and also crop the excess pixels in the largest dimension. Of course this would result in losing data, but you can repeatedly shift the center of your crop. This would help your ...


7

It would be better if you could provide some code which allows us to reproduce at least part of your DataFrame, such as this: import pandas as pd df = pd.DataFrame({'code': ['6254', '5854, 5676, 7265, 6051', '5815']}) At the start, your dataframe looks like this: code 0 6254 1 5854, 5676, 7265, 6051 2 ...


7

Yes, the encoding would be lost. You should instead use sklearn OneHotEncoder and save the corresponding encoder instance so that you can re-load it on unseen data. One can do something along these lines: import pandas as pd import pickle from sklearn.preprocessing import OneHotEncoder def get_encoder_inst(feature_col): """ returns: an instance of ...


7

Ideally PCA should not be used as a part of pre-processing feature reduction. Anyhow regarding saving and reusing PCA model, sharing a basic code snippet which is working very fine in my case(as I'm not able to reproduce the error case). from sklearn.decomposition import PCA import pickle as pk pca = PCA(n_components=2) result = pca.fit_transform(X) # ...


7

Searching the source code of Sklearn for SimpleImputer (with strategy= "most_frequent"), the most frequent value is calculated within a loop in python, therefore that is the part of code that is so slow. In the source code of SimpleImputer there is also the comment that explains why they do not use the scipy.stats.mstats.mode, which is mush faster: scipy....


6

Welcome to the real world of data science. Here, the data sets are not as clean as you thought while doing those courses/tutorials online. Those are super polished and refined. But, the real world data is not so. The step where one does the cleaning and scrubbing is called the data pre-processing step. So, some nice data cleaning techniques, in addition ...


6

I want to subsample observations from training set which closely resembles test set. I am not sure you'd want to do that. The whole purpose is rather to train your algorithm so that it generalises well to unseen data. Usually, one should adapt its test data to its train data (e.g. standardising test data according to train data) and not the other way ...


6

It could be your sampling strategy If you are oversampling by just duplicating data from class 0, then it is likely that you are overfitting. The same datapoint will be seen over and over. You could try another oversampling strategy, for example, SMOTE or ADASYN. These techniques create data points that are closets to decision boundaries so you are less ...


5

If I understood them correctly, both Jeremy and Edmund's (first) solutions are the same, namely, plain euclidean distance in a 4-dimensional space of IP addresses.BTW, I think a very fast alternative to euclidean distance would be to calculate a hamming distance bit-wise. Edmund's first update would be better than his second. The reason is simple to state: ...


5

It seems that Embedding vector is the best solution here. However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick". In this variant, when the number of unique words is too large to be assigned a unique index in a dictionary, one may hash words of into vector of fixed size. One advantage in your use case is that you may ...


5

Good question. Your interpretation is adequate. Using a logarithmic function reduces the skewness of the target variable. Why does that matter? Transforming your target via a logarithmic function linearizes your target. Which is useful for many models which expect linear targets. Scikit-Learn has a page describing this phenomenon: https://scikit-learn.org/...


5

No, it does not make sense to do this. You model has learned how to map one input space to another, that is to say it is itself function approximation, and will likely not know what to for the unseen data. By not performing the same scaling on the test data, you are introducing systematic errors in the model. This was pointed out in the comments by nanoman ...


5

To check if a value has changed, you can use .diff and check if it's non-zero with .ne(0) (the NaN in the top will be considered different than zero), and then count the changes with .cumsum, like this: df['counter'] = df.diff().ne(0).cumsum() Afterward, you can create a second dataframe, where the indices are the groups of consecutive values, and the ...


5

By sampling randomly from the main dataset the percentages from the subsets should roughly equal the percentages from the main dataset. If you however want a more precise way of doing this look into using a stratified method, which allows you to keep class frequencies when splitting/sampling the data. The scikit-learn implementation of the train_test_split ...


4

That's a very interesting question. Similarity here should be computed component-wise, but the thing is from a "business logic" perspective, the similarity of the last 3 numbers doesn't matter if the other 3 sets of numbers are not the same. Keeping that in mind, I would probably do something like the following (there is probably a more elegant way of doing ...


Only top voted, non community-wiki answers of a minimum length are eligible