# Tag Info

## New answers tagged preprocessing

3

Assuming I understand it correctly, I think your process is ok this way but I'm not sure about step 3 "standardization": Steps 1 and 2 are ok since this cannot leak any kind of information from the test set to the training set. If the standardization step involves calculating values (e.g. mean and s.d.) over the whole data (including test set) and ...

0

You can also do this with the dplyr package. The dplyr package has the functions group_by to group your data by one or more variables and summarise to do some aggregation function. The dplyr package also supports the 'pipe' notation %>%. This notation means the output of the previous function is the first argument of the next function. Here is what it ...

0

I like to use the plyr library but there are other ways: library(plyr) ddply(mydata, c('Replicate','Node','Day'), nrow) the dd in ddply means that the input is a dataframe and the output is also a dataframe the rows are grouped by the values of columns given as second argument the last argument is the function to apply on every group, in this case nrow to ...

1

To Tokenise, clean up symbols (i.e. Normalise), etc. just use one of the widely used NLP libraries, they should be able to do most of the work for you. Examples include: NTLK Spacy SparkNLP .. and many more. Perhaps look up some articles comparing their strengths and weaknesses on Google to decide what's best with your project. As for the detecting English ...

0

To do this, simply create a column with the language of the review and filter non-English reviews. To detect languages, I'd recommend using langdetect. This would like something like this: import pandas as pd def is_english(text): // Add language detection code here return True // or False cleaned_df = df[is_english(df["review”])]

0

You can still use StandardScaler() as it will keep the negative values. If you think you have a few outliers, and want to reduce their influence, you can also look at RobustScaler().

0

You can use Normalization. Normalization rescale your mean to 0 and standard deviation to 1 containing both positive and negative value. $X_{Normalised} = \frac{X - \mu}{\sigma}$ Here $\mu$ is your original mean and $\sigma$ is your standard deviation.

0

There's not one right answer to this question, because what scaler works best really depends on the data and the algorithm you use to make the prediction. You should try different scalers combined with different algorithms to decide which preprocessing is best by comparing the cross validation results of each pipeline. Of course, you don't have unlimited ...

0

If it is a classification problem, then you will use sigmoid or softmax to make the output value in (0,1) and all the value must sum to 1 as per the rule of probability.

1

A quick experiment you can do is to once do the preprocessing steps that you usually do and then feed it to the model and get the results. And once feed the dataset as it is to the model to compare the difference. In my experience doing the preprocessing won't make any difference, based on the dataset it gave me 1 more or less percent difference in accuracy (...

0

That's not a good way to represent symbolic monophonic music. The biggest problem is that time should be modelled as a discrete and not a continuous variable. The reason for that is that music with notes that are off the beat sounds like crap. E.g if the tempo indicates that one beat is 250 ms long and one beat randomly is only 200 ms long it will sound very ...

1

I would try two different approaches: interpolate the missing values on a user level. work with the sunset of rows for which we actually have the glucose level. Then, I would compare the test accuracy of the model built with both methods. Remember that your test set has to be composed of rows for which you have the glucose level - you cannot build it with ...

0

For any data science project when put into production. One has to follow the same preprocessing techniques used before training in production & testing. There is no difference in techniques while performing prediction in testing/production. Let's assume you have used this function in preprocessing same can be used in production also as it is just before ...

Top 50 recent answers are included