4
$\begingroup$

I have data that I'm compressing with AutoEncoders (3-layer neural network) and I would like to normalize my data first. I would like to try to use the coded latent vector and feed it into an anomaly detection algorithm and see what happens.

I would like to normalize the data for the autoencoder so my values are either between 0,1 or -1,-1 because my output activation function will either be a sigmoid or tanh. This way my algorithm can train and the input will be in the same range as the output values of the NN.

However, when I normalized with

x(i)-xmean/(xmax-xmin) 

I ended up dividing by 0 in several features of the data which gave NaN. Is is possible to normalize my data so it is between -1,1 or 0,1 while avoiding dividing by 0 for my data?

$\endgroup$
  • 1
    $\begingroup$ I just realized that if my max and min are the same value, which is why I would get zero in thd denominator then I should just remove those columns. $\endgroup$ – zipline86 Sep 28 '18 at 16:37
0
$\begingroup$

While you could do this manually, Python also has a handy little function called MinMaxScaler, which will automatically apply max-min normalization to scale data between 0 and 1.

Assume we have an array of 200 values for variables s and t:

import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler

mu, sigma = 20, 10 # mean and standard deviation
s = np.random.normal(mu, sigma, 200)
t = np.random.normal(mu, sigma, 200)

Reshape your variables if necessary:

s=np.reshape(s,(-1,1))
t=np.reshape(t,(-1,1))

Now, you can see that we are forming two new variables, snew and tnew, which we are scaling using MinMaxScaler.

scaler = MinMaxScaler()
print(scaler.fit(s))
print(scaler.fit(s))
snew=scaler.transform(s)
tnew=scaler.transform(t)

Here is a sample of our new variables:

>>> snew
array([[0.24896606],
       [0.63121206],
       [0.60448469],
       .......
       [0.49044733],
       [0.28131596],
       [0.32909155]

>>> tnew
array([[0.91224005],
       [0.74540598],
       [0.3938718 ],
       .......
       [0.75749275],
       [0.80709325],
       [0.19440844]
| improve this answer | |
$\endgroup$
0
$\begingroup$

As others pointed out, you can normalize or standardize your data using the following steps. I'm sure other libraries have similar functions but I think this is efficient.

Since you requested normalization, I'll cover that topic in this post. As others alluded, data normalization is the process in which researchers or data science practitioners make all the values in a given dataset be proportionally spread between 0 and 1.

To implement normalization, follow the steps below:

from sklearn.datasets import load_iris
from sklearn import preprocessing

iris = load_iris()
print(iris.data.shape)

X_data = iris.data
y_labels = iris.target

normalized_X_data = preprocessing.normalize(X_data)
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.