1
$\begingroup$

I am reading, a machinelearningmastery.com post about improving neural network performance, and I am attempting to normalize my own data set with the tips mentioned in the post using sklearn preprocessing MinMaxScaler.

In my code I am reading a CSV file directly into pandas.

#read CSV file
df = pd.read_csv('C:\\Users\\desktop\\EC\\data.csv', index_col='Date', parse_dates=True)

The snip below is what the data looks like with column names: enter image description here

From the post, I normalize the data with this code below. The process appears to just return a numpy array, but I use Pandas during the machine learning fit process.

from sklearn.preprocessing import MinMaxScaler

# create scaler
scaler = MinMaxScaler()

# fit and transform in one step
df2 = scaler.fit_transform(df)

df2 = pd.DataFrame(df2)

What's happening, is my column names are stripped away and I use column names a lot in dropping & selecting. For example, I fit a lot of models like this process below to differentiate target and input variables.

#Test random Forest

import numpy as np
from sklearn import preprocessing, cross_validation, neighbors
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.externals import joblib
import math

rmses = []
for i in range(2):

    X = np.array(df2.drop(['Demand'],1))
    y = np.array(df2['Demand'])

    offset = int(X.shape[0] * 0.7)
    X_train, y_train = X[:offset], y[:offset]
    X_test, y_test = X[offset:], y[offset:]

    clf = RandomForestRegressor(n_estimators=120, min_samples_split=20)

    clf.fit(X_train, y_train)
    mse = mean_squared_error(y_test, clf.predict(X_test))
    rmse = math.sqrt(mse)
    print("rmse: %.4f" % rmse)
    rmses.append(rmse)

print(sum(rmses)/len(rmses))

joblib.dump(clf, 'rfrModel.pkl')

But maybe this isn't a bid deal... Sorry not a lot of wisdom here any tips help. Is there another method to madness that I don't need to dependent on column names? If the df2 = scaler.fit_transform(df) leaves columns in place but just removes names, I could just use a column number to .drop - differentiate target & input variables.. Demand is the name of my target variable, and I could just call the second column..., right??

enter image description here

$\endgroup$
1
$\begingroup$

You can just use the previous column names for creating a new dataframe

# Use the previous column names from df and assign it to df2   
df2 = pd.DataFrame(df2, columns=df.columns)
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.