I am reading, a machinelearningmastery.com post about improving neural network performance, and I am attempting to normalize my own data set with the tips mentioned in the post using sklearn preprocessing MinMaxScaler.
In my code I am reading a CSV file directly into pandas.
#read CSV file
df = pd.read_csv('C:\\Users\\desktop\\EC\\data.csv', index_col='Date', parse_dates=True)
The snip below is what the data looks like with column names:
From the post, I normalize the data with this code below. The process appears to just return a numpy array, but I use Pandas during the machine learning fit process.
from sklearn.preprocessing import MinMaxScaler
# create scaler
scaler = MinMaxScaler()
# fit and transform in one step
df2 = scaler.fit_transform(df)
df2 = pd.DataFrame(df2)
What's happening, is my column names are stripped away and I use column names a lot in dropping & selecting. For example, I fit a lot of models like this process below to differentiate target and input variables.
#Test random Forest
import numpy as np
from sklearn import preprocessing, cross_validation, neighbors
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.externals import joblib
import math
rmses = []
for i in range(2):
X = np.array(df2.drop(['Demand'],1))
y = np.array(df2['Demand'])
offset = int(X.shape[0] * 0.7)
X_train, y_train = X[:offset], y[:offset]
X_test, y_test = X[offset:], y[offset:]
clf = RandomForestRegressor(n_estimators=120, min_samples_split=20)
clf.fit(X_train, y_train)
mse = mean_squared_error(y_test, clf.predict(X_test))
rmse = math.sqrt(mse)
print("rmse: %.4f" % rmse)
rmses.append(rmse)
print(sum(rmses)/len(rmses))
joblib.dump(clf, 'rfrModel.pkl')
But maybe this isn't a bid deal... Sorry not a lot of wisdom here any tips help. Is there another method to madness that I don't need to dependent on column names? If the df2 = scaler.fit_transform(df) leaves columns in place but just removes names, I could just use a column number to .drop - differentiate target & input variables.. Demand is the name of my target variable, and I could just call the second column..., right??
