I am reading, a machinelearningmastery.com post about improving neural network performance, and I am attempting to normalize my own data set with the tips mentioned in the post using sklearn preprocessing MinMaxScaler.
In my code I am reading a CSV file directly into pandas.
#read CSV file df = pd.read_csv('C:\\Users\\desktop\\EC\\data.csv', index_col='Date', parse_dates=True)
From the post, I normalize the data with this code below. The process appears to just return a numpy array, but I use Pandas during the machine learning fit process.
from sklearn.preprocessing import MinMaxScaler # create scaler scaler = MinMaxScaler() # fit and transform in one step df2 = scaler.fit_transform(df) df2 = pd.DataFrame(df2)
What's happening, is my column names are stripped away and I use column names a lot in dropping & selecting. For example, I fit a lot of models like this process below to differentiate target and input variables.
#Test random Forest import numpy as np from sklearn import preprocessing, cross_validation, neighbors import pandas as pd from sklearn.ensemble import RandomForestRegressor from sklearn.externals import joblib import math rmses =  for i in range(2): X = np.array(df2.drop(['Demand'],1)) y = np.array(df2['Demand']) offset = int(X.shape * 0.7) X_train, y_train = X[:offset], y[:offset] X_test, y_test = X[offset:], y[offset:] clf = RandomForestRegressor(n_estimators=120, min_samples_split=20) clf.fit(X_train, y_train) mse = mean_squared_error(y_test, clf.predict(X_test)) rmse = math.sqrt(mse) print("rmse: %.4f" % rmse) rmses.append(rmse) print(sum(rmses)/len(rmses)) joblib.dump(clf, 'rfrModel.pkl')
But maybe this isn't a bid deal... Sorry not a lot of wisdom here any tips help. Is there another method to madness that I don't need to dependent on column names? If the
df2 = scaler.fit_transform(df) leaves columns in place but just removes names, I could just use a column number to
.drop - differentiate target & input variables..
Demand is the name of my target variable, and I could just call the second column..., right??