I have a data set on predicting solar power generation, the dataset has 20 independent var and 1 dependent. The accuracy of my model is stuck at 60%. I have tried several models but this accuracy is the best I could get other sucks even more. Here is my code:

data_path = r'drive/My Drive/Proj/S.P.F./solarpowergeneration.csv'
dts = pd.read_csv('solarpowergeneration.csv')
X = dts.iloc[:, :-1].values
y = dts.iloc[:, -1].values
print(X.shape, y.shape)
y = np.reshape(y, (-1,1))
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
from sklearn.preprocessing import StandardScaler
sc= StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
y_train = sc.fit_transform(y_train)
y_test = sc.transform(y_test)

import keras.backend as K  
def calc_accu(y_true, y_pred):
  return K.mean(K.equal(K.round(y_true), K.round(y_pred)))

def get_spfnet():
  ann = tf.keras.models.Sequential()

  ann.add(Dense(X_train.shape[1], activation='relu'))
# ann.add(BatchNormalization())
  ann.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
 # ann.add(BatchNormalization())
  ann.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
 # ann.add(BatchNormalization())

              metrics=[tf.keras.metrics.RootMeanSquaredError(), calc_accu])
  return ann

spfnet = get_spfnet()
hist = spfnet.fit(X_train, y_train, batch_size=32, epochs=250, verbose=2)

the accuracy and loss graphs are:

plt.title('Model Accuracy')
plt.title('Model error')

enter image description here enter image description here

after 50 epochs nothing seems to improve, neither curve seems to overfit on the data I tried other models like reducing layers and removing kernel regularizes, using

kernel_initlizers='normal' and 'he-normal'  

but they perform poorly stuck at 20%.


enter image description here enter image description here


1 Answer 1


There are two issues here.

First, you are in a regression setting, where accuracy is meaningless - see What function defines accuracy in Keras when the loss is mean squared error (MSE)?

Although the linked thread is not directly applicable here (since you use your own custom calc_accu function), the essence of the argument remains: assessing K.equal(K.round(y_true), K.round(y_pred)) is not the correct way to measure the performance of a regression model. In such settings often the loss (MAE, MSE, RMSE) and the metric are the same; what you do here (accuracy aside), i.e having an MSE loss and an RMSE metric is also valid.

Second, there is no way to tell if an RMSE of ~ 0.5 is "good" or "bad" in itself, even more since here you scale your output y before feeding it to the model, hence the calculated RMSE is on these scaled values, and not in the true, unscaled ones. What you have to do is calculate the "true" RMSE in the original unscaled values, and then decide if it is good enough for your purpose or not. Essentially what you have to do is:

  1. Get predictions for your (scaled) test data
  2. Transform back your predictions to the initial scale of the original test data
  3. Get the RMSE between these invert-transformed predictions and the original data

To do so (and a good practice in general), it is necessary to define different scalers for your x and y, i.e.:

from sklearn.preprocessing import StandardScaler

# feature scaling
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test = sc_X.transform(X_test)

# outcome scaling:
sc_y = StandardScaler()
y_train = sc_y.fit_transform(y_train)    
y_test = sc_y.transform(y_test)

Now, assuming the data have been scaled as I show above, here is how you should calculate your RMSE on the original (unscaled) test data:

from sklearn.metrics import mean_squared_error

y_pred = model.predict(X_test) # get model predictions (scaled inputs here)
y_pred_orig = sc_y.inverse_transform(y_pred) # unscale the predictions
y_test_orig = sc_y.inverse_transform(y_test) # unscale the true test outcomes

RMSE_orig = mean_squared_error(y_pred_orig, y_test_orig, squared=False)

It is this RMSE_orig that you should decide if it is satisfactory for your problem; plotting the original unscaled predictions y_pred_orig against the original (unscaled) outcomes y_test_pred might also be useful and informative.

See own answer in SO thread How to interpret MSE in Keras Regressor for a similar case.

Other than that, it is speculative to say anything more since we do not have your data; nevertheless, a general rule is that we don't use dropout by default - only when we suspect that our model might overfit the data. So, I suggest experimenting with all dropout layers removed, and possibly putting them back in only gradually, and only in the case you observe overfitting.

  • $\begingroup$ i ran the algorithm again now i am getting weird results, r2scores test=0.7028 train=0.8951198650591028 and rmse of test=476.133 train=280.747 here is the colab link $\endgroup$ Sep 28, 2020 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.