0
$\begingroup$

Excuse me for this brief description of the problem, as I'm very bound on time, I'll try to sum up as much as I can.

I have a multivariate time-series, that I trained using an RNN, there are periods and repeating time indexes, from 2013-01 to 2016-09, steps are months, by repeating, I mean various subsets ordered from January to December, many times for the same year, for hundreds of times, and I am predicting the next year knowing other features. Predicting using Keras on real test data expecting the same shape as train data I trained using LSTM, on 3 years, and trying to predict also repeating time-series for the year 2017. I used fixed batch size, and one last layer for binary target value so I used such a basic neural network:

model = Sequential()
model.add(LSTM(10, batch_input_shape=(12, train_X.shape[1], train_X.shape[2]), return_sequences=True, stateful=True, activation='sigmoid', inner_activation='hard_sigmoid'))
model.add(Dropout(0.2))
# model.add(LSTM(70))
# model.add(Dropout(0.3))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])

The batch size is 12 ( I chose) for 12 months, the target in train is very unbalanced with

pd.value_counts(test_y)
1.0 163781
0.0 5551
dtype: int64

Yet, I waited for some low probability for one in other words predicts of zeros.

res = model.predict(t_e_s_t, batch_size=12)

res
array([[0.9633749 ],
[0.79078996],
[0.99266464],
...,
[0.9891131 ],
[0.7582535 ],
[0.95778626]], dtype=float32)

All values of probability are above 0.5 and near 1 that means no probability for any entry to be zero. What could be wrong?

[Edit]

I added

from sklearn.utils import class_weight

class_weights = class_weight.compute_class_weight('balanced',
                                                 np.unique(train_y),
                                                 train_y) 

and passed class_weights to fit method, still zero values under 0.5.

$\endgroup$
1
$\begingroup$

In unbalanced scenarios such as this, often the probability estimates are skewed massively in favour of the majority class, but the model is still learning to assign lower probabilities to the minority class, so it can be a good idea to choose a threshold higher than 0.5 to determine your final classification. It would be a good idea to look at a precision-recall curve (implemented in scikit-learn) which considers all thresholds between 0 and 1 and plots the precision and recall against each other for each threshold. You should pick the threshold that gives the best trade-off for your problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.