Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am working on an air-gapped PC and can only access the SciKit-Learn machine learning library. Unfortunately, its MLPClassifier doesn't include a Dropout hyperparameter, which severely limits the size of network I'm able to build.
How can I extend the Class and add Dropout to the algorithm/ model?
To implement this correctly you need to understand exactly how Dropout works, and where to change SciKit-Learn's MLPClassifier class to implement it. I'll start with an explanation of Dropout followed by the specifics of implementation. Skip to the end if you just want the code snippet.
The image below shows dropout visually, and is taken from the canonical Dropout paper by Srivistava, and Hinton et al.
The method is a form of regularisation that approximates averaging the predictions of hundreds to hundreds of thousands of smaller networks without massive computational overhead. For more details, read the paper above, for even more details read this paper and this paper. For a simple overview, watch Andrew Ng's video on the topic.
Put simply, Dropout switches off nodes in a Neural Net's layers, creating a series of smaller networks for your model to train on. It does this whilst preserving the overall architecture of your Neural Net. To create the smaller networks, each node is switched off randomly with a probability, $P$, a new hyperparameter, which makes the architecture of the smaller networks random. Training on varying, randomised architectures prevents nodes from Co-Adapting and overfitting the noise in your data.
Dropout is only applied during training, once the network is trained it uses its complete architecture to make predictions.
From a technical point of view there are two things to consider, the Forward Pass, and Backpropagation.
In the forward pass the smaller network is created by building a Dropout Mask that sets the activation function of each switched off node to $0$, and multiplies the activation function of the remaining nodes by $1/P$.
Multiplying the remaining nodes' activation functions by $1/P$ is known as Inverted Dropout. This method ensures that the network sees the same scale of activations (nodes' output values) in training as it does in testing. Without this scaling, switching off nodes with a probability $P$ will scale the size of the activations by $(1-P)$. If this isn't counteracted, the network will be trained to make predictions on lower activation levels (node output values). This becomes an issue at test time, when the entire network is used, which makes the activations, on average, $1/P$ times larger, ruining the networks predictive ability.
Once the Dropout mask has been applied, the input is passed through the network normally.
To understand how backpropagation works with Dropout, it's worth adding an excerpt from the canonical Dropout paper.
It states that backpropagation should be performed on the thinned network only, which means we have to prevent the algorithm updating the weights of switched off nodes' forwards and backwards connections as well as their biases (intercepts). This prevents nodes being trained on predictions that they contributed nothing towards.
To implement this model, you need to understand SciKit-Learn's model design. This webpage from SciKit-Learn goes over the development of new estimators. However, developing a whole new estimator is not necessary, instead we'll create a subclass of MLPClassifier called MLPDropout. I have assumed a base level of understanding in Python in Classes, Objects, Methods, Functions, etc. so if you don't understand Python I suggest you learn that first. I will post the entire code snippet at the end of this post, so skip to the end if you just want the answer. Keep reading if you're interested in the specific changes you have to make.
To subclass MLPClassifier you will need the following imports, add them to the top of your file or Jupyter notebook.
from sklearn.neural_network import MLPClassifier
from sklearn.neural_network._stochastic_optimizers import AdamOptimizer
from sklearn.neural_network._base import ACTIVATIONS, DERIVATIVES, LOSS_FUNCTIONS
from sklearn.utils import shuffle, gen_batches, check_random_state, _safe_indexing
from sklearn.utils.extmath import safe_sparse_dot
import warnings
from sklearn.exceptions import ConvergenceWarning
To control dropout in your new model you'll have to add a Dropout hyperparameter to the class's initialisation. This can be done by adding a new parameter to your subclass's initialisation function, which you can see at the bottom of def __init__(...) in the following code snippet.
class MLPDropout(MLPClassifier):
def __init__(
self,
... ( Parameters not included, see full code below.)
dropout = None,
):
'''
Additional Parameters:
----------
dropout : float in range (0, 1), default=None
Dropout parameter for the model, defines the percentage of nodes
to remove at each layer.
'''
self.dropout = dropout
super().__init__( ... see parameters in full code below.)
The additional parameter is set as an internal attribute of the MLPDropout estimator, and super().__init__(...) calls MLPClassifiers initialisation function with the remaining parameters, as normal.
layer_units)This next step is not strictly necessary, but seems to follow SciKit-Learn's design principles. layer_units is a variable instantiated by MLPClassifer that defines the node architecture of the Neural Net. To create the Dropout mask we need to pass this variable to the forward pass and backpropagation methods. Their are a couple of class methods that call forward pass and backpropagation, but, assuming you're using Stochastic Gradient Descent, the method we're interested in changing is _fit_stochastic, which is passed the layer_units parameter.
In _fit_stochastic's call to _backprop, the function used by MLPClassifer to perform the forward pass and backpropagation, we add layer_units parameter.
# (DROPOUT ADDITION) layer_units passed forward to help build dropout mask.
batch_loss, coef_grads, intercept_grads = self._backprop(
X_batch,
y_batch,
activations,
layer_units,
deltas,
coef_grads,
intercept_grads,
)
accumulated_loss += batch_loss * (
batch_slice.stop - batch_slice.start
)
To see this in context, skip to the complete code snippet below. You also need to edit _backprop so that it can accept layer_units:
def _backprop(self, X, y, activations, layer_units, deltas, coef_grads, intercept_grads):
"""Compute the MLP loss function and its corresponding derivatives
with respect to each parameter: weights and bias vectors.
Parameters
----------
... (Parameters above cut off, see full code below)
layer_units (DROPOUT ADDITION) : list, length = n_layers
The layer units of the neural net, this is the shape of the
Neural Net model. This is used to build the dropout mask.
... (Parameters below cut off, see full code below)
To create the dropout mask you can use the layer_units variable, and the self.dropout hyperparameter to create a list of numpy arrays that mask the activation functions of the nodes.
# Create the Dropout Mask (DROPOUT ADDITION)
if self.dropout != None:
if 0 < self.dropout < 1:
keep_probability = 1 - self.dropout
dropout_masks = [np.ones(layer_units[0])]
# Create hidden Layer Dropout Masks
for units in layer_units[1:-1]:
# Create inverted Dropout Mask, check for random_state
if self.random_state != None:
layer_mask = (self._random_state.random(units) < keep_probability).astype(int) / keep_probability
else:
layer_mask = (np.random.rand(units) < keep_probability).astype(int) / keep_probability
dropout_masks.append(layer_mask)
else:
raise ValueError('Dropout must be between zero and one. If Dropout=X then, 0 < X < 1.')
This creates a dropout mask that matches the size of each layer and adds it to a list. The first mask is for the input, which is set to all 1's in this implementation. Sometimes dropout is performed on the input nodes, but usually only in tasks where loss of a percentage of input data is minimally important. For example, image recognition. For tabular data this should be avoided.
The _forward_pass method is called from inside of _backprop and must be updated to accept a dropout_masks variable. First within _backprop:
# Forward propagate
activations = self._forward_pass(activations, dropout_masks)
and then in the method definition:
def _forward_pass(self, activations, dropout_masks=None):
"""Perform a forward pass on the network by computing the values
of the neurons in the hidden layers and the output layer.
Parameters
----------
activations : list, length = n_layers - 1
The ith element of the list holds the values of the ith layer.
dropout_mask : list, length = n_layers - 1
The ith element of the list holds the dropout mask for the ith
layer.
"""
dropout_masks has a default of None so that MLPDropout can be used exactly like MLPClassifier if you wish to have a dropout of zero. Assuming you are using Dropout, the dropout_masks are applied within _forward_pass by adding the following code to the forward pass for loop:
# Apply Dropout Mask (DROPOUT ADDITION)
if (i + 1) != (self.n_layers_ - 1) and dropout_masks != None:
check1 = activations[i].copy()
activations[i+1] = activations[i+1] * dropout_masks[i+1][None, :]
_forward_pass's for loop iteratively assigns activations to a list. Multiplying the layer's activations by its corresponding mask makes the algorithm behave as if the masked nodes have been switched off.
Finally, the mask must be applied again in backpropagation, as MLPClassifier includes a regularisation term that will update the weights of every node in a network regardless of their inclusion in the forward pass. The _backprop function calculates these gradients for you, and then passes them back to _fit_stochastic, which updates the model's coefficients (weights and biases). To follow the algorithm exactly, and update the thinned network only, you must zero out the coefficient gradients that correspond to switched off nodes. You can do this using the dropout_masks by adding the following code to the end of _backprop:
# Apply Dropout Masks to the Parameter Gradients (DROPOUT ADDITION)
if dropout_masks != None:
for layer in range(len(coef_grads)-1):
mask = (~(dropout_masks[layer+1] == 0)).astype(int)
coef_grads[layer] = coef_grads[layer] * mask[None, :]
coef_grads[layer+1] = (coef_grads[layer+1] * mask.reshape(-1, 1))
intercept_grads[layer] = intercept_grads[layer] * mask
The first line, in the for loop, defines a new mask that no longer includes a scaling factor. The second line zeroes out the coefficient gradients for the weights connecting to switched off nodes in the current layer from the previous layer. The third line zeroes out the coefficient gradients of all connection weights passing from switched off nodes in the current layer to the next layer. The fourth line zeroes out the bias (SciKit-Learn calls it intercept) gradients of all switched off nodes in the current layer.
At this stage, all the changes you need to make are complete. Please see the complete code for a new MLPDropout class below. Basic tests show that this model is marginally better than MLPClassifier.
# Creating a custom MLPDropout classifier
from sklearn.neural_network import MLPClassifier
from sklearn.neural_network._stochastic_optimizers import AdamOptimizer
from sklearn.neural_network._base import ACTIVATIONS, DERIVATIVES, LOSS_FUNCTIONS
from sklearn.utils import shuffle, gen_batches, check_random_state, _safe_indexing
from sklearn.utils.extmath import safe_sparse_dot
import warnings
from sklearn.exceptions import ConvergenceWarning
class MLPDropout(MLPClassifier):
def __init__(
self,
hidden_layer_sizes=(100,),
activation="relu",
*,
solver="adam",
alpha=0.0001,
batch_size="auto",
learning_rate="constant",
learning_rate_init=0.001,
power_t=0.5,
max_iter=200,
shuffle=True,
random_state=None,
tol=1e-4,
verbose=False,
warm_start=False,
momentum=0.9,
nesterovs_momentum=True,
early_stopping=False,
validation_fraction=0.1,
beta_1=0.9,
beta_2=0.999,
epsilon=1e-8,
n_iter_no_change=10,
max_fun=15000,
dropout = None,
):
'''
Additional Parameters:
----------
dropout : float in range (0, 1), default=None
Dropout parameter for the model, defines the percentage of nodes
to remove at each layer.
'''
self.dropout = dropout
super().__init__(
hidden_layer_sizes=hidden_layer_sizes,
activation=activation,
solver=solver,
alpha=alpha,
batch_size=batch_size,
learning_rate=learning_rate,
learning_rate_init=learning_rate_init,
power_t=power_t,
max_iter=max_iter,
shuffle=shuffle,
random_state=random_state,
tol=tol,
verbose=verbose,
warm_start=warm_start,
momentum=momentum,
nesterovs_momentum=nesterovs_momentum,
early_stopping=early_stopping,
validation_fraction=validation_fraction,
beta_1=beta_1,
beta_2=beta_2,
epsilon=epsilon,
n_iter_no_change=n_iter_no_change,
max_fun=max_fun,
)
def _fit_stochastic(
self,
X,
y,
activations,
deltas,
coef_grads,
intercept_grads,
layer_units,
incremental,
):
params = self.coefs_ + self.intercepts_
if not incremental or not hasattr(self, "_optimizer"):
if self.solver == "sgd":
self._optimizer = SGDOptimizer(
params,
self.learning_rate_init,
self.learning_rate,
self.momentum,
self.nesterovs_momentum,
self.power_t,
)
elif self.solver == "adam":
self._optimizer = AdamOptimizer(
params,
self.learning_rate_init,
self.beta_1,
self.beta_2,
self.epsilon,
)
# early_stopping in partial_fit doesn't make sense
early_stopping = self.early_stopping and not incremental
if early_stopping:
# don't stratify in multilabel classification
should_stratify = is_classifier(self) and self.n_outputs_ == 1
stratify = y if should_stratify else None
X, X_val, y, y_val = train_test_split(
X,
y,
random_state=self._random_state,
test_size=self.validation_fraction,
stratify=stratify,
)
if is_classifier(self):
y_val = self._label_binarizer.inverse_transform(y_val)
else:
X_val = None
y_val = None
n_samples = X.shape[0]
sample_idx = np.arange(n_samples, dtype=int)
if self.batch_size == "auto":
batch_size = min(200, n_samples)
else:
if self.batch_size < 1 or self.batch_size > n_samples:
warnings.warn(
"Got `batch_size` less than 1 or larger than "
"sample size. It is going to be clipped"
)
batch_size = np.clip(self.batch_size, 1, n_samples)
try:
for it in range(self.max_iter):
if self.shuffle:
# Only shuffle the sample indices instead of X and y to
# reduce the memory footprint. These indices will be used
# to slice the X and y.
sample_idx = shuffle(sample_idx, random_state=self._random_state)
accumulated_loss = 0.0
for batch_slice in gen_batches(n_samples, batch_size):
if self.shuffle:
X_batch = _safe_indexing(X, sample_idx[batch_slice])
y_batch = y[sample_idx[batch_slice]]
else:
X_batch = X[batch_slice]
y_batch = y[batch_slice]
activations[0] = X_batch
# (DROPOUT ADDITION) layer_units passed forward to help build dropout mask.
batch_loss, coef_grads, intercept_grads = self._backprop(
X_batch,
y_batch,
activations,
layer_units,
deltas,
coef_grads,
intercept_grads,
)
accumulated_loss += batch_loss * (
batch_slice.stop - batch_slice.start
)
# update weights
grads = coef_grads + intercept_grads
self._optimizer.update_params(params, grads)
self.n_iter_ += 1
self.loss_ = accumulated_loss / X.shape[0]
self.t_ += n_samples
self.loss_curve_.append(self.loss_)
if self.verbose:
print("Iteration %d, loss = %.8f" % (self.n_iter_, self.loss_))
# update no_improvement_count based on training loss or
# validation score according to early_stopping
self._update_no_improvement_count(early_stopping, X_val, y_val)
# for learning rate that needs to be updated at iteration end
self._optimizer.iteration_ends(self.t_)
if self._no_improvement_count > self.n_iter_no_change:
# not better than last `n_iter_no_change` iterations by tol
# stop or decrease learning rate
if early_stopping:
msg = (
"Validation score did not improve more than "
"tol=%f for %d consecutive epochs."
% (self.tol, self.n_iter_no_change)
)
else:
msg = (
"Training loss did not improve more than tol=%f"
" for %d consecutive epochs."
% (self.tol, self.n_iter_no_change)
)
is_stopping = self._optimizer.trigger_stopping(msg, self.verbose)
if is_stopping:
break
else:
self._no_improvement_count = 0
if incremental:
break
if self.n_iter_ == self.max_iter:
warnings.warn(
"Stochastic Optimizer: Maximum iterations (%d) "
"reached and the optimization hasn't converged yet."
% self.max_iter,
ConvergenceWarning,
)
except KeyboardInterrupt:
warnings.warn("Training interrupted by user.")
if early_stopping:
# restore best weights
self.coefs_ = self._best_coefs
self.intercepts_ = self._best_intercepts
def _backprop(self, X, y, activations, layer_units, deltas, coef_grads, intercept_grads):
"""Compute the MLP loss function and its corresponding derivatives
with respect to each parameter: weights and bias vectors.
Parameters
----------
X : {array-like, sparse matrix} of shape (n_samples, n_features)
The input data.
y : ndarray of shape (n_samples,)
The target values.
activations : list, length = n_layers - 1
The ith element of the list holds the values of the ith layer.
layer_units (DROPOUT ADDITION) : list, length = n_layers
The layer units of the neural net, this is the shape of the
Neural Net model. This is used to build the dropout mask.
deltas : list, length = n_layers - 1
The ith element of the list holds the difference between the
activations of the i + 1 layer and the backpropagated error.
More specifically, deltas are gradients of loss with respect to z
in each layer, where z = wx + b is the value of a particular layer
before passing through the activation function
coef_grads : list, length = n_layers - 1
The ith element contains the amount of change used to update the
coefficient parameters of the ith layer in an iteration.
intercept_grads : list, length = n_layers - 1
The ith element contains the amount of change used to update the
intercept parameters of the ith layer in an iteration.
Returns
-------
loss : float
coef_grads : list, length = n_layers - 1
intercept_grads : list, length = n_layers - 1
"""
n_samples = X.shape[0]
dropout_masks = None
# Create the Dropout Mask (DROPOUT ADDITION)
if self.dropout != None:
if 0 < self.dropout < 1:
keep_probability = 1 - self.dropout
dropout_masks = [np.ones(layer_units[0])]
# Create hidden Layer Dropout Masks
for units in layer_units[1:-1]:
# Create inverted Dropout Mask, check for random_state
if self.random_state != None:
layer_mask = (self._random_state.random(units) < keep_probability).astype(int) / keep_probability
else:
layer_mask = (np.random.rand(units) < keep_probability).astype(int) / keep_probability
dropout_masks.append(layer_mask)
else:
raise ValueError('Dropout must be between zero and one. If Dropout=X then, 0 < X < 1.')
# Forward propagate
# Added dropout_makss to _forward_pass call (DROPOUT ADDITION)
activations = self._forward_pass(activations, dropout_masks)
# Get loss
loss_func_name = self.loss
if loss_func_name == "log_loss" and self.out_activation_ == "logistic":
loss_func_name = "binary_log_loss"
loss = LOSS_FUNCTIONS[loss_func_name](y, activations[-1])
# Add L2 regularization term to loss
values = 0
for s in self.coefs_:
s = s.ravel()
values += np.dot(s, s)
loss += (0.5 * self.alpha) * values / n_samples
# Backward propagate
last = self.n_layers_ - 2
# The calculation of delta[last] here works with following
# combinations of output activation and loss function:
# sigmoid and binary cross entropy, softmax and categorical cross
# entropy, and identity with squared loss
deltas[last] = activations[-1] - y
# Compute gradient for the last layer
self._compute_loss_grad(
last, n_samples, activations, deltas, coef_grads, intercept_grads
)
inplace_derivative = DERIVATIVES[self.activation]
# Iterate over the hidden layers
for i in range(self.n_layers_ - 2, 0, -1):
deltas[i - 1] = safe_sparse_dot(deltas[i], self.coefs_[i].T)
inplace_derivative(activations[i], deltas[i - 1])
self._compute_loss_grad(
i - 1, n_samples, activations, deltas, coef_grads, intercept_grads
)
# Apply Dropout Masks to the Parameter Gradients (DROPOUT ADDITION)
if dropout_masks != None:
for layer in range(len(coef_grads)-1):
mask = (~(dropout_masks[layer+1] == 0)).astype(int)
coef_grads[layer] = coef_grads[layer] * mask[None, :]
coef_grads[layer+1] = (coef_grads[layer+1] * mask.reshape(-1, 1))
intercept_grads[layer] = intercept_grads[layer] * mask
return loss, coef_grads, intercept_grads
def _forward_pass(self, activations, dropout_masks=None):
"""Perform a forward pass on the network by computing the values
of the neurons in the hidden layers and the output layer.
Parameters
----------
activations : list, length = n_layers - 1
The ith element of the list holds the values of the ith layer.
dropout_mask : list, length = n_layers - 1
The ith element of the list holds the dropout mask for the ith
layer.
"""
hidden_activation = ACTIVATIONS[self.activation]
# Iterate over the hidden layers
for i in range(self.n_layers_ - 1):
activations[i + 1] = safe_sparse_dot(activations[i], self.coefs_[i])
activations[i + 1] += self.intercepts_[i]
# For the hidden layers
if (i + 1) != (self.n_layers_ - 1):
hidden_activation(activations[i + 1])
# Apply Dropout Mask (DROPOUT ADDITION)
if (i + 1) != (self.n_layers_ - 1) and dropout_masks != None:
check1 = activations[i].copy()
activations[i+1] = activations[i+1] * dropout_masks[i+1][None, :]
# For the last layer
output_activation = ACTIVATIONS[self.out_activation_]
output_activation(activations[i + 1])
return activations