1
$\begingroup$

I have programmed a MLP for a dataset (~500 rows) containing the length (L) and width (W) of an organism and the output of biomass (the organisms weight in pounds, B).

            mlp = MLPRegressor((5, 5), max_iter=1000)

I have trained the model with features

# Model 1
# Input = Feature 1: Length, Feature 2: Width. Output = Biomass
df = {'length': [60.1, 59.2, 59.4, 58.5], 'width': [15.4, 16.2, 14.9, 15.7], 'weight': [8.34, 7,65, 7.89, 7.14]}


# Model 2
# Input =  Feature 1: Length * Width^2. Output = Biomass
df = {'length*height^2': [60.1, 59.2, 59.4, 58.5], 'weight': [14253.31, 15536.44, 13187.39, 14419.66]}

The overall accuracy of my model with one feature is over 95%, however the accuracy with the features separated is about 85%.

My understanding of an MLP is that Model 1 should do better than model 2 as it will basically find the best combination of length and height to biomass, However my 1 feature model is doing significantly better. I have also tried standardizing the dataset with a scaler with no luck.

scaler = StandardScaler()
$\endgroup$
1
$\begingroup$

$h$ - height
$w$ - width
$p$ - weight

In real world we know that weight is volume times density. And volume is product of height, width and depth. If you assume constant depth and density the weight is just some constant multiplied by width and height i.e. $p = c \times hw$. If you have $hw$ as a feature the NN quickly learns $c$. This gives good performance.

Lets say your NN had identity as activation function i.e. no activation function. All it can learn is linear combination of input features.

So it will just learn $p \approx h w_h + w w_w + b$ where $w_h, w_w$ are weights and $b$ is bias.

If you have activation function like ReLU/Sigmoid it still learn linear combinations of outputs of ReLU/Sigmoid. Inputs to those is just linear combination of $h$ and $w$. In all that there is never a simple $hw$ term. This makes model complicated without having the important feature that is volume(area).

I think that is why the performance of $hw$ model is better.


If you have a NN with 2 neurons in the hidden layer with sigmoid activation, and 1 output neuron with no activation, then this is how $hw$ term shows up.

$\frac{1}{1+\exp(hw_{11} +w w_{12} + b_1)} + \frac{1}{1+exp(hw_{21} +w w_{22} + b_2)} = \frac{\ldots}{\ldots + \exp(w_{11} w_{22} hw + w_{12}w_{21}hw) } $

Thats very tricky place to show up if you just want to learn a constant multiplier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.