Neural network for Multiple integer output

Question

I have a data set that contains 135 input features and 132 output values to be predicted. The input features are all numeric floating point values and each output value would be an integer between [0,1,2,3,4].

I am basically new to data science and machine learning, therefore I need to understand what kind of neural network model (regression or classification) would fit best for this kind of data. On one hand the output values represent different classes but I am not sure how a single neural network can predict multiple classes from the same input data.

Answer 1

welcome to the site!

I think the key word you need to know that defines your task is: multi-target classification or regression.

You can find an explanation and some possible techniques at this link.

For neural networks:

The key is to remember that the last layer should have linear activations (i.e. no activation at all).

As per your requirements, the shape of the input layer would be a vector (135,) and the output (132,).

The usual loss function used for regression problems is mean squared error (MSE). Here's an example of multidimensional regression using Keras:

model = Sequential()
model.add(Dense(200, input_dim = (135,)))
model.add(Activation('relu'))
model.add(Dense(200))
model.add(Activation('relu'))
model.add(Dropout(0.3))
model.add(Dense(132))
model.compile(loss='mean_absolute_error', optimizer='Adam')

Answer 2

Take a look at the softmax activation function for the output layer for classification.

Answer 3

Keras functional API allows to easily add more than one target.

It is a little daunting to try this out, since there are not too many good examples for this technique online. Below I post a simple model based on Boston data, which shows how multi-target (or multi-output) models can be implemented using Keras. See also here.

import numpy as np
import pandas as pd
from keras.datasets import boston_housing
(train_data, train_targets), (test_data, test_targets) =  boston_housing.load_data()

# Standardise data
mean = train_data.mean(axis=0)
train_data -= mean
std = train_data.std(axis=0)
train_data /= std
test_data -= mean
test_data /= std

# Add an additional target (just add some random noise to the original one)
import random
train_targets2 = train_targets + random.uniform(0, 0.1)
test_targets2   = test_targets + random.uniform(0, 0.1)

# https://keras.io/models/model/
from keras import models
from keras import layers
from keras.layers import Input, Dense
from keras.models import Model
from keras import regularizers
from keras.layers.normalization import BatchNormalization

# Input and model architecture
Input_1=Input(shape=(13, ))
x = Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.05))(Input_1)
x = Dense(512, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)

# Outputs
out1 = Dense(1)(x)
out2 = Dense(1)(x)

# Compile/fit the model
model = Model(inputs=Input_1, outputs=[out1,out2])
model.compile(optimizer = "rmsprop", loss = 'mse')
# Add actual data here in the fit statement
model.fit(train_data, [train_targets,train_targets2], epochs=500, batch_size=4, verbose=0, validation_split=0.8)

# Predict / check type and shape
preds = np.array(model.predict(test_data))
#print(type(preds), preds.shape)
# is a 3D numpy array

# get first part of prediction (column/row/3D layer)
preds0 = preds[0,:,0]
# second part
preds1 = preds[1,:,0]

# Check MAE
from sklearn.metrics import mean_absolute_error
print(mean_absolute_error(test_targets, preds0))
print(mean_absolute_error(test_targets2, preds1))