Imagine I have the following dataset:

|      Feature 1      |  Feature 2       | possible labels  |
|          1          |         5        |    {1, 4}        |
|          2          |         2        |     {2}          |
|          4          |         3        |  {3, 5, 6}       |

and my goal is to be able to predict a value that is the closest to the set of possible labels. That is, this is a regression problem.

(A sub-optimal solution I would also accept as an answer would be for my goal to be to predict a value that belongs to the set of possible labels, not taking into account the distance between the predicted value and the elements of the set.)

Does this have a name? Are there any Python libraries (for deep learning, if possible) that solve this? Or should I define my custom loss function? If I have to define my custom loss function, could you suggest me a library/point me in the right direction? My main problem with that is that the set of possible labels has variable size! Thanks!

  • $\begingroup$ Mhm, I could maybe add 'invalid' elements to each list (maybe tensorflow has infinity?) so that they are of fixed size: stackoverflow.com/questions/40450506/… $\endgroup$ – Guillermo Mosse Dec 26 '19 at 11:30
  • 1
    $\begingroup$ Hi, what's the reason you want to tackle this with ML in general and Deep Learning specifically? $\endgroup$ – Sammy Dec 26 '19 at 11:39
  • $\begingroup$ I strongly suspect there is a (possibly, but not necessarily) linear correlation between the list of features and a subset of the valid labels for each data point. I'd like to know if this could be learned because I haven't discovered it myself so far. $\endgroup$ – Guillermo Mosse Dec 26 '19 at 12:07
  • $\begingroup$ I'm not sure about whether trying to predict the full list of possible labels is a good idea, because the space of possible labels is gigantic (they are binary strings) $\endgroup$ – Guillermo Mosse Dec 26 '19 at 12:09
  • 2
    $\begingroup$ Can you elaborate on what exactly do you mean by "closest to the set of possible labels", what is the actual metric you're optimizing? Would an ideal system output a set of multiple labels, or would an ideal system output a number that (hopefully) matches one of the numbers in that set? Are these outputs equidistant labels as in classification or numeric values as in regression, i.e. it the correct label is {1,4} would outputting 3 be 'just as bad' as outputting 10, or would 10 be much worse? In essence, what's your loss function? $\endgroup$ – Peteris Dec 27 '19 at 1:56

This could be treated as a multiclass & multilabel classification problem. If you are able to get your label lists in some good shape, you could check the Keras functional API for multitarget regression/classification models.

The main linitation is that you would need a list of outputs with equal length. I guess the best way is to do one-hot (aka dummy encoding) of your labels and to predict "classes" (labels). However, if the number of possible labels/classes is really large, you may get into trouble with this approach.

Using the Keras functional API is relatively straight forward. You simply need to define your outputs. Here is a minimal example. In order to do classification you would need to change a few things (loss function etc) but this should be no problem:

import numpy as np
import pandas as pd
from keras.datasets import boston_housing
(train_data, train_targets), (test_data, test_targets) =  boston_housing.load_data()

# Standardise data
mean = train_data.mean(axis=0)
train_data -= mean
std = train_data.std(axis=0)
train_data /= std
test_data -= mean
test_data /= std

# Add an additional target (just add some random noise to the original one)
import random
train_targets2 = train_targets + random.uniform(0, 0.1)
test_targets2   = test_targets + random.uniform(0, 0.1)

# https://keras.io/models/model/
from keras import models
from keras import layers
from keras.layers import Input, Dense
from keras.models import Model
from keras import regularizers
from keras.layers.normalization import BatchNormalization

# Input and model architecture
Input_1=Input(shape=(13, ))
x = Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.05))(Input_1)
x = Dense(512, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)

# Outputs
out1 = Dense(1)(x)
out2 = Dense(1)(x)

# Compile/fit the model
model = Model(inputs=Input_1, outputs=[out1,out2])
model.compile(optimizer = "rmsprop", loss = 'mse')
# Add actual data here in the fit statement
model.fit(train_data, [train_targets,train_targets2], epochs=500, batch_size=4, verbose=0, validation_split=0.8)

# Predict / check type and shape
preds = np.array(model.predict(test_data))
#print(type(preds), preds.shape)
# is a 3D numpy array

# get first part of prediction (column/row/3D layer)
preds0 = preds[0,:,0]
# second part
preds1 = preds[1,:,0]

# Check MAE
from sklearn.metrics import mean_absolute_error
print(mean_absolute_error(test_targets, preds0))
print(mean_absolute_error(test_targets2, preds1))
| improve this answer | |
  • $\begingroup$ It's not what I'm looking for as the amount of labels is quite job, but this is a really helpful answer :-) $\endgroup$ – Guillermo Mosse Dec 27 '19 at 15:17

One very simple way to tackle this could be to introduce a substitute value for your label sets to which you then apply a regression model of your choice. For example using the mean:

| Feature 1 | Feature 2 | possible labels | substitute target variable 
|     1     |     5     |     {1, 4}      |            2.5             |
|     2     |     2     |       {2}       |            2.0             |
|     4     |     3     |    {3, 5, 6}    |            4.67            |

However, if that makes sense will depend on your domain. For example it implies that $\{2, 4, 6\}$ is being treated the same as $\{1, 2, 3, 4, 5, 6, 7\}$ since they will have the same substitute target value (mean). Moreover, without rounding it will not give you an integer in general. If that is a problem or not again depends on your domain.

| improve this answer | |
  • $\begingroup$ I like your idea but this doesn't make much sense for my domain, as my goal is to predict a target variable that is as close as possible to a possible label. For example, if they are {0, 1000}, predicting the value 500 is extremely unhelpful. $\endgroup$ – Guillermo Mosse Dec 26 '19 at 16:00
  • $\begingroup$ For some context, I want to use my predicted value as a starting point for afterwards search for a valid label "walking" from that starting point. If I predicted 500, I would use that to check if the following are valid labels: 500, 499, 501, 498, 502, etc (this is a simplification, my search space is more complex than just integers, but it's a metric space anyway) $\endgroup$ – Guillermo Mosse Dec 26 '19 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.