1
$\begingroup$

In my setup I have one feature which is a sparse list representing categories. For example, let's say that we have M categories in the interval [0 .. M - 1]. My examples look like:

  • ex 1: [0, 2, 1]
  • ex 2: [3, 0]
  • ex 3: [1, 4, 5]

The order doesn't matter. So, let's suppose that we already know $f: s \rightarrow L$, where $s$ is the sequence and $L$ is the label, for example, lets consider just a toy function $f$ (in a real setting I don't know what $f$ looks like):

def toy_func(set):
    return int(
        0 in set
        and M // 2 in set
    )

So, I am trying to train a neural network, currently I have tried the following model:

class SimpleMultiCategoricalClassifier(torch.nn.Module):
    def __init__(self, num_embeddings, embedding_dim):
        super().__init__()

        self.embeddings = nn.EmbeddingBag(
                num_embeddings=num_embeddings,
                embedding_dim=embedding_dim, mode="mean")

        self.net = nn.Sequential(
            nn.Linear(embedding_dim, 1),
            nn.ReLU(),
            nn.Sigmoid(),
        )
        
    def forward(self, input, offset):
        x = self.embeddings(input, offset)
        return self.net(x)

Basically I represent each "category" as an embedding, and then do pooling on the entire sequence to learn the function.

However my model is not learning anything. Is there a better way to frame the problem so I can learn the latent function?

$\endgroup$

1 Answer 1

1
$\begingroup$

Typical setup for this would be to one hot encode each unordered set, so we have $M$ boolean indicator variables. And then we might train on cosine distance, to predict each label.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.