1
$\begingroup$

I'm writing a demo code to predict a 2-class classification for a dataset of 10-D inputs. Below, function _data generates the data:

import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

def _data(dimension, num_examples):
    # Create a simulated 10-dimensional training dataset consisting of 1000 labeled
    # examples, of which 800 are labeled correctly and 200 are mislabeled.
    num_mislabeled_examples = 20
    # We will constrain the recall to be at least 90%.
    recall_lower_bound = 0.9

    # Create random "ground truth" parameters for a linear model.
    ground_truth_weights = np.random.normal(size=dimension) / math.sqrt(dimension)
    ground_truth_threshold = 0

    # Generate a random set of features for each example.
    features = np.random.normal(size=(num_examples, dimension)).astype(
        np.float32) / math.sqrt(dimension)
    # Compute the labels from these features given the ground truth linear model.
    labels = (np.matmul(features, ground_truth_weights) >
              ground_truth_threshold).astype(np.float32)
    # Add noise by randomly flipping num_mislabeled_examples labels.
    mislabeled_indices = np.random.choice(
        num_examples, num_mislabeled_examples, replace=False)
    labels[mislabeled_indices] = 1 - labels[mislabeled_indices]

    return torch.tensor(labels), torch.tensor(features)

The code below shows my attempt, where predictor is the model and the loss function is chosen to be Hinge loss.

import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

dim = 10
N = 100
target, features = _data(dim, N)

class predictor(nn.Module):
    def __init__(self):
        super(predictor, self).__init__()
        self.f_1 = nn.Linear(10, 1)

    def forward(self, features):
        return self.f_1(features)

model = predictor()

optimizer = optim.Adam(model.parameters(), lr=1e-2)
loss = torch.nn.HingeEmbeddingLoss(margin=1.0, size_average=None, reduce=None, reduction='mean')

running_loss = 0
for _ in range(1000):
    optimizer.zero_grad()
    output = model(features)
    objective = loss(output, target)
    objective.backward()
    running_loss += objective.item()
    optimizer.step()

    print(running_loss)

My questions:

  1. I see my loss increase from zero to 20 and then dives deep into negative realm. I was wondering if my implementation is correct.
  2. I was trying to implement my predictor without using nn.Linear by defining the computations myself as:
class predictor(nn.Module):
    def __init__(self):
        super(predictor, self).__init__()
        self.weights = torch.zeros(dim, dim, requires_grad=True)
        self.threshold = torch.zeros(1, 1, requires_grad=True)

    def forward(self, features):
        return torch.matmul(self.weights, features) - self.threshold

but in the optimization process,

model = predictor()
optimizer = optim.Adam(model.parameters(), lr=1e-3) 

I get the following error:

ValueError: optimizer got an empty parameter list

I would appreciate direction or comments on how to fix these issues. Thanks.

$\endgroup$
1
$\begingroup$
  1. Choose lr of optimizer something very small. It might be because of exploding gradient.

  2. In self.weight use nn.Parameter() then pass your torch.zeros() to make it a model parameter.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, could you alter my class predictor(nn.Module) with nn.Parameter() so I can understand? $\endgroup$ – Blade Jun 20 at 21:06
  • 1
    $\begingroup$ Refer to this article. You will find detailed explanation. stackoverflow.com/a/51027227/11961138 $\endgroup$ – SrJ Jun 20 at 21:10
  • $\begingroup$ It seems that in order for this predictor to work, I would have to know the batch size in advance. In other words, when doing torch.matmul(self.weights, features) - self.threshold I need the batch size to set the dimensionality of self.weights in advance. However, when using nn.Linear(10, 1) I don't have to deal with the batch size anymore. Is this true or should I change something? $\endgroup$ – Blade Jun 21 at 0:34
  • $\begingroup$ Nvm, I just set the dimensions wrong. Thank you! $\endgroup$ – Blade Jun 21 at 1:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.