2
$\begingroup$

I'm working on training a network to do direction of arrival prediction and I'm having the issue that no matter what my network is (ResNet 18 - 101, CRNN, CNN, etc...) my results tend toward one small range of values as seen in the image below

Histogram of Ground Truth and Predictions

which leads obviously to the following errors:

Polar Error Diagram

I have attempted to just "wait it out" until my network finally learns, but my validation loss diverges pretty much immediately. An example can be seen below.

Training and Validation Loss

The strange thing is that even my training loss doesn't go to 0. I would think if my network would overfit, it would simply learn my dataset perfectly, but that doesn't happen, regardless of how complex I make my model. The only thing I can think of is my feature representation is completely nonsensical or I've got a typo somewhere in my training function and something bizarre is happening.

I've tried messing around with the loss function, tried different activation functions at the end of the network such as Tanh, sigmoid, ReLU and no activation function at all. At the moment I've simplified my training data as much as possible and am working with an 8-channel 1s long Chirp signal which can be found (at least temporarily) here: https://file.re/2021/06/20/chirp/

Like mentioned above, I've tried a standard ResNet of all sizes, and various different feature representations, with the most recent being taking the complex STFT of all 8 channels, stacking the magnitudes vertically and adding the angle information to the X-axis, as seen below:

Sample feature

If anybody has any ideas, I'm more than happy to try them. At the moment I'm using a CNN with vertical convolutions and no pooling operations in an attempt to conserve the time information.

My main training method can be seen below:

def train(self):
        steps, losses, metrics = TrainingUtilities.get_training_variables(self.parameters)
        patience_counter = 0
        best_epoch_data = None
        best_epoch_validation_loss = 999
        best_epoch = 0
        exit_training = False
        try:
            for epoch in range(steps, self.epoch_count):
                epoch_metrics = TrainingUtilities.initialize_metrics(self.mode)
                if exit_training:
                    break
                for idx, phase in enumerate(['train', 'val']):
                    if phase == 'train':
                        self.model.train()
                    else:
                        self.model.eval()
                    for _, data in tqdm(enumerate(self.data_loaders[idx])):

                        self.optimizer.zero_grad()
                        inputs, labels = data

                        outputs = self.model(inputs.to(self.device))
                        # labels = azi_class.squeeze_().to(self.device)
                        loss = self.criterion(outputs.squeeze(), labels.to(self.device))

                        epoch_metrics = TrainingUtilities.get_epoch_metrics(
                            outputs, labels, loss, epoch_metrics, phase, self.mode)

                        if phase == 'train':
                            loss.backward()
                            self.optimizer.step()

                    TrainingUtilities.report_metrics(self.writer, epoch_metrics, epoch, phase, self.parameters, self.mode)
                    if phase == "val":
                        TrainingUtilities.step_scheduler(
                            self.scheduler, np.mean(epoch_metrics[0][phase]), self.parameters)
                  
                losses.append(epoch_metrics[0])
                metrics.append(epoch_metrics)
                if epoch % self.epoch_save_count == 0:
                    TrainingUtilities.save_checkpoint(self.model, losses, metrics, self.training_dir, epoch, self.mode, self.model_name, self.size)

                steps += 1

        except (KeyboardInterrupt, RuntimeError) as error:
            print(f"Error: {error}")
            TrainingUtilities.save_checkpoint(self.model, losses, metrics, self.training_dir, steps, self.mode, self.model_name, self.size)

Its very possible that i've just missed something but the main idea is , I think quite simple. I've abstracted a lot of things due to having tested a lot of different models with different datasets as well as wanting to easily switch between classification and regression.

$\endgroup$
0
$\begingroup$

I can't access the file. If you can't share it, could you share a small sample?

Some ideas:

1- Did you try to normalize your data? Some data with unusual values needs to be normalized in order to be trained correctly.

2- There could be also an issue in the training, maybe an overfitting. Did you apply a dropout to increase the generalization.

3- If there is the same error with any model you use, maybe there is something wrong in your learning process: maybe an initial data transformation that you don't apply on the prediction data.

$\endgroup$
5
  • $\begingroup$ tmpfiles.org/dl/57437/chirp.wav Here is another link to the chirp. Its apparently only for 120 minutes. I'll have to find a different hosting service I guess. 1. Yes I normalized the data. I've tried with and without normalization. 2. I've applied dropout (varying between 0.1 and 0.9) and increased my weight decay for L2 normalization. 3. The training data are randomly taken from the complete dataset (around 90k files) with a split of 0.8. The 80% that are taken for training are then excluded when randomly selecting the validation split. There are no further transformations $\endgroup$
    – user25758
    Jun 20 at 7:39
  • $\begingroup$ Still unable to get the file :-/ Have you tried to debug the code in small pieces to see what it is going wrong? Having a view of your code might be helpful to investigate further. $\endgroup$ Jun 20 at 15:28
  • $\begingroup$ Yeah that link will have expired. Here is one that should work for another day or so: file.re/2021/06/20/chirp I also added some code to perhaps assist in any discoveries $\endgroup$
    – user25758
    Jun 20 at 15:51
  • $\begingroup$ I can't see any big mistake in your code. Have you read those tips? stats.stackexchange.com/questions/352036/… The problem could be due to a business understanding. About the sound arrival prediction, what are you trying to learn? Is it a moving object in space and you try to predict his movement? Is it in a plane or in space? I ask that because you model should fit to the dimensions of your data. $\endgroup$ Jun 21 at 7:55
  • $\begingroup$ If you consider the answers somewhat usefull, don't hesitate to upvote them as acknowledgment :) $\endgroup$ Jul 22 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.