0
$\begingroup$

From about 19:20 in the video here: https://www.youtube.com/watch?v=IHZwWFHWa-w

it shows the difference in value of the cost function for randomly labelled data vs. properly labelled data.

What do they mean by randomly labelled? That the labels change on every batch or epoch? Or that they are just assigned some random label which are then fixed throughout training?

Then how is it possible for a classification network with a softmax layer for the output to do any better on the properly labelled data? They are just labels and meaningless to the network it seems.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

They are just comparing learning rates between random labels and organised labels, to understand better how the neural networks behave. The lesson is that neural networks are better suited to learn on structured data. It can also learn on random data but much slower.

Improve this answer
$\endgroup$
6
  • $\begingroup$ This does not answer the question. $\endgroup$
    – mLstudent33
    Commented Jun 24, 2021 at 17:32
  • $\begingroup$ To answer exactly to your question, they are random labels that are fixed for the training, otherwise it would be impossible to converge progressively. $\endgroup$
    – Nicolas Martin
    Commented Jun 24, 2021 at 18:04
  • $\begingroup$ You mean a neural network can actually associate "cat" better to a picture of a cat than "car"? The only difference is the last letter. $\endgroup$
    – mLstudent33
    Commented Jun 24, 2021 at 18:43
  • $\begingroup$ In this case, "car" or "cat" are just labels: it would be the same with "car" and "eagle". The neural network learns pictures and associate them labels. Structured data means that all photos of cats are associated to a unique label. Random data means that all photos of cats are associated to random labels (car, pc, ant, bird,...). Which means that learning on random labels is much more difficult because there is no meaningful group: the nn won't find correlations between pictures. $\endgroup$
    – Nicolas Martin
    Commented Jun 24, 2021 at 19:51
  • $\begingroup$ Note: in structured data, you'll have group of cats, dogs, cars, birds,etc. where each group will have a specific label. If you name the group of cats as "dolphin", the neural network will learn correctly the cats pictures but it will associate them the "dolphin" label. I hope it is clear enough. $\endgroup$
    – Nicolas Martin
    Commented Jun 24, 2021 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.