0
$\begingroup$

I am trying to implement a Dependency Parsing model using the transformer model in here with a few changes. On the training, my loss has decreasing trend; but the predictions at the end of 20 epochs are not meaningful.

What could be the problems?

Thanks for any help in advance.

Outputs are below here:

The training:

Epoch 0
Epoch Step: 1 Loss: 12.520 Tokens per Sec: 156.228836
Epoch Step: 51 Loss: 9.549 Tokens per Sec: 144.077591
Epoch Step: 1 Loss: 2.845 Tokens per Sec: 139.670441
Epoch Step: 51 Loss: 7.709 Tokens per Sec: 148.557648
tensor(0.0452)
Epoch 1
Epoch Step: 1 Loss: 18.100 Tokens per Sec: 132.820221
Epoch Step: 51 Loss: 10.469 Tokens per Sec: 151.330307
Epoch Step: 1 Loss: 8.830 Tokens per Sec: 141.060349
Epoch Step: 51 Loss: 9.055 Tokens per Sec: 146.867462
tensor(0.0348)
Epoch 2
Epoch Step: 1 Loss: 11.577 Tokens per Sec: 143.026764
Epoch Step: 51 Loss: 4.532 Tokens per Sec: 133.869629
Epoch Step: 1 Loss: 1.540 Tokens per Sec: 166.832520
Epoch Step: 51 Loss: 0.617 Tokens per Sec: 133.209915
tensor(0.0191)
Epoch 3
Epoch Step: 1 Loss: 5.193 Tokens per Sec: 101.057816
Epoch Step: 51 Loss: 3.458 Tokens per Sec: 125.874458
Epoch Step: 1 Loss: 4.809 Tokens per Sec: 146.683258
Epoch Step: 51 Loss: 8.953 Tokens per Sec: 132.177780
tensor(0.0413)
Epoch 4
Epoch Step: 1 Loss: 11.321 Tokens per Sec: 115.404495
Epoch Step: 51 Loss: 3.964 Tokens per Sec: 129.871109
Epoch Step: 1 Loss: 5.374 Tokens per Sec: 153.859207
Epoch Step: 51 Loss: 5.484 Tokens per Sec: 119.325813
tensor(0.0279)
Epoch 5
Epoch Step: 1 Loss: 7.173 Tokens per Sec: 115.805588
Epoch Step: 51 Loss: 1.819 Tokens per Sec: 123.073883
Epoch Step: 1 Loss: 0.960 Tokens per Sec: 170.719009
Epoch Step: 51 Loss: 1.785 Tokens per Sec: 125.758141
tensor(0.0167)
Epoch 6
Epoch Step: 1 Loss: 0.949 Tokens per Sec: 143.089218
Epoch Step: 51 Loss: 5.166 Tokens per Sec: 119.562157
Epoch Step: 1 Loss: 1.305 Tokens per Sec: 145.756500
Epoch Step: 51 Loss: 3.682 Tokens per Sec: 124.026428
tensor(0.0146)
Epoch 7
Epoch Step: 1 Loss: 3.911 Tokens per Sec: 138.809906
Epoch Step: 51 Loss: 5.827 Tokens per Sec: 122.372780
Epoch Step: 1 Loss: 0.494 Tokens per Sec: 137.776642
Epoch Step: 51 Loss: 2.666 Tokens per Sec: 131.723038
tensor(0.0121)
Epoch 8
Epoch Step: 1 Loss: 2.576 Tokens per Sec: 136.597214
Epoch Step: 51 Loss: 2.826 Tokens per Sec: 126.036842
Epoch Step: 1 Loss: 2.027 Tokens per Sec: 126.010918
Epoch Step: 51 Loss: 3.742 Tokens per Sec: 130.350937
tensor(0.0090)
Epoch 9
Epoch Step: 1 Loss: 1.823 Tokens per Sec: 142.635910
Epoch Step: 51 Loss: 2.043 Tokens per Sec: 125.544189
Epoch Step: 1 Loss: 0.558 Tokens per Sec: 133.733505
Epoch Step: 51 Loss: 1.802 Tokens per Sec: 129.989548
tensor(0.0087)
Epoch 10
Epoch Step: 1 Loss: 1.817 Tokens per Sec: 128.555618
Epoch Step: 51 Loss: 5.115 Tokens per Sec: 121.563423
Epoch Step: 1 Loss: 0.555 Tokens per Sec: 145.560699
Epoch Step: 51 Loss: 2.464 Tokens per Sec: 135.761688
tensor(0.0088)
Epoch 11
Epoch Step: 1 Loss: 0.869 Tokens per Sec: 159.163788
Epoch Step: 51 Loss: 2.623 Tokens per Sec: 124.412704
Epoch Step: 1 Loss: 0.772 Tokens per Sec: 148.348251
Epoch Step: 51 Loss: 0.438 Tokens per Sec: 127.859993
tensor(0.0078)
Epoch 12
Epoch Step: 1 Loss: 2.377 Tokens per Sec: 123.196060
Epoch Step: 51 Loss: 1.719 Tokens per Sec: 122.183594
Epoch Step: 1 Loss: 1.415 Tokens per Sec: 124.849358
Epoch Step: 51 Loss: 1.805 Tokens per Sec: 132.770676
tensor(0.0080)
Epoch 13
Epoch Step: 1 Loss: 1.472 Tokens per Sec: 124.035645
Epoch Step: 51 Loss: 0.777 Tokens per Sec: 127.847717
Epoch Step: 1 Loss: 0.605 Tokens per Sec: 140.335785
Epoch Step: 51 Loss: 0.380 Tokens per Sec: 133.239578
tensor(0.0061)
Epoch 14
Epoch Step: 1 Loss: 1.652 Tokens per Sec: 116.933731
Epoch Step: 51 Loss: 2.005 Tokens per Sec: 131.188080
Epoch Step: 1 Loss: 0.057 Tokens per Sec: 162.092529
Epoch Step: 51 Loss: 0.840 Tokens per Sec: 131.577316
tensor(0.0066)
Epoch 15
Epoch Step: 1 Loss: 0.573 Tokens per Sec: 144.241257
Epoch Step: 51 Loss: 1.395 Tokens per Sec: 133.703445
Epoch Step: 1 Loss: 0.922 Tokens per Sec: 150.064331
Epoch Step: 51 Loss: 1.862 Tokens per Sec: 133.412491
tensor(0.0062)
Epoch 16
Epoch Step: 1 Loss: 1.002 Tokens per Sec: 141.052292
Epoch Step: 51 Loss: 0.367 Tokens per Sec: 127.523247
Epoch Step: 1 Loss: 0.792 Tokens per Sec: 137.230988
Epoch Step: 51 Loss: 1.554 Tokens per Sec: 132.268616
tensor(0.0057)
Epoch 17
Epoch Step: 1 Loss: 1.520 Tokens per Sec: 139.651688
Epoch Step: 51 Loss: 0.979 Tokens per Sec: 121.396866
Epoch Step: 1 Loss: 0.826 Tokens per Sec: 125.688881
Epoch Step: 51 Loss: 1.198 Tokens per Sec: 138.106827
tensor(0.0057)
Epoch 18
Epoch Step: 1 Loss: 0.738 Tokens per Sec: 142.477478
Epoch Step: 51 Loss: 0.467 Tokens per Sec: 128.806183
Epoch Step: 1 Loss: 0.153 Tokens per Sec: 155.590027
Epoch Step: 51 Loss: 1.191 Tokens per Sec: 136.467148
tensor(0.0050)
Epoch 19
Epoch Step: 1 Loss: 0.456 Tokens per Sec: 133.751083
Epoch Step: 51 Loss: 0.315 Tokens per Sec: 128.097702
Epoch Step: 1 Loss: 0.584 Tokens per Sec: 151.586060
Epoch Step: 51 Loss: 0.524 Tokens per Sec: 129.905441
tensor(0.0051)

The predictions:

> the reality is that it stands for and operationalizes us power , in cooperation with america 's closest allies . 
Eisner Prediction:  root 's 's 's 's 's 's 's 's 's 's 's 's 's 's 's . 's 's the 
Prediction: 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 
Target: reality is root stands stands is stands operationalizes stands power operationalizes stands cooperation stands allies allies america allies cooperation is 
----
> there are parallel shortcomings in many other markets . 
Eisner Prediction:  root many many many many . many many there 
Prediction: 1609 1609 1609 1609 1609 1609 1609 1609 1609 
Target: are root shortcomings are markets markets markets are are 
----
> the victorians produced his plays as lavish spectacles on a grand scale . 
Eisner Prediction:  root plays plays plays the . . . . . . . the 
Prediction: 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 
Target: victorians produced root plays produced spectacles spectacles produced scale scale scale produced produced 
----
> what is the minimum password strength ? 
Eisner Prediction:  root what strength strength strength what strength 
Prediction: 1609 1609 1609 1609 1609 1609 1609 
Target: root what strength strength strength what what 
----
> license . 
Eisner Prediction:  root license 
Prediction: 1609 1609 
Target: root license 
----
> if you already have a facebook account , you can log in to your account from the same page . 
Eisner Prediction:  root the the the the the the the the the the the the the the the page the if if 
Prediction: 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 
Target: have have have log account account have have log log root log account account log page page page log log 
----
> there is no oppression of russian speakers in ukraine , and there never has been . 
Eisner Prediction:  root . . . . . . . . . . . . . . there 
Prediction: 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 
Target: is root oppression is speakers speakers oppression ukraine is been been been been been is is 
----
> ( the house rose and observed a minute 's silence ) . 
Eisner Prediction:  root . . . . . . . . . . ( 
Prediction: 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 1609 
Target: rose house rose root observed rose minute silence minute observed rose rose
$\endgroup$
3
  • $\begingroup$ It's almost impossible to guess what's wrong without a deep investigation of the details. First did you try to replicate the original system exactly with the same data that they used? That would be a good basis to start from, and then you could see which of your changes causes the problem. $\endgroup$
    – Erwan
    Dec 24 '19 at 11:57
  • $\begingroup$ @Erwan Of course, I don't expect a 100% correct solution. But, as you know, there are plenty of things like normalization, data preprocessing, etc. What are the most possible options? What should I check and in what order? I know that there is no recipe :) $\endgroup$
    – stuck
    Dec 24 '19 at 12:01
  • $\begingroup$ I would follow this strategy: first make sure you have a version which works, typically following exactly the original system you mention; you make sure it works with the original dataset, then you run it with your data to have a "safe" baseline. Then you add the changes you want progressively, in order to see at which point things go wrong. Hopefully that should help locating the problem... good luck! $\endgroup$
    – Erwan
    Dec 24 '19 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.