0
$\begingroup$

I am using this to do some Byte Pair Encoding (BPE). My corpus looks like this.

When I run the learn_bpe, I get a vocabulary that looks like this.

e r
r e
o n
o r
t i
) ;</w>
a c
n t
' ,</w>
er r
a l
r o
h e
m e

When I try to combine it again to see if it worked with ubword-nmt apply-bpe -c data/jsvocab.txt < data/javascript.txt > tst.txt, the resulting file has a lot of strange @ characters.

const p@@ re@@ F@@ or@@ m@@ at@@ t@@ e@@ d@@ B@@ l@@ o@@ c@@ k@@ N@@ a@@ me@@ s = {
    '@@ ap@@ i@@ -@@ p@@ ro@@ j@@ ect@@ s@@ '@@ : '@@ A@@ P@@ I P@@ ro@@ j@@ ect@@ s@@ ',
    '@@ b@@ a@@ s@@ i@@ c@@ -@@ c@@ ss@@ '@@ : '@@ B@@ a@@ s@@ i@@ c C@@ S@@ S@@ ',
    '@@ b@@ a@@ s@@ i@@ c@@ -@@ h@@ t@@ m@@ l@@ -@@ and@@ -@@ h@@ t@@ m@@ l@@ 5@@ '@@ : '@@ B@@ a@@ s@@ i@@ c H@@ T@@ M@@ L an@@ d H@@ T@@ M@@ L@@ 5@@ ',
    '@@ c@@ ss@@ -@@ f@@ le@@ x@@ b@@ o@@ x@@ '@@ : '@@ C@@ S@@ S F@@ le@@ x@@ b@@ o@@ x@@ ',
    '@@ c@@ ss@@ -@@ g@@ r@@ i@@ d@@ '@@ : '@@ C@@ S@@ S G@@ r@@ i@@ d@@ ',
    de@@ v@@ o@@ p@@ s@@ : '@@ D@@ e@@ v@@ O@@ p@@ s@@ ',
    e@@ s@@ 6@@ : '@@ E@@ S@@ 6@@ ',
    '@@ in@@ f@@ or@@ m@@ ation@@ -@@ se@@ c@@ ur@@ i@@ t@@ y@@ -@@ w@@ i@@ th@@ -@@ he@@ l@@ me@@ t@@ j@@ s@@ '@@ : '@@ I@@ n@@ f@@ or@@ m@@ a@@ ti@@ o@@ n S@@ ec@@ ur@@ i@@ t@@ y w@@ i@@ t@@ h H@@ e@@ l@@ me@@ t@@ J@@ S@@ ',
    j@@ q@@ u@@ er@@ y@@ : '@@ j@@ Q@@ u@@ er@@ y@@ ',
    '@@ j@@ s@@ on@@ -@@ ap@@ i@@ s@@ -@@ and@@ -@@ a@@ j@@ a@@ x@@ '@@ : '@@ J@@ S@@ O@@ N A@@ P@@ I@@ s an@@ d A@@ j@@ a@@ x@@ ',
    '@@ m@@ on@@ g@@ o@@ d@@ b@@ -@@ and@@ -@@ m@@ on@@ g@@ o@@ o@@ se@@ '@@ : 'M@@ on@@ g@@ o@@ D@@ B an@@ d M@@ on@@ g@@ o@@ o@@ se@@ ',
    '@@ t@@ he@@ -@@ d@@ o@@ m'@@ : '@@ T@@ h@@ e D@@ O@@ M@@ ',
    '@@ ap@@ i@@ s@@ -@@ and@@ -@@ m@@ i@@ c@@ ro@@ serv@@ i@@ c@@ e@@ s@@ '@@ : '@@ A@@ P@@ I@@ s an@@ d M@@ i@@ c@@ ro@@ serv@@ i@@ c@@ e@@ s@@ ',
    '@@ ap@@ i@@ s@@ -@@ and@@ -@@ m@@ i@@ c@@ ro@@ serv@@ i@@ c@@ e@@ s@@ -@@ p@@ ro@@ j@@ ect@@ s@@ '@@ : '@@ A@@ P@@ I@@ s an@@ d M@@ i@@ c@@ ro@@ serv@@ i@@ c@@ e@@ s P@@ ro@@ j@@ ect@@ s@@ '
}@@ ;

And so on. I'm not sure what I'm missing, but it seems that it didn't fully reconstruct the text from the vocabulary?

$\endgroup$
1
  • $\begingroup$ How many merge operations did you specify in your invocation to learn-bpe.py? $\endgroup$
    – noe
    Jun 17 '19 at 15:05
1
$\begingroup$

Your BPE vocabulary is quite small given how the strings you want to segment look like. The biggest problem here is that BPE expects tokenized sentences with tokens separated by spaces. The BPE model is not aware of JS syntax and you feed it with JS code, so there is no wonder that it does not learn anything syntactically plausible.

You can interpret the BPE file a log of what subwords got merged and you can see that the longest string you get is something like:

m').valueOf
H:mm').valueOf

I would recommend some preprocessing, perhaps some JS lexer, so it respects the syntax and perhaps bigger vocabulary.

$\endgroup$
2
  • $\begingroup$ Ideally, shouldn't BPE handle all of the tokenization and syntax? $\endgroup$
    – Shamoon
    Jun 17 '19 at 19:23
  • $\begingroup$ No, it is designed to work with tokenized text. It only gradually merges most frequent pairs of subword units starting from characters. In JS code, these are obviously pairs like ); or .( that cross the token boundaries. $\endgroup$
    – Jindřich
    Jun 18 '19 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.