0
$\begingroup$

I have probelms understanding bpemb's encode_ids_with_eos() or similar. When I run the following code i get none-word like segmentations (rather syllalbus based or letters grouped of two in a seemingly arbitrary fashin). But how is that useful? And how is it useful when assigning ids?

print(bpemb_en.encode_ids_with_eos([" , ", ". "]))
print(bpemb_en.encode_ids([" , ", ".   "] ))
print(bpemb_en.encode_with_eos([" , ", ".   "] ))
print(bpemb_en.encode_with_eos(["Canvas and grouped People", ".   "] ))
print(bpemb_en.encode_with_eos(["Canvas and grouped people", ".   ", "I went to China to eat a roll of meat", "I am just an ordinary person"] ))
print(bpemb_en.encode_ids_with_bos_eos(["Olla , ", ".   ", "_"]))
print(bpemb_en.encode_with_bos_eos(["Olla , ", ".   ", "_"]))
print(bpemb_en.encode_with_bos_eos(["<PAD>", "Olla , ", ".   ", "_"]))


OUTPUT:
[[912, 934, 2], [896, 2]]
[[912, 934], [896]]
[['▁', ',', '</s>'], ['▁.', '</s>']]
[['▁can', 'v', 'as', '▁and', '▁group', 'ed', '▁people', '</s>'], ['▁.', '</s>']]
[['▁can', 'v', 'as', '▁and', '▁group', 'ed', '▁people', '</s>'], ['▁.', '</s>'], ['▁i', '▁w', 'ent', '▁to', '▁ch', 'ina', '▁to', '▁e', 'at', '▁a', '▁ro', 'l', 'l', '▁of', '▁me', 'at', '</s>'], ['▁i', '▁am', '▁j', 'ust', '▁an', '▁or', 'd', 'in', 'ary', '▁pers', 'on', '</s>']]
[[1, 13, 922, 922, 914, 912, 934, 2], [1, 896, 2], [1, 912, 976, 2]]
[['<s>', '▁o', 'l', 'l', 'a', '▁', ',', '</s>'], ['<s>', '▁.', '</s>'], ['<s>', '▁', '_', '</s>']]
[['<s>', '▁', '<', 'p', 'ad', '>', '</s>'], ['<s>', '▁o', 'l', 'l', 'a', '▁', ',', '</s>'], ['<s>', '▁.', '</s>'], ['<s>', '▁', '_', '</s>']]

$\endgroup$

1 Answer 1

1
$\begingroup$

The way the sentences are grouped in seemingly arbitrary tokens is based on the vocabularay size of the bpemb when instantiated/initialized. They represent the byte-paring that are the most frequent.

See https://www.youtube.com/watch?v=HEikzVL-lZU for an explanatory tutorial.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.