I want to use the twitter datasets in a project and the tweet contents look something like this:

tweet_ID         tweet_text

12324124         some text here that has been twitted bla bla bla
35325323         some other text, trump, usa , merica ,etc.
56743563         bla bla text whatever tweet bla bla

Now I would like to end-up with a file that contains tweet_IDs and some vector encodings. I was reading about BERT, ROBERTA, etc. Is there a way to simply generate these encodings without writing a huge amount of boilerplate code?


With Huginface's Transformers, it should be doing with not much boilerplate.

A minimum example using PyTorch:

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

Depending on how data you want to process, you might want to speed it up a little bit. In that case, you should do it batches which would require doing padding and passing the padding mask to the model.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.