0
$\begingroup$

The problem is implementation of Bertweet in a topic-modeling project with understandable output like BERTopic, i want to use it on a relatively large (20k tweets) unlabelled dataset to segment it into topics, number of which is either user-specified or pre-defined by the model.

I've read a documentation of Bertweet and it's too short to be cohesive for a someone like me without previous experience with neural networks and transformers. (For safety here's the whole example code from the link above)

import torch
from transformers import AutoModel, AutoTokenizer

bertweet = AutoModel.from_pretrained("vinai/bertweet-base")

# For transformers v4.x+:
tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base", use_fast=False)

# For transformers v3.x:
# tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base")

# INPUT TWEET IS ALREADY NORMALIZED!
line = "SC has first two presumptive cases of coronavirus , DHEC confirms HTTPURL via @USER :cry:"

input_ids = torch.tensor([tokenizer.encode(line)])

with torch.no_grad():
    features = bertweet(input_ids)  # Models outputs are now tuples

# With TensorFlow 2.0+:
# from transformers import TFAutoModel
# bertweet = TFAutoModel.from_pretrained("vinai/bertweet-base")

From documentation example the output is:

with torch.no_grad():
    features = bertweet(input_ids)

which is transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions class, that contains attributes and information i don't understand.

And few not less important questions:

  1. Is this suitable for unsupervised learning tasks like topic-modeling?
  2. How to unpack this class to a format similar to document - assigned_topic?
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.