I am trying to tokenize sentences of a document for aspect-based sentiment analysis. There are some sentences that consist of more than one topic. For example, " The touch screen is good but the battery is weak" or " Their smartphones are great and their TVs are perfect". I want to tokenize sentences based on these conjunctions. Is there any pre-trained model for this task? Are there any other solutions? Thank you all for your help.
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
I recommend just use a typical tokenizer (e.g. sentencepiece or BPE). And, use a multi-label classification head on top of you model (i.e. sentence encoder) for the task.
-
$\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$– Community BotJan 3 at 20:17
$\begingroup$
$\endgroup$
Tokenizing sentences based on conjunctions like "but" and "and" can be challenging because these conjunctions often indicate a contrast or conjunction between two separate thoughts, rather than a single, cohesive thought. One way to handle this is to split the sentence at the conjunction and treat the resulting two sentences as separate units. You can then tokenize each of these units separately.
If you want to use a pre-trained model for this task, you might want to look into using a pre-trained language model like BERT or RoBERTa. These models can be fine-tuned for a variety of natural language processing tasks, including sentence tokenization.
Another solution is to use a rule-based approach to identify conjunctions in the text and split the sentences accordingly. For example, you could use regular expressions to identify instances of "but" and "and" in the text and split the sentences at those points.
Regardless of which approach you choose, it's important to keep in mind that tokenization is a difficult task and there is no one-size-fits-all solution. You may need to experiment with different approaches and fine-tune your model or rule-based approach to get the best results for your specific use case.
