0
$\begingroup$

Bert is pre-trained model which can be fine-tuned for the text classification. How to extract local features using BERT

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

First, it is different to fine-tune BERT than extracting features from it.

  • In feature extraction, you normally take BERT's output together with the internal representation of all or some of BERT's layers, and then train some other separate model on those features.

  • In fine-tuning, you re-train the whole BERT model on the specific task you are interested in (e.g. classification). Normally, you choose a very low learning rate to avoid catastrophic forgetting.

To choose which approach is appropriate in your case, you can check this article (basically you will probably get better performance with fine-tuning).

BERT can given you both sentence-level representations of token-level representations. For the sentence-level representations, you just take the BERT representations at the first token position, which is the special token [CLS] (see this). For token-level representations, you just take the representations of the token at the position you need. Take into account that BERT uses subword-level tokens, so it won't give you word-level representations (see this).

Regardless of the approach you choose, you can use the Huggingface Transformers python library, which has a lot of examples on the internet (e.g. fine-tuning and feature extraction).

Improve this answer
$\endgroup$
10
  • 1
    $\begingroup$ My question is about feature extraction and I have read that transformers extract global features. But Does BERT also extracts global features or it incudes local features. For example consider the sentence "This is my first sentence. This is my second sentence" . Now How Bert extracts the features. $\endgroup$
    – SS Varshini
    Aug 2, 2021 at 14:25
  • $\begingroup$ I added some clarifications to describe what kind of representations you can get with BERT. $\endgroup$
    – noe
    Aug 2, 2021 at 14:30
  • 1
    $\begingroup$ As I comment in the answer, the features BERT gives you are the hidden states of the self-attention layers at specific positions: the first position is for sentence-level representations, while the rest of the positions belong to the token at that position. $\endgroup$
    – noe
    Aug 2, 2021 at 14:31
  • 1
    $\begingroup$ when i have multiple sentences in text then the first position has sentence level representation or the whole text represnetation? $\endgroup$
    – SS Varshini
    Aug 2, 2021 at 14:37
  • 1
    $\begingroup$ BERT extracts context-aware representations, meaning that the obtained vectors depend on the rest of the tokens, not just the token at the specific position. Nevertheless, a different representation is computed for each token in the sentence (apart from the described [CLS] representation). $\endgroup$
    – noe
    Aug 2, 2021 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.