Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?

Question

I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it based on DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter paper? This is published in March 2020. I am looking for links between this paper and Sentence-BERT (sentence-transformers). Original sentence-bert paper is published in Aug 2019. I wanted to try pre-trained model using S-BERT model and hence tried distilbert-base-nli-mean-tokens model. After implementation I found out that it's much faster than other pre-trained models available on sentence-transformer website. While studying it's paper I realised original paper do not mention this pre-trained model.

I found Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation this paper published by same author which do mention DistilmBERT but not DistilBert Can someone please help me solve this mystery?

Answer 1

The distilbert-base-nli-mean-tokens model is the DistilBERT base model for natural language inference (NLI) using the MEAN pooling strategy for CLS tokens. For more information on the DistilBERT base model, have a look at the original paper. The pooling strategy is simply the way the model combines the different embeddings (and thus information) from the different layers.