Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
N. Reimers, und I. Gurevych. (2019)cite arxiv:1908.10084Comment: Published at EMNLP 2019.
Zusammenfassung
BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new
state-of-the-art performance on sentence-pair regression tasks like semantic
textual similarity (STS). However, it requires that both sentences are fed into
the network, which causes a massive computational overhead: Finding the most
similar pair in a collection of 10,000 sentences requires about 50 million
inference computations (~65 hours) with BERT. The construction of BERT makes it
unsuitable for semantic similarity search as well as for unsupervised tasks
like clustering.
In this publication, we present Sentence-BERT (SBERT), a modification of the
pretrained BERT network that use siamese and triplet network structures to
derive semantically meaningful sentence embeddings that can be compared using
cosine-similarity. This reduces the effort for finding the most similar pair
from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while
maintaining the accuracy from BERT.
We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning
tasks, where it outperforms other state-of-the-art sentence embeddings methods.
Beschreibung
[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
%0 Journal Article
%1 reimers2019sentencebert
%A Reimers, Nils
%A Gurevych, Iryna
%D 2019
%K bert embedding sentence
%T Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
%U http://arxiv.org/abs/1908.10084
%X BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new
state-of-the-art performance on sentence-pair regression tasks like semantic
textual similarity (STS). However, it requires that both sentences are fed into
the network, which causes a massive computational overhead: Finding the most
similar pair in a collection of 10,000 sentences requires about 50 million
inference computations (~65 hours) with BERT. The construction of BERT makes it
unsuitable for semantic similarity search as well as for unsupervised tasks
like clustering.
In this publication, we present Sentence-BERT (SBERT), a modification of the
pretrained BERT network that use siamese and triplet network structures to
derive semantically meaningful sentence embeddings that can be compared using
cosine-similarity. This reduces the effort for finding the most similar pair
from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while
maintaining the accuracy from BERT.
We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning
tasks, where it outperforms other state-of-the-art sentence embeddings methods.
@article{reimers2019sentencebert,
abstract = {BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new
state-of-the-art performance on sentence-pair regression tasks like semantic
textual similarity (STS). However, it requires that both sentences are fed into
the network, which causes a massive computational overhead: Finding the most
similar pair in a collection of 10,000 sentences requires about 50 million
inference computations (~65 hours) with BERT. The construction of BERT makes it
unsuitable for semantic similarity search as well as for unsupervised tasks
like clustering.
In this publication, we present Sentence-BERT (SBERT), a modification of the
pretrained BERT network that use siamese and triplet network structures to
derive semantically meaningful sentence embeddings that can be compared using
cosine-similarity. This reduces the effort for finding the most similar pair
from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while
maintaining the accuracy from BERT.
We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning
tasks, where it outperforms other state-of-the-art sentence embeddings methods.},
added-at = {2020-06-30T15:14:11.000+0200},
author = {Reimers, Nils and Gurevych, Iryna},
biburl = {https://www.bibsonomy.org/bibtex/2b7d36a45cf983489b8f3ccaf2daeb5f2/jannaom},
description = {[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
interhash = {762e8dacdc867460a7b200c6b4cd1b5c},
intrahash = {b7d36a45cf983489b8f3ccaf2daeb5f2},
keywords = {bert embedding sentence},
note = {cite arxiv:1908.10084Comment: Published at EMNLP 2019},
timestamp = {2020-06-30T15:14:11.000+0200},
title = {Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
url = {http://arxiv.org/abs/1908.10084},
year = 2019
}