Inproceedings,
Improving Cross-Lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation
Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, page 1048–1056. New York, NY, USA, Association for Computing Machinery, (2023)
DOI: 10.1145/3539597.3570468
Abstract
Benefiting from transformer-based pre-trained language models, neural ranking models have made significant progress. More recently, the advent of multilingual pre-trained language models provides great support for designing neural cross-lingual retrieval models. However, due to unbalanced pre-training data in different languages, multilingual language models have already shown a performance gap between high and low-resource languages in many downstream tasks. And cross-lingual retrieval models built on such pre-trained models can inherit language bias, leading to suboptimal result for low-resource languages. Moreover, unlike the English-to-English retrieval task, where large-scale training collections for document ranking such as MS MARCO are available, the lack of cross-lingual retrieval data for low-resource language makes it more challenging for training cross-lingual retrieval models. In this work, we propose OPTICAL: <u>Op</u>timal <u>T</u>ransport dist<u>i</u>llation for low-resource <u>C</u>ross-lingual information retrieval. To transfer a model from high to low resource languages, OPTICAL forms the cross-lingu<u>al</u> token alignment task as an optimal transport problem to learn from a well-trained monolingual retrieval model. By separating the cross-lingual knowledge from knowledge of query document matching, OPTICAL only needs bitext data for distillation training, which is more feasible for low-resource languages. Experimental results show that, with minimal training data, OPTICAL significantly outperforms strong baselines on low-resource languages, including neural machine translation.
