HuggingFace introduces DilBERT, a distilled and smaller version of Google AI’s Bert model with strong performances on language understanding. DilBert s included in the pytorch-transformers library.
Y. Yue, C. Wang, K. El-Arini, and C. Guestrin. Proceedings of the 23rd international conference on World wide web, page 75--84. International World Wide Web Conferences Steering Committee, (2014)