We observed that generally the embedding representation is very rich and information dense. For example, reducing the dimensionality of the inputs using SVD or PCA, even by 10%, generally results in worse downstream performance on specific tasks.
J. Zhang, Y. Dong, Y. Wang, J. Tang, and M. Ding. Proceedings of the 28th International Joint Conference on Artificial Intelligence, page 4278–4284. AAAI Press, (Aug 10, 2019)
E. Nie, S. Liang, H. Schmid, and H. Schütze. Findings of the Association for Computational Linguistics: ACL 2023, page 8320--8340. Toronto, Canada, Association for Computational Linguistics, (July 2023)
X. Liu, T. Zhu, H. Tan, and R. Zhang. The Semantic Web--ISWC 2022: 21st International Semantic Web Conference, Virtual Event, October 23--27, 2022, Proceedings, page 284--302. Springer, (2022)
A. Boggust, B. Carter, and A. Satyanarayan. 27th International Conference on Intelligent User Interfaces, page 746–766. New York, NY, USA, Association for Computing Machinery, (2022)
Q. Le, and T. Mikolov. Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, page 1188--1196. Bejing, China, PMLR, (June 2014)