We observed that generally the embedding representation is very rich and information dense. For example, reducing the dimensionality of the inputs using SVD or PCA, even by 10%, generally results in worse downstream performance on specific tasks.
Short introduction to Vector Space Model (VSM) In information retrieval or text mining, the term frequency - inverse document frequency also called tf-idf, is
L. Li, H. Wang, T. Chin, D. Suter, and S. Zhang. Image and Signal Processing (CISP), 2010 3rd International Congress on, 4, page 1586 -1589. (October 2010)