I made an introductory talk on word embeddings in the past and this write-up is an extended version of the part about philosophical ideas behind word vectors.
C. Xu, Y. Bai, J. Bian, B. Gao, G. Wang, X. Liu, and T. Liu. Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, page 1219--1228. ACM, (2014)
M. Baroni, and R. Zamparelli. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, page 1183--1193. Stroudsburg, PA, USA, Association for Computational Linguistics, (2010)