R. Kidambi, P. Netrapalli, P. Jain, and S. Kakade. (2018)cite arxiv:1803.05591Comment: 28 pages, 10 figures. Appears as an oral presentation at International Conference on Learning Representations (ICLR), 2018. Code implementing the ASGD method can be found at https://github.com/rahulkidambi/AccSGD.