This explains several ways to use consistent colors across plots (same way works with scales). However, the easy way explained in one of the alternative answers does not seem to work for some versions of R/ggplot2. Opting for the accepted answer always works though.
S. Lukin, K. Bowden, C. Barackman, and M. Walker. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, European Language Resources Association (ELRA), (May 2016)