Comparing machine learning methods and selecting a final model is a common operation in applied machine learning.
Models are commonly evaluated using resampling methods like k-fold cross-validation from which mean skill scores are calculated and compared directly. Although simple, this approach can be misleading as it is hard to know whether the difference between mean skill scores is real or the result of a statistical fluke.
Working group for social relation and opinion mining
Research Group Social Computing
Department of Informatics
Technical University of Munich (TUM)
http://www.social.in.tum.de/ghagerer