Machine learning as an experimental science (revisited)
C. Drummond. In AAAI Workshop on Evaluation Methods for Machine Learning, page 1--5. (2006)
Abstract
In 1988, Langley wrote an influential editorial in the journal Machine Learning titled "Machine Learning as an Experimental Science," arguing persuasively for a greater focus on performance testing. Since that time the emphasis has become progressively stronger. Nowadays, to be accepted to one of our major conferences or journals, a paper must typically contain a large experimental section with many tables of results, concluding with a statistical test. In revisiting this paper, I claim that we have ignored most of its advice. We have focused largely on only one aspect, hypothesis testing, and a narrow version at that. This version provides us with evidence that is much more impoverished than many people realize. I argue that such tests are of limited utility either for comparing algorithms or for promoting progress in our field. As such they should not play such a prominent role in our work and publications.
%0 Conference Paper
%1 drummond2006machine
%A Drummond, Chris
%B In AAAI Workshop on Evaluation Methods for Machine Learning
%D 2006
%K evaluation experiment learning machine ml research science
%P 1--5
%T Machine learning as an experimental science (revisited)
%U https://www.aaai.org/Library/Workshops/2006/ws06-06-002.php
%X In 1988, Langley wrote an influential editorial in the journal Machine Learning titled "Machine Learning as an Experimental Science," arguing persuasively for a greater focus on performance testing. Since that time the emphasis has become progressively stronger. Nowadays, to be accepted to one of our major conferences or journals, a paper must typically contain a large experimental section with many tables of results, concluding with a statistical test. In revisiting this paper, I claim that we have ignored most of its advice. We have focused largely on only one aspect, hypothesis testing, and a narrow version at that. This version provides us with evidence that is much more impoverished than many people realize. I argue that such tests are of limited utility either for comparing algorithms or for promoting progress in our field. As such they should not play such a prominent role in our work and publications.
@inproceedings{drummond2006machine,
abstract = {In 1988, Langley wrote an influential editorial in the journal Machine Learning titled "Machine Learning as an Experimental Science," arguing persuasively for a greater focus on performance testing. Since that time the emphasis has become progressively stronger. Nowadays, to be accepted to one of our major conferences or journals, a paper must typically contain a large experimental section with many tables of results, concluding with a statistical test. In revisiting this paper, I claim that we have ignored most of its advice. We have focused largely on only one aspect, hypothesis testing, and a narrow version at that. This version provides us with evidence that is much more impoverished than many people realize. I argue that such tests are of limited utility either for comparing algorithms or for promoting progress in our field. As such they should not play such a prominent role in our work and publications.},
added-at = {2021-05-19T16:51:53.000+0200},
author = {Drummond, Chris},
biburl = {https://www.bibsonomy.org/bibtex/2b4509a2c41d50960a3c8379ca12627e2/jaeschke},
booktitle = {In AAAI Workshop on Evaluation Methods for Machine Learning},
interhash = {364c0c155a8d0e8f74dca9bea9bcfe1a},
intrahash = {b4509a2c41d50960a3c8379ca12627e2},
keywords = {evaluation experiment learning machine ml research science},
pages = {1--5},
timestamp = {2021-05-19T16:51:53.000+0200},
title = {Machine learning as an experimental science (revisited)},
url = {https://www.aaai.org/Library/Workshops/2006/ws06-06-002.php},
year = 2006
}