In practical classification, there is often a mix of learnable and unlearnable classes and only a classifier above a minimum performance threshold can be deployed.
N. Kaplan, and Y. Chisik. JCDL '05: Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, page 235--243. New York, NY, USA, ACM Press, (2005)
S. Dumais, E. Cutrell, and H. Chen. CHI '01: Proceedings of the SIGCHI conference on Human factors in computing systems, page 277--284. New York, NY, USA, ACM, (2001)
A. Zhang, J. Chen, W. Chai, J. Xu, L. Hong, and E. CHI. ACM Trans. Interact. Intell. Syst., 8 (2):
14:1--14:28(June 2018)Interesting attempt to engineer a multi-step process for app search cluster evaluation using crowdsourcing. In some cases crowd performed better than experts..
D. Zein, and C. da Costa Pereira. Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, page 170-178. ACM, (July 2022)