A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
K. Hashimoto, C. Xiong, Y. Tsuruoka, and R. Socher. (2016)cite arxiv:1611.01587Comment: Accepted as a full paper at the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017).
Abstract
Transfer and multi-task learning have traditionally focused on either a
single source-target pair or very few, similar tasks. Ideally, the linguistic
levels of morphology, syntax and semantics would benefit each other by being
trained in a single model. We introduce a joint many-task model together with a
strategy for successively growing its depth to solve increasingly complex
tasks. Higher layers include shortcut connections to lower-level task
predictions to reflect linguistic hierarchies. We use a simple regularization
term to allow for optimizing all model weights to improve one task's loss
without exhibiting catastrophic interference of the other tasks. Our single
end-to-end model obtains state-of-the-art or competitive results on five
different tasks from tagging, parsing, relatedness, and entailment tasks.
Description
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
%0 Generic
%1 hashimoto2016joint
%A Hashimoto, Kazuma
%A Xiong, Caiming
%A Tsuruoka, Yoshimasa
%A Socher, Richard
%D 2016
%K deep_learning multitask nlp
%T A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
%U http://arxiv.org/abs/1611.01587
%X Transfer and multi-task learning have traditionally focused on either a
single source-target pair or very few, similar tasks. Ideally, the linguistic
levels of morphology, syntax and semantics would benefit each other by being
trained in a single model. We introduce a joint many-task model together with a
strategy for successively growing its depth to solve increasingly complex
tasks. Higher layers include shortcut connections to lower-level task
predictions to reflect linguistic hierarchies. We use a simple regularization
term to allow for optimizing all model weights to improve one task's loss
without exhibiting catastrophic interference of the other tasks. Our single
end-to-end model obtains state-of-the-art or competitive results on five
different tasks from tagging, parsing, relatedness, and entailment tasks.
@misc{hashimoto2016joint,
abstract = {Transfer and multi-task learning have traditionally focused on either a
single source-target pair or very few, similar tasks. Ideally, the linguistic
levels of morphology, syntax and semantics would benefit each other by being
trained in a single model. We introduce a joint many-task model together with a
strategy for successively growing its depth to solve increasingly complex
tasks. Higher layers include shortcut connections to lower-level task
predictions to reflect linguistic hierarchies. We use a simple regularization
term to allow for optimizing all model weights to improve one task's loss
without exhibiting catastrophic interference of the other tasks. Our single
end-to-end model obtains state-of-the-art or competitive results on five
different tasks from tagging, parsing, relatedness, and entailment tasks.},
added-at = {2018-04-27T11:48:19.000+0200},
author = {Hashimoto, Kazuma and Xiong, Caiming and Tsuruoka, Yoshimasa and Socher, Richard},
biburl = {https://www.bibsonomy.org/bibtex/21ecd28e778c5d57b6a9ca3559754323a/dallmann},
description = {A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks},
interhash = {0026f760ecbd215930337535ac042fc4},
intrahash = {1ecd28e778c5d57b6a9ca3559754323a},
keywords = {deep_learning multitask nlp},
note = {cite arxiv:1611.01587Comment: Accepted as a full paper at the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017)},
timestamp = {2018-04-27T11:48:19.000+0200},
title = {A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks},
url = {http://arxiv.org/abs/1611.01587},
year = 2016
}