Artificial neural networks suffer from catastrophic forgetting when they are trained sequentially on multiple tasks. Many continual learning (CL) strategies aim to overcome this problem. One of the most effective is the hypernetwork-based approach. The hypernetwork generates the weights of a target model based on the task’s identity. The model’s main limitation is that, in practice, the hypernetwork can produce completely different architectures for subsequent tasks. To address this issue, we employ the lottery ticket hypothesis, which posits the existence of sparse subnetworks—referred to as “winning tickets”—that preserve the performance of the entire network. In this paper, we propose a method called HyperMask, which dynamically filters a target network depending on the CL task. The hypernetwork generates semi-binary masks to obtain dedicated target subnetworks. Furthermore, based on the lottery ticket hypothesis, we can use a single network with weighted subnets. Depending on the task, the importance of certain weights may be dynamically increased, while that of others may be decreased. HyperMask achieves competitive results on several CL datasets and, in some scenarios, outperforms state-of-the-art scores, both with derived and unknown task identities.
Authors: Kamil Książek, Przemysław Spurek