I am designing a convolutional neural network that I believe requires transfer learning to function in practice. The network will be a character level CNN for text classification, more specifically, authorship identification of an author given unknown texts. The initial model will be trained on millions of texts from thousands of authors. In practice, if I want to be able to determine the authorship of a new given author/class not trained upon originally, I need to use transfer learning.
The structure of the network involves 6 convolutional layers and 3 fully connected layers. Given that the amount of data of the new author/class will be minimal in most cases, which layers should I replace and retrain for the new class for it to be the most effective? Or are there other methods I could consider to solve this problem?