How large should the corpus be to optimally retrain the GPT-2 model?

Artificial Intelligence Asked by Andreas Toresäter on September 20, 2020

I just started working with the GPT-2 models and want to retrain one on a pretty narrow topic, so I have problems finding training material.

How large should the corpus be to optimally retrain the GPT-2 model? And what is the bare minimum size? Should it simply be as large as possible or can it flip over and make the model worse in some way?

I am also not certain how many steps you should let the retraining run. I have been using 6000 steps when testing, and it seems not much happens after that, loss only moved from 0.2 to 0.18 last 1000 steps.

deep learning gpt natural language processing tensorflow training

How large should the corpus be to optimally retrain the GPT-2 model?

Add your own answers!

Related Questions

Can GANs be used to generate something other than images?

What should the output of a neural network that needs to classify in an unsupervised fashion XOR data be?

Choosing a policy improvement algorithm for a continuing problem with continuous action and state-space

Why is the policy loss the mean of $-Q(s, mu(s))$ in the DDPG algorithm?

Are tabular reinforcement learning methods obsolete (or getting obsolete)?

How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?

How does the target network in double DQNs find the maximum Q value for each action?

Understanding the loss function in deep Q-learning

Is a reward given at every step or only given when the RL agent fails or succeeds?

Ways to keep up with the latest developments in Machine Learning and AI?

What is the expectation of an empirical model in model based RL?

How can I change observation states’ values in OpenAI gym’s cartpole environment?

Why can’t we train neural networks in a peer-to-peer manner?

What does the term $|mathcal{A}(s)|$ mean in the $epsilon$-greedy policy?

Do the order of the features ie channel matter for a 1d convolutional network?

What is convergence analysis, and why is it needed in reinforcement learning?

Correct dimensionality of parameter vector for solving an MRP with linear function approximation?

How can I convert a simple CLI RPG to a compatible environment for training an RL agent via stable-baselines?

What is the amount of test data needed to evaluate a CNN?

What is the Turing test?

Ask a Question