Transformer-CTP: Current token prediction using cross-attention of queries with current position information

초록

Transformers and pre-trained language models have significantly advanced a wide range of artificial intelligence tasks. Typically, transformer decoders for language generation are trained with the language modeling (LM) loss, which enables autoregressive generation by predicting the next token from previous tokens through self-attention. However, self-attention often assigns high attention weights to self-tokens, causing the transformer decoder to overfocus on the previous token, t-1, when predicting the current token, t, which can lead to performance degradation. Our proposed method prevents the transformer decoder from overfocusing on token t-1 when predicting token t. Instead of predicting token t using the output of token t-1, we use a new input to predict token t. We also introduce a Current-Token Prediction (CTP) module into the transformer decoder, which prevents token t-1 from being used as the attention query by applying cross-attention with the new input. Experiments on machine translation and document summarization demonstrate that our method mitigates the overfocusing problem and improves performance. Our approach not only enhances performance but also distributes attention more evenly across tokens. The code for the experiments is available on our GitHub.1

키워드