FourierQK research explores optimal attention filter shapes for language models

VERA Brief

AI-generated. Grounded in the article and its cited sources.

Research published on arXiv explores filter shapes for frequency-collapse attention in language models. The study found that an oscillatory bandpass structure is important for performance and identified specific filter properties that reduce leakage.

Key facts

• The study investigates optimal filter shapes for frequency-collapse attention in language models.
• DC and Nyquist components were found to be detrimental to performance.
• An oscillatory bandpass structure is highlighted as important.
• Narrowband filters showed cleaner results than wideband filters.
• Causal time-domain Morlet filters at character scale did not outperform the baseline.