Expire-Span

1 Post

Graph showing Expire-span which enables attention to ignore tokens that aren’t useful to the task at hand
Expire-Span

Sharper Attention

Self-attention enables transformer networks to track relationships between distant tokens — such as text characters — in long sequences, but the computational resources required grow quadratically with input size.
2 min read

Subscribe to The Batch

Stay updated with weekly AI News and Insights delivered to your inbox