BoTNet
Vision Models Get Some Attention: Researchers add self-attention to convolutional neural nets.
Self-attention is a key element in state-of-the-art language models, but it struggles to process images because its memory requirement rises rapidly with the size of the input. New research addresses the issue with a simple twist on a convolutional neural network.