Login

Recent searches

No recent searches

Search options

Only available when logged in.
fosstodon.org is one of the many independent Mastodon servers you can use to participate in the fediverse.
Fosstodon is an invite only Mastodon instance that is open to those who are interested in technology; particularly free & open source software. If you wish to join, contact us for an invite.

Administered by:

Server stats:

9.9K
active users

fosstodon.org: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.7

#multi

4 posts3 participants0 posts today
Public
N-gated Hacker News

Oh joy, yet another paper with a title that sounds like a new #crypto #fad 🤦‍♂️. Thank goodness for the #Simons Foundation's donations, because without them, we'd clearly be lost in a sea of #academic gobbledygook 🤷‍♀️. Can't wait to "multi-token" my way to #enlightenment - or was it ennui? 📚🔍
arxiv.org/abs/2504.00927 #Foundation #research #multi-token #HackerNews #ngated

arXiv logo
arXiv.orgMulti-Token AttentionSoft attention is a critical mechanism powering LLMs to locate relevant parts within a given context. However, individual attention weights are determined by the similarity of only a single query and key token vector. This "single token attention" bottlenecks the amount of information used in distinguishing a relevant part from the rest of the context. To address this issue, we propose a new attention method, Multi-Token Attention (MTA), which allows LLMs to condition their attention weights on multiple query and key vectors simultaneously. This is achieved by applying convolution operations over queries, keys and heads, allowing nearby queries and keys to affect each other's attention weights for more precise attention. As a result, our method can locate relevant context using richer, more nuanced information that can exceed a single vector's capacity. Through extensive evaluations, we demonstrate that MTA achieves enhanced performance on a range of popular benchmarks. Notably, it outperforms Transformer baseline models on standard language modeling tasks, and on tasks that require searching for information within long contexts, where our method's ability to leverage richer information proves particularly beneficial.
Public
Hacker News

Multi-Token Attention

arxiv.org/abs/2504.00927

#HackerNews #Multi-Token #Attention #research #AI #advancements #attentionmechanism #deeplearning #NLP

arXiv logo
arXiv.orgMulti-Token AttentionSoft attention is a critical mechanism powering LLMs to locate relevant parts within a given context. However, individual attention weights are determined by the similarity of only a single query and key token vector. This "single token attention" bottlenecks the amount of information used in distinguishing a relevant part from the rest of the context. To address this issue, we propose a new attention method, Multi-Token Attention (MTA), which allows LLMs to condition their attention weights on multiple query and key vectors simultaneously. This is achieved by applying convolution operations over queries, keys and heads, allowing nearby queries and keys to affect each other's attention weights for more precise attention. As a result, our method can locate relevant context using richer, more nuanced information that can exceed a single vector's capacity. Through extensive evaluations, we demonstrate that MTA achieves enhanced performance on a range of popular benchmarks. Notably, it outperforms Transformer baseline models on standard language modeling tasks, and on tasks that require searching for information within long contexts, where our method's ability to leverage richer information proves particularly beneficial.
Public
gee Ⓐ⚑

La dernière mise à jour de « Sales Gosses ! » vous permet de sauvegarder des skins multiples, de choisir différents modes de jeu… et vous offre une jolie atmosphère de printemps, avec ses papillons et ses décos de Pâques !

studios.ptilouk.net/sales-goss

#SalesGosses #JeuxVidéo #JeuVidéo #Miseàjour #Casual #Fun #PVP #jeu #multi
studios.ptilouk.net/sales-goss

Sales Gosses !Sales Gosses !
More from Ptilouk.net Studios
ExploreLive feeds
About