Repository navigation
🚨 [CB] 🚨 Little fixes before removing "paged" - #49069
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
vasqu
left a comment
vasqu
left a comment
There was a problem hiding this comment.
It is breaking but we have to live with that 🚨
The only part I dont understand is why we change the cache contract?
|
Ha yes mb I wanted to add it but I forgot the 🚨 -- done! We kind of have to live with it + I would wager it is not used anywhere |
remi-or
changed the title
|
TBH I am also going to break the cache update contract in the next PR in this stack just so you know |
remi-or
changed the title
remi-or
changed the title
vasqu
left a comment
vasqu
left a comment
There was a problem hiding this comment.
Nice just one nit but looks good 🫡
470f755 to
d05d617
Compare
d05d617 to
999a86e
Compare
CI recapDashboard: View test results in Grafana |
In order to later remove the
paged|attention implementations, this PR makes a few changes to CB:max_seqlen_qtomax_length_qto align with the non-paged attention APIlayer_typeto each allocator, which will help when routing thecache.updatecache.updateinput and output shape with the regular flash attention: it takes in[1, seqlen, num_heads, head_dim]and returns the same format (in CB, batch size is always 1)flash_paged,sdpa_pagedandeager_pagedto align with these changes🚨 Breaking 🚨 because it changes the cache format