Repository navigation
Community contribution - BetterTransformer integration for more models! #20372
Description
BetterTransformer integration for more models!
BetterTransformer API provides faster inference on CPU & GPU through a simple interface!
Models can benefit from very interesting speedups using a one liner and by making sure to install the latest version of PyTorch. A complete guideline on how to convert a new model has been created on the BetterTransformer documentation!
Here is a list of models that could be potentially supported, pick one of the architecture below and let's discuss about the conversion!
Text models 🖊️ :
- FSMT - FSMTEncoderLayer / @Sumanth077 [BT] Add
Bettertransformersupport for FSMT optimum#494 - MobileBERT - MobileBertLayer / @raghavanone [BetterTransformer] Add
MobileBERTsupport forBetterTransformeroptimum#506 - MBart - MBartEncoderLayer + M2M100EncoderLayer / Add
MBartsupport forBetterTransformeroptimum#516 @ravenouse - ProphetNet - ProphetNetEncoderLayer
- RemBert - RemBertLayer / @hchings Add BetterTransformer support for RemBERT optimum#545
- RocBert - RocBertLayer / @shogohida Add RoCBert support for Bettertransformer optimum#542
- RoFormer - RoFormerLayer
- Tapas - TapasLayer / Added support for Tapas Model optimum#520
Vision models 📷 :
- Detr - DetrLayer
- Flava - FlavaLayer / Add BetterTransformer support for FlavaModel optimum#538
- GLPN - GLPNLayer (cannot be supported)
- ViLT - ViLTLayer / [BT] add
BetterTransformersupport for ViLT architecture optimum#508
Audio models 🔉 :
- Speech2Text - Speech2TextLayer
- NEW: Audio Speech Transformer - ASTLayer / @ravenouse Adds
ASTLayersupport forBetterTransformeroptimum#548
Let us also know if you think that some architectures can be supported that we missed. Note that for encoder-decoder based models below, we expect to convert the encoder only.
Support for decoder-based models coming soon!
Reacted by Michael Benayoun, Daniel van Strien, Hamish Dickson, fxmarty, Hamid Shojanazeri and RoboTuan
Reacted by Michael Benayoun, Zhiyong Wang and Bram Vanroy
Activity
hamishdickson commentedon Nov 22, 2022
It's not on your list, but would you complain if I did this for DebertaV2Model?
michaelbenayoun commentedon Nov 22, 2022
It is not in the list because
DebertaV2does not have a regular attention mechanism, so it is not possible to use it with BetterTransformer.younesbelkada commentedon Nov 22, 2022
Yes I second what @michaelbenayoun said, please see related: huggingface/optimum#487
hamishdickson commentedon Nov 22, 2022
makes a lot of sense - sorry I should have thought about that a bit harder before posting!
GenVr commentedon Nov 23, 2022
I noticed that Better Transformers for the T5 model has not been implemented yet. Will it be implemented in the future (if possible)? Thanks.
younesbelkada commentedon Nov 23, 2022
Hi @GenVr
Thanks a lot for your reply! Unfortunately
T5cannot be supported because of the nature of its attention mechanism. In factT5uses attention bias and this is not supported forBetterTransformerThanks!
RJZauner commentedon Nov 25, 2022
Hi :) I would like to work on the implementation for RemBertLayer.
What are the next steps in getting started?
Thank you!
younesbelkada commentedon Nov 25, 2022
Hey @RJZauner !
Thanks so much for your interest in helping us integrating more models for
BetterTransformer!RemBert seems to use the same attention mechanism as BERT, the only difference should be on the Embedding layer, which is fine for us! So I would say you can move ahead and start forking optimum library, create a new branch and open a draft PR. Feel free to have some inspiration from what has been done by huggingface/optimum#494 and huggingface/optimum#508 to see what exactly needs to be done ;) Ping us (myself, @michaelbenayoun & @fxmarty) whenever you feel that you need help!
shogohida commentedon Nov 25, 2022
Hi @younesbelkada, I would like to work on the easiest of the models mentioned above. Which one do you recommend? What I said might sound a bit weird but I want to tackle a simple one since I'm not very familiar with these models 🙏
JuheonChu commentedon Nov 25, 2022
Hello, I would like to tackle the implementation for TapasLayer.
May I ask you how I can start the further steps?
Thank you for your time.
michaelbenayoun commentedon Nov 25, 2022
Hi @shogohida and @JuheonChu ,
You can read this page for learning how to contribute. You can then open a PR with your code, and ask questions there, we will be glad to help!
Also @shogohida, I think they are all similar in terms of difficulty, so do not block on that, maybe choose a model with the modality the most familiar to you.
younesbelkada commentedon Nov 25, 2022
Seconding what @michaelbenayoun said, feel free to check some example PRs huggingface/optimum#508 or huggingface/optimum#494 for reference!
@shogohida , you can take RocBERT, actually it copies from Bert so the conversion will be very easy :)
shogohida commentedon Nov 25, 2022
Thanks guys for your replies! I will take RocBERT then!
JuheonChu commentedon Nov 25, 2022
Thanks @michaelbenayoun ! I will take TapasLayer !
54 remaining items
dewasahu2003 commentedon May 4, 2023
@younesbelkada Hey 👋
I have submitted the pr for
BetterTransformerfor detrI mentioned you there PR
From next time i would keep in mind to ask pr authors
mobley-trent commentedon May 8, 2023
Hi, @younesbelkada I'd like to work on
ProphetNet😀mszsorondo commentedon May 8, 2023
Go for it! Sorry for the delay
Jack-Chuang commentedon May 27, 2023
Hi @younesbelkada, @michaelbenayoun, and @fxmarty,
I would like to work on Speech2TextLayer.
What are the next steps in getting started?
Thank you!
jucamohedano commentedon May 27, 2023
Hi! @younesbelkada @michaelbenayoun @fxmarty
I'm interested in adding support for one of the models in the list. Although, I believe that the only model left might be Speech2TextLayer and has been claimed by @Jack-Chuang
mobley-trent commentedon May 28, 2023
Hello @younesbelkada @fxmarty and @michaelbenayoun
I would like to work on the
RoFormerlayer since I saw that someone had already worked onProphetNet. Has the model been claimed ?RoboTuan commentedon Jun 11, 2023
Hello @younesbelkada @fxmarty and @michaelbenayoun
I would love to help you with the integration of more models for
BetterTransformer! I'm happy to take what is left since a lot of developers are already contributing to most of the models I think. Let me know if I can still help with something!mohammedElfatihSalah commentedon Jun 29, 2023
@younesbelkada is there anything I can help with in this issue?
deepwilson commentedon Sep 20, 2023
@younesbelkada please could you update the original list of pending items?
Or has this project been stalled?
sam-h-bean commentedon Sep 27, 2023
Is splade possible?
adeepbiswas commentedon Oct 6, 2023
Hi @younesbelkada,
Can I take up ProphetNet task? I'm new to open source and might take some time but eager to try my hands at this.
younesbelkada commentedon Oct 6, 2023
Hi everyone,
Sorry for the delay in replying to this issue and community contribution - we had some internal discussion and we decided to migrate the
BetterTransformerAPI into transformers core by directly supportingtorch.scaled_dot_product_attentionin the modeling files. Check out this issue: #26557 for more details and this PR for the PoC: #26572We will possibly open a community contribution for that to extend the support for all architectures but not sure. I will keep you all posted!
Thanks again for all your effort and amazing contribution! 🎉
vu0607 commentedon Jan 24, 2024
Hi @younesbelkada, @michaelbenayoun, and @fxmarty
The model type vision-encoder-decoder is not yet supported to be used with BetterTransformer !!!
Hope you to support soon <3