Repository navigation
Conversation
danieldk
changed the title
Liger Kernels] Sync changes for transformers
|
Note that until we fully switch to the bot, the PR title must have the format Also see the CI result: |
|
/build-bot build liger-kernels |
|
Sorry guys I was delayed by the gemma release, I will try to get to this today:
|
| # NOTE: We have this intentional graph break as we encounter issues such as IMAs and Cublas errors. | ||
| # We know that this is an optimized kernel already so there is less ways to | ||
| # fuse it either way; we rely on torch compile to go through the base model to optimize. | ||
| @torch.compiler.disable |
There was a problem hiding this comment.
I would like to keep this for now. If we were to change this with fake ops etc, it would be something that would need to be done upstream tbh because it touches the core layers
| unsqueeze_dim: int = 1, | ||
| ) -> tuple[torch.Tensor, torch.Tensor]: | ||
| """Apply standard rotary positional embedding to ``q`` and ``k``.""" | ||
| return LigerRopeFunction.apply(q, k, cos, sin, position_ids, unsqueeze_dim) |
There was a problem hiding this comment.
Honestly we could also use this instead of rotary or FA one's - these will be at least properly maintained cc @sayakpaul
| gate_proj: nn.Linear | ||
| up_proj: nn.Linear | ||
| down_proj: nn.Linear | ||
| num_shards: int |
There was a problem hiding this comment.
If we want to use these as opt in then we don't have this attribute in transformers but let me double check if things work as expected. This could be the training version
| "LigerGroupNorm", | ||
| "LigerDyT", | ||
| "LigerCrossEntropyLoss", | ||
| "LigerFusedLinearCrossEntropyLoss", | ||
| "LigerJSD", | ||
| "LigerKLDIVLoss", | ||
| "LigerTVDLoss", |
There was a problem hiding this comment.
I deleted a bunch because their exact usage is not too obv imo especially since a lot of these modules should rather be functions (losses). We can always get another version or progressively update/add(?)
|
Closing in favor of #948 to have a non-fork branch to be used (seems like some gh actions are dependent on write access so they fail here) |
As per title, I need a few changes for transformers to work properly
Also dependent on huggingface/kernels#607 as this enables functions with torch compile (but it won't break without the other PR either way, it would only unlock the feature)
NOTE: This assumes full transformers usage and only adjusts the modules that we need, nothing else