Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE]: Is it Possible to integrate Liger-Kernel? #6047

Open
ericxsun opened this issue Sep 6, 2024 · 6 comments
Open

[FEATURE]: Is it Possible to integrate Liger-Kernel? #6047

ericxsun opened this issue Sep 6, 2024 · 6 comments
Assignees
Labels
enhancement New feature or request

Comments

@ericxsun
Copy link
Contributor

ericxsun commented Sep 6, 2024

Describe the feature

https://github.com/linkedin/Liger-Kernel

Liger Kernel is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduce memory usage by 60%. We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy, and more to come.

@ericxsun ericxsun added the enhancement New feature or request label Sep 6, 2024
@Edenzzzz
Copy link
Contributor

Edenzzzz commented Sep 8, 2024

Seems like a pretty light-weight library. cc @ver217 @isky-cd Any take on this? 😃

@isky-cd
Copy link
Contributor

isky-cd commented Sep 9, 2024

Liger Kernel is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduce memory usage by 60%. We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy, and more to come.

I think this is a good attempt.

@ver217
Copy link
Member

ver217 commented Sep 10, 2024

Does it compare with apex's implementation? We've integrate some apex cuda kernels and some of them are also implemented in Liger-kernel.

@Edenzzzz
Copy link
Contributor

Edenzzzz commented Sep 14, 2024

I think Apex only provides fused RMSNorm and LayerNorm kernels? They have some more
image

@ericxsun
Copy link
Contributor Author

ericxsun commented Oct 9, 2024

Any good news? Thanks a lot

@Edenzzzz
Copy link
Contributor

Edenzzzz commented Oct 9, 2024

I think they are short-handed in wrapping up Zero Bubble, hybrid seq parallel and then they will focus on accelerate intergration?
Feel free to ask other members to clarify further, but it'd be great if the community can make an initial PR on this, then we can help/comment. This is an open-source initiative after all, and we always welcome contributions🙂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants