[MoE/ZeRO] fix .github conflict with main branch. #5827
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* [fix/example] fix llama inference loading dtype * revise loading dtype of benchmark llama3
* [release] update version * [devops] update compatibility test * [devops] update compatibility test * [devops] update compatibility test * [devops] update compatibility test * [test] fix ddp plugin test * [test] fix gptj and rpc test * [devops] fix cuda ext compatibility * [inference] fix flash decoding test * [inference] fix flash decoding test
* [fix] branch for fix testcase; * [fix] fix test_analyzer & test_auto_parallel; * [fix] remove local change about moe; * [fix] rm local change moe;
…5755) * [ci/tests] simplify some test case to reduce testing time * [ci/tests] continue to remove test case to reduce ci time cost * restore some test config * [ci/tests] continue to reduce ci time cost
* [misc] update dockerfile * [misc] update dockerfile
* Add Streaming LLM * add some parameters to llama_generation.py * verify streamingllm config * add test_streamingllm.py * modified according to the opinions of review * add Citation * change _block_tables tolist
* remove fp16 from lamb * remove d2h copy in checking states --------- Co-authored-by: Edenzzzz <[email protected]>
* [test] smaller gpt2 test case * [test] reduce test cases: tests/test_zero/test_gemini/test_zeroddp_state_dict.py * [test] reduce test cases: tests/test_zero/test_gemini/test_grad_accum.py * [test] reduce test cases tests/test_zero/test_gemini/test_optim.py * Revert "[test] smaller gpt2 test case" Some tests might depend on the size of model (num of chunks) This reverts commit df705a5. * [test] reduce test cases: tests/test_checkpoint_io/test_gemini_checkpoint_io.py * [CI] smaller test model for two mwo the two modifid cases * [CI] hardcode gpt model for tests/test_zero/test_gemini/test_search.py since we need a fixed answer there
* [fix] branch for fix testcase; * [fix] fix test_analyzer & test_auto_parallel; * [fix] remove local change about moe; * [fix] rm local change moe; * [fix] fix test_deepfm_model & test_dlrf_model； * [fix] fix test_hf_albert & test_hf_gpt;
* [gemini] optimize reduce scatter d2h copy * [fix] fix missing reduce variable * [refactor] remove legacy async reduce scatter code * [gemini] missing sync * Revert "[refactor] remove legacy async reduce scatter code" This reverts commit 58ad76d. * [gemini] further optimize with async all reduce * [fix] pass flag from manager to chunk
Added FORCE_CUDA environment variable support, to enable building extensions where a GPU device is not present but cuda libraries are.
* fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…or ppo, sp is buggy
for more information, see https://pre-commit.ci
[Inference] Refactor modeling attention layer by abstracting attention backends
* refactor baichuan * remove unused code and add TODO for lazyinit
[ColossalChat] Colossalchat upgrade
* Fix torch int32 dtype Signed-off-by: char-1ee <[email protected]> * Fix flash-attn import Signed-off-by: char-1ee <[email protected]> * Add generalized model test Signed-off-by: char-1ee <[email protected]> * Remove exposed path to model Signed-off-by: char-1ee <[email protected]> * Add default value for use_flash_attn Signed-off-by: char-1ee <[email protected]> * Rename model test Signed-off-by: char-1ee <[email protected]> --------- Signed-off-by: char-1ee <[email protected]>
…5781) * use async stream to prefetch and h2d data moving * Remove redundant code
* [gemini] quick fix on possible async operation * [gemini] quick fix on possible async operation
* [shardformer]upgrade transformers for gpt2/gptj/whisper (hpcaitech#5807) * [shardformer] fix modeling of gpt2 and gptj * [shardformer] fix whisper modeling * [misc] update requirements --------- Co-authored-by: ver217 <[email protected]> * [shardformer]upgrade transformers for mistral (hpcaitech#5808) * upgrade transformers for mistral * fix * fix * [shardformer]upgrade transformers for llama (hpcaitech#5809) * update transformers fix * fix * fix * [inference] upgrade transformers (hpcaitech#5810) * update transformers fix * fix * fix * fix * fix * [gemini] update transformers for gemini (hpcaitech#5814) --------- Co-authored-by: ver217 <[email protected]>
…ero working/master params bug
* support tp + sp + pp * remove comments --------- Co-authored-by: Edenzzzz <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📌 Checklist before creating the PR
[doc/gemini/tensor/...]: A concise descriptionpip install pre-commit && pre-commit install🚨 Issue number
📝 What does this PR do?
💥 Checklist before requesting a review
⭐️ Do you enjoy contributing to Colossal-AI?
Tell us more if you don't enjoy contributing to Colossal-AI.