* [fix/example] fix llama inference loading dtype

* revise loading dtype of benchmark llama3

* [release] update version

* [devops] update compatibility test

* [devops] update compatibility test

* [devops] update compatibility test

* [devops] update compatibility test

* [test] fix ddp plugin test

* [test] fix gptj and rpc test

* [devops] fix cuda ext compatibility

* [inference] fix flash decoding test

* [inference] fix flash decoding test

* [fix] branch for fix testcase;

* [fix] fix test_analyzer & test_auto_parallel;

* [fix] remove local change about moe;

* [fix] rm local change moe;

…5755)

* [ci/tests] simplify some test case to reduce testing time

* [ci/tests] continue to remove test case to reduce ci time cost

* restore some test config

* [ci/tests] continue to reduce ci time cost

* [misc] update dockerfile

* [misc] update dockerfile

* Add Streaming LLM

* add some parameters to llama_generation.py

* verify streamingllm config

* add test_streamingllm.py

* modified according to the opinions of review

* add Citation

* change _block_tables tolist

* remove fp16 from lamb

* remove d2h copy in checking states

---------

Co-authored-by: Edenzzzz <[email protected]>

* [test] smaller gpt2 test case

* [test] reduce test cases: tests/test_zero/test_gemini/test_zeroddp_state_dict.py

* [test] reduce test cases: tests/test_zero/test_gemini/test_grad_accum.py

* [test] reduce test cases tests/test_zero/test_gemini/test_optim.py

* Revert "[test] smaller gpt2 test case"

Some tests might depend on the size of model (num of chunks)

This reverts commit df705a5.

* [test] reduce test cases: tests/test_checkpoint_io/test_gemini_checkpoint_io.py

* [CI] smaller test model for two mwo the two modifid cases

* [CI] hardcode gpt model for tests/test_zero/test_gemini/test_search.py since we need a fixed answer there

* [fix] branch for fix testcase;

* [fix] fix test_analyzer & test_auto_parallel;

* [fix] remove local change about moe;

* [fix] rm local change moe;

* [fix] fix test_deepfm_model & test_dlrf_model；

* [fix] fix test_hf_albert & test_hf_gpt;

* [gemini] optimize reduce scatter d2h copy

* [fix] fix missing reduce variable

* [refactor] remove legacy async reduce scatter code

* [gemini] missing sync

* Revert "[refactor] remove legacy async reduce scatter code"

This reverts commit 58ad76d.

* [gemini] further optimize with async all reduce

* [fix] pass flag from manager to chunk

Added FORCE_CUDA environment variable support, to enable building extensions where a GPU device is not present but cuda libraries are.

* fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…or ppo, sp is buggy

for more information, see https://pre-commit.ci

Signed-off-by: char-1ee <[email protected]>

Signed-off-by: char-1ee <[email protected]>

Signed-off-by: char-1ee <[email protected]>

Signed-off-by: char-1ee <[email protected]>

Signed-off-by: char-1ee <[email protected]>

Signed-off-by: char-1ee <[email protected]>

[Inference] Refactor modeling attention layer by abstracting attention backends

* refactor baichuan

* remove unused code and add TODO for lazyinit

[ColossalChat] Colossalchat upgrade

…seless test

* Fix torch int32 dtype

Signed-off-by: char-1ee <[email protected]>

* Fix flash-attn import

Signed-off-by: char-1ee <[email protected]>

* Add generalized model test

Signed-off-by: char-1ee <[email protected]>

* Remove exposed path to model

Signed-off-by: char-1ee <[email protected]>

* Add default value for use_flash_attn

Signed-off-by: char-1ee <[email protected]>

* Rename model test

Signed-off-by: char-1ee <[email protected]>

---------

Signed-off-by: char-1ee <[email protected]>

…class member

…5781)

* use async stream to prefetch and h2d data moving

* Remove redundant code

* [gemini] quick fix on possible async operation

* [gemini] quick fix on possible async operation

* [shardformer]upgrade transformers for gpt2/gptj/whisper (hpcaitech#5807)

* [shardformer] fix modeling of gpt2 and gptj

* [shardformer] fix whisper modeling

* [misc] update requirements

---------

Co-authored-by: ver217 <[email protected]>

* [shardformer]upgrade transformers for mistral (hpcaitech#5808)

* upgrade transformers for mistral

* fix

* fix

* [shardformer]upgrade transformers for llama (hpcaitech#5809)

* update transformers

fix

* fix

* fix

* [inference] upgrade transformers (hpcaitech#5810)

* update transformers

fix

* fix

* fix

* fix

* fix

* [gemini] update transformers for gemini (hpcaitech#5814)

---------

Co-authored-by: ver217 <[email protected]>

…Hz188/ColossalAI into feature/moe

…ero working/master params bug

* support tp + sp + pp

* remove comments

---------

Co-authored-by: Edenzzzz <[email protected]>