Sync release to main for rhoai-2.14 #172
Commits on Sep 11, 2024
-
[Speculative Decoding] Test refactor (vllm-project#8317)
Co-authored-by: youkaichao <[email protected]>
-
Co-authored-by: Roger Wang <[email protected]>
-
Commits on Sep 12, 2024
-
-
-
-
[Bugfix] Fix InternVL2 inference with various num_patches (vllm-proje…
…ct#8375) Co-authored-by: DarkLight1337 <[email protected]>
-
[Model] Support multiple images for qwen-vl (vllm-project#8247)
Signed-off-by: Alex-Brooks <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: DarkLight1337 <[email protected]>
3 people committedSep 12, 2024 -
[Bugfix] Offline mode fix (vllm-project#8376)
Signed-off-by: Joe Runde <[email protected]>
-
-
[Hotfix][Core][VLM] Disable chunked prefill by default and prefix cac…
…hing for multimodal models (vllm-project#8425)
Commits on Sep 13, 2024
-
-
-
-
-
-
-
-
-
[Hardware][intel GPU] bump up ipex version to 2.3 (vllm-project#8365)
Co-authored-by: Yan Ma <[email protected]>
Commits on Sep 14, 2024
-
[Model] support minicpm3 (vllm-project#8297)
Co-authored-by: DarkLight1337 <[email protected]>
-
-
Commits on Sep 15, 2024
-
[Bugfix][Model] Fix Python 3.8 compatibility in Pixtral model by upda…
…ting type annotations (vllm-project#8490)
Commits on Sep 16, 2024
-
[Kernel] Enable 8-bit weights in Fused Marlin MoE (vllm-project#8032)
Co-authored-by: Dipika <[email protected]>
-
Commits on Sep 17, 2024
-
[Bugfix] Fix 3.12 builds on main (vllm-project#8510)
Signed-off-by: Joe Runde <[email protected]>
-
-
[Frontend] Improve Nullable kv Arg Parsing (vllm-project#8525)
Signed-off-by: Alex-Brooks <[email protected]>
-
[Misc] Limit to ray[adag] 2.35 to avoid backward incompatible change (v…
…llm-project#8509) Signed-off-by: Rui Qiao <[email protected]>
-
[Benchmark] Support sample from HF datasets and image input for bench…
…mark_serving (vllm-project#8495)
-
[Encoder decoder] Add cuda graph support during decoding for encoder-…
…decoder models (vllm-project#7631)
-
[Model] Add mistral function calling format to all models loaded with…
… "mistral" format (vllm-project#8515) Co-authored-by: Cyrus Leung <[email protected]>
-
[Bugfix] Fix TP > 1 for new granite (vllm-project#8544)
Signed-off-by: Joe Runde <[email protected]>
-
[doc] improve installation doc (vllm-project#8550)
Co-authored-by: Andy Dai <[email protected]>
-
[Kernel] Change interface to Mamba causal_conv1d_update for continuou…
…s batching (vllm-project#8012)
Commits on Sep 18, 2024
-
-
-
[CI/Build] Update Ruff version (vllm-project#8469)
Signed-off-by: Aaron Pham <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>
-
[Core][Bugfix][Perf] Introduce
MQLLMEngineto avoidasyncioOH (v……llm-project#8157) Co-authored-by: Nick Hill <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: Simon Mo <[email protected]>
5 people committedSep 18, 2024 -
[Core] zmq: bind only to 127.0.0.1 for local-only usage (vllm-project…
…#8543) Signed-off-by: Russell Bryant <[email protected]>
-
[Model] Support Solar Model (vllm-project#8386)
Co-authored-by: Michael Goin <[email protected]>
-
[AMD][ROCm]Quantization methods on ROCm; Fix _scaled_mm call (vllm-pr…
…oject#8380) Co-authored-by: Alexei-V-Ivanov-AMD <[email protected]> Co-authored-by: Michael Goin <[email protected]>
3 people committedSep 18, 2024 -
[Kernel] Change interface to Mamba selective_state_update for continu…
…ous batching (vllm-project#8039)
-
[Bugfix] add
dead_errorproperty to engine client (vllm-project#8574)Signed-off-by: Joe Runde <[email protected]>
Commits on Sep 19, 2024
-
[Kernel] Remove marlin moe templating on thread_m_blocks (vllm-projec…
…t#8573) Co-authored-by: [email protected]
-
[Bugfix] [Encoder-Decoder] Bugfix for encoder specific metadata const…
…ruction during decode of encoder-decoder models. (vllm-project#8545)
-
Commits on Sep 20, 2024
-
-
[CI/Build] Removing entrypoints/openai/test_embedding.py test from RO…
…Cm build (vllm-project#8670)
-
[Doc] neuron documentation update (vllm-project#8671)
Signed-off-by: omrishiv <[email protected]>
-
[Hardware][AWS] update neuron to 2.20 (vllm-project#8676)
Signed-off-by: omrishiv <[email protected]>
Commits on Sep 21, 2024
-
-
[MISC] add support custom_op check (vllm-project#8557)
Co-authored-by: youkaichao <[email protected]>
-
-
[Kernel][Triton][AMD] Remove tl.atomic_add from awq_gemm_kernel, 2-5x…
… speedup MI300, minor improvement for MI250 (vllm-project#8646)
Commits on Sep 22, 2024
-
-
[Misc] Use NamedTuple in Multi-image example (vllm-project#8705)
Signed-off-by: Alex-Brooks <[email protected]>
-
[Model][VLM] Add LLaVA-Onevision model support (vllm-project#8486)
Co-authored-by: litianjian <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: DarkLight1337 <[email protected]>
5 people committedSep 22, 2024 -
Commits on Sep 23, 2024
-
[Bugfix] Fix CPU CMake build (vllm-project#8723)
Co-authored-by: Yuan <[email protected]>
-
-
[Core][Frontend] Support Passing Multimodal Processor Kwargs (vllm-pr…
…oject#8657) Signed-off-by: Alex-Brooks <[email protected]>
-
-
[VLM] Fix paligemma, fuyu and persimmon with transformers 4.45 : use …
…config.text_config.vocab_size (vllm-project#8707)
-
[CI/Build] use setuptools-scm to set __version__ (vllm-project#4738)
Co-authored-by: youkaichao <[email protected]>
-
[Kernel] (2/N) Machete - Integrate into CompressedTensorsWNA16 and GP…
…TQMarlin (vllm-project#7701) Co-authored-by: mgoin <[email protected]> Co-authored-by: Divakar Verma <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]>
4 people committedSep 23, 2024 -
[Core] Allow IPv6 in VLLM_HOST_IP with zmq (vllm-project#8575)
Signed-off-by: Russell Bryant <[email protected]>
-
Add output streaming support to multi-step + async while ensuring Req…
…uestOutput obj reuse (vllm-project#8335)
Commits on Sep 24, 2024
-
-
re-implement beam search on top of vllm core (vllm-project#8726)
Co-authored-by: Brendan Wong <[email protected]>
-
[Model] Expose Phi3v num_crops as a mm_processor_kwarg (vllm-project#…
…8658) Signed-off-by: Alex-Brooks <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: DarkLight1337 <[email protected]>
3 people committedSep 24, 2024 -
[Kernel] Split Marlin MoE kernels into multiple files (vllm-project#8661
) Co-authored-by: mgoin <[email protected]>
-
[Frontend] Batch inference for llm.chat() API (vllm-project#8648)
Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Roger Wang <[email protected]>
5 people committedSep 24, 2024 -
-
Commits on Sep 25, 2024
-
[Core][Bugfix] Support prompt_logprobs returned with speculative deco…
…ding (vllm-project#8047) Signed-off-by: Travis Johnson <[email protected]>
-
-
[Frontend] OpenAI server: propagate usage accounting to FastAPI middl…
…eware layer (vllm-project#8672)
-
[Bugfix] Ray 2.9.x doesn't expose available_resources_per_node (vllm-…
…project#8767) Signed-off-by: darthhexx <[email protected]>
-
-
-
[Model] Add support for the multi-modal Llama 3.2 model (vllm-project…
…#8811) Co-authored-by: simon-mo <[email protected]> Co-authored-by: Chang Su <[email protected]> Co-authored-by: Simon Mo <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Roger Wang <[email protected]>
6 people committedSep 25, 2024 -
Commits on Sep 27, 2024
-
-
-
-
-
-
Dockerfile.ubi: remove vllm-nccl workaround
Fixed upstream in vllm-project#5091
-
-
fixes RHOAIENG-8043 Co-authored-by: Chih-Chieh-Yang <[email protected]> Signed-off-by: Thomas Parnell <[email protected]>
-
-
-
-
Dockerfile.ubi: misc improvements
- get rid cuda-devel stage, use cuda 12.4 - add build flags - remove useless installs
-
-
Dockerfile.ubi: use tensorizer (opendatahub-io#64)
add libsodium for tensorizer encryption --------- Signed-off-by: Prashant Gupta <[email protected]> Co-authored-by: Daniele <[email protected]>
-
-
-
-
-
Dockerfile.ubi: get rid of --distributed-executor-backend=mp
this is the default when `--worker-use-ray` is not provided and world-size > 1
-
-
-
-
-
-
-
fix: update setup.py to differentiate between fork and upstream
Signed-off-by: Nathan Weinberg <[email protected]>
-
-
-
-
-
Dockerfile.ubi: get rid of custom cache manager
fixed in vllm-project#6140 fixes https://issues.redhat.com/browse/RHOAIENG-8043
-
-
-
-
-
feat: allow long max seq length
Signed-off-by: Travis Johnson <[email protected]>
-
-
fix: enable logprobs during spec decoding by default
Signed-off-by: Travis Johnson <[email protected]>
-
-
This turns off tracking by default. If someone wants to, they can simply override this in yaml.
-
A review showed that nowhere in the Dockerfile.ubi do we do a microdnf -y update to pickup any known CVE and bugfixes. This patch adds this to the build process.
-
-
-
Libsodium is being built with default CFLAGS. This adds optimization on par with cmake release builds. It also adds security hardening flags suggested for RHEL 9 to protect against various issues.
-
Remove debug code.
-
-
- get rid of non-essential dependencies - consolidate package installs - do not copy wheels in final stage - fix ccache usage - use flashattention with triton backend by default: - clone main_perf branch - build rocm target - set up triton rocm env var - configure numba, outlines and triton cache directory -
-
-
-
-
-
Dockerfile.rocm.ubi: get rid of build triton stage
this is a torch dependency when installed from the pytorch/rocm6.1 index: https://download.pytorch.org/whl/nightly/rocm6.1
-
-
-
This sets the vllm build to a Release build type, builds libsodium,
fixes bad permissions on a logging directory, installs libsodium, and adds a LD_LIBRARY_PATH to fixup python bindings for 2 packages.
-
This moves the install up a couple lines.
-
Correct logging directory permissions
The /var/log/rocm_smi_lib/ directory was world writable. It is fixed now so that it is world readable. Similarly, the file in it's directory is also world writable, it is now world readable.
-
-
