New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Usage]: caching with different batches #8939

Open

1 task done

KevinZeng08 opened this issue Sep 29, 2024 · 0 comments

Open

1 task done

[Usage]: caching with different batches #8939

KevinZeng08 opened this issue Sep 29, 2024 · 0 comments

Labels

usage

KevinZeng08 commented Sep 29, 2024

Your current environment

Ubuntu 20.04, CUDA 12.1

How would you like to use vllm

Can vLLM do prefix caching with different batches?

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

KevinZeng08 added the usage label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment