Questions about lmdeploy version and GPU usage

[YoungjaeDev]

Hello, I have a few questions regarding the ShareGPT4Video project

1. The recommended version of lmdeploy is not specified. Which version should we use?
2. It appears that the internlm-xcomposer2-4khd model_name was removed 4 days ago. This seems to require code modifications. How should we address this?
   

   ShareGPT4Video/captioner/slide_captioner_lmdeploy.py

   Line 140 in 88426fd

   model = pipeline(args.model_name, chat_template_config=ChatTemplateConfig(model_name='internlm-xcomposer2-4khd'))

3. What single GPU was the slide_captioner_lmdeploy.py code tested on? I'm currently using multiple GPUs(24Gx2) due to OOM issues. Is it okay to use it this way?

[YoungjaeDev]

File "/home/user/res/lmdeploy/lmdeploy/serve/async_engine.py", line 521, in getprompt_input
    prompt = chat_template.messages2prompt(prompt,
  File "/home/user/res/lmdeploy/lmdeploy/model.py", line 223, in messages2prompt
    if len(messages) and messages[0]['role'] != 'system':
TypeError: string indices must be integers

The error suggests that it's expecting a message format with role-content structure, but our code simply uses (query, image) format.

[haodongze]

+1

[radna0]

How to do slide captioning on multiple gpus?

[YoungjaeDev]

@radna0

Use TurbomindEngineConfig

from lmdeploy import pipeline, ChatTemplateConfig, TurbomindEngineConfig

...

backend_config = TurbomindEngineConfig(tp=2)
model = pipeline('Lin-Chen/ShareCaptioner', backend_config=backend_config, chat_template_config=ChatTemplateConfig(model_name='internlm-xcomposer2-4khd'), log_level='INFO')

[radna0]

@YoungjaeDev Thank you, Do you have the slide captioning batch inference working for the ShareCaptioner-Video model? I'm looking at the code right now, trying to set up inference on a dataset and it seems like there is just no docs for that.

[YoungjaeDev]

Thank you, Do you have the slide captioning batch inference working for the ShareCaptioner-Video model? I'm looking at the code right now, trying to set up inference on a dataset and it seems like there is just no docs for that.

No. The current code doesn't run well because of package issues