Use_Case_1_Image_Short_Captions.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "code",
      "source": [
        "# Copyright 2024 Reddit, Inc.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "metadata": {
        "id": "tM8VsbOLyEnU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Use Case 1. Image Short Captions\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/reddit/kdd2024-tutorial-breaking-barriers/blob/master/Use_Case_1_Image_Short_Captions.ipynb)"
      ],
      "metadata": {
        "id": "7rlH-wrbyTRW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Overview\n",
        "\n",
        "This notebook provides a hands-on guide to deploying and prompting different multimodal LLMs to generate short, descriptive captions for images. The challenges and limitations of using LLMs for image captioning will be discussed."
      ],
      "metadata": {
        "id": "g-7W4aKIcQfR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Setting Up Google Colab\n",
        "Google Colab provides a convenient platform to run Python code in the cloud, with access to powerful computing resources, including GPUs. For this tutorial, it is recommended to enable GPU acceleration:\n",
        "\n",
        "1.   Click on *Runtime* in the top menu.\n",
        "2.   Select *Change runtime type*.\n",
        "3.   In the dialog that appears, under *Hardware accelerator*, choose **T4 GPU** (or any other GPU that you may have access to) if it is not already enabled.\n",
        "4.   Click *Save*."
      ],
      "metadata": {
        "id": "IxSl4uD2dDSS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Requirements\n",
        "\n",
        "The code in this notebook is based on Transformers, so we need to install all necessary requirements."
      ],
      "metadata": {
        "id": "86ix3ofBdkCs"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6huqOaKRb6ix"
      },
      "outputs": [],
      "source": [
        "# Install required Python packages\n",
        "!pip install -U transformers==4.41.2 bitsandbytes accelerate flash_attn"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Settings"
      ],
      "metadata": {
        "id": "-cnyD9XM1Hub"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run the following cells to make some convenient settings."
      ],
      "metadata": {
        "id": "JhKD91yvivOz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Disable Transformer warnings\n",
        "import logging\n",
        "logging.basicConfig(level=logging.INFO)\n",
        "\n",
        "import transformers\n",
        "transformers.logging.set_verbosity_error()\n",
        "\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "# Set GPU device\n",
        "import torch\n",
        "torch.set_default_device(\"cuda\") # or \"cpu\" is GPU is not available\n",
        "\n",
        "# Import Garbage Collector\n",
        "import gc"
      ],
      "metadata": {
        "id": "qHBXCw-KfD_e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run the following cell to get the run time on every cell execution:"
      ],
      "metadata": {
        "id": "HomTygaHjI46"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install ipython-autotime\n",
        "%load_ext autotime"
      ],
      "metadata": {
        "id": "9P6DMHXhrkSw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run the following cell to enable wrap when printing long strings:"
      ],
      "metadata": {
        "id": "tCmvlRdPspjG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from IPython.display import HTML, display\n",
        "\n",
        "def set_css():\n",
        "  display(HTML(\"\"\"\n",
        "  <style>\n",
        "    pre {\n",
        "        white-space: pre-wrap;\n",
        "    }\n",
        "  </style>\n",
        "  \"\"\"))\n",
        "get_ipython().events.register(\"pre_run_cell\", set_css)"
      ],
      "metadata": {
        "id": "nJJcC9czsX0K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Test Picture\n",
        "\n",
        "We will use the first image in the tutorial dataset."
      ],
      "metadata": {
        "id": "QGzcVG-qeIZc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Download images\n",
        "\n",
        "from PIL import Image\n",
        "import requests\n",
        "\n",
        "def download_image(url):\n",
        "  image = Image.open(requests.get(url, stream=True).raw)\n",
        "  return image"
      ],
      "metadata": {
        "id": "E_JfgDKdqQyh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "image = download_image(\"https://raw.githubusercontent.com/reddit/kdd2024-tutorial-breaking-barriers/main/media/image1.jpg\")\n",
        "image"
      ],
      "metadata": {
        "id": "ep6pWBFXgmwz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Get Ready!\n",
        "\n",
        "**❗Important❗** Despite our attempts to free the memory after testing each model, your GPU may get out of memory.\n",
        "\n",
        "If this happens, you will need to restart your Colab session. To do so, go to **Runtime** menu, click on *Restart session* and run the cells in the **Settings** and **Test Picture** sections. Then continue testing the next model."
      ],
      "metadata": {
        "id": "PS_TzgELgYaI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "## Model 1: LLaVA-1.5-7B\n",
        "\n",
        "LLaVa (proposed in [1] and improved in [2]) is an open-source auto-regressive language model, based on the transformer architecture, trained by fine-tuning Llama/Vicuna on GPT-generated multimodal instruction-following data. It is sometimes seen as an \"open source version of GPT4\".\n",
        "\n",
        "##### **References**\n",
        "\n",
        "[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. [arXiv:2304.08485](https://arxiv.org/abs/2304.08485) [cs.VCV]\n",
        "\n",
        "[2] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved Baselines with Visual Instruction Tuning. [arXiv:2310.03744](https://arxiv.org/abs/2310.03744) [cs.CV]"
      ],
      "metadata": {
        "id": "hcsr688ofDe1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load Transformer pipeline\n",
        "\n",
        "from transformers import BitsAndBytesConfig\n",
        "from transformers import pipeline\n",
        "\n",
        "quantization_config = BitsAndBytesConfig(\n",
        "    load_in_4bit=True,\n",
        "    bnb_4bit_compute_dtype=torch.float16\n",
        ")\n",
        "pipe = pipeline(\"image-to-text\",\n",
        "                model=\"llava-hf/llava-1.5-7b-hf\",\n",
        "                model_kwargs={\"quantization_config\": quantization_config})"
      ],
      "metadata": {
        "id": "xXtlldP3gruj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "It is important to prompt the model with a specific format, which is:\n",
        "```bash\n",
        "USER: <image>\\n<prompt>\\nASSISTANT:\n",
        "```"
      ],
      "metadata": {
        "id": "wro6iLLAg_db"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Set prompt and max output tokens\n",
        "prompt = \"USER: <image>\\nDescribe the image in detail\\nASSISTANT:\"\n",
        "max_new_tokens = 256\n",
        "\n",
        "# Prompt\n",
        "outputs = pipe(image, prompt=prompt, generate_kwargs={\"max_new_tokens\": max_new_tokens})\n",
        "\n",
        "# Get caption\n",
        "caption = outputs[0][\"generated_text\"][len(prompt)-5:]\n",
        "\n",
        "# Display caption\n",
        "print(caption)"
      ],
      "metadata": {
        "id": "gtiFvsP4hXfO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now test with different prompts. For instance:\n",
        "- `Generate a short caption for the image`\n",
        "- `Write a very short caption for the image with less than 20 words`\n",
        "- `Describe what you see in the picture`\n",
        "- `You are an assistant for a person with visual impairment. Describe the picture in detail so that the person can have\n",
        "a full idea of the contents. But make it short, without overwhelming the user with non important details of the picture.`\n"
      ],
      "metadata": {
        "id": "qttT1gFZs-7M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Clean memory (as far as possible)\n",
        "del pipe\n",
        "del outputs\n",
        "gc.collect()\n",
        "torch.cuda.empty_cache()"
      ],
      "metadata": {
        "id": "CG2p7hYBajSJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Model 2: nanoLLaVA\n",
        "\n",
        "[nanoLLaVA](https://huggingface.co/qnguyen3/nanoLLaVA) is a \"small but mighty\" 1B vision-language model designed to run efficiently on edge devices.\n",
        "\n",
        "It is based on the [Quyen-SE-v0.1](https://huggingface.co/vilm/Quyen-SE-v0.1) small base LLM combined with a [CLIP-SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384) vision encoder.\n",
        "\n",
        "Quyen-SE is a fine-tuned version of powerful small model Qwen-1.5–0.5B, with a 32k tokens context window. SigLIP is [CLIP](https://huggingface.co/docs/transformers/model_doc/clip), a multimodal model, with a better loss function."
      ],
      "metadata": {
        "id": "5F32mlnJjzno"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "\n",
        "# Load model\n",
        "model = AutoModelForCausalLM.from_pretrained(\n",
        "    \"qnguyen3/nanoLLaVA\",\n",
        "    torch_dtype=torch.float16,\n",
        "    device_map=\"auto\",\n",
        "    trust_remote_code=True)\n",
        "tokenizer = AutoTokenizer.from_pretrained(\n",
        "    \"qnguyen3/nanoLLaVA\",\n",
        "    trust_remote_code=True)"
      ],
      "metadata": {
        "id": "KS-t_BGSkaxm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The prompt uses the ChatML standard, however, without `\\n` at the end of `<|im_end|>`."
      ],
      "metadata": {
        "id": "7XtH89idln7x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Text prompt\n",
        "prompt = 'Describe the image in detail'\n",
        "\n",
        "# Build actual prompt\n",
        "messages = [\n",
        "    {\"role\": \"user\", \"content\": f'<image>\\n{prompt}'}\n",
        "]\n",
        "text = tokenizer.apply_chat_template(\n",
        "    messages,\n",
        "    tokenize=False,\n",
        "    add_generation_prompt=True\n",
        ")\n",
        "print(text)"
      ],
      "metadata": {
        "id": "MTvfAlEZmEHx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Process text\n",
        "text_chunks = [tokenizer(chunk).input_ids for chunk in text.split('<image>')]\n",
        "input_ids = torch.tensor(text_chunks[0] + [-200] + text_chunks[1], dtype=torch.long).unsqueeze(0)\n",
        "\n",
        "# Process image\n",
        "image_tensor = model.process_images([image], model.config).to(dtype=model.dtype)\n",
        "\n",
        "# Generate\n",
        "output_ids = model.generate(\n",
        "    input_ids,\n",
        "    images=image_tensor,\n",
        "    max_new_tokens=2048,\n",
        "    use_cache=True)[0]\n",
        "caption = tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True)\n",
        "\n",
        "# Display caption\n",
        "print(caption)"
      ],
      "metadata": {
        "id": "MOavIvKdl_BY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "It can be clearly seen that this model is much more verbose than the previous one, so the prompt has to be adapted."
      ],
      "metadata": {
        "id": "2JAZdXpUugpp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Clean memory (as far as possible)\n",
        "del model\n",
        "del tokenizer\n",
        "del text_chunks\n",
        "del input_ids\n",
        "del image_tensor\n",
        "del output_ids\n",
        "gc.collect()\n",
        "torch.cuda.empty_cache()"
      ],
      "metadata": {
        "id": "TTGQRmdzcHiU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Model 3: Phi-3-vision-128k-instruct\n",
        "\n",
        "The [Phi-3-Vision-128K-Instruct](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct) is a lightweight, state-of-the-art open multimodal model built upon datasets which include - synthetic data and filtered publicly available websites - with a focus on very high-quality, reasoning dense data both on text and vision. The model belongs to the Phi-3 model family, and the multimodal version comes with 128K context length (in tokens) it can support. The model underwent a rigorous enhancement process, incorporating both supervised fine-tuning and direct preference optimization to ensure precise instruction adherence and robust safety measures."
      ],
      "metadata": {
        "id": "-HDIHZJ4t8tp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load model\n",
        "\n",
        "from transformers import AutoModelForCausalLM, AutoProcessor\n",
        "\n",
        "model = AutoModelForCausalLM.from_pretrained(\n",
        "    \"microsoft/Phi-3-vision-128k-instruct\",\n",
        "    torch_dtype=\"auto\",\n",
        "    device_map=\"cuda\",\n",
        "    trust_remote_code=True,\n",
        "    _attn_implementation=\"eager\") # \"flash_attention_2\" to enable flash attention\n",
        "processor = AutoProcessor.from_pretrained(\n",
        "    \"microsoft/Phi-3-vision-128k-instruct\",\n",
        "    trust_remote_code=True)"
      ],
      "metadata": {
        "id": "Uh23FccluUN2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Prompt\n",
        "prompt = \"Describe the image in detail\"\n",
        "\n",
        "# Process inputs\n",
        "messages = [\n",
        "    {\"role\": \"user\", \"content\": f\"<|image_1|>\\n{prompt}\"}\n",
        "]\n",
        "prompt = processor.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)\n",
        "inputs = processor(prompt, [image], return_tensors=\"pt\").to(\"cuda\")\n",
        "\n",
        "# Generate\n",
        "generation_args = {\n",
        "    \"max_new_tokens\": 500,\n",
        "    \"temperature\": 0.0,\n",
        "    \"do_sample\": False,\n",
        "}\n",
        "generate_ids = model.generate(\n",
        "    **inputs,\n",
        "    eos_token_id=processor.tokenizer.eos_token_id,\n",
        "    **generation_args)\n",
        "\n",
        "# Remove input tokens\n",
        "generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:]\n",
        "caption = processor.batch_decode(\n",
        "    generate_ids,\n",
        "    skip_special_tokens=True,\n",
        "    clean_up_tokenization_spaces=False)[0]\n",
        "\n",
        "# Display caption\n",
        "print(caption)"
      ],
      "metadata": {
        "id": "-8aOVfamvShU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check out some other interesting use cases of this model:\n",
        "[6 Real-World Uses of Microsoft’s Newest Phi-3 Vision-Language Model](https://towardsdatascience.com/6-real-world-uses-of-microsofts-newest-phi-3-vision-language-model-8ebbfa317fe8)"
      ],
      "metadata": {
        "id": "jx5HhtO9xyoZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Clean memory (as far as possible)\n",
        "del model\n",
        "del processor\n",
        "del inputs\n",
        "del generate_ids\n",
        "gc.collect()\n",
        "torch.cuda.empty_cache()"
      ],
      "metadata": {
        "id": "vZyE8FSRdHpt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Model 4: imp-v1-3b\n",
        "\n",
        "The **Imp project** aims to provide a family of a strong multimodal small language models, and their [imp-v1-3b](https://huggingface.co/MILVLG/imp-v1-3b) is one of those models with only 3B parameters, build upon a small yet powerful Phi-2 (2.7B) and a powerful visual encoder SigLIP, and trained on the LLaVA-v1.5 training set. This model significantly outperforms the counterparts of similar model sizes, and even achieves slightly better performance than the strong LLaVA-7B model on various multimodal benchmarks."
      ],
      "metadata": {
        "id": "KQm0mJpJel51"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Getting Started"
      ],
      "metadata": {
        "id": "Ac1i9J5avzq8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "\n",
        "# Load model\n",
        "model = AutoModelForCausalLM.from_pretrained(\n",
        "    \"MILVLG/imp-v1-3b\",\n",
        "    torch_dtype=torch.float16,\n",
        "    device_map=\"auto\",\n",
        "    trust_remote_code=True)\n",
        "tokenizer = AutoTokenizer.from_pretrained(\n",
        "    \"MILVLG/imp-v1-3b\",\n",
        "    trust_remote_code=True)"
      ],
      "metadata": {
        "id": "Eydsva9bq845"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Text prompt\n",
        "#prompt = \"Describe the image in detail\"\n",
        "#prompt = \"Generate a short caption for the image\"\n",
        "prompt = \"Write a very short caption for the image with less than 20 words\"\n",
        "#prompt = \"Describe what you see in the picture\"\n",
        "\n",
        "messages = [\n",
        "    {\"role\": \"user\", \"content\": f'<image>\\n{prompt}'}\n",
        "]\n",
        "text = tokenizer.apply_chat_template(\n",
        "    messages,\n",
        "    tokenize=False,\n",
        "    add_generation_prompt=True\n",
        ")\n",
        "print(text)"
      ],
      "metadata": {
        "id": "TuaPZVSQrahb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Process text and image\n",
        "input_ids = tokenizer(text, return_tensors=\"pt\").input_ids\n",
        "image_tensor = model.image_preprocess(image)\n",
        "\n",
        "# Generate the answer\n",
        "output_ids = model.generate(\n",
        "    input_ids,\n",
        "    max_new_tokens=100,\n",
        "    images=image_tensor,\n",
        "    use_cache=True)[0]\n",
        "caption = tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True)\n",
        "\n",
        "# Display caption\n",
        "print(caption)"
      ],
      "metadata": {
        "id": "iazTBrohrsmP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Clean memory (as far as possible)\n",
        "del model\n",
        "del tokenizer\n",
        "del input_ids\n",
        "del image_tensor\n",
        "del output_ids\n",
        "gc.collect()\n",
        "torch.cuda.empty_cache()"
      ],
      "metadata": {
        "id": "OK2GjII_d2gC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Improved Implementation (MLLMv1)\n",
        "\n",
        "The following code shows a much better implementation that should be easier to integrate in a processing workflow."
      ],
      "metadata": {
        "id": "QDUtRxXCsKBM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "from PIL import Image\n",
        "import re\n",
        "\n",
        "class MLLMv1:\n",
        "\n",
        "  def __init__(self):\n",
        "    torch.set_default_device(\"cuda\")\n",
        "    self.vision_model = AutoModelForCausalLM.from_pretrained(\n",
        "      \"MILVLG/imp-v1-3b\",\n",
        "      torch_dtype=torch.float16,\n",
        "      device_map=\"auto\",\n",
        "      trust_remote_code=True)\n",
        "    self.vision_tokenizer = AutoTokenizer.from_pretrained(\n",
        "        \"MILVLG/imp-v1-3b\",\n",
        "        trust_remote_code=True)\n",
        "\n",
        "  def get_image_caption(self,\n",
        "                        image: Image,\n",
        "                        base_prompt=\"Write a very short caption for the image with less than 20 words\") -> str:\n",
        "    return self.prompt_llm(image, base_prompt)\n",
        "\n",
        "  def get_image_description(self,\n",
        "                            image: Image,\n",
        "                            base_prompt=\"Write a short description for the image\") -> str:\n",
        "    return self.prompt_llm(image, base_prompt)\n",
        "\n",
        "  def prompt_llm(self,\n",
        "                 image: Image,\n",
        "                 prompt: str,\n",
        "                 max_new_tokens: int = 256,\n",
        "                 temperature: float = 0.9,\n",
        "                 top_k: int = 50,\n",
        "                 top_p: float = 0.95) -> str:\n",
        "    if image:\n",
        "      text = self.vision_tokenizer.apply_chat_template(\n",
        "          [{\"role\": \"user\", \"content\": f\"<image>\\n{prompt}\"}],\n",
        "          tokenize=False,\n",
        "          add_generation_prompt=True\n",
        "      )\n",
        "      image_tensor = self.vision_model.image_preprocess(image)\n",
        "    else:\n",
        "      text = self.vision_tokenizer.apply_chat_template(\n",
        "          [{\"role\": \"user\", \"content\": f\"{prompt}\"}],\n",
        "          tokenize=False,\n",
        "          add_generation_prompt=True\n",
        "      )\n",
        "      image_tensor = None\n",
        "    input_ids = self.vision_tokenizer(text, return_tensors=\"pt\").input_ids\n",
        "    output_ids = self.vision_model.generate(\n",
        "      input_ids,\n",
        "      max_new_tokens=max_new_tokens,\n",
        "      images=image_tensor,\n",
        "      temperature=temperature,\n",
        "      do_sample=True,\n",
        "      top_k=top_k,\n",
        "      top_p=top_p,\n",
        "      use_cache=True)[0]\n",
        "    response = self.vision_tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True)\n",
        "    response = response.replace(\"\\n\", \" \").replace(\"  \", \" \").strip()\n",
        "    response = re.sub(r\"<\\|im_end\\|>.*$\", \"\", response)\n",
        "    return response"
      ],
      "metadata": {
        "id": "LNUctkD8em0o"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load model\n",
        "ic = MLLMv1()"
      ],
      "metadata": {
        "id": "WIPis_Qle4M9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Test one image\n",
        "image = download_image(f\"https://raw.githubusercontent.com/reddit/kdd2024-tutorial-breaking-barriers/main/media/image1.jpg\")\n",
        "display(image)\n",
        "print(ic.get_image_caption(image))"
      ],
      "metadata": {
        "id": "w3KGpzLge-Ku"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Test all images\n",
        "for image_name in [\"image1.jpg\", \"image2.png\", \"image3.png\", \"image4.png\", \"image5.png\", \"image6.png\"]:\n",
        "  image = download_image(f\"https://raw.githubusercontent.com/reddit/kdd2024-tutorial-breaking-barriers/main/media/{image_name}\")\n",
        "  display(image)\n",
        "  print(ic.get_image_caption(image))"
      ],
      "metadata": {
        "id": "7PriNM7u1zXQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Clean memory (as far as possible)\n",
        "del ic\n",
        "gc.collect()\n",
        "torch.cuda.empty_cache()"
      ],
      "metadata": {
        "id": "LcIqKxBgegHz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Other Models"
      ],
      "metadata": {
        "id": "stkSp60WnMht"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Qwen-VL\n",
        "\n",
        "Familiy of vision models [1], significantly upgraded for detailed recognition capabilities and text recognition abilities.\n",
        "\n",
        "- [GitHub project](https://github.com/QwenLM/Qwen-VL)\n",
        "- [Online demo](https://huggingface.co/spaces/Qwen/Qwen-VL-Max)\n",
        "\n",
        "##### **References**\n",
        "[1] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Jun-\n",
        "yang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-\n",
        "Language Model for Understanding, Localization, Text Reading, and Beyond.\n",
        "[arXiv:2308.12966](https://arxiv.org/abs/2308.12966) [cs.CV]\n"
      ],
      "metadata": {
        "id": "hnCefcaAxzd6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### CogVLM 2\n",
        "\n",
        "A new generation of strong open-source models [1] based on Meta-Llama-3-8B-Instruct, supporting a 8K content length, an image resolution up to 1344 * 1344 and output in both Chinese and English. **A GPT-4V Level Multimodal LLM on Your Phone**.\n",
        "\n",
        "- [GitHub project](https://github.com/THUDM/CogVLM2)\n",
        "- [Model in Huggingface Hub](https://huggingface.co/THUDM/cogvlm2-llama3-chat-19B)\n",
        "- [Online demo](https://huggingface.co/spaces/THUDM/CogVLM-CogAgent)\n",
        "\n",
        "##### **References**\n",
        "[1] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji,\n",
        "Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming\n",
        "Ding, and Jie Tang. 2023. CogAgent: A Visual Language Model for GUI Agents.\n",
        "[arXiv:2312.08914](https://arxiv.org/abs/2312.08914) [cs.CV]"
      ],
      "metadata": {
        "id": "XNDR6zs0x8nR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### MiniCPM-Llama3-V-2.5\n",
        "\n",
        "MiniCPM-V [1] is a series of end-side multimodal LLMs designed for vision-language understanding. MiniCPM-Llama3-V 2.5 is the latest and most capable model in the series. With a total of 8B parameters, the model surpasses proprietary models such as GPT-4V-1106, Gemini Pro, Qwen-VL-Max and Claude 3 in overall performance. The model can also support multimodal conversation for over 30 languages including English, Chinese, French, Spanish, German etc.\n",
        "\n",
        "- [Project in GitHub](https://github.com/OpenBMB/MiniCPM-V)\n",
        "- [Model in Huggingface Hub](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5)\n",
        "- [Online demo](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5)\n",
        "\n",
        "##### **References**\n",
        "\n",
        "[1] MiniCPM-V Team. MiniCPM-V 2.0: An Efficient End-side MLLM with Strong OCR and Understanding Capabilities. Online: https://openbmb.vercel.app/minicpm-v-2-en"
      ],
      "metadata": {
        "id": "HOz7eZYnx2qB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Florence-2\n",
        "\n",
        "Florence-2 [1] is an advanced vision foundation model trained by Microsoft with a sequence-to-sequence architecture that uses a prompt-based approach to handle a wide range of vision and vision-language tasks like captioning, object detection, and segmentation.\n",
        "\n",
        "- [Model in Huggingface Hub](https://huggingface.co/microsoft/Florence-2-large)\n",
        "- [Sample notebook](https://huggingface.co/microsoft/Florence-2-large/blob/main/sample_inference.ipynb)\n",
        "- [Online demo](https://huggingface.co/spaces/SixOpen/Florence-2-large-ft)\n",
        "\n",
        "##### **References**\n",
        "[1] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. [arXiv:2311.06242](https://arxiv.org/abs/2311.06242) [cs.CV]"
      ],
      "metadata": {
        "id": "kwzazvhYa9E9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "# Discussion: How Image Captioning with Multimodal LLMs can improve Accessibility in Social Media\n",
        "- **Automatic image captioning**: For visually impaired users, multimodal LLMs can automatically generate captions for images, making posts accessible and understandable.\n",
        "- **Extended captions with context**: LLMs can provide detailed descriptions that go beyond simple captions, explaining the context, emotions, and key details in an image. This enhances the user experience for everyone.\n",
        "- **Summarizing image content**: For users with limited time or attention spans, LLM-generated summaries of images can offer a quick overview of the content without having to examine the image in detail.\n",
        "- **Translation of captions**: LLMs can translate captions into multiple languages, making Reddit posts accessible to a wider international audience.\n",
        "- **Personalization based on user preferences**: LLMs can adapt their captions and descriptions based on user preferences and accessibility needs, offering a more personalized experience.\n",
        "- **Improving content moderation**: By analyzing image content and captions, LLMs can help identify and flag potentially offensive or inappropriate images, contributing to a safer and more inclusive community."
      ],
      "metadata": {
        "id": "dwP1sI0vpl4d"
      }
    }
  ]
}