LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model

Introduction

Try it out!

The base versions of the llava-gemma models are available on HuggingFace (HF) at Intel/llava-gemma-2b. While these checkpoints have been converted to the HF version of LLaVA, their usage currently requires a modified preprocessor (available here).

With preprocessing_llavagemma.py copied to the appropriate location (e.g. the directory you are running your script from), you can try out the llava-gemma checkpoint using the following code snippet:

import requests
from PIL import Image
from transformers import (
  LlavaForConditionalGeneration,
  AutoTokenizer,
  CLIPImageProcessor
)
from processing_llavagemma import LlavaGemmaProcessor # This is in this repo

checkpoint = "Intel/llava-gemma-2b"

# Load model
model = LlavaForConditionalGeneration.from_pretrained(checkpoint)
processor = LlavaGemmaProcessor(
    tokenizer=AutoTokenizer.from_pretrained(checkpoint),
    image_processor=CLIPImageProcessor.from_pretrained(checkpoint)
)

# Prepare inputs
# Use gemma chat template
prompt = processor.tokenizer.apply_chat_template(
    [{'role': 'user', 'content': "What's the content of the image?<image>"}],
    tokenize=False,
    add_generation_prompt=True
)
url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(text=prompt, images=image, return_tensors="pt")
inputs = {k: v.to('cuda') for k, v in inputs.items()}
      
# Generate
generate_ids = model.generate(**inputs, max_length=30)
output = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
print(output)

LLaVA

LLaVA (Large Language and Vision Assistant) is a lightweight and powerful framework for combining pretrained langauge and vision models into a multimodal chatbot capable of taking combined vision and language inputs.

The "recipe" consists of three components and two training steps. The three components are a pretrained language model, a pretrained vision encoder, and a connector. In the v1.5 recipe, the language model is Vicuna, the vision encoder CLIP ViT-L/14, and the connector a 2-layer perceptron. The first stage pretrains the MLP connector by training on a dataset of 595k vision-language samples filtered from CC3M. The second stage jointly finetunes the language modela nd connector using a mixture of 665 multimodal instruction tuning examples.

Gemma

Gemma is “[a] family of lightweight, state-of-the art open models built from the same research and technology used to create the Gemini models” released by Google in February 2024.

We use the instruction-tuned checkpoints available on HuggingFace (gemma-2b-it and gemma-7b-it).

Performance

Using the LLaVA-v1.5 recipe, the models achieve reasonable but unspectacular performance on a range of multimodal benchmarks, failing to beat the 7B-parameter LLaVA model. We are currently experimenting with various techniques to improve the performance of the Gemma-based LLaVA model, which we will update this page with as we publish results.