Vision Model Benchmark on GB10

Comparing four open-source vision-language models against our Qwen3-VL-32B baseline on a single NVIDIA GB10 (mailgb02 · 192.168.1.9 · 119.6 GB GPU memory). Run via vLLM 0.21 + LiteLLM, 2026-05-28.

🏆 TL;DR — Winner: Gemma-3-12B-FP8. Matches Qwen3-VL-32B accuracy on both tests, runs 2.5× faster (16 vs 6.5 t/s), needs only 13 GB on disk (vs 33 GB), and the same model also serves as the lab's text-chat backbone. Best quality/speed/footprint balance for the Dell AI Innovation Hub.

Test setup

Hardware: NVIDIA GB10 on mailgb02, 119.6 GB unified GPU memory, FP8 supported via Hopper-class tensor cores
Stack: K3s · vLLM 0.21 (single-pod) · LiteLLM proxy · 1 image per request
Test 1 (OCR): 500 × 280 PNG of a synthetic Dell server status card with 6 text fields including "NVIDIA GB10"
Test 2 (Chart): 400 × 300 PNG bar chart with 3 labeled bars showing tokens/sec values
Sampling: temperature 0.1, max_tokens 300, warmup run before timing

Models tested

Qwen3-VL-32B-FP8 baseline

Params32B

Disk33 GB

GPU at 32K ctx~55 GB

SourceAlibaba

Gemma-3-12B-FP8 winner

Params12B

Disk13 GB

GPU at 16K ctx~24 GB

SourceGoogle · RedHatAI

Qwen2.5-VL-7B-FP8 fastest

Params7B

Disk9.4 GB

GPU at 16K ctx~16 GB

SourceAlibaba · RedHatAI

InternVL2.5-8B broken

Params8B

Disk16 GB

StatusvLLM template issue

SourceOpenGVLab

Phi-4 Multimodal skipped

Params5.6B

Disk12 GB

Statusneeds scipy in vLLM image

SourceMicrosoft

Output speed (tokens/sec)

Qwen2.5-VL-7B

27.0

Gemma-3-12B

16.0

InternVL2.5-8B

13.5

Qwen3-VL-32B (baseline)

6.5

Average of OCR + chart test, single-stream, post-warmup. Higher is better.

Accuracy summary

Model	OCR test	Chart test	Key issue
Qwen3-VL-32B baseline	★★★★★	★★★★☆	Misread "Qwen-VL" as "Oven-VL" in chart
Gemma-3-12B-FP8 winner	★★★★★	★★★★★	None — perfect on both tests
Qwen2.5-VL-7B-FP8 fastest	★★★☆☆	★★★★★	Misread "NVIDIA GB10" as "GTX 1080"
InternVL2.5-8B broken	✗	✗	Gibberish output — vLLM chat-template incompatibility
Phi-4 Multimodal skipped	—	—	ImportError: missing scipy in vllm/vllm-openai image

Test 1 — Server card OCR

Read a small synthetic image containing 6 fields: Model, GPU type, Status, Temp, Memory, server line. The key value is "NVIDIA GB10" — a brand-new SKU not commonly seen in training corpora.

Qwen3-VL-32B (baseline) 6.4 t/s

The image displays a system status summary for a Dell PowerEdge server... - Manufacturer: Dell - Model: R760xa - GPU Type: NVIDIA GB10 ✓ - Status: ONLINE ✓ - GPU Temperature: 65°C ✓ - Total Memory: 119.6 GB ✓

Gemma-3-12B-FP8 16.1 t/s

- Server Model: Dell PowerEdge R760xa - GPU: NVIDIA GB10 x1 ✓ - GPU Temperature: 65°C ✓ - GPU Memory: 119.6 GB ✓ - Status: ONLINE ✓

Qwen2.5-VL-7B-FP8 27.1 t/s

- Model: R760xa - GPU: NVIDIA GeForce GTX 1080 x1 ✗ (misread GB10) Note: The image mentions "NVIDIA GeForce GTX 1080" but it seems to be a typo... - Status: ONLINE ✓ - GPU Temperature: 65°C ✓ - Memory: 119.6 GB ✓

Test 2 — Bar chart reading

Read 3 labeled bars showing tokens/sec values: Qwen-VL = 6.5, Gemma-2 = 18.5, Gemma-3 = 14.

Qwen3-VL-32B (baseline) 6.5 t/s

- Oven-VL: 6.5 tokens per second ✗ (Qwen misread as Oven) - Gemma-2: 18.5 ✓ - Gemma-3: 14 ✓

Gemma-3-12B-FP8 15.2 t/s

- Qwen-VL: 6.5 Tokens per Second ✓ - Gemma-2: 18.5 Tokens per Second ✓ - Gemma-3: 14 Tokens per Second ✓ In summary: Gemma-2 is the fastest model in this comparison...

Qwen2.5-VL-7B-FP8 24.2 t/s

- Qwen-VL: 6.5 tokens per second ✓ - Gemma-2: 18.5 tokens per second ✓ - Gemma-3: 14 tokens per second ✓

Recommendation

Adopt Gemma-3-12B-FP8 as the lab's default vision + chat model. It:

Reads "NVIDIA GB10" correctly (Qwen2.5-VL-7B doesn't)
Reads chart values correctly (Qwen3-VL-32B has a transcription typo)
Runs 2.5× faster than the current Qwen3-VL-32B baseline (16 vs 6.5 t/s)
Replaces both Gemma-2-9B (chat) and Qwen3-VL-32B (vision) — one model, two roles
Uses ~24 GB of GPU at 16K context, leaving ~95 GB free for RAG, embeddings, Whisper STT and Kokoro TTS
Has 128K native context (vs Gemma-2's 8K) — essential for RAG with chat history

Keep Qwen3-VL-32B in standby (scale-to-zero, not deleted). For demos requiring the absolute best OCR/chart accuracy, a 30-second kubectl scale --replicas=1 brings it back online. Cost: zero GPU when idle.

Reproducibility

## Models
RedHatAI/gemma-3-12b-it-FP8-dynamic
RedHatAI/Qwen2.5-VL-7B-Instruct-FP8-Dynamic
OpenGVLab/InternVL2_5-8B  (broken on vLLM 0.21)
microsoft/Phi-4-multimodal-instruct  (needs scipy)

## Deployment shape
runtimeClassName: nvidia
enableServiceLinks: false
gpu-memory-utilization: 0.40-0.50
max-model-len: 16384
max-num-seqs: 4
limit-mm-per-prompt: '{"image":4}'

## Bench script
/tmp/vision-bench.py <model>
# loops over 2 test images, prints prompt_tok, output_tok, tokens/sec, full answer

## Test images
/tmp/test1.png  — synthetic Dell server card (500×280, 7.8 KB)
/tmp/test2.png  — synthetic bar chart (400×300, 4.5 KB)