›Unified memory — CPU and GPU share the same 128 GB LPDDR5x pool at 273 GB/s. No PCIe penalty. Model weights, activations, and KV cache all live in the same physical memory.
›Native Blackwell FP8 tensor cores — hardware executes FP8 matrix operations without dequantisation overhead. Roughly half the memory of BF16 with <1% quality loss on most benchmarks.
⚠Qwen3 235B-A22B needs 2 nodes — full FP8 weight size is ~236 GB, exceeding 128 GB. Requires 2× GB10 with --tensor-parallel-size 2 (256 GB combined). Single-node sweet spot is Qwen3 72B FP8 (~72 GB).
›Four independent nodes in lab — each GB10 node runs a different model independently, or all four are pooled behind LiteLLM for high-throughput load-balanced serving.
Deployment Notes
⚠Must use NVIDIA NGC official vLLM image — GB10 is SM 12.1, unique to DGX Spark. Standard pip install vllm hits PTX compilation errors. NGC container resolves this automatically.
✓Set --gpu-memory-utilization 0.85 to reserve memory headroom for OS and vLLM runtime buffers.
⚠For 70B+ FP8 tight-fit models, add --max-model-len 32768 to prevent KV cache from exhausting remaining memory at long context.
✓Source .safetensors FP8 checkpoints from Hugging Face — search neuralmagic or nvidia namespaces for pre-quantised FP8 variants of popular models.
Recommended Role in Lab
›Primary inference node — largest single-node model is Qwen3 72B FP8 (~72 GB). For Qwen3 235B (~236 GB FP8), pair 2× GB10 nodes with --tensor-parallel-size 2. Best reasoning model: DeepSeek R1 Distill 70B.
›Multi-user API server — vLLM continuous batching handles concurrent users efficiently. Front all four GB10 nodes with LiteLLM for transparent horizontal scaling.
›POC demo backend — Latitude laptops connect to the GB10 vLLM API over LAN. Clients see the demo on a thin portable device; GB10 does all the compute.
›Full-precision evaluation — 128 GB headroom allows loading BF16 models for benchmark-accurate quality measurements before deciding on FP8 deployment.
Filter:
#
Model
Type
Quality Tier
Custom Score
FP8 / BF16 Size
GB10 Fit
Capabilities & Use Case
GPU Tier · vLLM + FP8
Dell Precision 5860 Tower
Intel Xeon W Sapphire Rapids · Dual NVIDIA RTX 4000 Ada · 40 GB combined VRAM · Ada Lovelace FP8 tensor cores · Tensor Parallel ×2
40 GB
Combined VRAM
360 GB/s
Per card BW
TP=2
Tensor parallel
FP8
Ada native
130W
Per card TDP
Dual GPU Setup
›Tensor parallelism via vLLM — weight matrices are split across both RTX 4000 Ada cards, creating a single 40 GB logical device. Launch with --tensor-parallel-size 2.
›Native FP8 on Ada Lovelace — both cards have 4th-gen Tensor Cores with FP8 support (E4M3 and E5M2 formats). Same precision path as GB10, different GPU generation.
›Single-card ceiling is ~18 GB — without TP=2 you are limited to 14B models. Dual GPU is what unlocks Qwen3 32B FP8 (~32 GB) as the workstation sweet spot.
›20 GB GDDR6 ECC per card at 360 GB/s — 6,144 CUDA cores, 192 Tensor Cores, 130W TDP each. Connected via PCIe Gen4 x16 through the Xeon W platform.
Critical Constraints
✗Hard VRAM ceiling at ~36 GB FP8 — beyond 36 GB there is no room for KV cache. Any model over 40 GB FP8 does not fit at all. Route those to GB10.
✗No Ollama + vLLM simultaneously — vLLM claims exclusive GPU access at launch. If Ollama is also running, it falls back to CPU-only inference (~5 tok/s).
⚠PCIe RAM spill kills performance — if a model overflows VRAM into system RAM, throughput collapses. PCIe bandwidth (64 GB/s) is ~5x slower than GDDR6 (360 GB/s).
⚠Tight-fit Qwen3 32B: add --max-model-len 8192 to ensure KV cache fits within the 6–8 GB of VRAM remaining after model load.
Recommended Role in Lab
›Workstation inference node — runs Qwen3 32B FP8 for a small concurrent user group at B-tier quality without consuming a GB10 slot.
›Developer sandbox — test prompts, run eval scripts, and iterate on fine-tuning experiments on real GPU hardware at workstation form factor and cost.
›Vision AI workloads — Gemma 4 26B-A4B (vision + tool calling, 256K context) runs well here. Ideal for document processing or visual inspection POCs.
›Complementary to GB10 — run a different model than the GB10 nodes simultaneously, enabling side-by-side capability comparisons during client demonstrations.
Filter:
#
Model
Type
Quality Tier
Custom Score
FP8 / BF16 Size
5860 Fit
Capabilities & Use Case
NPU Tier · llama.cpp + GGUF
Dell Latitude 7455
Qualcomm Snapdragon X Elite X1E-80-100 · 12 cores up to 4.0 GHz · 32 GB LPDDR5x unified · Hexagon NPU 45 TOPS · Adreno GPU · Copilot+ PC
32 GB
Unified RAM
45 TOPS
Hexagon NPU
12 cores
X Elite CPU
GGUF
Model format
ARM
Architecture
Hardware Profile
›Snapdragon X Elite X1E-80-100 — 12-core ARM CPU, dual-core boost to 4.0 GHz. Faster sustained inference throughput than the X Plus in the Latitude 5455.
›32 GB unified LPDDR5x — CPU and Hexagon NPU share one physical memory pool. No discrete VRAM concept. Full 32 GB available for model weights, OS, and KV cache.
›Hexagon NPU at 45 TOPS — accelerates ONNX/QNN-optimised models. Adreno GPU offload also available in llama.cpp via --ngl 99 flag (experimental path).
›Copilot+ PC, 20+ hour battery, 16.9 mm thin — Dell's thinnest Latitude. Supports Windows 11 ARM AI features natively including real-time translation and Recall.
Model Strategy
✓Sweet spot: Qwen3 14B Q8 (~15 GB) — fits with 17 GB headroom. Near full-precision quality at 8-bit. Best balance of quality and comfort on 32 GB RAM.
✓Qwen3 32B Q4 (~18 GB) — B-tier quality on a laptop. 14 GB headroom. Set context to 4096 tokens max. Best-quality model this machine can run reliably.
⚠Llama 3.3 70B Q4 (~40 GB) — technically fits but near-zero headroom for KV cache and OS. Only suitable for short single-turn queries, not multi-turn chat.
›Use Ollama ARM build for easiest setup. For NPU and Adreno GPU layer offload use llama.cpp direct with --ngl 99 and Adreno backend.
Recommended Role in Lab
›Premium field demo device — show clients fully offline on-device AI on a slim business laptop. No Wi-Fi, no cloud, no dependency on lab connectivity.
›Thin client to GB10 — point the app at the GB10 vLLM API over LAN for heavy workloads. The 7455 handles UI; GB10 runs the model transparently in the background.
›Standalone travel AI — run a 14B multilingual model locally for document summarisation, translation, or coding assist during client visits without network access.
›ARM compatibility testing — validate that application stacks run correctly on ARM Windows before deploying to Snapdragon-based client device environments.
Filter:
#
Model
Type
Quality Tier
Custom Score
GGUF Q4 / Q8 Size
7455 Fit
Capabilities & Use Case
NPU Tier · llama.cpp + GGUF
Dell Latitude 5455
Qualcomm Snapdragon X Plus X1P-64-100 · 10 cores up to 3.4 GHz · 16 GB LPDDR5x unified · Hexagon NPU 45 TOPS · Adreno GPU · Copilot+ PC · 22h battery
16 GB
Unified RAM
45 TOPS
Hexagon NPU
10 cores
X Plus CPU
22 hrs
Battery life
GGUF
Model format
Hardware Profile
›Snapdragon X Plus X1P-64-100 — 10-core ARM CPU up to 3.4 GHz. Efficiency-focused design. Prioritises battery endurance over peak-performance throughput.
›16 GB unified LPDDR5x — tight ceiling for LLM inference. OS and background apps consume ~3–4 GB, leaving roughly 12 GB usable for model weights and context window.
›Hexagon NPU at 45 TOPS — same NPU TOPS as the Latitude 7455. Copilot+ certified. Supports Windows 11 ARM AI features including real-time translation natively.
›22-hour battery life — the standout differentiator of this machine. Full-day field use without a charger or power bank. Lightest Copilot+ Latitude in the fleet.
Model Strategy
✓Sweet spot: Qwen3 8B Q4 (~4.5 GB) — fits cleanly with ~7 GB headroom. Fast on ARM, supports 29+ languages. Best all-around pick for this 16 GB machine.
✓Llama 3.1 8B Q4 (~4.5 GB) — same footprint as Qwen3 8B. Largest community fine-tune ecosystem. Good for Modelfile-customised task-specific deployments.
⚠14B models are tight — Q4 (~8 GB) leaves only ~6 GB for OS and KV cache. Limit context to 2048 tokens. Avoid long or complex multi-turn conversations.
✗32B+ models do not fit — Q4 exceeds available RAM entirely. Route those workloads to the Latitude 7455, Pro Micro 5, or GB10.
Recommended Role in Lab
›Primary field demo device — ultra-portable, 22-hour battery. Demonstrate on-device AI to clients at any location with no reliance on Wi-Fi or the lab network.
›Thin client to GB10 — for heavier workloads, connect to GB10 vLLM API over LAN. The 5455 handles the client-facing UI; GB10 runs the model.
›Copilot+ showcase — demonstrate Windows 11 ARM AI features (real-time translation, Cocreator) powered by the on-chip Hexagon NPU to clients evaluating AI PC adoption.
›Offline / disconnected AI — run small models at client sites with no internet. Factory floor, warehouse inspection, remote field work — no cloud dependency.
Filter:
#
Model
Type
Quality Tier
Custom Score
GGUF Q4 / Q8 Size
5455 Fit
Capabilities & Use Case
NPU Tier · llama.cpp + GGUF
Dell Pro Micro 5
Intel Core Ultra 7 366H vPro · Panther Lake · 16 cores (4P+8E+4LP) up to 4.8 GHz · 64 GB DDR5 · Intel NPU5 50 TOPS · Xe3 iGPU · 1.2L form factor
64 GB
DDR5 RAM
50 TOPS
NPU5
16 cores
Panther Lake
Intel 18A
Process node
1.2L
Form factor
Hardware Profile
›Panther Lake Core Ultra 7 366H vPro — Intel 18A process node. 16-core hybrid: 4 Cougar Cove P-cores (4.8 GHz) + 8 Darkmont E-cores + 4 LP E-cores. Intel's newest architecture as of 2026.
›64 GB DDR5 SO-DIMM — largest RAM pool in the NPU-tier group. Socketed and upgradeable. Supports DDR5-7200 MT/s. Enough to run 70B GGUF Q4 models entirely in memory.
›Intel NPU5 at 50 TOPS — highest NPU TOPS in the entire fleet. Accelerates ONNX models via Intel OpenVINO. Exceeds the 40 TOPS Copilot+ threshold by 25%.
›1.2-litre form factor — 182 × 36 × 178 mm — wall-mountable, desk-mountable, cabinet-integrated. Runs 24/7 quietly as a permanent inference node.
Model Strategy
✓Sweet spot: Qwen3 32B Q4 (~18 GB) — fits with 46 GB headroom. Best quality available on CPU inference. Interactive speeds (~8–12 tok/s) at Q4_K_M quantisation.
✓Llama 3.3 70B Q4 (~40 GB) — A-tier quality on a 1.2L mini PC. 24 GB headroom. ~3–6 tok/s on 16 CPU cores — sufficient for non-interactive or batch processing.
✓DeepSeek R1 70B Q4 distilled (~40 GB) — best reasoning model on the NPU tier. Same footprint as 70B dense. Strong for math, analysis, and step-by-step tasks.
›Use Ollama for easiest setup. For NPU5 acceleration use the Intel OpenVINO backend in llama.cpp or the IPEX-LLM framework for Intel-optimised inference.
Recommended Role in Lab
›Always-on edge inference node — runs 24/7 at low power, serves a small team via Ollama REST API on the local network without consuming a GB10 or 5860 resource.
›Dedicated reasoning endpoint — run DeepSeek R1 70B permanently for math, analysis, and chain-of-thought tasks that need deep reasoning available at any hour.
›Client leave-behind or POC unit — compact enough to ship or permanently install at a client site. Demonstrates A-tier AI on hardware smaller than a lunchbox, no cloud needed.
›Intel AI platform showcase — demonstrate NPU5 + OpenVINO stack for clients evaluating Intel-based edge AI infrastructure as a production alternative to cloud APIs.
Filter:
#
Model
Type
Quality Tier
Custom Score
GGUF Q4 / Q8 Size
Micro5 Fit
Capabilities & Use Case
Custom Score = Size 25% + Use Case 25% + Headroom 25% + Quality Tier 25% · Thresholds recalibrated per machine hardware · Ties broken by quality tier · May 2026