Does this calculator send my model details anywhere?
No. The VRAM math runs locally in your browser. Saved scenarios use namespaced browser storage through the Axle adapter, and no prompt text or model configuration is posted to a server.
Weights come from parameters and precision
Every model starts with its weight footprint: parameter count multiplied by bytes per parameter. An 8B model needs roughly 16 GB at FP16, 8 GB at INT8, and 4 GB at INT4 before any runtime reserve. The calculator applies the selected quantization to the full parameter count, including every mixture-of-experts expert that must stay resident.
KV cache grows with context and batch
The KV cache stores one key and one value vector per layer, per KV head, per token. Long context and larger batches multiply that cost quickly. A model with grouped-query attention keeps the cache smaller than a full multi-head model, which is why two 8B models can need very different memory at the same 32k context length.
Text encoders are part of the pipeline
Diffusion and multimodal pipelines often load a separate CLIP, T5, or LLM text encoder. CLIP-L is small, while T5-XXL or a 7B LLM encoder can add several gigabytes. When the encoder is unloaded after prompt encoding, peak memory becomes the larger of the resident model and the encoder phase instead of the sum of both.
Activations and runtime overhead
Activations scale with batch size, context length, and hidden size. Allocator reserves, CUDA context, fragmentation, and framework buffers add another layer. The default twelve percent overhead is a practical planning buffer for inference; raise it for training stacks, graph capture, or workloads with many concurrent requests.
Training and adapter tuning change the budget
Full fine-tuning needs FP32 master weights, gradients, and Adam optimizer moments, so it can require several times the inference footprint. LoRA or QLoRA tuning keeps most weights frozen and adds a much smaller adapter state. Mention training, fine-tuning, LoRA, or a GB budget in the primary input and the deterministic estimate adapts.
Read the verdict before you download
A comfortable result leaves at least fifteen percent of GPU memory free. A tight fit is below the hardware limit but leaves little room for another process or a longer prompt. An over-budget result needs a smaller quantization, a shorter context, fewer resident encoders, or more GPUs. Save scenarios to compare those trade-offs side by side.
Related tools and pages
- LLM Token Cost Calculator - Estimate API token spend for the same model workload.
- Token Usage Optimizer - Reduce prompt and context tokens before they reach your GPU.
- MCP Server Pricing Calculator - Plan the operating cost of an MCP server deployment.
- LLM Learning Planner - Build a structured study plan for local model work.
- LLM Content Planner - Turn a content brief into a complete LLM content plan.
- Cosmo Wise Tools - Browse more free browser-based tools.
- Privacy
- Contact
Frequently Asked Questions
Why is my KV cache larger than expected?
KV cache scales with layers, KV heads, head dimension, context length, batch size, and cache precision. Reducing context or batch has the largest effect. Switching the cache from FP16 to INT8 halves it, and grouped-query attention models already use fewer KV heads.
Should I unload the text encoder after prompt encoding?
Yes for most diffusion pipelines. If the encoder is only needed to turn a prompt into embeddings, unloading it frees its weights before the denoising or generation phase. Keep it resident when the pipeline calls the encoder repeatedly during one run.
How accurate is the overhead percentage?
It is a planning buffer, not a measurement. Twelve percent covers common allocator, CUDA context, and fragmentation costs for inference. Raise it for training, graph capture, multi-process serving, or frameworks that keep extra copies of weights.
Does the GPU count multiply memory?
The calculator treats the selected GPUs as one pooled budget, which is the optimistic case for tensor-parallel or pipeline-parallel serving. Real multi-GPU setups add communication buffers and may replicate some weights, so leave extra headroom for production deployments.
Can I compare fine-tuning with inference?
Yes. Run the inference estimate first, save the scenario, then mention full fine-tuning or LoRA in the primary input. Full fine-tuning adds FP32 master weights, gradients, and optimizer moments; LoRA adds a much smaller adapter state.