Will an AI model fit on your graphics card? This calculator estimates the memory needed to run a model locally and indicates, machine by machine, if it fits and at what speed it will generate text.
How to use it
- Enter the model size in billions of parameters; for a MoE model, also add the active parameters.
- Choose the quantization, context size, KV cache type, and number of parallel requests.
- Specify the number of identical GPUs, system RAM and its bandwidth.
- Check the table: each machine is classified as ‘fits in memory’, ‘spills over to RAM’ or ‘does not fit’.
The principles behind the calculation
- Weights occupy parameters × bits per weight ÷ 8: quantization directly reduces this component.
- The KV cache grows with context and the number of simultaneous requests: 2 × layers × KV heads × head dimension × bytes per value × context × parallelism. The tool uses a typical architecture based on model size.
- A fixed overhead of 0.8 GB plus 5% of the model weights is added for buffers and runtime.
- Generation is limited by memory bandwidth: speed ≈ 60% of bandwidth ÷ active weights. If the model spills over to RAM, the calculation combines the GPU portion and the RAM portion, which is significantly slower.
Limitations to know
These are estimates for a typical architecture; models with efficient attention require less cache. The speed excludes prompt processing. Machine values are those from manufacturers.
