> ## Documentation Index
> Fetch the complete documentation index at: https://cerebrium-mintlify-9f15f516.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Using GPUs

> Configure GPU hardware in cerebrium.toml, compare available GPU models by VRAM and plan tier, and size GPUs for your model's memory needs.

GPUs accelerate computational workloads through parallel processing. Originally designed for graphics rendering, modern GPUs are essential for AI models, large-scale data processing, and other compute-intensive applications.

Cerebrium provides GPU access through configuration in the `cerebrium.toml` file, without requiring infrastructure management.

## Specifying GPUs

Configure GPUs in the `[cerebrium.hardware]` section of `cerebrium.toml`, specifying the type (`compute` parameter) and quantity (`gpu_count`). Additional deployment and scaling considerations are covered in the sections below.

## Available GPUs

The platform offers GPUs ranging from cost-effective development options to high-end enterprise hardware.

| GPU Model    | Identifier         | VRAM (GB) | Max GPUs | Plan required |
| ------------ | ------------------ | --------- | -------- | ------------- |
| NVIDIA B300  | BLACKWELL\_B300    | 262       | 8        | Enterprise    |
| NVIDIA B200  | BLACKWELL\_B200    | 180       | 8        | Enterprise    |
| NVIDIA H200  | HOPPER\_H200       | 141       | 8        | Enterprise    |
| NVIDIA H100  | HOPPER\_H100       | 80        | 8        | Enterprise    |
| RTX PRO 6000 | BLACKWELL\_RTX6000 | 96        | 8        | Standard      |
| NVIDIA A100  | AMPERE\_A100\_80GB | 80        | 8        | Standard      |
| NVIDIA A100  | AMPERE\_A100\_40GB | 40        | 8        | Standard      |
| NVIDIA L40s  | ADA\_L40           | 48        | 8        | Hobby+        |
| NVIDIA L4    | ADA\_L4            | 24        | 8        | Hobby+        |
| NVIDIA A10   | AMPERE\_A10        | 24        | 8        | Hobby+        |
| NVIDIA T4    | TURING\_T4         | 16        | 8        | Hobby+        |
| AWS Trainium | TRN1               | 32        | 8        | Hobby+        |

<Info>
  The identifier is used in the `cerebrium.toml` file. It consists of the GPU
  model generation and model name to avoid ambiguity.
</Info>

<Tip>
  GPU selection is also possible using the `--compute` and `--gpu-count` flags
  during application initialization.
</Tip>

## Choosing a GPU for Your Model

Pick a GPU based on the total VRAM your workload needs at runtime, not just the on-disk size of the model files. Total VRAM has to fit the model weights plus activations, the KV cache (for LLMs), and any intermediate buffers your framework allocates.

As a rule of thumb, size VRAM to roughly **1.2×–2× the loaded weight size**, depending on batch size, context length, and precision. For example:

| Loaded weight size | Typical VRAM to target | Example single-GPU options                                                      |
| ------------------ | ---------------------- | ------------------------------------------------------------------------------- |
| Under 8 GB         | 16–24 GB               | `TURING_T4`, `AMPERE_A10`, `ADA_L4`                                             |
| 8–20 GB            | 24–48 GB               | `AMPERE_A10`, `ADA_L4`, `ADA_L40`                                               |
| 20–40 GB           | 48–80 GB               | `ADA_L40`, `AMPERE_A100_80GB`                                                   |
| 40–70 GB           | 80–96 GB               | `AMPERE_A100_80GB`, `BLACKWELL_RTX6000`                                         |
| 70 GB and above    | 96 GB or multi-GPU     | `BLACKWELL_RTX6000`, `HOPPER_H100` (multi-GPU), `HOPPER_H200`, `BLACKWELL_B200` |

If a model does not fit on a single GPU, set `gpu_count` to shard weights across multiple GPUs of the same type. Multi-GPU inference requires framework support (for example, vLLM tensor parallelism or `device_map="auto"` in Transformers).

<Tip>
  Quantized formats such as GGUF Q4/Q5, FP8, and INT8 reduce weight size
  significantly. A Q4 quantized 7B model, for example, fits comfortably on a 24
  GB GPU. Consult the model card for its runtime VRAM requirements at the
  precision you plan to use — Cerebrium does not maintain per-model
  requirements.
</Tip>

Match RAM to VRAM as a starting point so your framework can stage weights from CPU to GPU without swapping. See [CPU and memory](/hardware/cpu-and-memory) for details.

## Multi-GPU Configuration

Multiple GPUs are configured in the `cerebrium.toml` file:

```toml theme={null}
[cerebrium.hardware]
compute = "AMPERE_A100_80GB"
gpu_count = 4        # Number of GPUs needed
cpu = 8
memory = 128.0
```

GPU availability varies by region and provider. Narrowing the provider and region constraints increases the likelihood of request queuing.
For guaranteed burst capacity, contact the [enterprise plan](mailto:sales@cerebrium.ai) team.