GPU Selection — Architecture Generation, Product Tier, and Memory Tech, Understanding Compute Matching

About This Handbook

This handbook is organized by architecture generation and product tier, not by brand model. The reasoning matches the SSD post: GPU models refresh every generation, and listing them one by one is both verbose and reads like brand endorsement — once the vendor updates, the article loses value. But GPU underlying dimensions — architecture generation (Ampere, Hopper, Blackwell), product tier (consumer / workstation / datacenter), memory tech (HBM vs GDDR), interconnect tech (NVLink vs PCIe), precision tiers (FP16/FP8/FP4) — stay stable for 2–3 years. Understand these dimensions, and you can judge any GPU generation on the market.

Three things to clarify before buying a GPU: is your workload training or inference (sets VRAM and interconnect requirements), personal research or commercial service (decides whether consumer cards are allowed), and single-card or multi-card (sets platform and power). This handbook is organized along these three dimensions.

Quick Selection Table

Start with a table to locate your need. This table is organized by scenario → technical requirement; models are just current-generation representatives. After a refresh, use the same technical requirement to find new models.

Tier notes: Consumer prices at JD/Tmall mainstream channels; workstation and datacenter cards at NVIDIA MSRP or mainstream distributor quotes. Actual procurement prices vary significantly by channel and volume.

VRAM Is the First Constraint, Not TFLOPS

This is the point most easily misled by spec sheets. NVIDIA’s product pages put FP16/FP8 compute (TFLOPS) in the most prominent position, but for training scenarios VRAM capacity is the hard gate — insufficient compute can be split across cards, batch size reduced, or gradients accumulated; insufficient VRAM means the model won’t fit at all.

Common misconception correction: TFLOPS does not equal training speed. Two cards with similar TFLOPS can differ by 2× or more in actual training iteration time, because it also depends on VRAM bandwidth, interconnect bandwidth, architecture generation (Tensor Core efficiency), and software stack optimization. Choosing cards by TFLOPS is a common pitfall — look at the actual iteration time of your target framework (PyTorch/DeepSpeed/vLLM) at your model size.

Model Size to VRAM Mapping

VRAM usage = model weights + KV cache (inference) or optimizer state (training) + activations + framework overhead. FP16 training’s optimizer state (Adam) is roughly 2× the model parameter count.

Actual usable VRAM is 1–3 GB less than advertised: CUDA context, driver, and display output (if a monitor is attached) all consume VRAM. Plan VRAM by usable value, not advertised value.

NVIDIA Architecture Generation Evolution

The first dimension of GPU selection is the architecture generation. Each generation brings jumps in IPC, efficiency, Tensor Core, and memory technology, and generation characteristics stay stable for 2–3 years.

flowchart TD
    A["Ampere<br/>2020"] --> B["Hopper<br/>2022"]
    B --> C["Blackwell<br/>2024"]
    A --> A1["A100<br/>HBM2e"]
    B --> B1["H100<br/>HBM3"]
    C --> C1["B200<br/>HBM3e"]
    style A fill:#bbdefb,stroke:#2196F3
    style B fill:#c8e6c9,stroke:#4CAF50
    style C fill:#fff3e0,stroke:#FF9800

Source: NVIDIA architecture whitepapers for each generation (Ampere GA100, Hopper GH100, Ada AD102, Blackwell GB202), docs.nvidia.com.

Why Generation Matters More Than Model

Different models within the same generation share the same compute core design, Tensor Core generation, and precision support. When buying a GPU, first pick the generation (decides what precision compute and framework features you can use), then choose a model within that generation by VRAM capacity and form factor.

Precision tiers are the core generational differentiator:

• Ampere and earlier: FP16 is the main precision
• Hopper: introduces FP8 Transformer Engine, roughly doubling training throughput over FP16 (but model and framework must support it)
• Blackwell: further introduces FP4, again roughly doubling throughput (same framework requirement)

⚠️ Precision tiers require framework support. Hopper’s FP8 and Blackwell’s FP4 are not plug-and-play — PyTorch 2.3+, Transformer Engine library, vLLM, and other frameworks must support the corresponding precision. Old code training in FP16 won’t use the H100 and B200’s FP4/FP8 compute.

Product Tier: Consumer, Workstation, Datacenter

The second GPU dimension is product tier. The same generation’s architecture is made into three product tiers, differing in memory type, interconnect, ECC, vGPU, and driver licensing — these differences are decided by product positioning and don’t vanish with model refreshes.

Hard Limits of Consumer Cards

⚠️ GeForce EULA prohibits datacenter commercial use. NVIDIA’s driver end-user license agreement explicitly prohibits GeForce deployment in “data centers,” and a 2023 update further restricted GeForce use for commercial large-model inference services. Personal research, teaching, and prototyping are unaffected; commercial inference services for clients legally require workstation or datacenter cards.

⚠️ Consumer cards removed NVLink from the 40 series. All GeForce (RTX 40/50 series) do not support NVLink. Multi-card data exchange can only go over PCIe (≤ 64 GB/s), 1/14 of NVLink 4.0 (900 GB/s). Multi-card training has substantial AllReduce communication overhead.

Workstation Card Positioning

Workstation cards target studios and enterprises: NVLink support (some models), ECC VRAM, commercial deployment allowed, power draw within workstation PSU range (250–300W). 48 GB GDDR6 ECC can run 34B FP16 on a single card — the sweet spot for single-machine training.

⚠️ Workstation card NVLink requires a bridge. The workstation card’s NVLink is not the SXM onboard interconnect; it requires a separately purchased NVLink Bridge. Without the bridge, multi-card training falls back to PCIe.

⚠️ Only the flagship retains NVLink among workstation cards. Among Ada workstation cards, only the RTX 6000 Ada retains NVLink; the 5000 and below all lost it. Workstations needing multi-card NVLink must choose the 6000 Ada or datacenter cards.

Datacenter Card Scenarios

Datacenter cards have HBM memory, onboard NVLink interconnect, and passive cooling (SXM form) or active cooling (PCIe form). They are the standard for large-scale training and multi-instance inference.

⚠️ SXM cards can’t go in workstations. The H100 SXM, B200, and A100 SXM are all passively-cooled modules that must be installed in server chassis with high-static-pressure fans (HGX baseboards). Installing them in a workstation chassis causes thermal throttling within minutes.

⚠️ Passive PCIe cards need chassis airflow confirmation. Datacenter PCIe cards (e.g., L40S, A10) are passively cooled and require server-chassis front-to-back through airflow and high-speed fans. To install in a workstation, choose the actively-cooled version.

⚠️ vGPU requires a software license. Datacenter cards support vGPU in hardware (virtualizing one card to multiple VMs), but require separately purchased NVIDIA AI Enterprise or vGPU software license. Hardware support does not mean plug-and-play.

Memory Technology: The HBM vs GDDR Bandwidth Gulf

Memory type is a key GPU performance dimension, and the gap between HBM and GDDR is an order of magnitude.

The HBM vs GDDR bandwidth gulf. The H100 SXM’s HBM3 bandwidth of 3350 GB/s is 3.5× the RTX 6000 Ada’s GDDR6 (960 GB/s). Both have 80 GB-class VRAM, but the training throughput gap comes from bandwidth. Bandwidth-sensitive training tasks (large batch, long sequences) show HBM’s advantage clearly.

⚠️ VRAM capacity ≠ VRAM bandwidth. The RTX 5090’s GDDR7 has 1792 GB/s bandwidth (close to the A100’s HBM2e), but consumer cards lack ECC and NVLink. The RTX 6000 Ada has ECC and NVLink but GDDR6 bandwidth of only 960 GB/s. “Enough VRAM” doesn’t mean “fast enough for training” — bandwidth is equally a hard constraint.

Interconnect Technology: NVLink vs PCIe

The core of multi-card training is the bandwidth of inter-card data exchange. The gap between NVLink and PCIe is an order of magnitude.

⚠️ Consumer multi-GPU goes over PCIe. From the RTX 40 series on, consumer cards have no NVLink; dual-card training can only use ×8+×8 PCIe, with substantial AllReduce overhead. Workflows needing NVLink multi-card training must use workstation or datacenter cards.

⚠️ Workstation card NVLink needs a bridge. Workstation card NVLink connects via a bridge, not the SXM baseboard integration. Dual-card configs require a separately purchased NVLink Bridge.

Source: NVIDIA NVLink and NVSwitch technology page (nvidia.com/nvlink), NVLink generation specs whitepapers.

Pitfalls You Must Know

⚠️ GeForce EULA prohibits datacenter commercial use. Commercial inference services must use workstation or datacenter cards.

⚠️ Consumer cards have no NVLink. RTX 40/50 series all lack it; multi-card training over PCIe has high overhead.

⚠️ The HBM vs GDDR bandwidth gulf. Training throughput gaps come from bandwidth, not just VRAM capacity.

⚠️ SXM cards can’t go in workstations. Passive-cooling modules require server chassis.

⚠️ Passive PCIe cards need chassis airflow confirmation. L40S, A10, and similar passive cards require server-chassis airflow.

⚠️ vGPU requires a software license. Hardware support doesn’t mean plug-and-play.

⚠️ Transient power trips OCP. High-TDP GPUs (e.g., RTX 5090’s 575W, H100’s 700W) can draw 1.4–1.6× rated TDP in transient current when transitioning from idle to full load; cheap PSU OCP will falsely trip and shut down.

⚠️ Secondhand mining card risk. RTX 4090 and similar were mining mainstays; the secondhand market is flooded with mining cards. Identification: GPU-Z BIOS version, FurMark 30-min stress test temperature curve, 3DMark scores vs average. Production environments should avoid secondhand cards.

⚠️ Secondhand datacenter card sourcing risk. “Pulled H100” cards are mostly retired from mining farms or defunct cloud providers, possibly with flashed unofficial vBIOS and expired warranties. Production procurement should go through authorized channels.

⚠️ The 12V-2×6 connector must use the right cable. RTX 50 series use 12V-2×6 (an improved 12VHPWR); legacy 8pin adapters have limited power capacity and will overheat and melt if forced.

⚠️ Precision tiers require framework support. Hopper’s FP8 and Blackwell’s FP4 require PyTorch 2.3+, Transformer Engine, and other framework support to be used.

Common misconception correction: VRAM capacity ≠ usable VRAM. An H100 advertised at 80 GB has about 76–78 GB actually usable after CUDA context + driver + display output. Plan VRAM by usable value.

Engineering for Multi-GPU Training Platforms

The endpoint of GPU selection isn’t “which card to buy” but “multiple cards in a chassis running stably.”

Power Budget Calculation

System peak power = GPU power × N + CPU power + motherboard/memory/storage/fans + 20% headroom.

flowchart TD
    A["Power budget"] --> B["GPU × N"]
    A --> C["CPU 250–350W"]
    A --> D["Board+RAM+NVMe 80W"]
    A --> E["Fans+AIO 50W"]
    A --> F["+ 20% headroom"]
    style A fill:#e3f2fd,stroke:#2196F3
    style B,C,D,E,F fill:#e8f5e9,stroke:#4CAF50

Power tiers for common configs:

PCIe Lane Configuration

Multi-card training requires each card to run full PCIe ×16. The CPU’s total PCIe lanes determine how many cards you can hang:

Dual-card and above training requires workstation or server platforms. See the CPU selection post.

Cooling Airflow

GPUs under load exceed 80°C; insufficient multi-card spacing causes the upper card to ingest the lower card’s hot exhaust, hitting thermal throttling. Key points:

• Workstation chassis: choose models with 2+ slots of spacing between GPUs
• Actively-cooled cards can be installed compactly; passively-cooled cards require server chassis
• Consistent airflow direction, avoiding turbulence
• Chassis fan speed linked to GPU temperature

Acceptance Testing

After building a new machine, run these acceptance tests before going live:

Selection Comparison Summary

References

1. NVIDIA Ampere architecture whitepaper (GA100): https://docs.nvidia.com/cuda/ampere-tuning-guide/
2. NVIDIA Hopper architecture whitepaper (GH100): https://docs.nvidia.com/cuda/hopper-tuning-guide/
3. NVIDIA Ada Lovelace architecture whitepaper: https://www.nvidia.com/en-us/geforce/ada-architecture/
4. NVIDIA Blackwell architecture whitepaper: https://www.nvidia.com/blackwell-architecture/
5. NVIDIA Tensor Core technology page: https://www.nvidia.com/data-center/tensor-cores/
6. NVIDIA Transformer Engine documentation: https://docs.nvidia.com/deeplearning/transformer-engine/
7. NVIDIA NVLink and NVSwitch technology page: https://www.nvidia.com/nvlink/
8. NVIDIA HBM memory technology notes (within each architecture whitepaper): https://docs.nvidia.com/
9. NVIDIA driver EULA (GeForce datacenter restrictions): https://www.nvidia.com/drivers/license/
10. NVIDIA vGPU technology page: https://www.nvidia.com/data-center/vgpu/
11. NVIDIA GeForce product family (consumer card spec reference): https://www.nvidia.com/geforce/
12. NVIDIA RTX workstation product family: https://www.nvidia.com/rtx/
13. NVIDIA Data Center product family: https://www.nvidia.com/data-center/
14. MLPerf Training Benchmark: https://mlcommons.org/benchmarks/training/
15. TechPowerUp GPU database (measured data): https://www.techpowerup.com/gpu-specs/
16. Tom’s Hardware GPU reviews: https://www.tomshardware.com/gpus/
17. ServeTheHome datacenter GPU reviews: https://www.servethehome.com/
18. NVIDIA HGX and DGX system specs: https://www.nvidia.com/data-center/hgx/