Matching the GPU to the Job
Both GPUs remain in wide production use, and with Blackwell-generation hardware now absorbing most of the demand for cutting-edge training, Hopper and Ampere have settled into clearly defined roles. The A100 is a mature, heavily discounted workhorse for fine-tuning and inference. The H100 is the practical default for large-scale training and FP8-optimized inference.
01 Architecture at a Glance
The A100 is built on NVIDIA's Ampere architecture (GA100 die), released in 2020. It introduced third-generation Tensor Cores, Multi-Instance GPU (MIG) partitioning, and structured sparsity support — features that made it the standard data center GPU for several years and still make it a dependable choice today.
The H100 is built on the newer Hopper architecture (GH100 die), released in 2022. It carries forward MIG and sparsity but adds fourth-generation Tensor Cores, native FP8 precision, and the Transformer Engine. The Transformer Engine is purpose-built to accelerate transformer-based models like LLMs by automatically mixing FP8 and FP16 precision during training.
That Transformer Engine is the single biggest practical difference between the two cards. If your workload is transformer-heavy — which most modern LLM and diffusion workloads are — it's where the H100's advantage actually comes from, not just a generational clock speed bump.
02 Specifications Side by Side
| Specification | A100 (80GB, SXM) | H100 (80GB, SXM) |
|---|---|---|
| Architecture | Ampere | Hopper |
| CUDA Cores | 6,912 | 16,896 |
| Tensor Cores | 432 (3rd gen) | 528 (4th gen) |
| Memory | 80GB HBM2e | 80GB HBM3 |
| Memory Bandwidth | ~2.0 TB/s | 3.35 TB/s |
| FP32 (dense) | 19.5 TFLOPS | 67 TFLOPS |
| TF32 (dense) | 156 TFLOPS | 989 TFLOPS |
| FP16/BF16 (dense) | 312 TFLOPS | 989 TFLOPS |
| FP8 (dense) | Not supported | 1,979 TFLOPS |
| NVLink Bandwidth | 600 GB/s | 900 GB/s |
| Max TDP | Up to 400W (SXM) | Up to 700W (SXM) |
A note on these performance specs: NVIDIA's published TFLOPS figures are typically listed "with sparsity," which effectively doubles the number on paper. The dense figures above reflect what most production workloads will actually see, since structured 2:4 sparsity requires specific model preparation most teams don't do. PyTorch also automatically routes FP32 matrix multiplications through the Tensor Cores using the TensorFloat-32 (TF32) format, making the TF32 throughput the real-world "FP32" speed for deep learning tasks in practice.
03 Where the A100 Still Makes Sense
The A100 isn't obsolete — it's a mature, well-supported card, and with Blackwell now soaking up demand for frontier-scale training, A100 rental and spot pricing has only gotten more favorable. It's a solid fit for:
04 Where the H100 Pulls Ahead
The H100 earns its higher price in workloads that specifically exercise Hopper's architectural additions:
05 The FP8 Software Tax & Form Factor Traps
Historically, unlocking the H100's FP8 speeds required integrating NVIDIA's Transformer Engine or Megatron-LM directly. Modern PyTorch now natively supports FP8 data types, and libraries like torchao provide out-of-the-box workflows. However, if your codebase is older, highly custom, or relies on niche frameworks without native FP8 support, the H100's primary advantage stays locked — and you're effectively overpaying for compute you can't use.
The H100 PCIe memory trap: While the H100 SXM delivers 3.35 TB/s of bandwidth, the H100 PCIe falls back to older HBM2e memory, capping it at around 2.0 TB/s — the same memory bandwidth as an A100. Renting a PCIe H100 for memory-bound inference means paying a Hopper premium for Ampere-level memory throughput.
Node-to-node networking: If you're renting a multi-node cluster, confirm your provider offers InfiniBand or high-speed RoCE. Renting H100s with standard networking will strangle the GPUs during distributed training.
06 Cost-to-Performance & The Blackwell Factor
Hourly rental prices for an H100 are noticeably higher than an A100, but comparing hourly rates alone misses the point — what matters is cost per completed job. A training run that finishes several times faster on FP8 with the H100 can end up cheaper in total dollars spent. Conversely, a small fine-tuning job won't benefit proportionally, costing more overall.
NVIDIA's Blackwell generation (B100/B200/B300) is now the tier absorbing demand for the largest training runs. That shift has been good news for Hopper and Ampere renters: H100s are no longer hard to find, and A100s are widely available at discounted rates. Both cards are now mature, accessible choices rather than the bleeding edge.
Quick Decision Guide
- ✅ Training a large model (30B+ params) or need FP8: Choose H100
- ✅ Standard inference or fine-tuning under 30B params: A100 is highly cost-effective
- ✅ Inference for large 70B–100B models (quantized 4-bit/8-bit): Choose A100
- ✅ Multi-GPU training cluster: Choose H100 SXM (requires InfiniBand/RoCE networking)
- ✅ Latency-sensitive production inference at scale: Choose H100 SXM
- ✅ Frontier-scale pretraining: Consider Blackwell-generation hardware instead
Frequently Asked Questions
Ready to Deploy?
Now that you know exactly which architecture fits your workload and budget, it's time to provision your compute. Whether you need the cost-effective reliability of an A100 for fine-tuning or the FP8-optimized raw power of an H100 for large-scale training, Fit Servers has you covered.
