Large language models are no longer defined simply by how many parameters they contain. Two models can have very different hardware requirements even when their parameter counts look similar, because the model architecture determines how much computation is performed for each token and how the model's weights are distributed across GPUs.
Two important approaches are dense models and Mixture-of-Experts (MoE) models.
A dense model uses essentially the same set of model parameters for every input token. An MoE model contains multiple expert networks but uses a routing mechanism to activate only a subset of those experts for each token. That difference has major consequences for GPU infrastructure.
For organizations running AI inference or training on dedicated GPU servers, the important questions are not simply "How many parameters does the model have?" but also:
- How many parameters must be stored in GPU memory?
- How many parameters are active for each token?
- How much computation is required?
- How many GPUs are needed?
- Does the workload require fast GPU-to-GPU communication?
- How does batching affect utilization?
- Is the model being trained or only served for inference?
Understanding dense and MoE architectures makes it easier to match an AI workload with an appropriate GPU server configuration.
01 What Is a Dense AI Model?
A dense model uses its parameterized layers for every input token.
Consider a simplified model containing 70 billion parameters. In a dense architecture, the model's major computational layers operate using that complete set of parameters for each token.
Conceptually:
There is no expert-selection stage deciding which part of the feed-forward network should process a particular token.
This makes dense models relatively straightforward to map onto GPUs. If the model fits on one GPU, inference can potentially be performed there. If it does not, the model can be distributed across multiple GPUs using techniques such as tensor or pipeline parallelism.
The important point is that a dense model's parameter count is closely tied to the amount of model computation performed per token, although the exact compute requirement also depends on the architecture, sequence length, attention implementation, precision, and other factors.
02 What Is a Mixture-of-Experts Model?
A Mixture-of-Experts model takes a different approach.
Instead of having one feed-forward network process every token, an MoE architecture contains multiple expert networks. A learned router determines which experts should process each token.
A simplified representation looks like this:
The router does not normally activate every expert for every token. Instead, it selects a limited number of experts according to the model's routing strategy.
Modern MoE implementations commonly use top-k routing, where k represents the number of selected experts for a token. The exact routing design varies between architectures.
This creates an important distinction: Total parameters ≠ parameters activated for every token.
03 Total Parameters vs. Active Parameters
This is one of the most important concepts when comparing MoE and dense models.
Suppose an MoE model has:
- 100 billion total parameters
- 10 billion parameters active for a particular token
It does not mean the GPU only needs to store 10 billion parameters.
If the complete model is resident in GPU memory, the model's weights can still require memory corresponding to the full set of parameters, including the inactive experts.
The advantage is that only the selected experts participate in the computation for that token. This is why an MoE model can have a very large total parameter count while requiring substantially less computation per token than a dense model with the same total parameter count. Hugging Face's explanation of MoE architectures makes the same distinction between total stored parameters and the smaller subset involved in each token's computation.
04 A Real Example: DeepSeek-V3
DeepSeek-V3 provides a useful real-world example of this distinction. Its technical report describes the model as having 671 billion total parameters with 37 billion activated for each token.
Those two numbers describe different things. The 671B figure describes the model's overall parameter scale. The 37B figure describes the amount of parameters activated for each token under its reported architecture.
Therefore, describing DeepSeek-V3 simply as a "37B model" would be misleading. Likewise, treating it as equivalent to a dense 671B model would also misrepresent its computational characteristics. This distinction is particularly important when planning GPU infrastructure.
05 Dense vs. MoE: The Fundamental Difference
The simplest comparison is:
| Characteristic | Dense Model | MoE Model |
|---|---|---|
| Parameters | All major model parameters participate | Only selected experts participate per token |
| Total parameter storage | Based on complete model | Usually based on complete model when experts are resident |
| Compute per token | Uses the dense network | Uses shared components plus selected experts |
| Routing | No expert router | Router selects experts |
| GPU memory | Strongly affected by model size | Strongly affected by total resident model size |
| GPU communication | Depends on parallelism | Can be particularly important because of expert routing |
| Scaling | Relatively straightforward | More complex at multi-GPU scale |
| Training | Dense computation pattern | Requires expert routing and load balancing |
This is why active parameter count should never be used as the only number when sizing GPU memory.
06 What Changes for GPU Memory?
GPU memory is often the first constraint encountered when deploying large AI models.
A simplified weight-memory calculation is: Weight memory ≈ Number of parameters × bytes per parameter
For example, ignoring metadata, temporary buffers, and other memory requirements:
100B parameters × 2 bytes ≈ 200 GB
This represents roughly the weight storage requirement when using a 16-bit representation such as FP16 or BF16. However, this is only a rough calculation.
Actual GPU memory requirements can also include: KV cache, activations, temporary computation buffers, framework overhead, CUDA/runtime allocations, quantization metadata, communication buffers, and optimizer states during training. Training requires substantially more memory than inference because optimizer states, gradients, and training activations can become major contributors.
Why MoE Does Not Automatically Mean Less VRAM
This is one of the most common misconceptions about MoE models. Suppose an MoE model has 500B total parameters but only a fraction of those parameters are activated for each token. It would be incorrect to conclude: "Only the active parameters need to be loaded, so the GPU only needs memory for the active parameters."
That is not generally how a standard fully resident MoE deployment works. If the model's experts are distributed across GPUs, the complete set of model weights may still need to be available across the GPU cluster. The compute savings come from not executing every expert for every token, rather than automatically eliminating the memory requirement for the inactive experts. Hugging Face's MoE documentation explicitly highlights this difference: only some experts are activated for inference, while the complete set of expert parameters can still need to be loaded into memory.
07 GPU Count Can Become a Different Problem
With a dense model, distributing the model across multiple GPUs often revolves around splitting the model's layers or tensor operations. MoE introduces another dimension: expert parallelism.
With expert parallelism, different experts can be placed on different GPUs. For example:
When a token is routed to an expert located on another GPU, the token's data needs to be communicated to that GPU. The selected expert performs its computation, and the result is then communicated back as required by the implementation.
This can introduce significant communication requirements. Hugging Face's current documentation describes expert parallelism as placing expert feed-forward layers on different accelerators, with the router dispatching tokens to the appropriate devices and gathering their results.
08 GPU Interconnect Matters More for Large MoE Deployments
This is where GPU server architecture becomes particularly important. Suppose the GPUs are inside one server:
Fast GPU-to-GPU communication can reduce the cost of moving data between devices. By contrast, distributing experts across separate machines introduces communication over the network, which is significantly slower.
For MoE workloads, the routing pattern can therefore make network and GPU interconnect performance an important part of the system design. This is why a GPU server with multiple accelerators is not simply a collection of independent GPUs. The topology and communication path between those GPUs can affect real-world performance.
09 What Is Expert Parallelism?
Expert parallelism distributes different experts across accelerators. A router determines where tokens should go. The system then dispatches tokens to the appropriate experts.
This can allow a very large MoE model to be distributed across multiple GPUs without requiring every GPU to hold every expert. Modern frameworks support dedicated mechanisms for this. Hugging Face's Transformers documentation, for example, describes expert parallelism in which expert weights are distributed across devices and tokens are routed to the devices containing the selected experts.
10 Why Inter-GPU Communication Can Become a Bottleneck
MoE models can create an all-to-all communication pattern. Imagine eight GPUs, each holding different experts. A batch of tokens arrives:
- GPU 1 → Experts on GPU 3
- GPU 2 → Experts on GPU 7
- GPU 3 → Experts on GPU 1
- GPU 4 → Experts on GPU 5
The exact pattern depends on the architecture and parallelization strategy, but the fundamental problem is the same: Tokens may need to move to the GPUs where their selected experts reside.
If computation is extremely fast but communication is slow, the GPUs can spend more time waiting for data. This is why MoE performance cannot be predicted simply by looking at the theoretical FLOPS of the GPUs. Memory bandwidth, GPU interconnects, communication libraries, batch size, routing behavior, and software implementation all matter.
11 Dense Models Can Also Require Multiple GPUs
It would be wrong to assume that only MoE models need multi-GPU systems. Large dense models can exceed the memory of a single GPU.
The model can be distributed using approaches such as tensor parallelism, pipeline parallelism, data parallelism, or other distributed strategies. The exact approach depends on the model, framework, batch size, and workload. The difference is that MoE adds another useful dimension: expert parallelism.
12 Inference: Dense vs. MoE
For inference, the distinction between total and active parameters becomes particularly important. A dense model processes its main parameterized computation for every token. An MoE model can activate only a subset of experts.
Therefore, comparing a Dense 100B against an MoE 100B total / 20B active as if both perform exactly the same amount of computation would be misleading. The MoE model has a 100B-parameter total model, but only part of its expert computation is activated for each token.
However, that does not mean the MoE model will always be faster. Real inference performance depends on active experts, sequence length, GPU architecture, precision, quantization, expert placement, GPU interconnect, kernel implementation, batch size, and routing efficiency.
Modern MoE implementations therefore rely heavily on optimized expert kernels and communication strategies. Current Hugging Face documentation, for example, lists specialized grouped-GEMM and DeepGEMM backends for different GPU architectures and MoE workloads.
13 Batch Size Changes the Picture
GPU workloads generally benefit from enough parallel work to keep the hardware busy. MoE introduces an additional complication.
Suppose a batch contains many tokens and the router sends most of them to one expert. One GPU may become heavily loaded while others have less work. This is known as an expert-load imbalance problem.
MoE systems therefore use routing and load-balancing techniques to prevent excessive concentration of tokens on particular experts. The underlying routing and capacity problem has been studied extensively in MoE research.
14 Training Is Different From Inference
GPU requirements become substantially more demanding when training or fine-tuning an MoE model. During training, the system needs to handle model weights, gradients, optimizer states, activations, routing, expert computation, and inter-GPU communication.
Large distributed MoE training therefore requires careful planning of parallelism and memory. DeepSeek-V3 is a useful example of the scale involved. Its technical report describes a 671B-total-parameter MoE architecture with 37B parameters activated per token and reports its training using 2.788 million H800 GPU hours. That figure should not be interpreted as a universal hardware requirement for training every 671B MoE model, but it highlights the scale.
15 GPU Memory Is Not the Only Specification
When choosing a GPU server for AI inference, it is tempting to focus only on VRAM. For large MoE workloads, that can be a mistake. A practical evaluation should consider at least:
- GPU memory capacity: Can the model weights and runtime data fit?
- GPU memory bandwidth: How quickly can the GPU move data?
- GPU compute capability: Does the accelerator provide the required performance?
- GPU-to-GPU interconnect: How quickly can GPUs communicate?
- System RAM: Substantial host memory is needed for loading/offloading.
- PCIe topology & Network bandwidth: Vital for multi-node setups.
- Software support: CUDA, PyTorch, communication libraries.
16 When Does a Dense Model Make More Sense?
Dense models can be attractive when the model fits comfortably within the available GPU memory and the workload does not require extremely large parameter counts.
A dense model can be simpler to deploy because there is no expert-routing layer distributing tokens among separate expert networks. For example, a workload that fits on a single high-memory GPU can avoid the complexity of multi-GPU expert placement and communication.
This is valuable for smaller inference deployments, development environments, and applications where simple deployment is important. This does not mean dense models are universally better, just easier to operate when the model fits the hardware.
17 When Does an MoE Model Make Sense?
MoE becomes particularly interesting when a model needs a large total parameter capacity without requiring every parameter to be computed for every token. The architecture can provide a way to scale parameter count while keeping per-token computation substantially lower than a dense model.
However, that benefit comes with infrastructure complexity. MoE is a good fit when the deployment has multiple GPUs, sufficient aggregate memory, fast GPU-to-GPU communication, appropriate inference software, and enough workload to keep accelerators busy.
18 A Practical GPU Server Comparison
Consider two hypothetical models: Model A (Dense, 100B parameters) and Model B (MoE, 400B total / 50B active).
It would be wrong to conclude: "Model B only needs the VRAM of a 50B model." The complete MoE model can still require memory for its total parameter set if the experts are resident.
But it would also be wrong to conclude: "Model B has four times the compute cost of Model A." Only a subset of the MoE experts is activated for each token.
| Metric | Dense (100B) | MoE (400B Total / 50B Active) |
|---|---|---|
| Total weights | 100B | 400B |
| Active experts | N/A | 50B* |
| Memory requirement | Large | Very large* |
| Per-token compute | Dense computation | Sparse expert computation* |
| Communication | Depends on setup | Potentially significant* |
* Exact behavior depends on architecture, precision, implementation, and deployment strategy.
19 Quantization Changes the Memory Calculation
Quantization can reduce the memory required to store model weights by representing them with fewer bits. For example, moving from a 16-bit representation toward 8-bit or 4-bit representations can substantially reduce weight storage.
A simplified calculation illustrates the idea:
- 100B parameters × 2 bytes (FP16) ≈ 200 GB
- 100B parameters × 1 byte (INT8) ≈ 100 GB
- 100B parameters × 0.5 bytes (INT4) ≈ 50 GB
These are theoretical weight-storage calculations, not complete GPU-memory requirements. Quantized models still require additional memory for runtime data, KV cache, and activations. For MoE models, quantization can be especially valuable because the total expert weight set can be very large.
20 A Useful Rule for GPU Planning
Do not choose a GPU server from the model's parameter count alone. Evaluate the model using four separate numbers: Total parameters, Active parameters per token, Precision / quantization, and Runtime memory requirements.
Dense models make model size and available memory the central concern. MoE models add model distribution, expert routing, and inter-GPU communication to that equation. That is why selecting hardware for modern AI workloads requires considering the architecture of the model and the GPU server together.
FAQ Frequently Asked Questions
What is the main difference between Dense and MoE AI models?
Does an MoE model automatically use less GPU VRAM than a Dense model?
What is expert parallelism in MoE models?
Why is GPU-to-GPU communication critical for MoE workloads?
Should I choose a GPU server based solely on active parameter count?
Need High-Performance GPU Infrastructure?
Fit Servers offers dedicated GPU servers with high-speed interconnects optimized for deploying and scaling complex Dense and Mixture-of-Experts AI workloads.
