16-Card GPU Mesh Interconnect
A non-blocking, high-density GPU interconnect solution for single-node AI inference and fine-tuning — 16 dual-slot GPUs woven into a full-mesh fabric by 144-lane PCIe 5.0 switch chips.
System Architecture & Testbed
The 16-GPU mesh fabric is realized on dual-switch GPU boards powered by 144-lane PCIe 5.0 switch chips with KFNative™ interconnect, and validated in a production test environment.
Mesh Topology
Four 144-lane PCIe 5.0 switch chips are fully interconnected via Fabric Links into a mesh. Each switch manages 4 GPUs, forming a true full-connectivity fabric where any GPU pair communicates with at most one routing hop — no CPU detour, no shared-bus contention.
| Port | Count per Switch | Connected To |
|---|---|---|
| USP | 1 | Host CPU |
| DSP | 4 | GPUs (4 per switch) |
| DSP | 1 | 400G NIC |
| Fabric Link | 3 | Peer switch chips |
Why 144 Lanes Is the Hard Requirement
A full 16-GPU interconnect needs 256 lanes just for the GPUs at x16 each — far beyond any CPU. Four 144-lane switches deliver 576 lanes, enough to keep every GPU at full speed, retain the NIC, and reserve Fabric Link bandwidth for the mesh. With 96-lane switches, only 48 lanes remain after inter-switch links — at most 3 GPUs can be attached. 144 lanes per chip is therefore the entry ticket for a non-blocking 16-card fabric.
Non-Blocking Mechanics
Within a switch
4 GPUs under the same chip plane talk at full line rate directly.
Across switches
Data flows through Fabric Link with a single routing hop — no CPU detour, no shared-bus congestion.
Measured on a production dual-switch board: the GPU-to-GPU P2P bidirectional bandwidth matrix is highly balanced at 111.3–111.4 GB/s for every pair — near the PCIe 5.0 x16 full-duplex limit — confirming the fabric is truly non-blocking end to end.
Testbed Configuration
| Item | Specification |
|---|---|
| Test board | MG-SW5A10-P5G216-V20 Dual-WLG5144 GPU Board (2 boards, each carries 2× WLG5144) |
| CPU | Intel Xeon Gold 6530 × 2 (32 cores/socket, 128 threads) |
| GPU | 16 × NVIDIA GeForce RTX 5090 |
| Switch | 4 × WLG5144 (PCIe 5.0, 144 lanes), full mesh |
| Interconnect | 1 Fabric Link between any two switches; each WLG5144 carries 4 × RTX 5090 |
| OS | Ubuntu 22.04.3 LTS |
| Memory | 251 GiB |
| NCCL | 2.19.7 (nccl-headers=22907, nccl-library=2.26.2+cuda12.8) |
Measured Data
| Metric | Measured Result |
|---|---|
| GPU-to-GPU P2P bidirectional bandwidth | ≈ 111.4 GB/s — every GPU pair in the 16×16 matrix lands in 111.33–111.44 GB/s, near the PCIe 5.0 x16 full-duplex limit |
| Self bandwidth (matrix diagonal) | ≈ 1528–1537 GB/s per GPU (NVIDIA P2P self-reference) |
| NCCL Allreduce throughput | ≈ 49.60 GB/s (18MB–4B message sizes) |
| Within-switch vs across-switch | Identical bandwidth — Fabric Link is not a bottleneck; the mesh is truly full-connectivity end to end |
P2P Bandwidth Matrix — Sample Pairs (GB/s)
Representative pairs from the full 16-GPU bidirectional matrix; all 120 pairs measured 111.3–111.4 GB/s with high uniformity.
| GPU Pair | P2P Bidirectional (GB/s) |
|---|---|
| GPU1 – GPU2 | 111.37 |
| GPU1 – GPU3 | 111.40 |
| GPU2 – GPU3 | 111.40 |
| GPU5 – GPU6 | 111.40 |
| GPU8 – GPU9 | 111.41 |
| GPU8 – GPU12 | 111.42 |
| GPU11 – GPU12 | 111.39 |
| GPU14 – GPU15 | 111.41 |
Target Scenarios
Single-Node Inference
High-density 16-GPU nodes in 10U chassis with dual-slot accelerators, built for large-model serving.
Fine-Tuning Density
Rising density demand for fine-tuning drives the shift from 4U 8–10 cards to 10U 16–20 dual-wide cards.
Full Model Residency
16 GPUs in one node keep large model parameters entirely in GPU memory — no cross-node sharding overhead.
Scaling Path
Mesh delivers the lowest latency for 6–32 GPU scale-up domains, at the cost of O(n²) fabric links. Beyond 64 GPUs, the topology switches to a two-tier CLOS (Spine-Leaf) architecture. For multi-node super-pods, shadow devices (virtual EP/NTB) provide global address translation to resolve BAR overlaps across nodes — reserved as the extension path of this design.