llama.cpp tests went well on my single 7900XTX Sapphire Nitro+ GPU. Enticed by that, I got the courage to buy two 20GB 7900XT Sapphire Pulse GPUs, which were available at a great price.
Once I had them, I first validated by just mounting the new ones, and it ran great with tensor parallel for Q8_K_XL quant of Qwen 3.8 27B Unsloth GGUF. This helped me test the new cards and also made me realise the physical PCI slot space constraints.
My motherboard is ASrock WRX90 WS EVO and it has 7 PCIe slots out of which 6 run at full PCIe 5.0 x16 speeds and one runs at PCIe 5.0 x8 (remember this bit for later). Unfortunately, consumer GPUs are usually sized at 2.5 slots to 3.5 slots.
Initial Satisfaction
The two new GPUs ran great at PCIe 4.0 x16 speeds (third one was removed temporarily), which is what the 7900 series of AMD cards supports. Tensor parallel was a beast at 1200 tokens per second prompt prefill (compared to 400 that I was getting with pipeline parallel).
Now, to add the third GPU in that space, I had to squeeze a riser cable in and pull it out to a vertical mount in the chassis (Phanteks Enthoo Pro II).
The GPUs had cost me a pretty penny, and I did not want to invest in a costly PCIe 5.0 riser card (which I assumed would be useless for the PCIe 4.0 GPUs anyway). The three-times lower price of the PCIe 3.0 riser card enticed me as this was my first time using a riser card in my life, and I wanted to validate the setup with one. The riser was inexpensive enough for me to take a risk on and accept the loss if it turned out to be a dud. If it worked, I could have reused the card in another, older, machine of mine and would have bought a faster riser card a few months down the line.
Disappointments
The riser card arrived in unassuming and unbranded packaging. As soon as I had the riser card installed, my anticipation was high to try tensor parallel in the new setup. Naively, I had believed that everything would be fine as soon as I pressed the power button on the first attempt. After a few failed boots and unstable operating system behaviour, I remembered to downgrade the PCIe link speeds from 5.0 to 3.0 in the BIOS. This change got rid of all instabilities, and I was at least able to run all the GPUs at full sustained loads.
This, however, wasn’t enough to help me experience a significant speedup. In fact, there was a strange drop in the inference speed from 1200 tokens PP to 1000 tokens PP. My first instinct was that PCIe 3.0 x16 had really limited the speedup I expected from the extra GPU in place. The greed for speed almost had me regretting cutting corners on the riser card despite reading recommendations about getting good ones on multiple internet forums.
Debugging through RCCL
After a few days, on a free weekend, I decided to give it another shot. I had read about RCCL but had not considered it as a potential bottleneck for inference work. Also, my llama.cpp build was already compiled with RCCL support, and it was orchestrating the inter-GPU traffic in llama.cpp’s tensor-parallel mode — what I was missing was a way to measure it. I hadn’t given much thought to how much it would matter in a home setup with consumer GPUs. I decided to investigate it after exhausting other avenues. AMD provides a repository for rccl-tests which can be built locally, and we can test the inter-PCIe communication speeds. Building it from source was simple enough, and the first run gave me the following results:
$ ./all_reduce_perf -b 256M -e 2G -f 2 -g 3
# -b 256M: start buffer size = 256 MiB
# -e 2G: end buffer size = 2 GiB
# -f 2: multiply size by 2 each step
# -g 3: use 3 GPUs
# Collective test starting: all_reduce_perf
# nThread 1 nGpus 3 minBytes 268435456 maxBytes 2147483648 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
rccl-tests: Version develop_deprecated:40b1b17
# Using devices
# Rank 0 Group 0 Pid 1798494 on n-machine-threadripper device 0 [0000:f3:00] AMD Radeon RX 7900 XTX
# Rank 1 Group 0 Pid 1798494 on n-machine-threadripper device 1 [0000:d3:00] AMD Radeon RX 7900 XT
# Rank 2 Group 0 Pid 1798494 on n-machine-threadripper device 2 [0000:bc:00] AMD Radeon RX 7900 XT
#
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
268435456 67108864 float sum -1 57891 4.64 6.18 0 57120 4.70 6.27 0
536870912 134217728 float sum -1 114810 4.68 6.23 0 116521 4.61 6.14 0
1073741824 268435456 float sum -1 239450 4.48 5.98 0 237240 4.52 6.03 0
2147483648 536870912 float sum -1 476912 4.50 6.00 0 474130 4.53 6.04 0
# Errors with asterisks indicate errors that have exceeded the maximum threshold.
# Out of bounds values : 0 OK
# Avg bus bandwidth : 6.109
#
# Collective test concluded: all_reduce_perf
Now, this was fascinating. I was getting a speed average which corresponded to PCIe 3.0 x8 instead of full PCIe 3.0 x16. My motherboard had been already downgraded from PCIe 5.0, so I was sure that it wouldn’t have any trouble running at PCIe 3.0 x16 speeds. I immediately doubted the riser card again. That was the only new and untested component in my system.
Further Investigations
I had almost given up, resigned to the frustrating assumption that I would simply have to accept these lower inference speeds. However, a final stubborn desire to maximise the value of my investments sparked a renewed effort. It was time to pull out the rubber ducks and do some deep, meticulous diagnostics. After much deliberation, I revisited the physical slot configuration of the motherboard itself. And that is when the entire picture suddenly clicked into place!
I realised I had downgraded all the primary slots from PCIe 5.0 to PCIe 3.0. So the x16 5.0 slots went down to x16 3.0. BUT, there was one crucial slot that was originally only x8. When I downgraded the entire system, that specific slot had been downgraded from x8 5.0 to x8 3.0! This oversight on my part was the source of the slowdown.
Checking my GPU connection profiles confirmed the truth: one of the GPUs was running at PCIe 3.0 x8. This single bottleneck caused the entire ring topology of my GPUs to downgrade the inter-GPU communications to the slower x8 speed. Here is what a ring topology looks like:
+-------------------------------------------------------------+
| Inter-GPU Communication Ring |
| |
| +------------+ Direct PCIe +------------+ |
| | Rank 0 | --------------> | Rank 1 | |
| | 7900 XTX | (P2P/Ptr) | 7900 XT | |
| +------------+ +------------+ |
| ^ | |
| | Direct PCIe Direct | |
| | (P2P/Ptr) PCIe | |
| | (P2P/Ptr) v |
| +------------------------ +------------+ |
| | Rank 2 | |
| | 7900 XT | |
| +------------+ |
| |
+-------------------------------------------------------------+
In a ring all-reduce every chunk traverses every hop, so throughput is set by the slowest link. I immediately rebooted and changed the following in my BIOS (for this motherboard, go to BIOS settings, then Advanced → PCIe Subsystem Settings):
- The directly connected GPU in the x8 slot: Changed from PCIe 3.0 x8 to PCIe 4.0 x8.
- Confirmed the riser cable connected GPU staying at PCIe 3.0 x16.
- Upgraded the second directly connected GPU in the x16 slot: Changed from PCIe 3.0 x16 to PCIe 4.0 x16.
The last GPU adjustment wasn’t going to change the overall topology, but I made it to ensure future compatibility when I upgrade.
Finally, all three cards could run at effective PCIe 3.0 x16 OR PCIe 4.0 x8 (which are equivalent in speed).
Reckoning
It was time to run all reduce again.
.
.
.
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
268435456 67108864 float sum -1 29585 9.07 12.10 0 29088 9.23 12.30 0
536870912 134217728 float sum -1 58684 9.15 12.20 0 59782 8.98 11.97 0
1073741824 268435456 float sum -1 122910 8.74 11.65 0 121789 8.82 11.76 0
2147483648 536870912 float sum -1 244569 8.78 11.71 0 243093 8.83 11.78 0
# Errors with asterisks indicate errors that have exceeded the maximum threshold.
# Out of bounds values : 0 OK
# Avg bus bandwidth : 11.933
#
# Collective test concluded: all_reduce_perf
We are at stable speeds now! Firing up llama.cpp again in TP (tensor parallel) mode, and voilà, I was getting early PP (prompt prefill) speeds up to 1900 tokens per second on a dense model. The setup finally delivered the speed I’d paid for.
Summary
Here is a comparison table showing the performance improvements through different PCIe configuration stages:
| Setup | GPU Link Configuration | Bottleneck | Avg Bus Bandwidth (GB/s) | Prompt Processing @ 4k ctx |
|---|---|---|---|---|
| Two-card setup | PCIe 4.0 x16 PCIe 4.0 x16 |
None; both GPUs at full native 7900 XT(X) speed | — | ~1200 t/s |
| Initial three-card setup | PCIe 3.0 x8 PCIe 3.0 x16 PCIe 3.0 x16 |
PCIe 3.0 x8 link; ring topology limited by slowest GPU link | 6.109 | ~1000 t/s |
| Mixed-speed fix | PCIe 4.0 x8 PCIe 3.0 x16 PCIe 4.0 x16 |
None significant; PCIe 4.0 x8 and PCIe 3.0 x16 are effectively equivalent | 11.933 | ~1900 t/s |
| 1st Sep update | PCIe 4.0 x16 PCIe 4.0 x16 PCIe 4.0 x16 |
None; all GPUs at full native 7900 XT(X) speed | 23.897 | ~2500 t/s |
Future improvements
My ultimate goal is to move beyond internal chassis limitations. I plan to invest in high-quality PCIe Retimer cards paired with MCIO cables to mount the GPUs externally in legacy bitcoin mining rigs. Achieving this setup will allow me to run all these cards at their intended full PCIe 5.0 and 4.0 speeds, unlocking the final layer of performance potential.
Another big learning is that PCIe link speeds are not irrelevant in their impact on inference speeds in multi GPU setups. It does not bring linear speedups, but it does make a noticeable impact on wall time.
Stay tuned for updates on that in a later post!
Update 1st September 2026:
Finally got a PCIe Riser that supports up to 5.0! I do not have a card that supports PCIe 5.0 at the moment, but the all-reduce results using these cards doubled!
In my refreshed setup, all slots are configured at PCIe 4.0 x16. I also moved the x8-slot GPU into a full x16 slot. All three cards now negotiate PCIe 4.0 x16 — the full speed the 7900 XT(X) cards support.
.
.
.
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
268435456 67108864 float sum -1 14905 18.01 24.01 0 14894 18.02 24.03 0
536870912 134217728 float sum -1 29759 18.04 24.05 0 29759 18.04 24.05 0
1073741824 268435456 float sum -1 59594 18.02 24.02 0 59605 18.01 24.02 0
2147483648 536870912 float sum -1 121891 17.62 23.49 0 121899 17.62 23.49 0
# Errors with asterisks indicate errors that have exceeded the maximum threshold.
# Out of bounds values : 0 OK
# Avg bus bandwidth : 23.8967
#
# Collective test concluded: all_reduce_perf
As of now this is the maximum optimisation I can achieve through hardware configuration with my current setup, and I am quite satisfied with it (PP jumped to ~2.5k t/s!). Further fine-tuning will have to be achieved through software.
Glossary
All-Reduce — A collective communication operation where all GPUs contribute data, perform a reduction operation (like sum or average), and all GPUs receive the final result. Critical for distributed training and inference.
MCIO (Mini Cool-edge I/O) — A high-density connector standard used for PCIe connections, commonly found in server-grade hardware and external GPU enclosures.
P2P (Peer-to-Peer) — Direct GPU-to-GPU communication without going through the CPU or system memory. Enables faster data transfers between GPUs on the same PCIe fabric.
PCIe (Peripheral Component Interconnect Express) — A high-speed interface standard for connecting hardware components like GPUs to the motherboard. Each generation (3.0, 4.0, 5.0) roughly doubles the bandwidth of the previous one.
PCIe Lanes (x8, x16) — The number of data lanes available for a PCIe connection. More lanes provide higher bandwidth; x16 has twice the bandwidth of x8 at the same generation.
PCIe Riser Card — An adapter that extends a PCIe slot to a different physical location, allowing GPUs to be mounted in non-standard positions. Quality and PCIe generation support vary significantly.
PCIe Retimer — An active signal conditioning device that regenerates PCIe signals, enabling longer cable runs and maintaining signal integrity over extended distances.
Pipeline Parallel — A parallelization strategy where different layers of a neural network are distributed across multiple GPUs, processing data sequentially through the pipeline.
Prompt Prefill (PP) — The initial phase of LLM inference where the entire input prompt is processed in parallel before token generation begins. Measured in tokens per second.
RCCL (ROCm Communication Collectives Library) — AMD's library for multi-GPU communication, equivalent to NVIDIA's NCCL. Handles data synchronization and collective operations across GPUs.
**Ring Topology** — A communication pattern where GPUs are logically arranged in a circle, with each GPU communicating directly with its two neighbors. The slowest link in the ring determines overall performance. **Tensor Parallel (TP)** — A parallelization strategy where model weights are split across multiple GPUs, with each GPU computing part of the same layer simultaneously. Requires high-bandwidth inter-GPU communication. **Token** — The basic unit of text processing in LLMs. A token can be a word, part of a word, or a character, depending on the tokenization scheme used.Disclaimer: All posts (except glossaries) are handwritten. All views and opinions are my own and do not reflect those of my employer or any other organisation. Feedback welcome: nandeepmali@gmail.com.