Reference
Poros — Official Validation Results (NVIDIA A100 80GB PCIe)
training peakQwen2.5-32B
Poros uses 17% of the dense load
These are the official, reproducible benchmark numbers for the architecture
families measured in this A100 run (qwen2, qwen3, gemma3, gemma4) on an
A100 80GB PCIe (sm_80, PCIe Gen4). Every cell is measured — no number is
interpolated or estimated.
This is the clean, same-attention-path matrix the B300 report planned:
all three methods (poros / resident / unsloth) run on the same default fast
attention path (deterministic: false), so the slowdown column is the pure
cost of blockwise residency with no math-vs-flash gap mixed in. It is also
fully standardized on the synthetic dataset (license-free, zero download).
Configuration
| Field | Value |
|---|---|
| Hardware | NVIDIA A100 80GB PCIe (80 GB VRAM, PCIe Gen4) |
| Precision | NF4 (the only bitwise path in the validated matrix) |
| Steps × passes | 200 steps × 3 passes (warmup discarded; median ± std across passes) |
| Sequence length | 512 |
| Batch / grad-accum | 1 / 1 (so 1 step = 1 optimizer step for every method) |
| LoRA | rank 16, alpha 32, dropout 0.0 |
| Block size (poros) | 4 |
| Dataset | synthetic (deterministic, license-free random tokens) |
| Methods | poros (blockwise residency), resident (full QLoRA), unsloth 2026.6.1 |
| Software stack | torch 2.10.0+cu128, transformers 5.5.0, bitsandbytes 0.49.2, peft 0.19.1 |
| Attention path | same default fast path for all methods (deterministic: false) |
Bitwise parity is a separate gate (tests/), run on the
deterministic path that the parity guarantee ships on. It asserts
torch.equal(resident, blockwise) — exact, not allclose. 0.00e+00 =
bit-for-bit identical.
Bitwise parity
Full bitwise parity is gated separately per the paper's Table 3: Qwen2.5 and Qwen3
across the datacenter/consumer matrix, and Gemma3/Gemma4 on RTX 5090
(block-streamed loader; receipts under
validation/). This A100
run reports throughput and VRAM.
Benchmark matrix
| Model | Arch | poros s/step (median±std) | resident s/step | unsloth s/step | Slowdown vs resident | vs unsloth | poros VRAM | resident VRAM | unsloth VRAM | VRAM cut vs resident | vs unsloth |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-32B | qwen2 | 2.0489±0.0091 | 0.7662 | 0.9086 | 2.67× | 2.25× | 10.95 GB | 34.72 GB | 20.82 GB | 68.4% | 47.4% |
| Qwen3-32B | qwen3 | 2.1832±0.0040 | 0.8157 | 0.9287 | 2.68× | 2.35× | 11.03 GB | 36.00 GB | 20.77 GB | 69.4% | 46.9% |
| Gemma3-27B | gemma3 | 1.8933±0.0048 | 0.7835 | 0.9365 | 2.42× | 2.02× | 9.95 GB | 33.17 GB | 22.06 GB | 70.0% | 54.9% |
| Gemma4-31B | gemma4 | 2.2537±0.0055 | 0.9889 | 1.0255 | 2.28× | 2.20× | 10.65 GB | 38.75 GB | 20.57 GB | 72.5% | 48.2% |
The raw per-cell artifacts (with full provenance: torch/cuda/transformers/bitsandbytes
versions, GPU name, per-pass timings) are in
validation/.
Takeaways
- ~68–73% VRAM cut vs resident, ~47–55% vs unsloth. Every 27–32B model fine-tunes in 10–11 GB instead of 33–39 GB resident / 20–22 GB unsloth.
- Same-path slowdown is 2.3–2.7× vs resident on the A100 — measured with every method on the identical attention path, so this is the pure streaming cost. It confirms the slowdown ratio is hardware-dependent: ~2.3–2.7× on A100 (PCIe Gen4) vs ~3.5–4.3× on the B300 (whose resident baseline is so fast that the fixed PCIe streaming cost dominates the ratio).
- This run adds the first published 3-method A100 benchmark numbers for the Gemma rows. Their bitwise parity is gated on RTX 5090 (see Bitwise parity above).
Reproduce
# bitwise parity gate (per arch)POROS_GPU_TEST_MODEL=Qwen/Qwen2.5-32B pytest -m gpu tests/test_gpu_parity.pyPOROS_GPU_TEST_MODEL=Qwen/Qwen3-32B pytest -m gpu tests/test_gpu_parity.pyPOROS_GPU_TEST_MODEL=google/gemma-3-27b-pt pytest -m gpu tests/test_gpu_parity.pyPOROS_GPU_TEST_MODEL=google/gemma-4-31B-it pytest -m gpu tests/test_gpu_parity.py# 3-method benchmark matrix (per model; gemma repos are gated — needs HF_TOKEN)poros bench matrix configs/bench/official_a100_qwen25_32b.yaml -o out/poros bench matrix configs/bench/official_a100_qwen3_32b.yaml -o out/poros bench matrix configs/bench/official_a100_gemma3_27b.yaml -o out/poros bench matrix configs/bench/official_a100_gemma4_31b.yaml -o out/Downstream capability check
One Qwen2.5-32B NF4 adapter was trained end-to-end with Poros Blockwise (oasst1, 100 steps, rank-16 LoRA; the recorded run is on the RTX PRO 6000) and evaluated with the LM Evaluation Harness: zero-shot MMLU is preserved (81.6% with the adapter vs 83.0% base on a fixed 1,824-question subset, about 1.1 combined standard errors). Raw harness JSONs, the per-step training record, and the full protocol (including a note on format-sensitive generative benchmarks) are in docs/validation/downstream-eval/.