tl;dr — Poros fine-tunes a 72B model end-to-end on one 32 GB RTX 5090, and a 32B in 11.61 GB. The adapter it produces is bit-identical to a full-VRAM run:torch.equal, notallclose. It does it by streaming the frozen base through the GPU one block at a time instead of keeping it resident.
Why the frozen base sits in VRAM
Most standard single-GPU QLoRA paths (PEFT, Unsloth, hand-rolled loops) keep the frozen base resident on the GPU for the whole run. At 32B, that is roughly 18 GB of NF4 weights sitting in VRAM every step, receiving no gradients and never changing.
The base is read-only. It exists in the computation only because the forward pass needs to read its weights, block by block, as the input passes through. Between one block's forward and the next, those weights are idle. And during the optimizer step, they are completely irrelevant.
Backpropagation doesn't require full-base residency. QLoRA inherited that default from full fine-tuning, where the base weights actually change and need to stay on-device. Frozen-base training kept it after the reason for it was gone.
Moving model state down the memory hierarchy is not new: ZeRO-Offload pushes optimizer state and parameters to the CPU, FlexGen schedules placement across GPU, CPU, and disk for inference, and FSDP+QLoRA shards a quantized base across several GPUs. The case Poros targets is single-GPU training, where the backward pass itself has to be bounded.
Poros replaces persistent residency with streamed residency. Each transformer block's weights are needed exactly twice per step: once in the forward pass, once in the backward pass. Poros loads a block for the forward, holds activations at block boundaries, then reloads and recomputes each block during the backward in reverse order. Between those two moments, the weights are not on the GPU.
The result: 32B training drops from a 44.88 GB peak to 11.61 GB, and 72B from 81.57 GB to 18.87 GB. And the adapter that comes out is not approximately the same as a resident run's. It is the same, bit for bit. Loss, adapter weights, gradient norms, and optimizer state match a fully-resident run exactly across 200 steps, checked with torch.equal, not allclose: the difference is 0.00e+00. At 72B we verify loss and weights directly; the rest follows, since they are deterministic functions of weights that already match.
Why getting an identical trajectory requires more than just streaming weights
Streaming the frozen base reduces memory, but matching the resident trajectory bit-for-bit is a harder problem.
The catch is dropout. When Poros reruns a block's forward during the backward pass to recover its activations, that rerun resamples the dropout masks — different masks than the original forward drew. The gradients from that point on belong to a graph that never actually ran forward, so the adapter drifts from a resident run. And nothing warns you: the loss curve looks completely normal while the adapter quietly diverges.
global RNG stream → ┆ ┆ ┆ ░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒▓▓▓▓▓▓▓▓▓ ▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▓▓▓▓▓▓▓▓▓ ✗ drifts past P ┆ ┆ ┆ ┌────────────┐ ░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒┆ │ ▓▓▓▓▓▓▓▓ │ ▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░▒░┆ │ ▓▓▓▓▓▓▓▓ │ ✓ held at P → 0.00e+00 ┆ └────────────┘ PnaivePoros→fork_rng
Poros closes this with RNG state capture. Before each block's forward pass, the engine snapshots CPU and CUDA RNG state. During the backward, each block is reloaded once and its forward is recomputed inside torch.random.fork_rng with that exact snapshot restored. The recomputed forward reproduces the original exactly, down to the bit.
PyTorch's activation checkpointing handles the same subtlety the same way when it recomputes segments, by preserving per-segment RNG state. Poros applies that discipline at block granularity, in a runtime where the weights themselves are not resident, and verifies the result bit-for-bit end to end.
The rest of the training step is straightforward once that fix is in place. The forward streams frozen blocks through a small GPU window with asynchronous prefetch on CUDA streams; activations at block boundaries are retained. The backward walks blocks in reverse, reloads each one, recomputes with restored RNG, and propagates gradients only to the adapters and block inputs. Frozen weights are never differentiated. Then one optimizer step, identical to the step a resident run would have taken.
FORWARD snapshot CPU+CUDA RNG before each block ┌──────●──────┐ ┌──────●──────┐ ┌──────●──────┐ ┌──────●──────┐ ┌──────●──────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ ┆ ┆ │ ┆ ┆ ┆ ┆ │ fork_rng ┆ ┆ ┆ ┆ │ restore RNG ┆ ┆ ┆ ┆ │ = identical mask ┆ ┆ ┆ ┆ │ ┆ ┆ ┆ ┆ │ ┆ ┆ ┌──────○──────┐ ┌──────○──────┐ ┌──────○──────┐ ┌──────○──────┐ ┌──────○──────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ BACKWARD reverse order · reload · restore RNG · recompute B1B1B2B2B3B3B4B4B5B5
The learning problem itself is unchanged: the same attention kernels, optimizer, LoRA configuration, data order, and loss function as a resident run.
Block-major gradient accumulation combined with dropout is not supported: it would reorder RNG consumption across micro-batches and break bitwise parity silently, so Poros raises an error instead of producing a non-exact result.
Parity validation across the full audit matrix
We checked parity across a matrix of conditions rather than a single run: dropout off and on, synthetic and real Alpaca data, 200-step gates and 1000-step extended audits, multiple seeds, and gradient accumulation at 2 and 4 micro-batches. Every combination reports 0.00e+00 on all four quantities. The run artifacts are committed in docs/validation/.
The 1000-step audits hold exact parity at every checkpoint, ruling out slow drift as distinct from a step-200 match. The mode-sweep gate re-runs parity with deterministic algorithms off, default fast SDPA, TF32 enabled, bf16 autocast, and torch.compile active; the faster kernels don't disturb it, and parity holds exactly across all four families in the paper's matrix, in every mode.
loss │ ▓ Resident = Poros │ ▓ ▓▓ ▓ ▓ ├ ░▓░░▓░▓▓░▓▓▓ ▓ │ ░░░░░░░░░░░░▓░▓▓ │ ░░░░░░░░░░░░░░░░▓▓ ├ ░░░░░░░░░░░░░░░░░░▓▓ │ ░░░░░░░░░░░░░░░░░░░░▓▓▓▓ │ ░░░░░░░░░░░░░░░░░░░░░░░░▓▓▓ │ ░░░░░░░░░░░░░░░░░░░░░░░░░░░▓▓▓▓▓▓▓ ├ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ └ ──────────────────────────────────────────────────────────────────────────── 0 difference · Resident − Poros ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 0 ──────────────────────────────────────────────────────────────────────────── 0.00e+00 ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ 121314200 steps
Bitwise parity is verified per model family on real hardware:
- Qwen2.5 (7B–72B, incl. DeepSeek-R1-Distill-Qwen): RTX 5090 and RTX 3090 at 7B; A100, B300, PRO 6000, and RTX 5090 at 32B; B300 at 72B
- Qwen3 (32B): B300, RTX 5090
- Qwen3.5 / Qwen3.6 dense (0.8B–27B): RTX 4090
- Gemma3 (27B): RTX 5090
- Gemma4 (31B): RTX 5090
Each result is exact within its own hardware and software stack.
Memory savings in practice
At 32B, with a rank-16 LoRA adapter, NF4, and sequence length 512, peak training memory drops from 44.88 GB to 11.61 GB, which fits a 16 GB budget with 4.39 GB to spare. The same run on a consumer RTX 5090 peaks a touch lower, at 11.02 GB.
Block size trades memory for step time: at 32B, a block size of 1 peaks at 10.48 GB, 2 at 10.73, 4 at 11.61, 8 at 14.01, and 16 at 18.67; larger blocks buy back step time at the cost of a higher peak.
Sequence length moves the peak sub-linearly, because block weights rather than activations dominate at single-block residency: at 7B, going from sequence length 128 to 1024 raises the peak from 6.13 GB to 8.13 GB. The headline numbers are at 512, and parity is additionally gated at sequence length 4096 at 32B.
At 72B, peak training memory drops from 81.57 GB to 18.87 GB. Qwen2.5-72B loads and trains end to end on a single RTX 5090. Today it needs the 5090's 32 GB rather than 24: a one-time startup self-check briefly peaks around 30.5 GB before training settles, and streaming that check too is what will bring it under 24 GB. (The bitwise-parity gate at 72B ran on a B300; the 5090 run establishes the memory envelope.)
The same-card comparison at 32B on an RTX 3090 (24 GB):
│
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ · OOM
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
│
│
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ · OOM
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
│
│
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ │
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ·│· · · · 22.29 GB
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ │
│
│
▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ │
▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ · · · · · · · · · · · · ·│· · · · 11.61 GB
▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ │ 24 GB cardPEFT QLoRAAccelerateUnsloth QLoRAPorosThe Accelerate result follows from how hook-based offload works: it changes where the weights live, but the backward pass is still standard autograd, and its working set grows with model size until it exceeds the card. Poros bounds the backward as well, rematerializing and releasing each block; that bounded backward is what makes the 11.61 GB possible. That is 48% less VRAM than Unsloth here and 46.9–54.9% less on the equivalent A100 run. The two solve different problems: Unsloth optimizes the resident path, Poros changes what has to be resident. Where a model already fits comfortably resident, the resident path remains the right choice.
░ Resident QLoRA ▓ Poros ├ │ ░░░░░░ │ ░░░░░░ │ ░░░░░░ │ ░░░░░░ │ ░░░░░░ │ ░░░░░░ │ ░░░░░░ ├ ░░░░░░ ░░░░░░ │ ░░░░░░ ░░░░░░ │ ░░░░░░ ░░░░░░ │ ░░░░░░ ░░░░░░ ├ ░░░░░░ ░░░░░░ │ ░░░░░░ ░░░░░░ ░░░░░░ ▓▓▓▓▓▓ │ ░░░░░░ ░░░░░░ ▓▓▓▓▓▓ ░░░░░░ ▓▓▓▓▓▓ │ ░░░░░░ ▓▓▓▓▓▓ ░░░░░░ ▓▓▓▓▓▓ ░░░░░░ ▓▓▓▓▓▓ ├───────────────────────────────────────────────────────────────────────── peak GB024459015.246.907B−55%44.8811.6132B−74%81.5718.8772B−77%
The streamed NF4 loader
The training-time savings assume the model can be loaded onto the card at all. The standard NF4 load materializes the full quantized base on GPU before the block manager can evict it; at 72B that load peak is 38.80 GB, so a 24 GB card runs out of memory during load, before training ever starts. This is what motivated a loader built on the same streaming idea as training.
The block-streamed NF4 loader stages one tensor at a time, quantizes it, and evicts before loading the next. Load-time GPU peak:
- 0.27 GB at 7B (vs 5.30 GB standard)
- 0.81 GB at 32B (vs 18.16 GB standard)
- 1.63 GB at 72B (vs 38.80 GB standard)
The output is byte-identical to a standard load, checked against sha256 manifests at 0.5B, 7B, 32B, and 72B across the four families in the paper's matrix. The 72B byte gate passed on A100 across 3,765 tensors, 0 mismatched. Training parity re-gates at 0.00e+00 under the streamed loader at 7B and 32B.
This ships as the default loader in v0.1. nf4_load_strategy="auto" selects streamed load with standard fallback.
Host RAM follows the same pattern: the standard path stages the whole checkpoint through host memory while it quantizes, and streaming removes that transient.
┌─────────────┬─────────────────────────┬────────────────────────┐ │ Scale │ Streamed RSS │ Standard RSS │ ├─────────────┼─────────────────────────┼────────────────────────┤ │ 7B │ 12.1 GB │ 15.3 GB │ │ │ │ │ │ 32B │ 34.7 GB │ 62.0 GB │ │ │ │ │ │ 72B │ 48.2 GB │ 136.5 GB │ └─────────────┴─────────────────────────┴────────────────────────┘
The cost: step time
Poros is slower than resident training. On cards where both can run, expect roughly 1.6× to 2.7× per step depending on the hardware; the ratio looks worse on a B300 only because the resident baseline there is already so fast that any fixed streaming cost inflates it. The more common case is that the resident baseline does not run at all: 32B on an RTX 3090 OOMs resident, and Poros trains it at 3.21 s/step. The full per-card timing matrix is in the paper. In exchange, Poros trains at scales where PEFT and Accelerate cannot run, with a trajectory that matches a resident run exactly.
Most of the overhead is transfer, and it is predictable. A step streams the base twice: once in the forward, once in the rematerialized backward. On an RTX PRO 6000, that transfer is about 716 ms of a 1,101 ms step at 32B. The full per-card timings are in the paper.
Host memory is the other cost: provision for the streamed-loader peaks (34.7 GB at 32B, 48.2 GB at 72B). Steady state settles lower once the load transient passes.
step time, s/step 7B PEFT QLoRA ░░░░░░ 0.51 Accelerate ░░░░░░░░░ 0.73 Unsloth ░░░░░░░ 0.53 Poros ▓▓▓▓▓▓▓▓▓▓▓ 0.87 32B PEFT QLoRA × OOM Accelerate × OOM Unsloth ░░░░░░░░░░░░░░░░░░░░░░░░░░ 2.07 Poros ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 3.21 peak VRAM11.61 GB22.29 GB11.61 GB
Future work
We're working on two things next.
Full-precision streaming. The same blockwise residency works without quantization. The repo includes an experimental BF16 streaming runtime that passes the same bitwise training gates at 32B: an 11.40 GB peak against roughly 65 GB of resident bf16 weights, with bitwise parity at both 0.5B and 32B. It is currently unoptimized and slow; making it fast is the goal for the next release.
Multi-GPU and additional architectures. v0.1 targets a single GPU and the validated dense families. Both extensions are held to the same requirement before anything is marked validated: byte-identical load and bitwise training parity, with the run artifacts committed.
Frozen-base fine-tuning never needed the whole model in VRAM. Poros gives it up, pays for it in step time, and proves with torch.equal that nothing else changed. It runs on the card you already own: pip install poros-train[ml], star it on GitHub, and the paper has the rest.
Get running
pip install "poros-train[ml]"
One line on top of your existing PEFT script:
import poros model = poros.prepare(model) # your loop, your config, untouched
Poros prepared model
model Qwen/Qwen2.5-7B
model_type qwen2
architecture validated
adapter lora (validated)
precision NF4
stack validated
block_size 4
guarantee official bitwise path (NF4, deterministic)Poros is Apache-2.0, Linux/CUDA, single GPU in v0.1, dense models only. Requirements, supported families, and the full quickstart are in the repository; architecture requests go through issues.