Poros

Most standard single-GPU QLoRA paths (PEFT, Unsloth, hand-rolled loops) keep the frozen base resident on the GPU for the whole run. At 32B, that is roughly 18 GB of NF4 weights sitting in VRAM every step, receiving no gradients and never changing.

The base is read-only. It exists in the computation only because the forward pass needs to read its weights, block by block, as the input passes through. Between one block's forward and the next, those weights are idle. And during the optimizer step, they are completely irrelevant.

Backpropagation doesn't require full-base residency. QLoRA inherited that default from full fine-tuning, where the base weights actually change and need to stay on-device. Frozen-base training kept it after the reason for it was gone.

Moving model state down the memory hierarchy is not new: ZeRO-Offload pushes optimizer state and parameters to the CPU, FlexGen schedules placement across GPU, CPU, and disk for inference, and FSDP+QLoRA shards a quantized base across several GPUs. The case Poros targets is single-GPU training, where the backward pass itself has to be bounded.

Poros replaces persistent residency with streamed residency. Each transformer block's weights are needed exactly twice per micro-batch: once in the forward pass, once in the backward pass. Poros loads a block for the forward, holds activations at block boundaries, then reloads and recomputes each block during the backward in reverse order. Between those two moments, the weights are not on the GPU.

The result: 32B training drops from a 44.88 GB peak to 11.61 GB, and 72B from 81.57 GB to 18.87 GB. And the adapter that comes out is not approximately the same as a resident run's. It is the same, bit for bit. Loss, adapter weights, gradient norms, and optimizer state match a fully-resident run exactly across 200 steps, checked with torch.equal, not allclose: the difference is 0.00e+00. At 72B we verify loss and adapter weights directly, at every one of 200 steps; gradient norms and optimizer state are gated directly at 7B and 32B.

Streaming the frozen base reduces memory, but matching the resident trajectory bit-for-bit is a harder problem.

The catch is dropout. When Poros reruns a block's forward during the backward pass to recover its activations, that rerun resamples the dropout masks, different masks than the original forward drew. The gradients from that point on belong to a graph that never actually ran forward, so the adapter drifts from a resident run. And nothing warns you: the loss curve looks completely normal while the adapter quietly diverges.

Poros closes this with RNG state capture. Before each block's forward pass, the engine snapshots CPU and CUDA RNG state. During the backward, each block is reloaded once and its forward is recomputed inside torch.random.fork_rng with that exact snapshot restored. The recomputed forward reproduces the original exactly, down to the bit.

PyTorch's activation checkpointing handles the same subtlety the same way when it recomputes segments, by preserving per-segment RNG state. Poros applies that discipline at block granularity, in a runtime where the weights themselves are not resident, and verifies the result bit-for-bit end to end.

The rest of the training step is straightforward once that fix is in place. The forward streams frozen blocks through a small GPU window with asynchronous prefetch on CUDA streams; activations at block boundaries are retained. The backward walks blocks in reverse, reloads each one, recomputes with restored RNG, and propagates gradients only to the adapters and block inputs. Frozen weights are never differentiated. Then one optimizer step, identical to the step a resident run would have taken.

The learning problem itself is unchanged: the same attention kernels, optimizer, LoRA configuration, data order, and loss function as a resident run.

Block-major gradient accumulation combined with dropout is not supported: it would reorder RNG consumption across micro-batches and break bitwise parity silently, so Poros raises an error instead of producing a non-exact result.

We checked parity with targeted gates across these conditions rather than a single run: 200-step NF4 gates, gradient accumulation at 2 and 4 micro-batches, real Alpaca data, 1000-step extended audits, and dedicated dropout RNG gates. Every gate comes back 0.00e+00. The run artifacts are committed in docs/validation/.

The 1000-step audits hold exact parity at every checkpoint, ruling out slow drift as distinct from a step-200 match. The mode-sweep gate re-runs parity with deterministic algorithms off, default fast SDPA, TF32 enabled, bf16 autocast, and torch.compile active; the faster kernels don't disturb it. That sweep gates forward parity per mode; the 200-step gates above are the full four-surface audits.

Bitwise parity is verified per model family on real hardware:

• Qwen2.5 (7B–72B, incl. DeepSeek-R1-Distill-Qwen): RTX 5090 and RTX 3090 at 7B; A100, B300, PRO 6000, and RTX 5090 at 32B; datacenter hardware at 72B, with the end-to-end run on an RTX 5090
• Qwen3 (32B): B300, RTX 5090
• Qwen3.5 / Qwen3.6 dense: gated at 4B on an RTX 4090; the 0.8B–27B dense range shares that one validated implementation
• Gemma3 (27B): RTX 5090
• Gemma4 (31B): RTX 5090

Each result is exact within its own hardware and software stack.

At 32B, with a rank-16 LoRA adapter, NF4, and sequence length 512, peak training memory drops from 44.88 GB to 11.61 GB, which fits a 16 GB budget with 4.39 GB to spare. That budget fit is arithmetic, not a separate run; the smallest card we measured on directly is 24 GB. The same run on a consumer RTX 5090 peaks a touch lower, at 11.02 GB.

Block size trades memory for step time: at 32B, a block size of 1 peaks at 10.48 GB, 2 at 10.73, 4 at 11.61, 8 at 14.01, and 16 at 18.67; larger blocks buy back step time at the cost of a higher peak.

Sequence length moves the peak sub-linearly, because block weights rather than activations dominate at single-block residency: at 7B, going from sequence length 128 to 1024 raises the peak from 6.13 GB to 8.13 GB. The headline numbers are at 512, and parity is additionally gated at sequence length 4096 at 32B.

At 72B, peak training memory drops from 81.57 GB to 18.87 GB. Qwen2.5-72B loads and trains end to end on a single RTX 5090: the recorded run measured an 18.13 GB training peak, with a training loss that matches the same configuration on an RTX PRO 6000 to about 3e-5. The training peak fits a 24 GB budget; a measured 24 GB end-to-end run is future work. (The bitwise-parity gate at 72B ran on datacenter hardware; the 5090 run establishes the memory envelope.)

The same-card comparison at 32B on an RTX 3090 (24 GB):

The Accelerate result follows from how device-map offload works: it changes where the weights live, but the backward pass is still standard autograd, and its working set grows with model size until it exceeds the card. Poros bounds the backward as well, rematerializing and releasing each block; that bounded backward is what makes the 11.61 GB possible. That is 48% less VRAM than Unsloth here and 46.9–54.9% less on the equivalent A100 run. The two solve different problems: Unsloth optimizes the resident path, Poros changes what has to be resident. Where a model already fits comfortably resident, the resident path remains the right choice.

The training-time savings assume the model can be loaded onto the card at all. By default, Poros loads the model the way it trains it: streamed, so the full base never materializes on the GPU. The standard load does materialize it, staging the whole quantized base before the block manager can evict anything; at 72B that peak is 38.80 GB, enough to run a 24 GB card out of memory before training ever starts.

The block-streamed NF4 loader stages one tensor at a time, quantizes it, and evicts before loading the next. Load-time GPU peak:

• 0.27 GB at 7B (vs 5.30 GB standard)
• 0.81 GB at 32B (vs 18.16 GB standard)
• 1.63 GB at 72B (vs 38.80 GB standard)

The output is byte-identical to a standard load, checked against sha256 manifests at 0.5B, 7B, 32B, and 72B across the four families in the paper's matrix. The 72B byte gate passed on A100 across 3,765 tensors, 0 mismatched. Training parity re-gates at 0.00e+00 under the streamed loader at 7B and 32B.

This is the default. nf4_load_strategy="auto" selects streamed load with standard fallback.

Host RAM follows the same pattern: the standard path stages the whole checkpoint through host memory while it quantizes, and streaming removes that transient.

Poros is slower than resident training. On cards where both can run, expect roughly 1.6× to 2.7× per step depending on the hardware; the ratio looks worse on a B300 only because the resident baseline there is already so fast that any fixed streaming cost inflates it. The more common case is that the resident baseline does not run at all: 32B on an RTX 3090 OOMs resident, and Poros trains it at 3.21 s/step. In exchange, Poros trains at scales where PEFT and Accelerate cannot run, with a trajectory that matches a resident run exactly.

Most of the overhead is transfer, and it is predictable. Each micro-batch streams the base twice: once in the forward, once in the rematerialized backward. On an RTX PRO 6000 at gradient accumulation 1, that transfer is about 716 ms of a 1,101 ms step at 32B. The full per-card timings are in the paper.

Host memory is the other cost: provision for the streamed-loader peaks (34.7 GB at 32B, 48.2 GB at 72B). Steady state settles lower once the load transient passes.

We're working on two things next.

Full-precision streaming. The same blockwise residency works without quantization. The repo includes an experimental BF16 streaming runtime that passes the same bitwise training gates at 32B: an 11.40 GB peak against roughly 65 GB of resident bf16 weights, with bitwise parity at both 0.5B and 32B. It is currently unoptimized and slow; making it fast is the goal for the next release.

Multi-GPU and additional architectures. Poros targets a single GPU and the validated dense families today. Both extensions are held to the same requirement before anything is marked validated: byte-identical load and bitwise training parity, with the run artifacts committed.

Frozen-base fine-tuning never needed the whole model in VRAM. Poros gives it up, pays for it in step time, and proves with torch.equal that nothing else changed. It runs on the card you already own: pip install poros-train[ml], star it on GitHub, and the paper has the rest.

pip install "poros-train[ml]"

One line on top of your existing PEFT script:

import poros
model = poros.prepare(model)   # your loop, your config, untouched

Poros prepared model
        model  Qwen/Qwen2.5-7B
   model_type  qwen2
 architecture  validated
      adapter  lora (validated)
    precision  NF4
        stack  not checked (run poros doctor)
   block_size  4
    guarantee  bitwise parity (NF4, deterministic)

Poros is Apache-2.0, Linux/CUDA, single GPU, dense models only. Requirements, supported families, and the full quickstart are in the repository; architecture requests go through issues.