Skip to content
Read the paper
Search

Getting started

Quickstart

Copy pageMarkdown

Install, train, keep the adapter: three commands from zero to a trained model.

Get from zero to a trained adapter in under 5 minutes.

Option 1: PorosTrainer (5 lines)

from poros import PorosTrainer

trainer = PorosTrainer.from_pretrained(
    "Qwen/Qwen2.5-32B",
    quantization="nf4",
    lora_rank=16, lora_alpha=32,
)trainer.train("OpenAssistant/oasst1", steps=200, seq_len=512)  # any HF dataset, or "synthetic"trainer.save("./poros_output/qwen32b-lora")   # adapter + tokenizer + config

block_size defaults to the validated 4; pass it explicitly only to tune. For full configuration use PorosConfig (see the API reference).

Use your trained adapter

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import PeftModel
# Load the 32B base in NF4 4-bit — a dense load would need ~65 GB of VRAM.bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B", quantization_config=bnb, device_map="auto"
)model = PeftModel.from_pretrained(base, "./poros_output/qwen32b-lora")tok = AutoTokenizer.from_pretrained("./poros_output/qwen32b-lora")

Or merge into a standalone model for vLLM / TGI: poros export ./poros_output/qwen32b-lora -o ./merged. See examples/inference.py.

Merging dequantizes the base back to bf16, so poros export needs the whole model resident on the GPU — the one thing blockwise streaming does not help with, since it is a training technique. If you trained a model bigger than your card, keep serving the adapter on top of the 4-bit base (Option A in examples/inference.py); Poros tells you this with a clear error rather than an OOM.

from_pretrained loads the model in NF4 (4-bit), attaches LoRA adapters, and sets up blockwise residency. block_size=4 means 4 transformer layers are resident at a time; the rest stream from CPU pinned memory. If you omit block_size, Poros uses the validated default 4 — it is not guessed from available VRAM. block_size is a memory<->speed knob (larger = fewer transfers = faster but more resident VRAM; smaller = less VRAM but slower); it never affects parity.

Option 2: HF drop-in (one line)

If you already have a Hugging Face training script:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import get_peft_model, LoraConfig
import poros

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B", quantization_config=bnb
)
model = get_peft_model(    model, LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"]))

model = poros.prepare(model)   # the only Poros line
# standard HF Trainer now trains blockwise (the canonical 32B config peaks at 11.61 GB)

This is the same one line the README leads with. The prepared model is still a compliant nn.Module — HF Trainer, eval callbacks, loss handlers, and AMP all work unchanged. prepare() also detects precision, gates the adapter, and reports which guarantee applies; poros.patch_hf_model is the lower-level engine call it wraps, for when you want to skip the detection and gating.

Option 3: CLI

The CLI trains from a YAML config, and writes you a ready-to-run one:

poros init              # writes train.yaml: model, dataset, output_dirporos train train.yaml

The generated file has three active fields — everything omitted uses the validated defaults (NF4, rank-16 LoRA, block size 4) — plus the common knobs as comments, ready to uncomment. poros init -m <model> -d <data> pre-fills it; poros schema config prints every available field.

To train on your own .jsonl/.json/.csv file, see poros-trainer.md — a local path in the dataset: field works as-is. If a run is interrupted, poros train train.yaml --resume-from latest resumes from the newest checkpoint in output_dir (written every save_steps steps), bitwise-faithful to an uninterrupted run.

What to expect

The 32B configuration trains at 11.61 GB peak (measured), which fits a 16 GB VRAM budget with 4.39 GB of headroom; the smallest physical card we have measured on directly is 24 GB. The per-step slowdown vs fully-resident QLoRA is hardware- and scale-dependent — measured 1.7× at 7B on an RTX 3090, about 1.6× at 32B on an RTX PRO 6000, 2.28–2.68× on an A100, and ~3.5–4.3× on a B300 — the deliberate memory-for-time tradeoff. On a 24 GB card the resident 32B does not fit at all; Poros does.

The training output is bitwise identical to the same setup run fully resident: 0.00e+00 max difference on loss and adapter weights. (That is a claim about Poros's streaming, not about matching a different framework's numbers — see limitations.md.)

Next steps