CUDA LoRA Fine-Tune – a Literate, True-GPU Walkthrough
the connect-only "CUDA LoRA fine-tune" demo, run on an NVIDIA GPU
Table of Contents
NVIDIA/CUDA, true-GPU. Every device("cuda") block below
runs its forward pass, backward pass, and the Adam optimizer on the
NVIDIA GPU via candle. This
is not runnable in the public browser demo (WASM is CPU-only and has no
CUDA peer) -- which is exactly why it ships as a literate page: the blocks were
executed by a CUDA-enabled mlpl-repl on a Linux + NVIDIA host, and
the outputs below are that real run.
This recreates the playground's CUDA LoRA fine-tune demo as a literate
program. It is the CUDA analog of the MLX LoRA fine-tune walkthrough –
identical steps, only device("cuda") instead of device("mlx"). We
pretrain a tiny language model, wrap it with LoRA adapters, measure it,
fine-tune only the adapters on the NVIDIA GPU, and measure again –
showing concrete before vs after learning, not just a loss curve. The
blocks share one :session cuda, so state threads through exactly as if
typed into the REPL.
Step 1 – prepare the corpus
Load a snippet of tiny-Shakespeare, train a 280-token BPE tokenizer, encode to ids, and build next-token (X -> Y) training pairs with a context window of 32.
corpus = load_preloaded("tiny_shakespeare_snippet")
tok = train_bpe(corpus, 280, 0)
ids = apply_tokenizer(tok, corpus)
X = reshape(shift_pairs_x(ids, 32), [reduce_mul(shape(shift_pairs_x(ids, 32)))])
Y = reshape(shift_pairs_y(ids, 32), [reduce_mul(shape(shift_pairs_y(ids, 32)))])
shape(X)
1440
Step 2 – define and pretrain a tiny language model
A minimal decoder (embedding, one causal-attention block, a final norm, a linear head), pretrained for 20 Adam steps so it has some structure to specialize from.
V = 280 ; d = 32 ; h = 1
base = chain(embed(V, d, 0), residual(chain(rms_norm(d), causal_attention(d, h, 1))), rms_norm(d), linear(d, V, 4))
experiment "cuda_base" { train 20 { adam(cross_entropy(apply(base, X), Y), base, 0.001, 0.9, 0.999, 0.00000001); loss_metric = cross_entropy(apply(base, X), Y) } }
last_losses
8.748087078871551 8.556515224936417 8.374789080456024 8.2292430940222 8.088912927120974 7.9501537209206745 7.822384960039974 7.702769117559668 7.589426658793876 7.475659719397106 7.361660820476453 7.245777031342536 7.132938368249842 7.036058560056138 6.931899933236404 6.825627580515069 6.723534595757975 6.622502452566158 6.519077135403341 6.439220037847968
Step 3 – wrap with LoRA, then measure BEFORE fine-tuning
lora(base, 8, 16.0, 0) freezes every base weight and adds rank-8
adapters to the head (the adapter B inits to zero, so the student
starts identical to the base). We capture two baselines: perplexity
(exp(cross-entropy); lower is better) and a *sampled continuation* of
the prompt ="the ".
student = lora(base, 8, 16.0, 0) ppl_before = perplexity(apply(student, X), Y) ppl_before
625.9184167007902
seq = apply_tokenizer(tok, "the ")
repeat 24 { logits = apply(student, seq); last = last_row(logits); nxt = sample(top_k(last, 20), 0.8, step); seq = concat(seq, nxt) }
decode(tok, seq)
the CCCCCCCCCCCC
Step 4 – fine-tune the adapters on the GPU
Inside device("cuda"), the fine-tune step is one traceable candle
graph over the adapters, differentiated by backward, and the adapters
are updated by a candle Adam – forward, backward, and optimizer all on
the NVIDIA GPU. Only the adapters move; the base stays frozen.
device("cuda") { experiment "cuda_lora" { train 25 { adam(cross_entropy(apply(student, X), Y), student, 0.01, 0.9, 0.999, 0.00000001); loss_metric = cross_entropy(apply(student, X), Y) } } }
last_losses
6.349223708994379 6.213732804121422 6.035280688159431 5.820439835881928 5.584292973011285 5.351586547183255 5.148975384323652 4.98457387266443 4.845299113114293 4.723219366956176 4.620552020121121 4.539821354838601 4.474434202358667 4.412057126074641 4.346068450604383 4.279629049679461 4.217657050184844 4.1619442620057345 4.1114097642888 4.062399089324685 4.013292900438474 3.9659534316095195 3.9230260768053635 3.8854941164114183 3.8523719128068215
Step 5 – measure AFTER fine-tuning (the payoff)
Same model, same prompt, same sampling seeds – only the adapters changed. Compare these two numbers and the two continuations to the baselines in Step 3.
ppl_after = perplexity(apply(student, X), Y) ppl_after
47.10466178035338
gen = apply_tokenizer(tok, "the ")
repeat 24 { logits = apply(student, gen); last = last_row(logits); nxt = sample(top_k(last, 20), 0.8, step); gen = concat(gen, nxt) }
decode(tok, gen)
the �YYYYYYYYYYYHF i i i
Perplexity falls sharply: after fine-tuning the model assigns far higher probability to the true next token – a concrete before/after demonstration that the GPU fine-tune actually taught the adapters, not merely that a loss number dropped. (At d=32 on a tiny corpus the model is far too small for fluent prose, so both sampled continuations are repetitive; the rigorous signal here is the perplexity, while the sampled distribution does visibly shift.) The CUDA loss curve is parity-tested against the CPU path within fp32 tolerance.
Step 6 – the fine-tune loss curve
loss_curve(last_losses)
How this was run
This page needs a CUDA-enabled mlpl-repl (Linux + NVIDIA). Build it once
(this does NOT touch your installed binary):
CUDA_COMPUTE_CAP=120 cargo build -p mlpl-repl --features cuda --release # -> target/release/mlpl-repl
Then publish, pointing the Org-babel backend at that build:
CUDA_COMPUTE_CAP=120 MLPL_REPL_CMD="$PWD/target/release/mlpl-repl" \ ./examples/literate/publish.sh examples/literate/cuda-lora-finetune.org
Off a CUDA host, device("cuda") transparently falls back to CPU (the
numbers still match within fp32 tolerance), so the page still publishes –
it just won't be running on a GPU.