emufpga investigates whether immutable MoE expert weights can flow serially from cheap storage while a small or old GPU keeps attention, KV cache, and active state. CPU/System-RAM offload is a practicality comparison, not a speed target.
Current verdict: preliminary capacity success; coding reliability not yet measured.Route requests first, fetch only the experts they need, and reuse each fetched expert across concurrent requests. CPU, GPU, FPGA, storage, and control hardware can each do the work they suit.
Can an oversized model run? For 1/2/4/8 independent agents: correct tasks/hour, tokens/s, TTFT and p95 latency, RAM/VRAM, bytes moved, watts, and tasks/kWh—compared with CPU/RAM offload.
Serial staging should win when the model exceeds VRAM, selected experts avoid enough traffic, and batching reuses weights. It may lose when storage is slow, routes touch most experts, KV dominates, or transfer overhead exceeds saved compute.
Conclusion: serial execution is a capacity strategy, not an automatic speedup. It wins when weights do not fit near compute and compression, batching, caching, or route-aware fetch keep transferred bytes useful.
Goal: reuse a small/old GPU to run a model it cannot hold. The bounded prototype passed 45/45 short tasks at 4.17 GiB VRAM; speed and energy describe usability, not success.
All 1.024 GiB of model data near compute, plus KV. Lowest fetch latency; highest scarce-memory demand.
DERIVED CAPACITY FLOOR—not allocator peakForward-only tape: 42.21 MB/layer at batch 1; 25.2% of expert bytes are useful.
MEASURED BYTES; host timing10.65 MB/layer at batch 1, but routes and expert-addressable fetching must come first.
MEASURED COMPARISON LAYOUTAt short context, weights dominate. At long context, KV becomes the next capacity wall.
Loading empirical routing distribution…
Loading repeated timing distribution…
Selection gates 16 MAC cycles, but every blind-stream block still pays fetch and decode. Cycles become seconds only after physical clock measurement.
Direct I/O is a cache-bypass proxy, not a power-on cold read.
Double buffering must earn its thread and queue overhead.
| Batch | All-expert median (min–max) tok/s | Selected-union median (min–max) tok/s | Sample | ||
|---|---|---|---|---|---|
Warm Linux page cache, release scalar Rust, expert sweep only. Host throughput and hardware-shaped cycles are different experiments. Emission and cold-media time are excluded.
+ Lowest latency, wide local bandwidth, mature kernels.
− Entire model consumes VRAM; weights compete with contexts.
+ Tiny residency, simplest controller, no expert seek.
− Wastes 75% of expert traffic for top-8-of-32 at batch 1.
+ Avoids unused bytes and reuses experts across routed tokens.
− Indexed fetch or preprocessing; savings shrink as batches touch most experts.
Capacity: resident floor = 1,099,212,096 model bytes + context × 49,152 F16 KV bytes. Serial substitutes a measured 132,306-byte input path for resident model bytes. Runtime workspaces and activations are additional.
Speed: time ≥ max(bytes ÷ bandwidth, operations ÷ compute rate). Physical FPGA seconds remain unknown until synthesis establishes clock and memory bandwidth.
Memory ledger · Benchmark method · Correctness evidence · HLS4ML analysis · Other streams