coco — an architecture-agnostic LLM engine in Rust
⚠️ Experimental — work in progress. coco is an early-stage research engine under active development. It builds and the core pieces are validated, but APIs and coverage are unstable and it is not production-ready. It's here to learn from and hack on.

Most inference engines carry one big file per architecture. coco reduces every model to a config row plus a single pluggable per-layer seam — so supporting a new architecture is an afternoon, not a rewrite.

★ GitHub — KikoCis/coco ◆ How it works 📖 Blog
Rust · candle · Metal Apache-2.0 data-free turboquant

One seam, many mixers

Modern architectures only really differ in how they mix information between tokens. In coco everything else is generic code written once; the only swappable part is a SequenceMixer — a plug where any attention variant fits. Three fit today, dispatched per layer:

GQA — full attention MLA — latent attention GatedDeltaNet — linear

✓ What works today

  • 4 architectures end-to-end: Qwen2, Qwen3, Qwen3-MoE, DeepSeek V2/V3 (MLA)
  • turboquant: pure-Rust, data-free 3-bit — 3.15× smaller, byte-for-byte identical output
  • Weights stay GPU-resident & compressed; fused Metal kernels decode in-register
  • GatedDeltaNet oracle-validated to 1.8e-7

⚠ Honest gaps

  • Qwen3.5 recognized but not runnable yet (per-layer dispatch WIP)
  • Speed wins are short-context only (prefill still O(n²))
  • MLA runs the candle fallback (no fused kernel yet)
  • Thin test coverage — a real ≥1200-case suite (overflow + security) is planned, not written

turboquant fidelity (Qwen2.5-1.5B, vs fp16)

Compression only counts if the model still behaves. Whole model 2944 → 934 MB (3.15×), and the output is byte-for-byte identical to fp16 on the checked cases.

Try it

git clone https://github.com/KikoCis/coco
cd coco && cargo build --release
cargo run --release --bin gdn_ref # validate the GatedDeltaNet port vs its oracle

Read the story

coco is written up, for humans, on the Bitácora — the design decisions, the wins, and the honest dead-ends, each with interactive demos:

  1. Every new AI model breaks the tooling. I built an engine that shrugs it off.
  2. How I ported a new kind of "attention" to Rust without missing a single decimal
  3. Shrinking an AI model 3× with no data and no quality loss
  4. Why a 3 GB model runs faster than a 14 GB one (and when it stops)
  5. The GPU trick: multiplying straight from compressed data, without unpacking it
  6. I optimized the GPU and it came out 20× slower. The lesson.
  7. When the honest thing is for your program to say "not yet"