Bonsai 2 27B: The 5.9 GB Model That Retains 98.2% of Full-Precision Performance

PrismML's Bonsai 2 27B compresses a 27B multimodal model to 5.9 GB while retaining 98.2% of full-precision benchmark performance — a genuine engineering milestone for local-first AI deployment.

Common Visualization of Artificial Neural Network with Chip
Feature image: mikemacmarketing / original posted on flickr Liam Huang / clipped and posted on flickr / CC BY 2.0 / Wikimedia Commons.

Why This Matters

For two years, the AI industry has operated under an implicit contract: if you want frontier-class reasoning, coding, and multimodal capability, you need cloud API access or a multi-GPU server with hundreds of gigabytes of VRAM. PrismML's Bonsai 2 27B — released September 17, 2026 — breaks that contract.

At 5.9 GB on disk, it is more than 9× smaller than its full-precision parent (Qwen3.8 27B at 53.8 GB in FP16), yet retains 98.2% of the parent's aggregate benchmark performance across 20 tests spanning reasoning, math, coding, instruction following, vision, and agentic tool use. That is an improvement from the first Bonsai release, which retained roughly 95% — a gap that PrismML itself acknowledged was material for agentic workflows where small errors compound across steps.

This article examines what makes Bonsai 2 27B technically interesting, how its ternary compression works, what the benchmarks actually show, and where the model does and does not make sense to deploy today.


What Is Bonsai 2 27B?

Bonsai 2 27B is a ternary-weight version of Qwen3.8 27B, a hybrid-attention causal language model from Alibaba. The model has 27.36 billion total parameters: 24.35B for the language backbone, 0.47B for the vision tower, and 2.54B for embeddings and the LM head.

Unlike conventional quantization schemes that round weights to the nearest value in a fixed grid (e.g., 4-bit or 6-bit), ternary compression maps every weight to one of exactly three values: −1, 0, or +1. The trick is that these ternary values live in a fixed rotated basis (blockwise Hadamard rotation) and are paired with FP16 group-wise scaling factors, yielding an effective 1.76 bits per weight across the entire model.

The result: a 5.93 GB GGUF file that fits comfortably on a single consumer GPU, a Mac with unified memory, or even — in principle — a smartphone.

Key Specifications

Property Value
Parameters 27.36B total (24.35B language + 0.47B vision + 2.54B embedding/LM head)
Base model Qwen3.8 27B (hybrid attention, SwiGLU MLP, RoPE, RMSNorm)
Weight format Ternary g128 with FP16 group-wise scaling, blockwise Hadamard rotation
Effective bit width 1.72 bpw true ternary; 1.76 bpw as shipped (PTQ1_0)
Max context 262,144 tokens
Modalities Text + image in → text out (multimodal)
License Apache 2.0
File size 5.93 GB (PTQ1_0), 7.25 GB (PQ2_0)

How Ternary Compression Works — And Why It Is Not Naive Rounding

A naive approach to quantizing a model below 4 bits per weight typically degrades sharply. The reason is that floating-point weights have a distribution: most cluster near zero, but a small number are large outliers. Simply rounding all of them to the nearest value in a coarse grid destroys the signal carried by both the cluster and the outliers simultaneously.

PrismML's approach is more sophisticated:

  1. Rotation. The weight matrix is first rotated into a quantization-friendly basis using a blockwise Hadamard transform. This spreads the information more uniformly across all dimensions, reducing the variance within each quantization group.
  2. Ternarization. Each group of weights is mapped to one of {−1, 0, +1} based on its sign and magnitude relative to the group's scaling factor. The 0 class captures weights that are close enough to zero that they contribute negligibly — a form of structured sparsity.
  3. Error compensation. Per-group FP16 scaling factors absorb the residual error. Because the scales are stored at FP16 precision (not quantized), they provide a high-fidelity correction term that preserves the model's representational capacity far beyond what pure ternary arithmetic would allow.

The compression pipeline proceeds through three stages:

flowchart LR
  A[FP16 Weights] --> B[Blockwise Hadamard Rotation]
  B --> C[Ternarize to -1, 0, +1]
  C --> D[FP16 Scaling Factors]
  D --> E[Compressed GGUF 5.93 GB]

The net effect is a model where 98.2% of capability is preserved despite storing each weight at less than 2 bits. For comparison, a standard 4-bit GGUF quant of Qwen3.8 27B would occupy roughly 14 GB — still small by cloud standards, but more than twice the size of Bonsai 2.


Benchmarks: The Numbers

PrismML evaluated Ternary Bonsai 2 27B across a 20-benchmark thinking-mode suite spanning:

  • Reasoning and math: GSM8K, MATH, AIME, GPQA
  • Coding: SWE-bench Verified, LiveCodeBench, HumanEval+MBPP
  • Instruction following and alignment: IFEval, Arena-Hard
  • Vision: MMMU, MathVista
  • Agentic tool use: Multi-Agent Arena

The model achieves an aggregate score of 83.9 out of the parent's 85.4 — a retention rate of 98.2%. For context, the first Bonsai (released July 2026) retained ~95%, meaning PrismML closed roughly half the quality gap in just two months while simultaneously moving to a stronger base model (Qwen3.8 instead of Qwen3.6).

Throughput

Hardware Tokens/second (reported)
NVIDIA RTX 5090 Up to 143 tok/s
Apple M5 Max 46.8 tok/s

Independent testers reported a wider range — 7–44 tok/s on consumer hardware — and at least one tester observed the model entering a repetition loop under certain prompts. These are not dealbreakers, but they are important context: reported numbers come from PrismML's own infrastructure (including their custom llama.cpp fork with optimized kernels), and real-world throughput depends heavily on prompt length, KV cache management, and system memory bandwidth.


The Tooling Reality: Proprietary Fork Required

This is the most significant practical caveat for deployment today. Bonsai 2 27B does not run on stock llama.cpp. The rotated weight basis requires a runtime Walsh–Hadamard transform that PrismML has implemented in their fork but has not yet upstreamed.

On a stock build, attempting to load PTQ1_0 or PQ2_0 weights is refused outright. Loading the deprecated Q2_0 variant silently succeeds but produces gibberish output — a dangerous failure mode that could mislead an unwary user into thinking the model works when it does not.

The available runtimes are:

Format Repository Size Runtime
GGUF PTQ1_0 (1.76 bpw) prism-ml/Ternary-Bonsai-2-27B-gguf 5.93 GB PrismML llama.cpp fork (prism-b10658+)
GGUF PQ2_0 (2.16 bpw) Same repo 7.25 GB Same fork; simpler 2-bit unpacking, faster prompt processing
MLX (2.25 bpw) prism-ml/Ternary-Bonsai-2-27B-mlx-2bit 8.49 GB Stock MLX (no fork needed)

The MLX pack is notable: it runs on stock MLX without any PrismML fork, making it the easiest path for Mac users. The vision tower in this variant is included at full precision (hence the larger 8.49 GB size), whereas the GGUF variants ship a separate HQQ 4-bit mmproj (0.63 GB) or BF16 reference (0.93 GB).


Who Should Use Bonsai 2 27B?

✅ Good fit

  • Local-first developers who need reasoning and coding capability without API dependency or data egress.
  • Small teams and solo builders shipping products under Apache 2.0 with no commercial negotiation required.
  • Edge and IoT scenarios where a 5.9 GB model is the largest footprint that fits on available hardware.
  • Research into low-bit models, where Bonsai 2 demonstrates that ternary compression at this scale can approach practical utility.

⚠️ Use with caution

  • Agentic tool use and long-horizon workflows. PrismML itself acknowledges that the 2% performance gap compounds in multi-step agentic tasks. If your agent calls tools, reasons across steps, and writes code, the small error rate may manifest as intermittent failures that are difficult to debug.
  • Production deployments requiring deterministic output. The repetition loops observed by independent testers suggest the model can enter degenerate generation modes under certain conditions.

❌ Not yet a replacement for

  • Cloud-grade reasoning. Frontier models on API (GPT-4o, Claude Opus, Gemini Ultra) still outperform Bonsai 2 on the hardest reasoning and coding benchmarks. The 98.2% retention figure is aggregate — individual benchmarks show larger gaps, particularly in math and complex coding.
  • High-throughput serving. Without PrismML's optimized fork and specialized hardware, throughput on consumer hardware may not meet production SLAs.

The Bigger Picture: Where Compression Is Heading

Bonsai 2 27B arrives at a moment when local AI is transitioning from novelty to necessity. Several factors are converging:

  1. Privacy and compliance. GDPR, HIPAA, and sector-specific regulations make cloud API access impractical for sensitive workloads. A 5.9 GB model that runs locally eliminates data egress entirely.
  2. Cost. At 11 million+ downloads of the original Bonsai (per PrismML), there is clear market demand for free, self-hosted AI. Cloud API costs scale with usage; a local model has zero marginal cost per inference.
  3. Latency and reliability. Local inference has no network dependency, no rate limits, and no outage risk from upstream providers.
  4. Vendor momentum. Multiverse Computing in Spain is working the same problem. Rumors of discussions with Apple (unconfirmed) suggest silicon-level integration may be on the horizon.

Ion Stoica, advisor to PrismML and UC Berkeley professor, captured the significance well: "AI deployment has historically forced a choice between model quality and the resources required to run it. What is compelling about Bonsai 2 27B is how little capability is lost despite such a dramatic reduction in footprint."


Getting Started

The quickest path to running Bonsai 2 27B:

# Clone PrismML's demo repo
git clone https://github.com/prism-ml/bonsai-demo.git
cd bonsai-demo

# Start the OpenAI-compatible API server + chat UI
./scripts/start_llama_server.sh
# → Available at http://localhost:8080

Or generate directly with MLX on a Mac:

mlx_lm.generate --model prism-ml/Ternary-Bonsai-2-27B-mlx-2bit \
  --prompt "Explain KV cache growth in one paragraph."

The model is available on Hugging Face under Apache 2.0, and also accessible via API providers like OpenRouter for teams that want to try it before committing to self-hosting.


Final Thoughts

Bonsai 2 27B is not a silver bullet. The 2% performance gap matters in production agentic workflows, the proprietary runtime fork limits immediate accessibility, and independent throughput numbers are lower than PrismML's marketing materials suggest.

But 98.2% retention at 5.9 GB is a genuine engineering achievement. It demonstrates that ternary compression, when done with rotation and error compensation rather than naive rounding, can close the quality gap to a point where the remaining 2% is often invisible in practice. For local-first developers, small teams, and privacy-conscious deployments, Bonsai 2 27B is currently the best option available for running a 27B-class multimodal model on consumer hardware.

The real question is not whether Bonsai 2 27B is good enough today — it clearly is for many workloads — but whether the compression gap will continue closing at this rate. If it does, the distinction between "local" and "cloud" models may become increasingly academic.


Sources: PrismML announcement, technical documentation, explainx.ai analysis, TechCrunch coverage, Hugging Face

Feature image: mikemacmarketing / original posted on flickr Liam Huang / clipped and posted on flickr / CC BY 2.0 / Wikimedia Commons.