Needle 2, the 14MB Agentic Model — I Benchmarked It Against My Own AI Assistant

I run a resident AI assistant called Spark. I built it myself, and I use it every day. I talk to it, ask about the weather, play music, leave memos. When you live with a resident assistant, one question keeps coming back: can we push more of the AI down to the device itself?

Cloud models are powerful, no question. But a resident assistant doesn’t need a frontier model for every utterance.

“What’s the weather?”
“Stop the music.”
“Remind me in 3 minutes.”

Those could run on something much smaller. That line of thinking led me to an interesting model: Needle 2 from Cactus Compute.

45M parameters. A single 14MB binary. 28MB of session RAM. Purpose-built for tool calling and structured extraction. Small and fast — and maybe a fit for Spark’s pre-routing stage.

So I actually ran it. And to cut to the chase: Needle 2 itself is genuinely interesting. But for Spark specifically, I’m passing on it this round. This article covers what I tested and why.

What Needle 2 is

Needle 2 deliberately narrows what it can do. It’s not a general chat model. It does:

Text → structured tool calls
Schema-based data extraction
Grammar-constrained JSON
A confidence score on every output

In other words, it’s not “an AI that chats about anything.” It’s “an AI that reads text and turns it into something a program can consume.”

Give it “What’s the weather in Lagos right now?” and it returns a structured result — city = Lagos — that your code can act on directly.

That design makes a lot of sense for resident assistants and IoT.

Item	Value
Parameters	45M
Binary size	~14MB
Session RAM	~28MB
License	Apache 2.0
Strengths	Tool calling / structured extraction
Chat	Not supported
Target use	Edge, IoT, devices

A 14MB model is a different category from the local LLMs most people are used to.

Test environment

macOS, Apple Silicon
Python 3.14
cactus-needle 2.0.5
Official base model
A normal laptop — no GPU or special setup

The engine is fetched from Hugging Face once on first load, then runs fully local.

Test 1: Setup is almost annoyingly light

The official quickstart is one decorator away:

import needle

@needle.tool
def get_weather(city: str):
    "Get the current weather for a city."
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
print(agent.run("what's it like in Lagos right now?")["results"])

That’s it. Results, run three times:

[run] 0.06s [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]
[run] 0.06s [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]
[run] 0.06s [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

About 0.06 seconds per call after the initial load. And decode speed is genuinely aggressive for a 45M model:

prefill: ~3,001 tokens/s
decode:  ~1,572 tokens/s
peak RAM: ~40.7MB

This is not a “small but slow” model. It’s small → fast → low memory, which is exactly the shape you want at the edge.

Test 2: Argument extraction from a single tool

Next, I checked whether it can pull arguments out of natural language.

what is the weather in Tokyo   → Tokyo
tell me the weather in Osaka   → Osaka

It cleanly lifts the city out of the sentence and packs it into the city argument. This is where Needle 2 shines — it doesn’t need to understand language deeply, it needs to extract a string and map it to a typed field. That’s a job you don’t need a huge LLM for, and the thinking fits Spark well.

Test 3: Choosing between multiple tools

This is the real test for Spark. I gave it a set of tools — MediaPlay, MediaControl, Weather, TimeDate, Greeting — and asked it to pick the right one for each utterance:

OK  want=MediaPlay    got=MediaPlay      play some jazz
OK  want=MediaControl got=MediaControl   stop the music
OK  want=Weather      got=Weather        what is the weather in Tokyo?
NG  want=TimeDate     got=Weather        what time is it?
NG  want=Greeting     got=Weather        hello
OK  want=Weather      got=Weather        what's it like in Lagos right now?

4/6 with 5 tools. Music play, music stop, and weather all route cleanly. But “time” and “greeting” got absorbed into Weather — the model seems to over-index on weather-shaped questions.

Single-tool structuring is strong. Selecting among several candidates is a different, harder problem.

Test 4: What happens when you add more tools

I scaled up to ~15 tools resembling what Spark actually uses — weather, music, OCR, reminders, news, model switching, and so on:

play Takeuchi Mariya          → MediaPlay ✓
play cat videos on youtube    → MediaPlay ✓
stop the music                → MediaPlay ✗
what's the weather in Tokyo   → Weather   ✓
what does the screen say      → ScreenOCR ✓
what time is it               → TimeDate  ✓
remember this note            → Remember  ✓
remind me in 10 minutes       → Remind    ✓
switch the model              → ModelSwitch ✓
any news today                → News      ✓

11/20 across 15 tools. Precision drops as the tool count grows — which is almost expected. Understanding 15 tool definitions and picking the best one is a genuinely hard job for 45M parameters.

Test 5: Then I tried Japanese

This matters most for Spark, because Spark runs in Japanese. I tried natural commands close to real usage:

“スパーク、東京の天気教えて”
“音楽止めて”
“今何時？”

This is where the biggest gap showed up. Needle 2 struggles to handle natural Japanese commands for Spark’s use cases. It often can’t meaningfully parse the input, and using it as a Japanese tool router straight out of the box is not realistic.

That was the most significant finding of the whole test. It’s not that Needle 2 is bad — it’s that this model’s strengths live in an English-centric environment. Benchmarks can look great, but once you put a model into your actual environment, the capabilities you need can be different. That’s an important lesson for anyone integrating local AI.

Test 6: Determinism is genuinely nice

The same input should produce the same output every time. I hammered it five times each:

play jazz                    → MediaPlay ×5
what is the weather in Tokyo → Weather ×5

Deterministic, every run. For a router stage that’s important — generative models drift (“it did this last time, now it’s doing that”), and drift can cause real mistakes. A tool-calling model that’s deterministic is a genuinely interesting property.

Test 7: Structured extraction is its home turf

The other pillar is schema-based extraction. Give it a sentence:

Invoice from Acme Corp, $1,200.00, due 2026-09-01

And get back:

vendor   = Acme Corp
total    = 1200.0
due_date = 2026-09-01

Invoices, forms, logs, sensor data — anything that’s “read text, emit a typed record.” Output is constrained to the schema, so the program side gets clean, well-formed data. This direction is very promising.

The confidence score is also interesting

Needle 2 returns a confidence score with every output.

confidence: 0.98 → let it run
confidence: 0.01 → hand off to a bigger model

This pairs beautifully with a resident assistant:

Wake → Needle → confidence high → Rust executes directly
Wake → Needle → confidence low  → local/cloud LLM

Run a small model always-on, escalate only the hard cases to a big model. I think that’s going to be an important pattern for local AI.

So can Spark adopt it?

At this point you’d say “then just put it in Spark.” And I did think that at first. Spark already has a Rust-based direct executor, so a Wake → Needle → Tool + Args → Rust pipeline would be clean.

But comparing the two surfaced one problem.

The existing Rust router is strong

Spark already has router.rs. It is deterministic, and it is fast. And it’s purpose-built for Spark. The frequent, routable utterances — weather, music, reminders — are already handled in Rust today.

Comparing Rust Router vs. Needle 2:

Needle 2 needs:

Japanese support
Better multi-tool selection
Tuning for Spark’s specific toolset

The Rust router needs: none of those.

And Rust is fast. When I asked “what does adding Needle 2 actually improve?” the test results didn’t give a clear answer.

Result: passing this round

So Spark does not adopt Needle 2 this time. But that doesn’t mean Needle 2 is useless. What the test showed is that Needle 2’s strengths and Spark’s needs don’t quite overlap.

Needle 2 is small, fast, deterministic, great at structured extraction, and interesting at single-tool argument extraction.

Spark needed natural Japanese utterances, multi-tool selection, and Spark-specific intent classification. The existing Rust router wins there.

There’s room to grow it with LoRA

That’s not the end of the story. Needle 2 supports LoRA fine-tuning on a Mac. If I trained it on Spark’s actual utterances:

「天気どう？」  → Weather
「音楽止めて」 → MediaControl
「3分後に教えて」 → Remind

it could plausibly become a Spark-specific micro-router. 45M parameters makes that kind of experiment realistic. But I didn’t go there this round, for a simple reason: the current Rust router isn’t causing any pain. If it isn’t broken, replacing it needs to be a real improvement, not “this seems interesting.”

Verdict

Item	Rating
Size	★★★★★
RAM usage	★★★★★
Inference speed	★★★★★
Single-tool argument extraction	★★★★☆
Structured data extraction	★★★★☆
Determinism	★★★★★
Confidence	★★★★☆
Multi-tool selection	★★☆☆☆
Japanese commands	★☆☆☆☆
Fit for Spark	★★☆☆☆

The model itself is genuinely interesting. As a pre-router for Spark, though, it didn’t meet the bar this round.

Small is why I could try it

When I saw “45M? 14MB? really?” my reaction was to actually run it. And because it’s that small, running it was trivial. The result was a “no” for Spark, but now I know exactly where it works, where it breaks, what it’s good at, what Spark would need from it, and how it differs from the existing Rust router. That alone made the experiment worth it. I plan to keep testing models like this.

Big LLMs aren’t the only AI

Building Spark has reinforced one idea: you don’t have to make one giant model do everything.

Wake
 ↓
lightweight decision
 ↓
Rust executes directly
 ↓
small LLM if needed
 ↓
large LLM if still hard

Divide the work. Don’t send trivial jobs to a giant model. Let an AI handle the judgment that rules can’t, and escalate only what’s genuinely hard. A resident assistant becomes far more efficient that way.

Needle 2 was a very interesting candidate for that stack. Spark just doesn’t have a seat for it yet.

Summary

I tested Needle 2 by actually wiring it toward Spark.

The result: “Small. Fast. Interesting. But not adopting it this time.”

A 14MB binary. ~1,500 tokens/s. Tens of MB of RAM. Deterministic structured output. All genuinely attractive. But the natural-language intent classification in Japanese and multi-tool selection Spark needs didn’t hold up, and the existing Rust router is faster and more reliable in this environment.

Still, this isn’t the end. If Needle 2 strengthens its Japanese and tool-selection capabilities, it’s worth another look. And a Spark-specific LoRA tune is an interesting experiment in its own right.

Most of all: the fact that a model like this runs normally at 14MB is fascinating on its own. AI isn’t only “bigger is better.” There’s real room in the direction of giving small models small jobs. Needle 2 was a great way to confirm that by actually touching it.

Environment summary

Item	Value
Library	cactus-needle 2.0.5
Environment	macOS / Apple Silicon, Python 3.14
First load	a few seconds
Per call	~0.05–0.06s
Decode speed	~1,500–1,600 tokens/s
Peak RAM	~41MB
Result	Not adopted for Spark