Observations: Most AI Agents Don't Need General Intelligence

In the rush to adopt AI, most teams reach for large language models (LLMs) like GPT-4 or Claude as the default solution. It makes sense - LLMs are flexible, powerful, and broadly capable. But there's a quiet cost to this approach: high latency, opaque decision-making, spiraling API bills, and dependence on external cloud infrastructure.

The truth is, most AI agents don’t need to reason broadly. They need to operate consistently within narrow, well-defined tasks-think classification, information extraction, form filling, UI navigation, customer support triage.

This is where small language models (SLMs) shine.

A recent Stanford and Carnegie Mellon paper - "Small Language Models are the Future of Agentic AI"-backs this up with real data. SLMs (≤10B parameters) are:

• 2-5× faster at inference
• Up to 10-100× cheaper per token
• Fine-tuneable on consumer-grade GPUs
• Easier to control, audit, and deploy at the edge

So why aren’t they the default yet?

Analysis: Matching Tool to Task

The paper outlines a clear heuristic: match model size to task ambiguity.

• Use SLMs for structured, repetitive, and bounded tasks.
• Use LLMs for open-ended, exploratory, or ambiguous queries.

Examples:

In real-world terms, 80-90% of queries hitting production agents fall into the first category. Yet we often handle them using a commercial LLM that costs $10-$30 per million tokens, when a fine-tuned open-source SLM could deliver comparable results for as little as $0.001-$0.005 per million tokens.

Add to that: the ability to run SLMs on-device, with sub-500ms latency and zero data egress, and the architectural shift becomes obvious.

Options: How Others Are Solving This

‍

Apple’s Foundation Models prove this at scale. Their AI system uses:

• 3B parameter SLMs running on-device for summarization, writing assistance, command execution.
• A private fallback LLM hosted on Apple Silicon infrastructure for high-complexity tasks.

This mirrors the Stanford framework exactly: an SLM-first architecture, with an LLM safety net.

Their rationale?

• Keep data local (privacy)
• Reduce cost per interaction (scale)
• Improve responsiveness (UX)

In parallel, the Stanford team proposes a “LLM-to-SLM conversion algorithm”:

1. Log LLM outputs across key tasks.
2. Fine-tune a candidate SLM on that data.
3. Validate and gradually replace LLM calls.
4. Use fallback escalation only when confidence is low.

Suggestions for Implementation

If you're building or running agentic AI systems today, consider:

• Audit high-frequency use cases: Which tasks are deterministic and repetitive?
• Estimate per-token costs: Calculate total spend for those tasks under your current LLM contract.
• Select an open-source base model: Mistral, Phi-3, TinyLlama are viable starting points.
• Use synthetic data or LLM outputs to fine-tune an SLM with QLoRA or similar methods.
• Route traffic based on complexity: Use a simple controller to invoke LLMs only when SLMs fail confidence thresholds.
• Track fallback patterns to further refine or segment capabilities.

These aren’t theoretical improvements. In practice, teams see 50-90% cost reduction while maintaining accuracy and improving latency.

Conclusion: Smaller Models, Smarter Decisions

SLMs aren’t just a cost-saving trick. They enable better system design:

• Faster inference, especially at the edge
• Lower ongoing costs at scale
• More predictable behavior in high-volume workflows
• Easier privacy and compliance guarantees

There are trade-offs. SLMs require investment in task-specific fine-tuning. You need routing logic to manage fallbacks. And you still need LLMs for creative or ambiguous work.

But for any team building AI agents that serve repeatable functions, the ROI math is now clear:

• Fewer hallucinations
• Faster answers
• Lower bills

Don’t just scale AI up. Scale it smart.

Prompt for reflection:

What’s one agent in your system that you’re currently running on GPT-4 that could be just as effective (and 90% cheaper) on a fine-tuned SLM?

Further reading:

• Small Language Models are the Future of Agentic AI – Stanford/CMU (2024)
• Introducing Apple Foundation Models – Apple Machine Learning Research (2024)