LLM Response Caching

LeafMesh includes a response cache that can reduce API costs by 40-70% by serving identical requests from cache instead of making new LLM calls.

How It Works

When an agent makes an LLM call, LeafMesh checks whether an identical request has already been served. On a hit, the model's text is reused and no LLM API call is made; otherwise the LLM is called normally and its text is stored for future use.

What is (and is not) cached — the safety contract (2.4.107)

The cache boundary is the LLM call only — the one pure step of an agent's pipeline:

Reused on a hit: the model's text. Nothing else.
Always runs fresh, hit or miss: your agent function, @chain, @compose, dispatches, audit writes — every side effect re-executes on every request. A cached turn can never replay an old dispatch result or skip an audit entry.
Never cached: failed or empty responses (a transient error can't become "permanent for an hour").
Only answers that are safe to reuse (2.4.487+): the cache decides on its own, with nothing to opt in. An answer is stored only when both hold:
- every tool the turn used changes nothing: a read-only built-in tool, an MCP tool marked readOnlyHint: true, or a tool you declare with changes_nothing: @tool(..., changes_nothing=True) or @global_tool(..., changes_nothing=True) (2.4.491+), changes_nothing: true on a zapier, composio or n8n block (2.4.491+), or a BaseTool subclass that sets CHANGES_NOTHING = True. The default is false: a tool that does not say so is treated as changing something. A turn that used any other tool is never stored;
- the answer holds no personal data and no secret.
Visible: hits carry _llm_text_from_cache: true and _cached_at on the result, and cache_hit / cache_age_s attributes on the LLM span — a replayed text can never masquerade as a live call.
The cache key is computed after @pre_compose, so input shaping (including per-session or per-time salting) varies the key exactly as you'd expect.

Configuration

Enable caching in your project's YAML config under the cache section:

cache:
  enabled: true
  ttl: 3600          # Cache lifetime in seconds (default: 3600 = 1 hour)

TTL (Time to Live)

The ttl setting controls how long cached responses are kept before they expire automatically. Choose a TTL based on how frequently your data changes:

TTL	Use Case
`300` (5 min)	Real-time data that changes frequently
`3600` (1 hour)	General-purpose caching (default)
`86400` (24 hours)	Stable data, reference lookups
`604800` (7 days)	Static content, rarely changing knowledge

# Short TTL for time-sensitive agents
cache:
  enabled: true
  ttl: 300

# Long TTL for agents that process stable reference data
cache:
  enabled: true
  ttl: 86400

Deduplication

The cache automatically deduplicates identical requests, regardless of timestamps, session IDs, or other metadata. Requests that differ in model, prompt content, sampling parameters, or available tools are treated as distinct entries.

This means:

Same user asking the same question twice -- second call is served from cache
Different users asking the same question -- second call is served from cache
Same prompt but different sampling settings -- treated as separate requests
Same prompt but different tools available -- treated as separate requests

Checking Cache Statistics

Use Studio's Dashboard to see how the cache is performing:

stats = await Studio's Dashboard (HTTP)
print(stats)

Returns:

{
  "hits": 142,
  "misses": 58,
  "stores": 58,
  "evictions": 3,
  "total_lookups": 200,
  "hit_rate": 0.71,
  "ttl_seconds": 3600
}

Field	Description
`hits`	Number of requests served from cache
`misses`	Number of requests that required an LLM API call
`stores`	Number of responses written to cache
`evictions`	Number of entries manually invalidated
`total_lookups`	Total cache lookups (`hits + misses`)
`hit_rate`	Cache hit rate as a decimal (0.0 to 1.0)
`ttl_seconds`	Current TTL setting

A hit_rate above 0.5 typically indicates significant cost savings. Monitor this value to tune your TTL.

When Caching Helps

Caching provides the most value when:

Users ask similar questions repeatedly -- common in customer support, FAQ bots, and help desk agents
Multiple agents process the same data -- shared reference lookups, common classifications
Development and testing -- avoid burning API credits on the same test prompts during iteration
High-traffic applications -- reduce latency and cost for popular queries
Batch processing -- when processing items that share common sub-queries

Example: Customer Support

A support agent that handles "What are your business hours?" dozens of times per day benefits enormously from caching. The first request calls the LLM; subsequent identical requests return instantly from cache.

agents:
  faq_agent:
    name: "faq_agent"
    model: "gpt-4o-mini"
    prompt: |
      You answer frequently asked questions about our company.

cache:
  enabled: true
  ttl: 86400    # FAQ answers are stable for 24 hours

When to Disable Caching

Disable caching per agent with the cache field (2.4.107):

creative_agent:
  cache: false     # this agent always gets a fresh model call

Disable caching when responses must be unique every time:

Creative content generation -- each response should be different (stories, brainstorming)
Agents with high temperature settings -- you want the variability that temperature provides
Time-sensitive operations -- when the answer depends on the current moment (though ttl: 60 may be preferable to disabling entirely)
Personalized responses -- when each user's context makes every request unique (cache misses will dominate anyway)

cache:
  enabled: false

Next Steps

LLM Providers --- Supported models and provider configuration
Adaptive Execution --- Automatic model selection and optimization
Streaming --- Real-time streaming responses

LeafMesh --- Reduce LLM costs with intelligent response caching