Scaling & Load Balancing

Strategies for scaling LeafMesh deployments to handle production workloads.

Architecture for Scale

                Load Balancer
                     │
          ┌──────────┼──────────┐
          │          │          │
     ┌────┴────┐ ┌───┴─────┐ ┌──┴──────┐
     │LeafMesh │ │LeafMesh │ │LeafMesh │
     │ Pod 1   │ │ Pod 2   │ │ Pod 3   │
     └────┬────┘ └───┬─────┘ └──┬──────┘
          │          │          │
          └──────────┼──────────┘
                     │
               ┌─────┴──────┐
               │   Shared    │
               │   Backend   │
               └────────────┘

Horizontal Scaling

Scale by adding instances behind a load balancer. With the scaffolder's managed-container model — App Runner (--provider aws) or Container Apps (--provider azure) via leafmesh-deploy, or extra pods on Kubernetes — you scale by raising the instance/replica count. All instances share one managed Redis.

Running more than one instance requires workers/HA mode. Set the environment variable on every instance:

LEAFMESH_WORKERS_MODE=1

and point every instance at the same Redis. With workers mode on, a multi-instance deployment is safe and coordinated:

Exactly-once events — internal events are delivered to a single instance, not broadcast to all, so an event is handled once across the whole fleet (no duplicate processing).
Single-runner background jobs — run-once work (the agent scheduler, event listeners) is coordinated so only one instance runs each at a time. If that instance dies, another takes over automatically.
Cross-instance session locking — session state is lock-protected across instances, so two instances never mutate the same session concurrently.
Durable human-in-the-loop waits — a session paused waiting on a human survives an instance restart; another instance resumes it when the human responds.

Each instance still loads the same YAML configuration and can serve any session, so no sticky sessions / session affinity is required — a plain round-robin load balancer is fine.

Without LEAFMESH_WORKERS_MODE=1, run a single instance. The default single-instance path broadcasts internal events in-process and runs its own scheduler; with two or more instances on that path you would get duplicate event handling and duplicate scheduled-agent firing. Turn on workers mode before scaling past one instance.

In-Pod Worker Fleet (`LEAFMESH_WORKERS`)

Beyond scaling across instances, you can run multiple coordinated workers inside a single pod with one command. Set:

LEAFMESH_WORKERS=2 python main.py

The process becomes a lightweight supervisor that starts N workers, each on its own API port (base port, base+1, …), all sharing the same Redis or Postgres. Every worker runs the proven high-availability machinery, so there are no duplicates inside the pod — internal events are delivered to exactly one worker, and singleton background jobs (the scheduler, the manager) run on just one. Each worker behaves exactly like a lone single server.

This gives you redundancy within a pod: if a worker crashes, the supervisor respawns it (with backoff and a crash-loop ceiling) and a sibling reclaims its in-flight work in the meantime. Kubernetes still handles pod-level scale (replicas, one Service port per pod → worker-0's base port); LEAFMESH_WORKERS handles crash redundancy under one pod.

Notes:

The evolution/health API stays a singleton across the fleet — exactly one worker serves it at any time. No per-worker override needed.
LEAFMESH_WORKERS (in-pod fleet) and LEAFMESH_WORKERS_MODE=1 (cross-instance HA) compose: the fleet supervisor stamps workers-mode on each child automatically.

Redis Scaling

Single Redis → Redis Cluster

From 2.4.482 the store is set only in .env, never in the YAML, so moving to a cluster is an environment change. Setting REDIS_CLUSTER_NODES is what turns cluster mode on:

# .env (development)
LEAFMESH_AGENCY_NAME=my-agency
LEAFMESH_STORE=redis
REDIS_HOST=localhost
REDIS_PORT=6379

# .env (production)
LEAFMESH_AGENCY_NAME=my-agency
LEAFMESH_STORE=redis
REDIS_CLUSTER_NODES=redis-1:6379,redis-2:6379,redis-3:6379
REDIS_PASSWORD=your-password

TLS (REDIS_SSL and related variables) and Postgres (POSTGRES_*, including POSTGRES_POOL_MIN / POSTGRES_POOL_MAX for the connection pool) are set the same way; the full list is on Persistence. Studio shows whether the store is reachable under Infrastructure → Datastores, which also shows its latency, the server, the settings in force and which .env names set them — never the address or password.

Redis Performance Tuning

Connection pooling: LeafMesh maintains a connection pool to the backend
Pipeline commands: Batch Redis operations where possible
Key expiration: TTLs prevent unbounded storage growth
Cluster sharding: Data distributed across nodes by key hash

LLM Provider Scaling

Distribute agents across providers to avoid rate limits:

agents:
  # Spread across providers
  triage_1:
    name: "triage_1"
    model: "gpt-6-luna"          # OpenAI
  triage_2:
    name: "triage_2"
    model: "gemini-3.6-flash"    # Google
  triage_3:
    name: "triage_3"
    model: "deepseek-flash"      # DeepSeek
  specialist:
    name: "specialist"
    model: "claude-sonnet-5-5"   # Anthropic

Each provider has independent rate limits, increasing total throughput.

Performance Characteristics

Component	Latency	Scaling Factor
LeafMesh routing overhead	10-50ms	Per-instance
Backend operations	1-10ms	Scales with cluster
LLM calls	500ms-5s	Scales with providers
Event processing	Async	Per-instance

Bottleneck Analysis

Bottleneck	Symptom	Solution
Backend	High latency in state operations	Redis Cluster
LLM rate limits	429 errors from providers	Multi-provider, caching
CPU per instance	Event loop blocking	Add instances (workers mode + shared Redis)
Network	Slow external API calls	Pre-compose caching

Monitoring at Scale

Aggregated analytics, cache metrics, and per-session event streams are available in Studio's Dashboard and session inspector — operator UI, not part of the customer Python or HTTP surface.

Monitor backend health and overall throughput via the Studio dashboard. If end-to-end latency rises while per-instance CPU stays low, add more instances — the shared backend handles state coordination automatically.

Current Limitations

No built-in load balancer — front the instances with your platform's load balancer (App Runner / Container Apps do this for you; on Kubernetes use a Service + Ingress).
Multi-instance requires LEAFMESH_WORKERS_MODE=1 and a shared Redis; without it, run a single instance.
The shared Redis is the state bottleneck for extremely high-volume deployments — move to Redis Cluster before it saturates.

Next Steps

Kubernetes Deployment — Auto-scaling with K8s
Docker Deployment — Container deployment
Runtime Optimization — Performance tuning

LeafMesh — Scaling for production workloads