LLM Providers

LeafMesh provides a unified interface across multiple LLM providers. Specify any supported model in the model: field of an agent and the platform routes the request to the correct provider automatically. No provider-specific code is needed.

Three Ways to Reach a Model

Think of it like paying for electricity: you can have an account with each power company, use the supply that already comes with your building, or go through one reseller who bills you for all of them.

Each vendor with its own key (the default). You hold an account with OpenAI, Anthropic, Google, Moonshot and so on, and save each one's key. Calls go straight to that vendor and you pay them directly. Nothing sits in between.
Your own cloud. If your company already buys models through AWS, Google Cloud or Azure, route the agent through Amazon Bedrock (bedrock/…), Google Vertex AI (vertex/…) or Microsoft Foundry (foundry/…). Billing, network rules and access control stay in the cloud account you already have.
OpenRouter, one key for the long tail (optional). OpenRouter hosts hundreds of models from many labs behind a single key. Write the model as openrouter/<lab>/<model>. It is handy for trying a model you do not have an account for, and each answer comes back with what that call really cost, so the cost you see is the amount OpenRouter bills.

You can mix all three in one mesh: each agent picks its own model.

Connecting a provider

Every provider connects one of two ways:

Infrastructure page: in Studio, under Infrastructure → Models, add the provider and paste its key (some also take an optional Endpoint, for another region or a proxy). Press Test to check the key before you rely on it; Test works for every model provider. A saved key reaches the running mesh without a restart.
Environment variable: set the provider's key variable (listed below) in your .env or platform settings. A key set in the environment wins over one saved on the Infrastructure page.

Several accounts with one provider (`model_connection`)

2.4.486+. An LLM agent can name the vault record its model calls sign in with, so different agents can use different accounts of the same provider:

agents:
  writer:
    agent_type: llm
    model: gpt-4o-mini
    model_connection: openai-dev   # the vault record its model calls sign in with

Set on the agent, for LLM agents only (not mesh.models).
Unset: the provider's own record, as before.
If the record is missing, holds no key, or is for another provider (for example an Anthropic record on a GPT model), that model call fails with a plain message naming the record. It never falls back to another key.
For Bedrock, Vertex and Foundry, the record is resolved as that cloud's sign-in.

Supported Providers

Labs with their own API — the model name alone picks the provider:

Provider	Example Models	Key variable
OpenAI	gpt-6-astra, gpt-6.1-sol, gpt-6.1-terra, gpt-6-luna, gpt-5.6-terra, gpt-5.4-mini, gpt-4.1, gpt-4o-mini, o3, o4-mini	`OPENAI_API_KEY`
Anthropic	claude-fable-5-1, claude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5, claude-opus-4-8, claude-sonnet-4-6	`ANTHROPIC_API_KEY`
Google Gemini	gemini-4-argon, gemini-omni-1.2-flash, gemini-3.8-flash, gemini-3.6-flash, gemini-3.5-flash-lite, gemini-3.1-pro-preview, gemini-2.5-pro	`GOOGLE_API_KEY` (or `GEMINI_API_KEY`)
DeepSeek	deepseek-v4-pro, deepseek-flash	`DEEPSEEK_API_KEY`
Mistral	mistral-large-4-0, mistral-medium-3-5-26-04, mistral-small-4-0-26-03, codestral-2508	`MISTRAL_API_KEY`
xAI Grok	grok-4.7, grok-4.6, grok-4.3, grok-4.20-0309-reasoning, grok-build-0.1	`XAI_API_KEY` (or `GROK_API_KEY`)

Your own cloud — prefix the model with the cloud:

Provider	Example Models	Sign-in
Amazon Bedrock	bedrock/claude-sonnet-4-20250514, bedrock/llama3.1-70b, bedrock/mistral-large	Your AWS credentials
Google Vertex AI	vertex/gemini-2.5-flash, vertex/claude-sonnet-4-20250514, vertex/mistral-large	Your Google Cloud credentials
Microsoft Foundry	`foundry/<deployment name>`, e.g. foundry/gpt-4o, foundry/DeepSeek-R1	`AZURE_FOUNDRY_API_KEY`, or a service principal / managed identity
Local / self-hosted	llama3.1, mistral, codellama (or any model served by your own model server — see `mesh.local` below)	Usually none; `LOCAL_MODEL_API_KEY` if your server asks

Model vendors, one key each — see Other Model Vendors for the full list (Moonshot Kimi, Alibaba Qwen, Z.AI GLM, MiniMax, Groq, Cerebras, OpenRouter and ten more).

The tables above are an illustrative baseline. The live, authoritative list of selectable models, providers, connectors, and channels for your deployment is served by the SDK's GET /api/registry endpoint: it discovers each configured provider's current models live and merges them with a curated baseline, so newly released models appear without a LeafMesh upgrade. Query it (optionally with ?category=models) to see exactly what your running mesh can select. Older identifiers such as gpt-4-turbo, deepseek-chat, grok-4, or gemini-2.0-flash still work as free-text model names if a provider still serves them, but the model picker no longer offers them.

Provider Setup

Each section shows the key variable to set. Saving the same key on the Infrastructure page works just as well.

OpenAI

agents:
  openai_agent:
    name: "openai_agent"
    model: "gpt-6.1-sol"         # Strong all-rounder
    # model: "gpt-6-astra"       # Most capable OpenAI model
    # model: "gpt-6-luna"        # Fast and cheap
    # model: "gpt-5.6-terra"     # Balanced
    # model: "gpt-5.4-mini"      # Cost-effective option
    # model: "gpt-4.1"           # GPT-4.1
    # model: "gpt-4o-mini"       # Long-standing low-cost choice
    # model: "o3"                # Reasoning model
    # model: "o4-mini"           # Small reasoning model

GPT-6, GPT-5.6 and the -pro models use OpenAI's newer Responses API to call tools. LeafMesh switches to it automatically for those models, so the agent config is the same as for any other OpenAI model.

Environment variables:

OPENAI_API_KEY=sk-your-openai-api-key-here
OPENAI_BASE_URL=https://api.openai.com/v1  # Optional: custom endpoint

Anthropic

agents:
  claude_agent:
    name: "claude_agent"
    model: "claude-sonnet-5-5"   # Balanced flagship
    # model: "claude-fable-5-1"  # Most capable Claude
    # model: "claude-opus-5-5"   # Deep reasoning and long tasks
    # model: "claude-haiku-5-5"  # Fast, cost-effective
    # model: "claude-sonnet-4-6" # Previous generation, still offered

Environment variables:

ANTHROPIC_API_KEY=sk-ant-your-anthropic-api-key

Google Gemini

agents:
  gemini_agent:
    name: "gemini_agent"
    model: "gemini-4-argon"            # Most capable Gemini
    # model: "gemini-3.6-flash"        # Fast Gemini model
    # model: "gemini-3.5-flash-lite"   # Cost-effective Gemini
    # model: "gemini-2.5-pro"          # Gemini 2.5 Pro

Environment variables:

GOOGLE_API_KEY=your-google-api-key   # GEMINI_API_KEY also works

The key is a Gemini API key from Google AI Studio.

DeepSeek

agents:
  deepseek_agent:
    name: "deepseek_agent"
    model: "deepseek-flash"      # Fast and very cheap
    # model: "deepseek-v4-pro"   # DeepSeek's strongest model

Environment variables:

DEEPSEEK_API_KEY=your-deepseek-api-key
DEEPSEEK_BASE_URL=https://api.deepseek.com   # Optional: custom endpoint

DeepSeek charges by the time of day: calls in its busy hours (01:00–04:00 and 06:00–10:00 UTC on weekdays) cost twice the off-peak price. LeafMesh prices each call by when it was made. The older deepseek-chat and deepseek-reasoner ids are retired.

Mistral

agents:
  mistral_agent:
    name: "mistral_agent"
    model: "mistral-large-4-0"              # Flagship
    # model: "mistral-medium-3-5-26-04"     # Balanced
    # model: "mistral-small-4-0-26-03"      # Cost-effective
    # model: "codestral-2508"               # Code-focused

Environment variables:

MISTRAL_API_KEY=your-mistral-api-key
MISTRAL_BASE_URL=https://api.mistral.ai/v1   # Optional: custom endpoint

xAI Grok

agents:
  grok_agent:
    name: "grok_agent"
    model: "grok-4.7"                        # Flagship
    # model: "grok-4.3"                      # Faster, cheaper
    # model: "grok-4.20-0309-non-reasoning"  # Quick answers, no reasoning
    # model: "grok-build-0.1"                # Lowest-cost Grok

Environment variables:

XAI_API_KEY=your-xai-api-key                 # GROK_API_KEY also works
XAI_BASE_URL=https://api.x.ai/v1             # Optional: custom endpoint

Grok 4.5, 4.6 and 4.7 take a reasoning effort (low, medium, high; 4.6 and 4.7 also xhigh) when reasoning: true is set.

Other Model Vendors

Seventeen more vendors connect the same way: one key each, saved on the Infrastructure page or set as an environment variable. On the Infrastructure page each one also takes an optional Endpoint, only needed for another region or a proxy (the environment variable <NAME>_BASE_URL, e.g. QWEN_BASE_URL, does the same).

There are two kinds:

A vendor that serves its own models is picked by the model name, just like OpenAI or Anthropic: kimi-k3 goes to Moonshot, glm-5.3 goes to Z.AI.
A vendor that hosts other labs' models (OpenRouter, Groq, Cerebras) is picked by a prefix: openrouter/…, groq/…, cerebras/…. LeafMesh removes the prefix before sending, so the vendor gets the id it expects. The prefix is what says which host to use, since the same open model is often served by several.

Vendor	Model ids that route to it	Example	Key variable
OpenRouter	`openrouter/<lab>/<model>`	`openrouter/moonshotai/kimi-k3`	`OPENROUTER_API_KEY`
Groq	`groq/<model>`	`groq/openai/gpt-oss-120b`	`GROQ_API_KEY`
Cerebras	`cerebras/<model>`	`cerebras/<model id from Cerebras>`	`CEREBRAS_API_KEY`
Moonshot Kimi	`kimi-…`, `moonshot-…`	`kimi-k3`, `kimi-k3.1-pro`, `kimi-k3.1-pro-highspeed`	`MOONSHOT_API_KEY` (or `KIMI_API_KEY`)
Alibaba Qwen	`qwen…`	`qwen3.8-max`, `qwen3.7-plus`, `qwen3.7-flash`	`DASHSCOPE_API_KEY` (or `QWEN_API_KEY`)
Z.AI GLM	`glm-…`	`glm-5.3`, `glm-5.3-flash`	`ZAI_API_KEY` (or `GLM_API_KEY`)
MiniMax	`minimax-…`	`minimax-m3`, `minimax-m3.1-flash`	`MINIMAX_API_KEY`
Inception (Mercury)	`mercury-…`	`mercury-3`	`INCEPTION_API_KEY`
Meta (Muse)	`muse-…`	`muse-spark-1.3`, `muse-wonder-1.0`	`META_API_KEY` (or `MODEL_API_KEY`)
StepFun	`step-…`	`step-5-preview`, `step-3.7-flash`	`STEPFUN_API_KEY` (or `STEP_API_KEY`)
Arcee	`trinity-…`, `arcee-…`	`trinity-large-thinking`	`ARCEE_API_KEY` (or `ARCEEAI_API_KEY`)
Tencent Hunyuan	`hy3…`, `hy4…`, `hunyuan-…`	—	`HUNYUAN_API_KEY`
Xiaomi MiMo	`mimo-…`	—	`MIMO_API_KEY`
Upstage (Solar)	`solar-…`	—	`UPSTAGE_API_KEY`
Poolside (Laguna)	`poolside/…` (sent exactly as written)	`poolside/laguna-s-2.1`	`POOLSIDE_API_KEY`
Reka	`reka-…`	—	`REKA_API_KEY`
Cohere	`command-…`	—	`CO_API_KEY` (or `COHERE_API_KEY`)

Where the Example column is empty, use a model id from that vendor's own model list; with the key saved, the vendor's current models also appear in Studio's model picker. Poolside's ids already start with poolside/, and that is part of the real id, so it is sent unchanged.

agents:
  researcher:
    name: "researcher"
    model: "kimi-k3"                          # Moonshot, with your Moonshot key

  summarizer:
    name: "summarizer"
    model: "groq/openai/gpt-oss-120b"         # gpt-oss hosted on Groq

  explorer:
    name: "explorer"
    model: "openrouter/moonshotai/kimi-k3"    # the same Kimi model, through OpenRouter

Things LeafMesh handles for you:

Alibaba Qwen keys work only in the region they were created in. The default address is Alibaba's international (Singapore) endpoint; set the connection's Endpoint (or QWEN_BASE_URL) to use another region.
Xiaomi MiMo and Tencent Hunyuan only let the model decide for itself which tool to call. A config that pins a tool still works there; the model is simply left to choose.
Cohere rejects one of the usual request settings (the parallel tool calls switch), so LeafMesh leaves it out.
Groq does not accept a temperature of exactly 0; LeafMesh sends the smallest value it does accept.
A lab's own rules follow its model wherever it is hosted: GPT-5/GPT-6 and Claude 4.7 or later are never sent a temperature, through OpenRouter or Groq as much as directly.

Local Models (Ollama, vLLM, LM Studio)

Run models locally for data privacy and cost control. The local provider supports Ollama native, OpenAI-compatible (vLLM, LM Studio, LocalAI), Text Generation Web UI, and Hugging Face Inference endpoints.

Ollama

agents:
  local_agent:
    name: "local_agent"
    model: "llama3.1"            # Model name as shown by `ollama list`
    # model: "mistral"
    # model: "codellama"

Environment variables:

LOCAL_MODEL_ENDPOINT=http://localhost:11434/api/generate

Start Ollama and pull a model:

# Install Ollama (macOS)
brew install ollama

# Start the server
ollama serve

# Pull a model
ollama pull llama3.1

vLLM (OpenAI-compatible)

agents:
  vllm_agent:
    name: "vllm_agent"
    model: "mistral-7b"

Environment variables:

LOCAL_MODEL_ENDPOINT=http://localhost:8000/v1/chat/completions

Start vLLM:

python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.2 \
  --port 8000

LM Studio

agents:
  lmstudio_agent:
    name: "lmstudio_agent"
    model: "local-model"

Environment variables:

LOCAL_MODEL_ENDPOINT=http://localhost:1234/v1/chat/completions

LM Studio serves an OpenAI-compatible API by default. Start the server from the LM Studio UI, then point LeafMesh at it.

Supported Local Server Formats

Format	Examples	Features
OpenAI-compatible (chat completions)	vLLM, LM Studio, LocalAI, Ollama /v1	Tool calling, structured output, streaming
Text Generation Web UI	oobabooga	Basic text generation
Hugging Face Inference API	HF Inference Endpoints	Basic text generation
Ollama native	Standard Ollama	Basic text generation

Set LOCAL_MODEL_ENDPOINT to the URL of your local server -- the platform handles the payload format automatically.

Declaring the model server in config (`mesh.local`)

The model server is set once for the mesh, not per agent. Its address and any extra header rows are a record under Infrastructure → Models → Local / self-hosted; mesh.local names it:

mesh:
  local:
    connection: "local-models"   # the record holds the endpoint and any header rows
    server_type: "openai"        # optional: openai, ollama, textgen, huggingface — detected from the URL if left out
    timeout: 120                 # seconds (default 120; local models can be slow on CPU)

From 2.4.481 an endpoint or headers written inline in mesh.local is refused at load. LOCAL_MODEL_ENDPOINT and LOCAL_MODEL_SERVER_TYPE are read from .env, or filled from the Local / self-hosted record. A local model's request timeout is mesh.local.timeout (default 120 seconds). With no mesh.local block, the local model uses timeouts.llm_hard_timeout_s (default 90 seconds).

Fixed query values (2.4.486+). Some servers want the same value on every request's address, for example api-version=2024-06 on an Azure-style OpenAI endpoint, or a gateway's routing value. Add them on the Models page, on the model's add / edit form, under Query values (sent on every call) (API: POST /api/models with query: {name: value}). Keys never go here; a key stays on the vault record.

Upgrade note: custom_endpoint and custom_headers are no longer agent settings. An agent that still has them loads with a warning, and the values have no effect. Put the endpoint and headers on the Local / self-hosted record instead.

Amazon Bedrock

Route all LLM calls through AWS Bedrock for unified billing, VPC networking, and IAM-based access control. Bedrock hosts models from multiple providers (Anthropic, Meta, Mistral, Cohere, Amazon) behind a single API.

Use the bedrock/ prefix to route any model through Bedrock instead of calling the provider directly:

mesh:
  bedrock:
    connection: "bedrock-main"      # "AWS Bedrock" record under Infrastructure → Models: region (default us-east-1), credentials, any VPC endpoint
    # profile: "prod-profile"       # Optional: AWS CLI profile name

agents:
  analyst:
    name: "analyst"
    model: "bedrock/claude-sonnet-4-20250514"   # Claude via Bedrock

  condenser:
    name: "condenser"
    model: "bedrock/llama3.1-70b"               # Llama via Bedrock

  classifier:
    name: "classifier"
    model: "bedrock/mistral-large"       # Mistral via Bedrock

Authentication: Uses standard AWS credential chain -- environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), ~/.aws/credentials profile, or IAM role (EC2/ECS/Lambda).

Bedrock support (boto3) is included with pip install leafmesh.

Available aliases:

Alias	Bedrock Model ID
`claude-sonnet-4-20250514`	`anthropic.claude-sonnet-4-20250514-v1:0`
`llama3.1-70b`	`meta.llama3-1-70b-instruct-v1:0`
`claude-3.5-sonnet`	`anthropic.claude-3-5-sonnet-20241022-v2:0`
`claude-3-haiku`	`anthropic.claude-3-haiku-20240307-v1:0`
`claude-3-opus`	`anthropic.claude-3-opus-20240229-v1:0`
`llama3-70b`	`meta.llama3-70b-instruct-v1:0`
`llama3-8b`	`meta.llama3-8b-instruct-v1:0`
`mistral-large`	`mistral.mistral-large-2402-v1:0`
`titan-text-express`	`amazon.titan-text-express-v1`
`command-r-plus`	`cohere.command-r-plus-v1:0`

You can also pass full Bedrock model IDs directly: bedrock/anthropic.claude-3-sonnet-20240229-v1:0. For a newer model that has no alias, use the full model ID exactly as your Bedrock console lists it.

Google Vertex AI

Route all LLM calls through Google Cloud Vertex AI for unified billing, IAM, and access to models from multiple publishers (Google, Anthropic, Mistral, Meta) through a single control plane.

Use the vertex/ prefix to route any model through Vertex AI:

mesh:
  vertex:
    connection: "vertex-main"       # "Google Vertex AI" record under Infrastructure → Models: project ID, location (default us-central1)

agents:
  analyst:
    name: "analyst"
    model: "vertex/gemini-2.5-flash"            # Google Gemini via Vertex

  reviewer:
    name: "reviewer"
    model: "vertex/claude-sonnet-4-20250514"    # Claude via Vertex

  coder:
    name: "coder"
    model: "vertex/mistral-large"               # Mistral via Vertex

  condenser:
    name: "condenser"
    model: "vertex/llama-4-scout"               # Llama via Vertex

Authentication: Uses Google Cloud Application Default Credentials (ADC).

# Local development
gcloud auth application-default login

# Or set service account key
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json

# GKE / Cloud Run: automatic via attached service account

Vertex AI support (google-auth) is included with pip install leafmesh.

Available aliases:

Alias	Publisher	Vertex Model ID
`gemini-2.5-pro`	Google	`gemini-2.5-pro`
`gemini-2.5-flash`	Google	`gemini-2.5-flash`
`gemini-2.0-flash`	Google	`gemini-2.0-flash`
`claude-sonnet-4-20250514`	Anthropic	`claude-sonnet-4@20250514`
`claude-opus-4`	Anthropic	`claude-opus-4@20250514`
`claude-3.5-sonnet`	Anthropic	`claude-3-5-sonnet-v2@20241022`
`mistral-large`	Mistral	`mistral-large@2411`
`codestral`	Mistral	`codestral@2501`
`llama-4-scout`	Meta	`llama-4-scout-17b-16e-instruct-maas`
`llama-4-maverick`	Meta	`llama-4-maverick-17b-128e-instruct-maas`

The platform handles the request format for each publisher automatically -- Gemini, Anthropic, Mistral, and Meta models all work through the same vertex/ prefix. A model without an alias is sent with the ID you write after vertex/, so a newer model works as soon as your Google Cloud project offers it — use the ID shown in the Vertex AI Model Garden.

Microsoft Foundry

Route all LLM calls through Microsoft Foundry (Azure AI Foundry) for unified billing and access to any model deployed in your Azure AI resource -- OpenAI, DeepSeek, Llama, Mistral, Grok, Phi, Cohere, and more.

Use the foundry/ prefix followed by your deployment name:

mesh:
  foundry:
    connection: "foundry-main"      # "Azure AI Foundry" record under Infrastructure → Models: your endpoint and sign-in
    # api_version: "2024-10-21"     # Optional: use legacy versioned API

agents:
  analyst:
    name: "analyst"
    model: "foundry/gpt-4o"                    # OpenAI via Foundry

  reasoner:
    name: "reasoner"
    model: "foundry/DeepSeek-R1"               # DeepSeek via Foundry

  condenser:
    name: "condenser"
    model: "foundry/Meta-Llama-3.1-70B"        # Llama via Foundry

  coder:
    name: "coder"
    model: "foundry/Mistral-Large-2411"        # Mistral via Foundry

Authentication: API key or Entra ID (Bearer token).

# API key auth (recommended for getting started)
export AZURE_FOUNDRY_API_KEY="your-api-key"   # AZURE_OPENAI_API_KEY also works

# Or Entra ID / managed identity token
export AZURE_FOUNDRY_TOKEN="your-bearer-token"

Model keys in .env still work (a value in .env wins; the vault fills any gaps). On the Infrastructure page (Models), a Foundry record signs in one of three ways: a service principal (an Entra app registration — the default, and it works wherever your mesh runs), an API key (quickest to start), or a managed identity (only when the mesh runs inside your own Azure, with no secret at all).

No extra dependencies -- Foundry uses the OpenAI-compatible endpoint, and openai is already a core dependency.

How it works: Models are deployed in your Azure AI resource first (via Azure Portal or CLI). The deployment name becomes the model identifier after the foundry/ prefix. Foundry exposes an OpenAI-compatible /openai/v1/ endpoint, so all standard features (tool calling, structured output, streaming) work across every model.

Two API modes:

v1 (default, recommended): Omit api_version. Uses the new /openai/v1/ unified endpoint -- no version parameter needed.
Legacy versioned: Set api_version: "2024-10-21". Uses Azure's /openai/deployments/{name}/...?api-version=X path via the AzureOpenAI client. Use this for older Azure OpenAI resources or features only available in preview API versions.

Automatic Provider Routing

You never configure a provider explicitly -- just set the model. The platform routes the request to the correct provider automatically:

agents:
  solver:
    name: "solver"
    model: "gpt-5.4-mini"         # Routes to OpenAI

  analyst:
    name: "analyst"
    model: "claude-sonnet-5-5"    # Routes to Anthropic

  translator:
    name: "translator"
    model: "gemini-3.6-flash"     # Routes to Google

  coder:
    name: "coder"
    model: "deepseek-v4-pro"      # Routes to DeepSeek

  drafter:
    name: "drafter"
    model: "glm-5.3"              # Routes to Z.AI

  scout:
    name: "scout"
    model: "groq/openai/gpt-oss-120b"   # Routes to Groq (prefix removed before sending)

  private:
    name: "private"
    model: "llama3.1"             # Routes to a local model server

How a model name is read:

A lab's own model name (gpt-…, o3, claude-…, gemini-…, deepseek-…, mistral-…, codestral-…, grok-…, kimi-…, qwen…, glm-… and the other vendors' names) goes to that lab.
A host prefix (openrouter/, groq/, cerebras/) goes to that host, which gets the rest of the name.
A cloud prefix (bedrock/, vertex/, foundry/) goes through your cloud account.
Any other name goes to your local model server.

Custom providers may register their own model name prefixes via the SDK.

Startup Behavior

At startup, LeafMesh checks for API keys and connectivity for each provider in use. Providers that are not available (missing API keys, unreachable endpoints) are skipped with a warning but do not prevent the system from starting. At least one provider must be available.

Model Configuration

Per-Agent Settings

Each agent configures its own model parameters directly in YAML:

agents:
  precise_agent:
    name: "precise_agent"
    model: "gpt-4o"
    temperature: 0.1           # Low randomness for consistency
    max_tokens: 500            # Limit response length

  creative_agent:
    name: "creative_agent"
    model: "claude-sonnet-4-6"
    temperature: 0.8           # Higher randomness for creativity
    max_tokens: 2000           # Longer responses

Some models do not accept a temperature at all — GPT-5 and GPT-6, the o-series, and Claude 4.7 or later. On those, LeafMesh leaves the temperature out instead of failing the call, wherever the model is hosted.

Configuration Parameters

Parameter	Type	Default	Description
`model`	string	required	Model identifier (e.g., `"gpt-4o"`)
`temperature`	float	0.1	Sampling temperature (0.0-2.0)
`max_tokens`	int	1000	Maximum response tokens
`top_p`	float	1.0	Nucleus sampling threshold
`tool_choice`	string	`"auto"`	Tool calling mode
`max_tool_calls_per_message`	int	5	Maximum tool calls per execution (range 0-20)
`optimization_strategy`	string	none	Adaptive model selection: `"performance"`, `"cost"`, or `"speed"`

Tool calls on every provider

The tool loop behaves the same on every provider (OpenAI-compatible, Anthropic, Bedrock, Google, Vertex and local):

A garbled tool call is retried with a note. When a model returns a tool call it could not format (Gemini can do this), the retry tells it the call was not run and asks for valid arguments, up to two more times. A call that stays garbled is a failure, not an empty reply.
Tools ran, but no answer. When the model's reply comes back empty after it ran tools, it is told so with its tools still on, and asked to carry on with the next step or write its answer. Two empty replies in a row is a failure, never an empty success. Only when no tools are left to call (the tool-call limit is reached, the budget is spent, or a tool kept failing) is it asked for its answer with tools off.

See LLM Agents → Tool Execution Flow.

Multi-Provider Architecture

Different agents in the same system can use different providers. This reduces systematic bias from any single model, provides natural redundancy against provider outages, and enables cost optimization.

Practical Multi-Provider Configuration

name: "multi_provider_system"
architecture: "managed_mesh"

agents:
  # Fast, cheap classification
  triage:
    name: "triage"
    model: "gpt-4o-mini"
    temperature: 0.1
    prompt: |
      Classify the incoming request into one of: billing, technical, general.
    yields:
      category: "string"
      urgency: "number"
    can_call:
      - agent: "specialist"
        condition: "category == 'technical'"
      - agent: "billing_handler"
        condition: "category == 'billing'"

  # Strong reasoning for complex analysis
  specialist:
    name: "specialist"
    model: "claude-sonnet-4-6"
    temperature: 0.3
    prompt: |
      You are a technical specialist. Analyze the issue and provide a solution.
    yields:
      solution: "string"
      confidence: "number"

  # Cost-effective for structured tasks
  billing_handler:
    name: "billing_handler"
    model: "gpt-4o-mini"
    temperature: 0.0
    prompt: |
      You handle billing inquiries. Look up account details and resolve issues.
    yields:
      resolution: "string"
      amount: "number"

Provider Failure Isolation

If one provider experiences downtime, only agents using that provider are affected. Other agents continue operating normally:

Agents 1-3 use OpenAI    --> OpenAI outage --> Agents 1-3 unavailable
Agents 4-5 use Anthropic --> Unaffected    --> Continue operating
Agents 6-7 use Google    --> Unaffected    --> Continue operating

LeafMesh's self-healing detects elevated failure rates and tells your escalation contact; with self_healing.actions_enabled: true it also restarts an affected agent or starts a backup for it.

Custom Provider Registration

from leafmesh import LLMProvider, LLMRequest, LLMResponse

class InternalProvider(LLMProvider):
    def is_available(self) -> bool:
        # Check if your internal endpoint is reachable
        return True

    async def generate(self, request: LLMRequest) -> LLMResponse:
        # Forward request to your internal model serving
        import aiohttp
        async with aiohttp.ClientSession() as session:
            async with session.post(
                "https://internal-models.company.com/v1/generate",
                json={"prompt": request.prompt, "model": request.model}
            ) as resp:
                data = await resp.json()

        return LLMResponse(
            content=data["text"],
            model=request.model,
            usage={"prompt_tokens": data.get("prompt_tokens", 0),
                   "completion_tokens": data.get("completion_tokens", 0)},
            provider="internal",
            success=True
        )

# Register with model prefix routing
leafmesh.register_llm_provider("internal", InternalProvider, model_prefixes=["internal-", "company-"])

After registration, any agent with model: "internal-llama-70b" or model: "company-gpt" will automatically route to your custom provider.

LLM Cost Tracking

LeafMesh tracks the cost of every LLM call -- model used, token counts, cost, response time, and whether the response was served from cache. When an agent loops through tools, each model call in the loop has its own cost as well as the step's total.

How a call's cost is worked out. LeafMesh keeps each vendor's published prices (re-read every release) and prices a call the way that vendor bills it:

Size tiers. Some models charge more for a very long prompt: Gemini Pro and Grok above 200,000 prompt tokens, Claude Haiku 5.5 above 100,000, GPT-6 Astra, GPT-6.1 Sol and GPT-6.1 Terra above 272,000. A call past the line is priced at the higher rate.
Dated price changes. When a vendor announces a new price from a given day (for example Gemini 4 Argon from 1 December 2026, Gemini 3.8 and 3.6 Flash from 1 January 2027), calls from that day use the new price.
Off-peak pricing. DeepSeek charges by the time of day; each call is priced by when it was made.
Cached prompts are priced at the vendor's cheaper cache rate.
Image models (2.4.508) are billed by kind of token, from the vendor's own counts: on Gemini image models the image made is priced at the image rate and the text and thinking beside it at the text rate; on GPT Image an image given in and the text written are each priced at their own rate.
OpenRouter tells LeafMesh what each call cost, so its figure is exactly what OpenRouter bills — no price list to fall behind.
A model with no known price (a local model, or a vendor that publishes no per-token price) shows as cost unknown rather than a guessed figure or $0.

Aggregated analytics, cache metrics, and per-session event streams are available in Studio's Dashboard and session inspector — operator UI, not part of the customer Python or HTTP surface.

Identical requests are deduplicated through the response cache -- see LLM Response Caching.

Unified LLM Request

LeafMesh exposes a single LLM interface that works across every supported provider, handling differences in message format, tool calling syntax, and response parsing:

# Direct LLM call from an intelligence function
response = await leafmesh.call_llm(
    prompt="Summarize this data",
    model="gpt-4o",
    temperature=0.3,
    max_tokens=500
)

The unified request supports: messages, model, temperature, max_tokens, tools, tool_choice, response_format, stream flag, files, custom endpoints, and custom headers.

LLM Types Reference

These types are exported from the leafmesh package for custom provider implementation and direct LLM calls.

LLMRequest

Standard request structure sent to all LLM providers. Supports multimodal input, tool calling, streaming, and reasoning enhancement.

from leafmesh import LLMRequest

Field	Type	Default	Description
`prompt`	`str`	required	The prompt text
`model`	`str`	`""` — must be set	Model identifier. Empty by default; the SDK never substitutes one.
`temperature`	`float`	`0.1`	Sampling temperature
`max_tokens`	`int`	`800`	Maximum response tokens
`system_prompt`	`str`	`None`	System prompt
`context`	`dict`	`{}`	Additional context
`tools`	`list`	`[]`	Tool schemas for function calling
`tool_choice`	`str`	`None`	Tool calling mode (`"auto"`, `"required"`, `"none"`)
`tool_call_depth`	`int`	`0`	Current recursion depth in tool call loop
`max_tool_calls`	`int`	`5`	Maximum tool calls per conversation
`messages`	`list`	`[]`	Conversation messages array
`response_format`	`dict`	`None`	JSON schema for structured output
`stream`	`bool`	`False`	Enable streaming response
`files`	`list`	`[]`	File objects or URLs for multimodal input
`reasoning`	`bool`	`False`	Enable the provider's native extended thinking (reasoning-capable models only — Claude, Gemini 2.5+, o-series, GPT-5 and GPT-6, Grok 4.5+). Distinct from `thinking`, which turns on the SDK's own text-only reasoning tools for any model (it never runs code).

LLMResponse

Standard response structure returned by all LLM providers.

from leafmesh import LLMResponse

Field	Type	Default	Description
`content`	`str`	required	Response text content
`model`	`str`	required	Model that generated the response
`usage`	`dict`	required	Token usage (`{"prompt_tokens": N, "completion_tokens": N}`)
`provider`	`str`	required	Provider name (e.g. `"openai"`, `"anthropic"`)
`success`	`bool`	`True`	Whether the call succeeded
`error`	`str`	`None`	Error message if `success` is `False`
`tool_calls`	`list`	`None`	Tool calls requested by the LLM
`finish_reason`	`str`	`None`	Why generation stopped (e.g. `"stop"`, `"tool_calls"`)
`stream_chunks`	`list`	`None`	Collected chunks for streaming responses
`structured_output`	`dict`	`None`	Parsed JSON for structured responses

LLMProvider (Abstract Base Class)

Base class for implementing custom LLM providers.

from leafmesh import LLMProvider, LLMRequest, LLMResponse

Method	Signature	Description
`generate`	`async def generate(self, request: LLMRequest) -> LLMResponse`	Generate a response from the LLM (required)
`is_available`	`def is_available(self) -> bool`	Check if provider is properly configured (required)

The base class also handles automatic context truncation on provider context-length errors and routes native extended-thinking settings to the provider when reasoning=True is set on a request.

Model Selection Guide

Use Case	Recommended Model	Why
Triage / classification	`gpt-6-luna` or `claude-haiku-5-5`	Fast, cheap, accurate for structured tasks
Complex analysis	`gpt-6.1-sol` or `claude-sonnet-5-5`	Strong reasoning, broad capability
Hardest problems	`gpt-6-astra`, `claude-fable-5-1` or `gemini-4-argon`	Most capable models, priced to match
High-volume processing	`gemini-3.6-flash` or `deepseek-flash`	Fast response time, cost-efficient
Code generation	`gpt-5.6-terra` or `codestral-2508`	Strong code capabilities
Reasoning-heavy tasks	`o3`, `claude-opus-5-5` or `deepseek-v4-pro`	Built for multi-step reasoning
Sensitive data	Local model (Ollama/vLLM)	Data never leaves the network
Creative content	`claude-sonnet-5-5`	Strong creative writing
Budget-conscious	`gpt-6-luna`, `gemini-3.5-flash-lite` or `qwen3.7-flash`	Lowest per-token cost
Trying many models with one key	`openrouter/<lab>/<model>`	One account; real cost reported per call
Unified cloud billing	`bedrock/`, `vertex/`, or `foundry/*`	Single bill, IAM access control
Multi-model via AWS	`bedrock/claude-sonnet-4-20250514`	AWS VPC, IAM roles, CloudTrail
Multi-model via GCP	`vertex/gemini-2.5-flash`	GCP IAM, single project billing
Multi-model via Azure	`foundry/gpt-4o`	Azure IAM, unified Azure billing

Fast and cheap

When speed and price matter more than depth (triage, routing, short summaries, high volume), start with one of these:

OpenAI: gpt-6-luna, gpt-5.4-nano, gpt-4.1-nano
Anthropic: claude-haiku-5-5
Google: gemini-3.5-flash-lite, gemini-2.5-flash-lite
DeepSeek: deepseek-flash (cheapest off-peak)
Mistral: mistral-small-4-0-26-03
Others: qwen3.7-flash (Alibaba), glm-5.3-flash (Z.AI), minimax-m3.1-flash (MiniMax), mercury-3 (Inception, built for speed), step-3.7-flash (StepFun), or open models hosted on Groq (groq/…) or Cerebras (cerebras/…)

Set optimization_strategy: "cost" or "speed" on an agent to let LeafMesh choose among the models you have keys for.

Next Steps

Adaptive Execution -- Automatic model selection based on request characteristics
Agent Configuration -- Full YAML reference including model settings
Tools -- Extend agents with tool calling across providers
Architecture Guide -- How LLM calls fit into the control plane

LeafMesh -- One interface, any model, automatic routing

LLM Providers

Three Ways to Reach a Model

Connecting a provider

Several accounts with one provider (model_connection)

Supported Providers

Provider Setup

OpenAI

Anthropic

Google Gemini

DeepSeek

Mistral

xAI Grok

Other Model Vendors

Local Models (Ollama, vLLM, LM Studio)

Ollama

vLLM (OpenAI-compatible)

LM Studio

Supported Local Server Formats

Declaring the model server in config (mesh.local)

Amazon Bedrock

Google Vertex AI

Microsoft Foundry

Automatic Provider Routing

Startup Behavior

Model Configuration

Per-Agent Settings

Configuration Parameters

Tool calls on every provider

Multi-Provider Architecture

Practical Multi-Provider Configuration

Provider Failure Isolation

Custom Provider Registration

LLM Cost Tracking

Unified LLM Request

LLM Types Reference

LLMRequest

LLMResponse

LLMProvider (Abstract Base Class)

Model Selection Guide

Fast and cheap

Next Steps

Several accounts with one provider (`model_connection`)

Declaring the model server in config (`mesh.local`)