Self-Healing Networks

LeafMesh watches every agent's health and tells a person when one keeps failing. If you allow it, the mesh can also act on its own: restart the agent, or start a backup copy and route the agent's work to it. Watching is on by default; acting is off until you turn it on.

How self-healing works

Self-healing is event-driven. The platform listens for agent errors, mesh errors and session timeouts, and only reassesses an agent when one of those arrives. There is no extra cost during normal operation.

When the platform detects a problem:

Health metrics for the affected agent are updated (consecutive failures, error rate, latency, quality score).
The agent's health is reassessed against your self_healing: thresholds.
A healing.triggered event is raised and your escalation contact gets a high-priority Inbox notice (at most one per agent per hour).
Only if actions_enabled: true, and only within the cooldown and caps, the mesh restarts the agent or starts a backup for it.
Every step is recorded as a healing.* event.

Configuration

self_healing is a top-level block. Every field is optional; leave one out and the default below applies. A mesh that never mentions the block watches its agents and notifies, and never acts by itself.

self_healing:
  enabled: true                 # watch health, raise healing.* events, notify (default true)
  actions_enabled: false        # allow restart / backup (default false)
  consecutive_failures: 5       # failures in a row before an agent is critical
  error_rate: 0.3               # error share of recent runs before an agent is critical
  latency_factor: 3.0           # slower than 3x its own median latency = slow
  min_quality_score: 0.4        # reported quality below this = critical
  cooldown_s: 600               # wait after an action before acting on that agent again
  max_actions_per_hour: 3       # actions per agent per rolling hour
  max_backups_per_agent: 1      # backups that may run for one agent at a time
  monitoring_interval_s: 5      # how often health is sampled, seconds

Field	Default	Meaning
`enabled`	`true`	Watch agent health, raise `healing.*` events and send an Inbox notice.
`actions_enabled`	`false`	Let the mesh restart a failing agent or start a backup for it. Off by default because a backup is a second agent that spends on every call it takes.
`consecutive_failures`	`5`	Failures in a row before an agent is critical.
`error_rate`	`0.3`	Share of recent runs that failed before an agent is critical.
`latency_factor`	`3.0`	How much slower than its own median an agent must be to count as slow. Judged only once the agent has 20 runs behind it, so a slow-but-normal agent is never flagged.
`min_quality_score`	`0.4`	Quality score (0–1, when the agent reports one) below which it is critical.
`cooldown_s`	`600`	Seconds after an action before the same agent may be acted on again.
`max_actions_per_hour`	`3`	Actions allowed per agent per rolling hour.
`max_backups_per_agent`	`1`	Backup agents that may exist for one failing agent at a time.
`monitoring_interval_s`	`5`	How often agent health is sampled, in seconds.

The block is strict: an unknown key is refused at load. The old fields detection_interval, max_recovery_attempts and recovery_strategies no longer exist — a config that still carries them does not load.

Health assessment

Agent health states

Each agent has a health state that reflects whether it is operating normally, showing early-warning signs, in active trouble, non-responsive, or in the middle of a recovery action. Operators see these states in dashboards and the management API — you do not have to model the state machine yourself.

Metrics tracked

The platform collects these metrics for each agent. The thresholds that move an agent between states are configurable — see the configuration reference for tunable values.

Metric	Description
`response_time_ms`	Average response time
`error_rate`	Ratio of errors to total requests
`success_rate`	Ratio of successes to total requests
`consecutive_failures`	Failures without an intervening success
`quality_score`	Output quality from downstream validation
`memory_usage_mb`	Agent memory consumption
`last_heartbeat`	Timestamp of the most recent heartbeat

What the mesh does

The mesh picks one action per finding. With actions_enabled: false (the default) it is always notify.

Action	When it's chosen	What it does
`notify`	Any unhealthy agent short of repeated failure, and every finding while `actions_enabled` is false	Raises the events and sends the Inbox notice. Nothing in the mesh changes; a person decides.
`restart_agent`	The agent has failed outright	Re-initialises the agent with its own configuration and drops stale backups.
`spawn_backup`	The agent is critical with 3+ failures in a row, or has failed while carrying more than 10 active sessions	Starts a copy with the same configuration and the same model, and routes the original's work to it.

An action that would break the cooldown, the hourly cap or the backup cap is not taken; it is reported as healing.suppressed with the reason. Twenty-five errors never mean twenty-five backups.

Recovery events

All recovery actions are recorded as audit events:

Event	When
`healing.triggered`	A health issue is detected
`healing.action.taken`	A restart or backup begins
`healing.completed`	The action finished
`healing.failed`	The action did not succeed
`healing.suppressed`	An action was chosen but not taken (actions off, cooldown or cap)

These events create a persistent record for post-incident analysis.

Interaction with the Manager

Self-healing and the Manager are complementary systems:

System	Scope	Mechanism
Manager	Per-session protection	Tracks failed agents per session, prevents routing to known-failing agents
Self-healing	System-wide health	Notifies, and (when allowed) restarts agents or starts a backup

The Manager provides immediate per-session protection (stops routing to a failing agent within the current session). Self-healing provides global health: it tells a person, and when you allow it, restarts the agent so it works for all sessions.

Example: monitoring system with self-healing

name: "monitored_swarm"
version: "1.0.0"
architecture: "managed_mesh"

self_healing:
  actions_enabled: true         # let the mesh restart or back up a failing agent
  consecutive_failures: 3
  cooldown_s: 900

agents:
  collector:
    name: "collector"
    agent_type: "programmatic"
    communication_type: "chain"
    wake_up: "*/5 * * * *"
    yields:
      readings: "array"
      timestamp: "string"
    can_call:
      - agent: "analyzer"
        condition: "readings != []"

  analyzer:
    name: "analyzer"
    model: "gpt-4o-mini"
    temperature: 0.1
    communication_type: "chain"
    prompt: |
      Analyze numeric readings and detect anomalies.
      Flag values that deviate more than 2 standard deviations from the mean.
    yields:
      anomaly_count: "number"
      severity: "string"
    can_call:
      - agent: "alerter"
        condition: "anomaly_count > 0"

  alerter:
    name: "alerter"
    model: "gpt-4o-mini"
    temperature: 0.0
    communication_type: "chain"
    prompt: |
      Determine alert action based on severity.
      Low: log. Medium: notify. High: escalate to human.
    yields:
      action: "string"
      message: "string"
      escalate: "boolean"
    can_call:
      - agent: "supervisor"
        condition: "escalate == true"

  supervisor:
    name: "supervisor"
    agent_type: "human"
    is_human_powered: true
    human_interface: "api"
    human_timeout_seconds: 1800
    yields:
      decision: "string"
      notes: "string"

From 2.4.482 the store is set only in .env, never in the YAML:

# .env
LEAFMESH_STORE=redis
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_SESSION_TTL=86400

If the analyzer agent keeps failing (model timeouts, a provider outage), self-healing:

Raises healing.triggered and sends your escalation contact an Inbox notice naming the agent, its failure count and error rate.
Once it is critical with three failures in a row, starts one backup copy of analyzer (same config, same model) and routes its work there.
If the agent fails outright, restarts it with its original config.
Does nothing more to analyzer for 15 minutes (cooldown_s: 900), and at most 3 times an hour; anything it holds back is reported as healing.suppressed.

The collector keeps running on schedule throughout.

Observability for self-healing

Monitor the healing system itself through the observability dashboard:

observability:
  service_name: "my_swarm"
  metrics_retention_minutes: 1440  # 24 hours

The dashboard shows:

Agent health status
Recovery action history (what was tried, when, success/failure)
Error rates and response times per agent
Mesh communication patterns and failures

Configuration patterns

Watch only (the default)

self_healing:
  enabled: true
  actions_enabled: false

You are told when an agent is unhealthy; nothing in the mesh changes by itself.

Act, within tight limits

self_healing:
  actions_enabled: true
  consecutive_failures: 3
  cooldown_s: 900
  max_actions_per_hour: 2
  max_backups_per_agent: 1

Off (for example in a test sandbox)

self_healing:
  enabled: false

Key design decisions

Event-driven. The platform reacts when a failure signal arrives.

Told first, acting only when you allow it. Detection and the Inbox notice are on by default. Restarts and backups are off until you set actions_enabled: true, because a backup spends model budget on every call it takes.

Judged against the agent's own normal. Latency is compared with the agent's own median, so a researcher that usually takes six seconds is not flagged for taking six seconds.

Bounded. The cooldown, the hourly cap and the backup cap limit what the mesh may do to one agent; a held-back action is reported, not lost.

Audit trail. Every finding and action is a healing.* event with the agent, its health, the chosen action, the metrics and, for a backup, the backup's name.

Next Steps

Monitoring Patterns — Build monitoring systems with scheduled agents
Architecture Guide — How self-healing fits into the control plane
Agent Configuration — Full YAML reference

LeafMesh — Systems that detect, recover, and learn from failures