pal-e-docs

SOP: MCP Server Recovery

sop-mcp-server-recovery Sop

sop active
Project

SOP: MCP Server Recovery

Purpose: Teach agents how to diagnose and recover when MCP servers fail to load, time out, or behave unexpectedly. Absorbs findings from bug-mcp-silent-load-failure.
Traceability: plan-pal-e-agency → Phase 5 (Error Recovery SOPs)

Background

MCP servers are defined in ~/.mcp.json and loaded by Claude Code at session startup. They provide tools like mcp__pal-e-docs__*, mcp__forgejo__*, mcp__woodpecker__*, and mcp__notion__*. Claude Code provides zero observability into MCP server health — no startup logs, no error messages, no ~/.claude/logs/ directory.

Failure Modes

Symptom Likely Cause Recovery Steps
<strong>Silent load failure</strong> — MCP tools simply do not appear in the tool registry Transient timeout or dependency resolution delay during session startup. The private Forgejo PyPI index used by <code>pal-e-docs-sdk</code> and <code>ldraney-forgejo-sdk</code> may contribute to slower <code>uv run</code> startup times. Claude Code silently drops servers that fail to initialize. 1. Check if tools exist using <code>ToolSearch</code> (e.g., search for "pal-e-docs" or "forgejo"). 2. If tools are missing, the ONLY fix is to <strong>restart the session</strong>. 3. There is no way to reload MCP servers mid-session. 4. Report the failure to Betty Sue so it can be tracked.
<strong>Tool call times out</strong> — MCP tool hangs and eventually errors The MCP server process is alive but the backend service (pal-e-docs API, Forgejo API, etc.) is unreachable or slow. 1. Retry once — may be transient. 2. If persistent, the backend service may be down. Check pod status: <code>kubectl get pods -n NAMESPACE</code>. 3. For pal-e-docs: check <code>pal-e-docs</code> namespace. For Forgejo: check <code>forgejo</code> namespace.
<strong>Tool not found</strong> — a specific tool name is not recognized MCP server version mismatch. The local checkout may be on a feature branch from a previous agent session, missing the tool. 1. Check which version of the MCP server is running. 2. For pal-e-docs-mcp: <code>cd ~/pal-e-docs-mcp &amp;&amp; git status</code> — ensure it is on <code>main</code>. If on a feature branch: <code>git checkout main &amp;&amp; git pull</code>. 3. Restart the session to reload the MCP server with the correct version.
<strong>Tool returns unexpected errors</strong> (400, 404, 500) API schema mismatch between MCP server and the backend, or the resource does not exist. 1. For 404: verify the slug/ID you are passing is correct. 2. For 400: check parameter types and required fields. 3. For 500: the backend has a bug. Note the exact error and escalate.
<strong>Version mismatch</strong> — MCP server offers tools that the backend does not support MCP server was updated but the backend (pal-e-docs API) was not redeployed with the matching changes. 1. Check what version of the API is deployed: <code>kubectl get deploy -n pal-e-docs -o jsonpath='{.items[0].spec.template.spec.containers[0].image}'</code>. 2. Compare with what the MCP server expects. 3. If the API needs redeployment, escalate to Betty Sue.

Decision Tree

When MCP tools are missing or failing:
  • Verify the tools exist: Use ToolSearch to search for the expected tool prefix (e.g., "pal-e-docs", "forgejo").
  • If tools are completely missing: The MCP server failed to load silently. Restart the session. There is no mid-session fix.
  • If tools exist but fail: Check if the backend service is healthy. Retry once for transient errors.
  • If tool not found (specific tool): Check ~/pal-e-docs-mcp (or the relevant MCP repo) for branch status. Switch to main if on a feature branch. Restart session.
  • If errors persist: Escalate to Betty Sue with: which tool, exact error message, and whether the backend pod is running.

Prevention

  • After any agent work on MCP server repos, verify the local checkout is back on main: cd ~/pal-e-docs-mcp && git checkout main && git pull.
  • At session start, Betty Sue should verify critical MCP tools are available before dispatching agents.
  • A SessionStart hook to verify MCP server health is planned but not yet implemented (see bug-mcp-silent-load-failure).

Escalation Criteria

Escalate immediately (do NOT self-correct) when:
  • MCP server failed to load and you need it for your task (session restart is the only fix)
  • Backend service (pal-e-docs, Forgejo) is down — pods not running
  • Persistent 500 errors from the backend API
  • Version mismatch that requires a backend redeployment
  • bug-mcp-silent-load-failure — the original discovery of silent MCP load failures, with proposed SessionStart hook fix
  • sop-claude-config-development — MCP server configuration lives in ~/.mcp.json
  • agent-workflow — which agents have access to which MCP servers