Project Guide | StagePilot

## <img width="3168" height="1344" alt="StagePilot banner" src="https://github.com/user-attachments/assets/9a002988-e535-42ac-8baf-56ec8754410f" />

StagePilot

Tool-calling reliability runtime for LLMs. Parses, repairs, and retries malformed tool-call output so you don't have to. Lifts baseline success from 25% to 90% on a 60-case benchmark with 30 mutation modes.

Architecture pack: `docs/architecture-pack.md`

Provenance And Ownership

StagePilot is an independent reliability lab that extends an Apache-2.0 codebase originally published by Woonggi Min at minpeter/ai-sdk-tool-call-middleware. The upstream project owns and publishes the `@ai-sdk-tool/parser` npm package. StagePilot does not publish or control the @ai-sdk-tool/parser npm package.

This repository keeps the upstream copyright and license, adds the StagePilot runtime, BenchLab, adapters, service APIs, telemetry, and deployment assets, and is deliberately marked private in package.json to prevent accidental npm publication. See `NOTICE.md` for the boundary between upstream code and StagePilot additions.

Three-Minute Proof

1. Read the 25% to 90% benchmark claim, then open the mutation benchmark evidence.
2. Inspect the parser/retry path and one malformed tool-call fixture.
3. Run npm run verify or the equivalent pnpm command for checks, tests, and build.
4. Use pnpm api:stagepilot only after the package proof is clear.

System Overview

Evaluation Path

• Start here: Read the 25% to 90% benchmark claim, then inspect parser recovery and mutation fixtures.
• Local demo: Install with pnpm install, then run pnpm api:stagepilot and open http://127.0.0.1:8080/demo.
• Checks: Run npm run verify or the equivalent package-manager command for check, tests, and build.

Architecture Notes

• Architecture guide summarizes the system scope, first files to inspect, runtime commands, and known boundaries.
• Quality notes lists the local checks, CI surface, and release expectations for this repository.
• Enterprise readiness notes outlines security, data, operations, integration, and handoff expectations.

The Problem

Models without native tool support produce unreliable output — XML one turn, JSON the next, hallucinated tool names, missing args, type mismatches. On our benchmark, baseline success is 25%. Most workarounds are regex hacks or single-pass prompts. They break when the format drifts and give you no way to see what went wrong.

The Solution

StagePilot provides three composable pieces:

Architecture

flowchart TB
    subgraph Client["Client Application"]
        App[Your App / Agent]
    end

    subgraph Parser["@ai-sdk-tool/parser  —  npm package"]
        MW[Middleware Layer]
        MW --> Proto{Protocol Detection}
        Proto --> Hermes[Hermes JSON]
        Proto --> MorphXML[MorphXML]
        Proto --> YamlXML[YamlXML]
        Proto --> Qwen[Qwen3Coder]
        Hermes & MorphXML & YamlXML & Qwen --> RJSON[RJSON Parser]
        Hermes & MorphXML & YamlXML & Qwen --> RXML[RXML Parser]
        RJSON & RXML --> Coerce[Schema Coercion]
        Coerce --> Repair[Repair + Retry Loop]
    end

    subgraph Pipeline["StagePilot Runtime  —  5-Stage Pipeline"]
        direction LR
        E[Eligibility] --> S[Safety]
        S --> P[Planner]
        P --> O[Outreach]
        O --> J[Judge]
    end

    subgraph Observe["Observability"]
        OTel[OpenTelemetry Spans]
        Prom[Prometheus Metrics]
        DD[Datadog Dashboards]
    end

    subgraph Deploy["Deployment"]
        Docker[Docker]
        CR[GCP Cloud Run]
        K8s[Kubernetes + HPA]
        CF[Cloudflare Workers]
        Vercel[Vercel]
    end

    subgraph IaC["Infrastructure as Code"]
        TF[Terraform]
        Manifests[K8s Manifests]
    end

    App --> MW
    Repair --> Pipeline
    Pipeline --> Observe
    Pipeline --> Deploy
    Deploy --> IaC

    style Parser fill:#1a1a2e,stroke:#e94560,color:#fff
    style Pipeline fill:#16213e,stroke:#0f3460,color:#fff
    style Observe fill:#0f3460,stroke:#533483,color:#fff

Stage-Gated Pipeline Detail

sequenceDiagram
    participant C as Client
    participant MW as Parser Middleware
    participant E as EligibilityAgent
    participant S as SafetyAgent
    participant P as PlannerAgent
    participant O as OutreachAgent
    participant J as JudgeAgent
    participant T as Telemetry

    C->>MW: Raw model text
    MW->>MW: Protocol detect → Parse → Coerce → Repair
    MW-->>T: parse_span (protocol, latency, status)

    alt Parse failed + retry enabled
        MW->>MW: RALPH retry loop (max 2 attempts)
    end

    MW->>E: Normalized tool call
    E->>E: Scope check + program matching
    E-->>T: eligibility_span

    alt Not eligible
        E-->>C: Early rejection
    end

    E->>S: Eligible intake
    S->>S: Policy enforcement (DUI, duplicates, etc.)
    S-->>T: safety_span

    alt Safety blocked
        S-->>C: Block + reason
    end

    S->>P: Safe intake
    P->>P: Generate action plan + fallback route
    P-->>T: planner_span

    P->>O: Action plan
    O->>O: Generate outreach messages per agency
    O-->>T: outreach_span

    O->>J: Execution results
    J->>J: Quality score (0-100) + review
    J-->>T: judge_span

    alt Score < threshold
        J->>E: Trigger replay
    end

    J-->>C: Final result + audit trail

Benchmark Results

Source: `docs/benchmarks/stagepilot-latest.json` — 60 cases, 30 mutation modes.

30 Mutation Modes

Each mode simulates a real-world LLM output failure pattern:

⚠️ = Unrecoverable by retry (requires model-level fix → see tool-call-finetune-lab)

With The Upstream Middleware

The following installs the upstream package maintained by `minpeter`, not a package published by this repository.

pnpm add @ai-sdk-tool/parser

import { morphXmlToolMiddleware } from "@ai-sdk-tool/parser";
import { wrapLanguageModel, streamText } from "ai";

// Works with any AI SDK provider: OpenAI, Anthropic, Google, Ollama, etc.
const enhanced = wrapLanguageModel({
  model: anyModel,
  middleware: morphXmlToolMiddleware,
});

const result = await streamText({
  model: enhanced,
  prompt: "What is the weather in Seoul?",
  tools: {
    get_weather: {
      description: "Get weather for a city",
      parameters: z.object({ city: z.string() }),
      execute: async ({ city }) => `${city}: 22°C, sunny`,
    },
  },
});

As full runtime (API server)

git clone https://github.com/KIM3310/stage-pilot.git
cd stage-pilot
pnpm install
pnpm api:stagepilot

## → http://127.0.0.1:8080/demo

Middleware Variants

API Endpoints

pnpm api:stagepilot  # http://127.0.0.1:8080

Deployment

<details> <summary><strong>Docker</strong></summary>

docker build -t stagepilot-api .
docker run -p 8080:8080 -e GEMINI_API_KEY="$GEMINI_API_KEY" stagepilot-api

</details>

<details> <summary><strong>GCP Cloud Run</strong> (one command)</summary>

pnpm deploy:stagepilot

Infrastructure managed by Terraform:

cd infra/terraform
terraform init && terraform apply

</details>

<details> <summary><strong>Kubernetes</strong> (local/BYI manifests)</summary>

The checked-in Kubernetes manifests are bring-your-own-image deployment scaffolding. Before using them outside local validation, build the API image, push it to an operator-controlled registry, and substitute an immutable image reference such as a digest-pinned Artifact Registry or ECR image.

docker build -t "$REGISTRY/stagepilot-api:$GIT_SHA" .
docker push "$REGISTRY/stagepilot-api:$GIT_SHA"


## Replace the local placeholder with the pushed immutable image.
kubectl set image -f infra/k8s/deployment.yaml \
  stagepilot-api="$REGISTRY/stagepilot-api@$IMAGE_DIGEST" \
  --local -o yaml > /tmp/stagepilot-deployment.yaml

kubectl create namespace stagepilot
kubectl create secret generic stagepilot-secrets \
  --namespace stagepilot \
  --from-literal=gemini-api-key="$GEMINI_API_KEY"
kubectl apply -f /tmp/stagepilot-deployment.yaml
kubectl apply -f infra/k8s/configmap.yaml -f infra/k8s/service.yaml -f infra/k8s/hpa.yaml


## ConfigMap, liveness/readiness/startup probes

Do not treat infra/k8s/deployment.yaml as production-ready as checked in: it uses the local placeholder image stagepilot-api:latest with IfNotPresent for developer clusters. </details>

<details> <summary><strong>Vercel / Cloudflare Workers</strong></summary>

See vercel.json and wrangler.toml in the repo root. </details>

Observability Stack

┌─────────────────────────────────────────────────────┐
│                  StagePilot API                      │
│                                                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │
│  │ OTel SDK │  │Prometheus│  │  Datadog Agent   │  │
│  │  Spans   │  │ Counters │  │  (optional)      │  │
│  └────┬─────┘  └────┬─────┘  └────────┬─────────┘  │
│       │              │                 │            │
└───────┼──────────────┼─────────────────┼────────────┘
        │              │                 │
   ┌────▼────┐   ┌─────▼─────┐   ┌──────▼──────┐
   │  Jaeger  │   │ Grafana   │   │  Datadog    │
   │  Zipkin  │   │ Dashboard │   │  Dashboard  │
   └──────────┘   └───────────┘   └─────────────┘

• OpenTelemetry: Per-stage spans (eligibility_span, safety_span, planner_span, outreach_span, judge_span, parse_span)
• Prometheus: toolCallsTotal counter + toolCallParseDuration histogram, scraped via /v1/metrics
• Datadog: Pre-built dashboard + monitor configs in docs/datadog/

Project Layout

src/
  adapters/          # AWS S3/CloudWatch, GCP integrations
  api/               # HTTP server, Prometheus metrics, sessions
  bin/               # CLI entry points (stagepilot-api, benchlab-api)
  community/         # Community protocols (Sijawara, UI-TARS)
  core/              # Parser protocols (8 variants), prompts, utils
  rjson/             # Relaxed JSON parser with repair heuristics
  rxml/              # Relaxed XML parser with tokenizer + schema extraction
  schema-coerce/     # Type coercion engine
  stagepilot/        # 5-agent orchestrator, benchmark, insights, twin
  telemetry/         # OpenTelemetry + Prometheus instrumentation
  __tests__/         # ~174 unit test files
tests/               # ~13 integration test files
infra/
  k8s/               # Local/BYI Deployment, Service, HPA, ConfigMap
  terraform/         # GCP Cloud Run provisioning
docs/
  adr/               # Architecture Decision Records
  benchmarks/        # Benchmark artifacts + reports
  benchlab/          # BFCL experiment docs
  datadog/           # Dashboard + monitor configs
experiments/         # 5 BFCL experiment variants (Claude, Gemini, Grok, Kiro, OpenAI-compat)
scripts/             # Build, deploy, load-test (k6)
.github/workflows/   # CI/CD pipelines