Compare commits

...

2 Commits

Author SHA1 Message Date
johnlanni
8e426a5e99 feat: add higress-deep-troubleshooting skill
Distill the investigation methodology used in #4034 (Wasm CPU-spin root
cause across envoy + proxy-wasm-cpp-host forks) into a reusable skill
that future investigations can pick up and extend.

Structure:
- SKILL.md: entry point with trigger description and code map
- references/: investigation methodology + code-trace verification rules
- patterns/: accumulating case-study library; first entry is the #4034
  drain-to-local fix (covers both CPU-spin and #326 stage-mismatch)
- templates/: investigation report + state-trace table formats

The patterns/ directory is the living part — each major investigation
adds a pattern capturing the technical root cause plus meta-lessons
(what tripped up the first analysis round, what the maintainer caught,
what generalizes).

Origin: #4034 / #4064 issue-spec investigation session
s-6de6de055cd791bc3f3b.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-07 12:59:07 +00:00
澄潭
d5f2b624fb feat: initialize issue-spec workflow for Higress (#4059)
Signed-off-by: johnlanni <johnlanni@users.noreply.github.com>
Co-authored-by: johnlanni <johnlanni@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:55:53 +08:00
28 changed files with 1289 additions and 1 deletions

View File

@@ -0,0 +1,35 @@
---
name: issue-spec-apply
description: Implement PROCESS comments for an issue-spec change and keep PR traceability synchronized.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Apply
Use when the user asks for /issue-spec:apply, issue-spec apply, or implementing PROCESS/TASK scopes from an issue-spec change.
## Steps
1. Read proposal/design/implement issue context and list typed comments with issue-spec comment list --json.
2. Confirm issue-spec auth status --json includes the expected GitHub backend. Local gh-authenticated sessions can use the native gh backend; keep ISSUE_SPEC_TOKEN="$(gh auth token)" only as an older-version or forced-rest compatibility path.
3. Create or update PROCESS comments with owner agent, scope, dependencies, write ownership, and status.
4. Split non-trivial work into independent worker PROCESS nodes when file/module ownership does not overlap; execute independent workers in parallel when available.
5. Add dedicated review PROCESS nodes for non-trivial changes. Review PROCESS nodes should own review scopes such as CLI/API behavior, workflow docs, tests, compatibility, or security-sensitive surfaces.
6. Link each PROCESS to its TASK comments with issue-spec link.
7. Implement the code changes for one PROCESS scope at a time, or integrate completed worker outputs by dependency order.
8. Link every worker and review PROCESS to the PR with issue-spec pr link-process.
9. Add PR rationale comments on key changed lines with issue-spec pr rationale, each linked to a SPEC comment.
10. Mark PROCESS comments done only after implementation/review work and focused verification evidence exist.
## Coordinator DAG Execution
1. Build the ready set from PROCESS nodes whose dependencies are done.
2. Keep immediate blocking work local when the next step depends on it.
3. Spawn or assign independent worker agents only when their write ownership is disjoint.
4. Spawn or assign independent review agents only when their review scopes are disjoint.
5. Integrate completed outputs by dependency order and update PROCESS evidence before marking done.

View File

@@ -0,0 +1,24 @@
---
name: issue-spec-archive
description: Create the post-merge durable spec archive PR for an issue-spec change.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Archive
Use when the user asks for /issue-spec:archive, issue-spec archive, or creating the post-merge durable spec PR.
## Steps
1. Confirm the implementation PR is merged.
2. Create the durable spec PR:
issue-spec archive durable-spec --repo higress-group/higress --proposal <issue> --capability <capability> --create-pr --branch issue-spec/durable-spec-<capability> --json
3. Review the durable spec PR for long-lived behavior only. Do not copy process records, review findings, or verification logs into durable specs.
4. After durable spec PR merge, keep proposal/design/implement issues as audit history unless the project policy says to close them.

View File

@@ -0,0 +1,58 @@
---
name: issue-spec-github
description: Use GitHub CLI for GitHub issues, pull requests, CI runs, and API queries that issue-spec does not wrap.
license: MIT
compatibility: Requires GitHub CLI (gh).
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# GitHub CLI
Use the `gh` CLI to interact with GitHub repositories, issues, pull requests, CI, and API endpoints.
## When To Use
- Checking PR status, reviews, mergeability, or CI checks.
- Creating, viewing, updating, closing, or commenting on GitHub issues.
- Listing or inspecting pull requests, workflow runs, releases, labels, or repository metadata.
- Calling GitHub API endpoints with `gh api` when issue-spec does not provide a dedicated command.
## When Not To Use
- Local git operations such as commit, branch, fetch, merge, or push. Use `git` directly.
- Non-GitHub repositories. Use the matching provider CLI instead.
- Complex code review across local diffs. Read the repository files directly and use issue-spec review commands for traceable findings.
## Setup
```bash
gh auth login
gh auth status
```
## Common Commands
```bash
gh issue list --repo owner/repo --state open
gh issue view 42 --repo owner/repo --json number,title,state,url,body
gh issue comment 42 --repo owner/repo --body "Comment body"
gh pr list --repo owner/repo
gh pr view 17 --repo owner/repo --json number,title,state,headRefName,baseRefName,url
gh pr checks 17 --repo owner/repo
gh run list --repo owner/repo --limit 10
gh run view <run-id> --repo owner/repo --log-failed
gh api repos/owner/repo/labels --jq '.[].name'
```
## Notes
- Always pass `--repo owner/repo` when the current directory is not definitely inside the target repository.
- Use GitHub URLs directly when convenient, for example `gh pr view https://github.com/owner/repo/pull/17`.
- Prefer structured output with `--json` and `--jq` when another command or agent step consumes the result.
- issue-spec owns the proposal, design, implement, typed comment, review, verify, and archive workflow state. Use `gh` for adjacent GitHub operations that are outside issue-spec's command surface.

View File

@@ -0,0 +1,37 @@
---
name: issue-spec-propose
description: Create or continue proposal, SPEC, QUESTION, design, and TASK artifacts for an issue-spec change.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Propose
Use when the user asks for /issue-spec:propose, issue-spec propose, creating a change proposal, drafting SPEC comments, or preparing design/tasks after questions converge.
## Steps
1. Create the proposal issue:
issue-spec issue create proposal --repo higress-group/higress --change <change-name> --body-file <proposal.md>
2. If the proposal body needs revision after discussion, update it in place:
issue-spec issue update --repo higress-group/higress --issue <proposal-issue> --body-file <proposal.md> --summary "<what changed>"
3. Add SPEC comments with issue-spec comment upsert --type SPEC. SPEC comments must use MUST/SHALL and WHEN/THEN scenarios.
4. Add QUESTION comments for unresolved behavior with issue-spec question create and resolve blocking questions before design.
5. Create the design issue after SPEC/QUESTION convergence:
issue-spec issue create design --repo higress-group/higress --change <change-name> --proposal <proposal-issue-or-url> --body-file <design.md>
6. Add TASK comments with issue-spec comment upsert --type TASK and link every TASK to covered SPEC comments with issue-spec link.
7. Create the implement issue once tasks are ready:
issue-spec issue create implement --repo higress-group/higress --change <change-name> --proposal <proposal-issue-or-url> --design <design-issue-or-url> --body-file <implement.md>
8. Run issue-spec verify-links and fix missing backlinks before implementation.

View File

@@ -0,0 +1,32 @@
---
name: issue-spec-review
description: Review an issue-spec implementation PR, create PR line findings, reply after fixes, and sync REVIEW comments.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Review
Use when the user asks for /issue-spec:review, issue-spec review, or a PR review gate for an issue-spec implementation.
## Steps
1. Run issue-spec review sync --repo higress-group/higress --pr <number> --implement <issue> --id REVIEW-<n> --json to capture current rationale comments, findings, and checks.
2. For non-trivial PRs, spawn or assign dedicated review agents as review PROCESS owners. Multiple review agents can run in parallel when their review scopes are independent.
3. Give each review agent a concrete scope and expected output: actionable findings only, severity, file/line, linked SPEC, owner PROCESS, and suggested fix.
4. Create actionable PR line findings with issue-spec review finding. Use P0/P1 for blockers and P2 for non-blocking follow-up.
5. Assign every finding to a PROCESS owner. If no findings are found, record that result in REVIEW or VERIFY evidence.
6. After the worker fixes a finding, reply to the original thread with issue-spec review reply --status resolved.
7. Re-run review sync. P0/P1 findings must be resolved before final verify/archive.
## Review DAG Policy
1. Every non-trivial PR should have at least one dedicated review PROCESS node before final verify.
2. Use multiple review agents in parallel when scopes are independent, for example CLI/API behavior, workflow docs, tests, compatibility, or security-sensitive surfaces.
3. A review agent reports findings only; the coordinator converts actionable line findings into issue-spec review finding comments.
4. P0/P1 findings block final verify until the owner PROCESS fixes them and issue-spec review reply records the resolution on the original thread.
5. If a review agent finds no issues, record that result in REVIEW or VERIFY evidence before marking the review PROCESS done.

View File

@@ -0,0 +1,28 @@
---
name: issue-spec-verify
description: Run final issue-spec verification across traceability, questions, review findings, PR rationale, PR checks, and durable spec draft.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Verify
Use when the user asks for /issue-spec:verify, issue-spec verify, or final readiness evidence before merge/archive.
## Steps
1. Run focused project tests and record evidence in VERIFY comments.
2. Run issue-spec verify-links --repo higress-group/higress --proposal <issue> --design <issue> --implement <issue> --json.
3. Render a durable spec draft:
issue-spec archive durable-spec --repo higress-group/higress --proposal <issue> --capability <capability> --output /tmp/<capability>-spec.md --json
4. Run final verify:
issue-spec verify --repo higress-group/higress --proposal <issue> --design <issue> --implement <issue> --pr <pr> --durable-spec /tmp/<capability>-spec.md --json
5. Final verify must fail if blocking questions, missing links, missing PROCESS rationale, open P0/P1 findings, failed or pending PR checks, or durable spec omissions exist.

View File

@@ -0,0 +1,53 @@
---
name: issue-spec-workflow
description: Use issue-spec to run an issue-native OpenSpec-style workflow with GitHub issues, typed comments, PR review comments, final verification, and durable spec archive PRs.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Workflow
Use this skill for issue-native OpenSpec work. Active change artifacts live in GitHub issues and issue comments; durable specs are repository files created after implementation merge.
## Start
1. Run issue-spec auth status --json and confirm the active auth source and GitHub backend.
2. Run issue-spec status --repo higress-group/higress --proposal <issue> --design <issue> --implement <issue> --json when issues already exist.
3. For new work, create proposal, design, and implement issues with issue-spec issue create and pass --body-file with concrete markdown content.
4. When an issue body changes, update it in place with issue-spec issue update --body-file and include --summary for the human-readable audit trail.
5. Store requirements, tasks, process ownership, review, and verify evidence as typed comments.
## GitHub Backend
- Local agents may rely on native GitHub CLI support: when no ISSUE_SPEC_TOKEN, GH_TOKEN, GITHUB_TOKEN, keyring token, or issue-spec config token is present and gh auth status --active succeeds for the target host, issue-spec auto-selects the gh backend.
- Explicit env or stored issue-spec tokens keep the rest backend under auto selection. Set ISSUE_SPEC_GITHUB_BACKEND=rest or ISSUE_SPEC_GITHUB_BACKEND=gh only when a workflow needs deterministic backend selection.
- The gh backend proxies GitHub API operations through gh api and uses gh --hostname for Enterprise hosts. It does not replace local git commands.
- ISSUE_SPEC_API_URL applies to the rest backend. Forced gh mode should be used only with hosts that gh can address.
- Use ISSUE_SPEC_TOKEN="$(gh auth token)" only for older issue-spec versions or when deliberately forcing rest while sourcing the token from gh.
## Rules
- Create SPEC comments before design; each SPEC must be testable and include WHEN/THEN scenarios.
- Do not leave active proposal/design/implement issue bodies as TBD placeholders.
- Resolve blocking QUESTION comments before design/tasks, or explicitly record accepted assumptions.
- Link SPEC <-> TASK and TASK <-> PROCESS with issue-spec link.
- Link every PROCESS to the implementation PR with issue-spec pr link-process.
- For non-trivial changes, include review PROCESS nodes in the DAG; review agents are scheduled like worker agents and can run in parallel when their review scopes are independent.
- Small changes may stay coordinator-only, but record the serial execution decision in the implement or VERIFY evidence.
- Before human review, add PR rationale comments with issue-spec pr rationale for every active PROCESS.
- Use issue-spec review finding for PR line findings and issue-spec review reply to close the original thread.
- Run issue-spec review sync and issue-spec verify before declaring ready.
- After the implementation PR merges, create the separate durable spec PR with issue-spec archive durable-spec --create-pr.
## Coordinator DAG Execution
1. Treat PROCESS comments as DAG nodes with explicit owner, dependencies, write or review scope, PR link, and evidence.
2. Select ready PROCESS nodes whose dependencies are done and whose scopes do not overlap.
3. Dispatch independent worker PROCESS nodes in parallel when their file/module ownership is disjoint.
4. Dispatch independent review PROCESS nodes in parallel for non-trivial PRs after PR rationale exists.
5. Integrate completed worker outputs by dependency order; route P0/P1 review findings back to the owner PROCESS.
6. Mark PROCESS nodes done only after their implementation or review evidence is recorded and blocking findings are resolved.

View File

@@ -0,0 +1,31 @@
---
name: "Issue Spec: Apply"
description: "Implement PROCESS comments for an issue-spec change and keep PR traceability synchronized."
category: "Workflow"
tags: ["workflow", "issue-spec"]
---
# Issue Spec Apply
Use when the user asks for /issue-spec:apply, issue-spec apply, or implementing PROCESS/TASK scopes from an issue-spec change.
## Steps
1. Read proposal/design/implement issue context and list typed comments with issue-spec comment list --json.
2. Confirm issue-spec auth status --json includes the expected GitHub backend. Local gh-authenticated sessions can use the native gh backend; keep ISSUE_SPEC_TOKEN="$(gh auth token)" only as an older-version or forced-rest compatibility path.
3. Create or update PROCESS comments with owner agent, scope, dependencies, write ownership, and status.
4. Split non-trivial work into independent worker PROCESS nodes when file/module ownership does not overlap; execute independent workers in parallel when available.
5. Add dedicated review PROCESS nodes for non-trivial changes. Review PROCESS nodes should own review scopes such as CLI/API behavior, workflow docs, tests, compatibility, or security-sensitive surfaces.
6. Link each PROCESS to its TASK comments with issue-spec link.
7. Implement the code changes for one PROCESS scope at a time, or integrate completed worker outputs by dependency order.
8. Link every worker and review PROCESS to the PR with issue-spec pr link-process.
9. Add PR rationale comments on key changed lines with issue-spec pr rationale, each linked to a SPEC comment.
10. Mark PROCESS comments done only after implementation/review work and focused verification evidence exist.
## Coordinator DAG Execution
1. Build the ready set from PROCESS nodes whose dependencies are done.
2. Keep immediate blocking work local when the next step depends on it.
3. Spawn or assign independent worker agents only when their write ownership is disjoint.
4. Spawn or assign independent review agents only when their review scopes are disjoint.
5. Integrate completed outputs by dependency order and update PROCESS evidence before marking done.

View File

@@ -0,0 +1,20 @@
---
name: "Issue Spec: Archive"
description: "Create the post-merge durable spec archive PR for an issue-spec change."
category: "Workflow"
tags: ["workflow", "issue-spec"]
---
# Issue Spec Archive
Use when the user asks for /issue-spec:archive, issue-spec archive, or creating the post-merge durable spec PR.
## Steps
1. Confirm the implementation PR is merged.
2. Create the durable spec PR:
issue-spec archive durable-spec --repo higress-group/higress --proposal <issue> --capability <capability> --create-pr --branch issue-spec/durable-spec-<capability> --json
3. Review the durable spec PR for long-lived behavior only. Do not copy process records, review findings, or verification logs into durable specs.
4. After durable spec PR merge, keep proposal/design/implement issues as audit history unless the project policy says to close them.

View File

@@ -0,0 +1,33 @@
---
name: "Issue Spec: Propose"
description: "Create or continue proposal, SPEC, QUESTION, design, and TASK artifacts for an issue-spec change."
category: "Workflow"
tags: ["workflow", "issue-spec"]
---
# Issue Spec Propose
Use when the user asks for /issue-spec:propose, issue-spec propose, creating a change proposal, drafting SPEC comments, or preparing design/tasks after questions converge.
## Steps
1. Create the proposal issue:
issue-spec issue create proposal --repo higress-group/higress --change <change-name> --body-file <proposal.md>
2. If the proposal body needs revision after discussion, update it in place:
issue-spec issue update --repo higress-group/higress --issue <proposal-issue> --body-file <proposal.md> --summary "<what changed>"
3. Add SPEC comments with issue-spec comment upsert --type SPEC. SPEC comments must use MUST/SHALL and WHEN/THEN scenarios.
4. Add QUESTION comments for unresolved behavior with issue-spec question create and resolve blocking questions before design.
5. Create the design issue after SPEC/QUESTION convergence:
issue-spec issue create design --repo higress-group/higress --change <change-name> --proposal <proposal-issue-or-url> --body-file <design.md>
6. Add TASK comments with issue-spec comment upsert --type TASK and link every TASK to covered SPEC comments with issue-spec link.
7. Create the implement issue once tasks are ready:
issue-spec issue create implement --repo higress-group/higress --change <change-name> --proposal <proposal-issue-or-url> --design <design-issue-or-url> --body-file <implement.md>
8. Run issue-spec verify-links and fix missing backlinks before implementation.

View File

@@ -0,0 +1,28 @@
---
name: "Issue Spec: Review"
description: "Review an issue-spec implementation PR, create PR line findings, reply after fixes, and sync REVIEW comments."
category: "Workflow"
tags: ["workflow", "issue-spec"]
---
# Issue Spec Review
Use when the user asks for /issue-spec:review, issue-spec review, or a PR review gate for an issue-spec implementation.
## Steps
1. Run issue-spec review sync --repo higress-group/higress --pr <number> --implement <issue> --id REVIEW-<n> --json to capture current rationale comments, findings, and checks.
2. For non-trivial PRs, spawn or assign dedicated review agents as review PROCESS owners. Multiple review agents can run in parallel when their review scopes are independent.
3. Give each review agent a concrete scope and expected output: actionable findings only, severity, file/line, linked SPEC, owner PROCESS, and suggested fix.
4. Create actionable PR line findings with issue-spec review finding. Use P0/P1 for blockers and P2 for non-blocking follow-up.
5. Assign every finding to a PROCESS owner. If no findings are found, record that result in REVIEW or VERIFY evidence.
6. After the worker fixes a finding, reply to the original thread with issue-spec review reply --status resolved.
7. Re-run review sync. P0/P1 findings must be resolved before final verify/archive.
## Review DAG Policy
1. Every non-trivial PR should have at least one dedicated review PROCESS node before final verify.
2. Use multiple review agents in parallel when scopes are independent, for example CLI/API behavior, workflow docs, tests, compatibility, or security-sensitive surfaces.
3. A review agent reports findings only; the coordinator converts actionable line findings into issue-spec review finding comments.
4. P0/P1 findings block final verify until the owner PROCESS fixes them and issue-spec review reply records the resolution on the original thread.
5. If a review agent finds no issues, record that result in REVIEW or VERIFY evidence before marking the review PROCESS done.

View File

@@ -0,0 +1,24 @@
---
name: "Issue Spec: Verify"
description: "Run final issue-spec verification across traceability, questions, review findings, PR rationale, PR checks, and durable spec draft."
category: "Workflow"
tags: ["workflow", "issue-spec"]
---
# Issue Spec Verify
Use when the user asks for /issue-spec:verify, issue-spec verify, or final readiness evidence before merge/archive.
## Steps
1. Run focused project tests and record evidence in VERIFY comments.
2. Run issue-spec verify-links --repo higress-group/higress --proposal <issue> --design <issue> --implement <issue> --json.
3. Render a durable spec draft:
issue-spec archive durable-spec --repo higress-group/higress --proposal <issue> --capability <capability> --output /tmp/<capability>-spec.md --json
4. Run final verify:
issue-spec verify --repo higress-group/higress --proposal <issue> --design <issue> --implement <issue> --pr <pr> --durable-spec /tmp/<capability>-spec.md --json
5. Final verify must fail if blocking questions, missing links, missing PROCESS rationale, open P0/P1 findings, failed or pending PR checks, or durable spec omissions exist.

View File

@@ -0,0 +1,61 @@
---
name: higress-deep-troubleshooting
description: Deep-dive troubleshooting methodology for Higress gateway bugs that require code-trace evidence across the higress-group/envoy and higress-group/proxy-wasm-cpp-host forks — CPU spins, infinite loops, Wasm VM reentrancy, filter-chain stage-mismatch, callback-ordering issues, SaveRestoreContext/current_context_ bugs. Use when investigating a complex Higress/Wasm/proxy-wasm incident that needs root-cause analysis with file:line citations, when ramping up on a reported CPU-spin or crash, or when verifying whether a proposed fix covers related bug classes. Accumulates case-study patterns under patterns/ — add a new pattern after each major investigation.
---
# Higress Deep Troubleshooting
A methodology plus an accumulating pattern library for deep-diving into Higress gateway bugs that require code-level root cause analysis. The codebase spans two Higress-maintained forks — [`higress-group/envoy`](https://github.com/higress-group/envoy) (filter chain, Wasm `Context` integration) and [`higress-group/proxy-wasm-cpp-host`](https://github.com/higress-group/proxy-wasm-cpp-host) (Wasm VM host runtime, `WasmBase`, `ContextBase`) — and bugs often cross the boundary between them. This skill exists so that the next investigation doesn't start from scratch.
## When to use
Reach for this skill when ANY of these apply:
- A Higress issue reports a CPU spin, infinite loop, crash, or stage-mismatched behavior.
- The bug involves the Wasm plugin lifecycle, `addAfterVmCallAction` / `doAfterVmCallActions`, `SaveRestoreContext`, `current_context_`, or the filter chain reentry paths.
- You need to verify whether a one-repo fix also covers an upstream bug class (e.g., `proxy-wasm/proxy-wasm-cpp-host#326`).
- You're ramping up on an incident and need a structured way to collect evidence before proposing a fix.
- The maintainer is pushing back on a conclusion and you need to re-verify with code.
Do NOT reach for this skill for routine plugin development (`higress-wasm-go-plugin` covers that), config issues, or anything solvable by reading a single file without cross-repo context.
## Investigation methodology (brief)
1. **Symptom → code surface.** From perf data / repro steps / stack trace, identify which functions and files are involved. Note which fork each file lives in.
2. **Parallel investigation.** Dispatch focused worker agents for independent aspects (envoy side, proxy-wasm-cpp-host side, upstream references). Each worker returns file:line evidence — never narration.
3. **Code-trace verification.** Every claim must cite `file:line`. No "I think it does X" — only "see envoy context.cc:2085 where it calls ... synchronously".
4. **Synthesize root cause.** Combine worker findings into a numbered call-chain trace, showing program state (queue contents, `current_context_` value, etc.) at each hop.
5. **Fix design + side effects.** Propose a primitive, analyze what else it touches, check if it covers related bug classes upstream.
6. **Verify when challenged.** If the maintainer pushes back, re-verify with code — don't restate the conclusion. The earlier analysis is often wrong in some detail that only a fresh code read catches.
Full process: [`references/investigation-methodology.md`](references/investigation-methodology.md). Verification patterns: [`references/code-trace-verification.md`](references/code-trace-verification.md).
## Accumulated patterns
Each major investigation that uses this skill should add a pattern under `patterns/`. Patterns are the "living" part of this skill — they capture both the technical root cause AND the meta-lessons (what tripped up the analysis, what the maintainer caught, what the fix primitive generalizes to).
Current patterns:
- [`patterns/2026-07-wasm-doaftervmcallactions-reentry.md`](patterns/2026-07-wasm-doaftervmcallactions-reentry.md) — #4034 CPU 100% from `onRedisCallFailure` + `injectEncodedDataToFilterChain` nested reentry. Root cause: unguarded `while(!empty())` drain in `WasmBase::doAfterVmCallActions`. Fix: drain-to-local via `std::deque::swap`. Meta-lesson: the same primitive covers the upstream #326 stage-mismatch class, which the first analysis round missed.
To add a new pattern after an investigation, see [`patterns/README.md`](patterns/README.md).
## Templates
- [`templates/investigation-report.md`](templates/investigation-report.md) — structure for a new investigation's findings.
- [`templates/code-trace-table.md`](templates/code-trace-table.md) — step-by-step state-trace table format (very effective for communicating queue/state transitions to maintainers).
## Key code maps (quick orientation)
When investigating a Wasm-related Higress bug, the usual suspects:
| Concern | Repo | File |
| --- | --- | --- |
| Plugin callback entry points (`onRequestHeaders`, `onResponseBody`, `onHttpCallResponse`, etc.) | `higress-group/proxy-wasm-cpp-host` | `src/context.cc` |
| Deferred-action queue (`addAfterVmCallAction`, `doAfterVmCallActions`, `DeferAfterCallActions`) | `higress-group/proxy-wasm-cpp-host` | `include/proxy-wasm/wasm.h`, `src/context.cc` |
| `SaveRestoreContext` / `current_context_` | `higress-group/proxy-wasm-cpp-host` | `include/proxy-wasm/wasm_vm.h`, `src/exports.cc` |
| Higress envoy `Context` (filter chain integration, foreign functions) | `higress-group/envoy` | `source/extensions/common/wasm/context.cc` |
| Foreign-function registration (`proxy_inject_encoded_data_to_filter_chain`, etc.) | `higress-group/envoy` | `source/extensions/common/wasm/foreign.cc` |
| envoy fork's pinned proxy-wasm-cpp-host commit | `higress-group/envoy` | `bazel/repository_locations.bzl` |
The Higress envoy fork pins a specific `proxy-wasm-cpp-host` commit via bazel — always check which fork commit is pinned before reasoning about `WasmBase` behavior, because the fork diverges from upstream (RedisCall, rebuild/recover, weak_handle additions).

View File

@@ -0,0 +1,116 @@
# Wasm `doAfterVmCallActions` reentry — CPU-spin and stage-mismatch, fixed by drain-to-local
> Source: [#4034](https://github.com/higress-group/higress/issues/4034) — Wasm 插件死循环onRedisCallFailure + injectEncodedDataToFilterChain 嵌套导致 Envoy CPU 100% 跑满
> Upstream relative: [proxy-wasm/proxy-wasm-cpp-host#326](https://github.com/proxy-wasm/proxy-wasm-cpp-host/issues/326)
> Date: 2026-07
> Status: fix designed (SPEC-001), implementation pending
## Symptom
`release-2.2.2` Envoy pods CPU 100% on all 6 pods within ~3 minutes under concurrent load. Trigger: Redis unreachable + plugin calls `InjectEncodedDataToFilterChain` from an HTTP/Redis callback. perf:
| Overhead | Function |
| --- | --- |
| 39.21% | `Envoy::Extensions::Common::Wasm::Context::onRedisCallFailure` |
| 25.41% | `proxy_wasm::WasmBase::doAfterVmCallActions` |
| **合计** | **64.62%** |
## Root cause
`WasmBase::doAfterVmCallActions` (`higress-group/proxy-wasm-cpp-host` `include/proxy-wasm/wasm.h:158-169`) drains `after_vm_call_actions_` in an unguarded `while(!empty())` loop:
```cpp
while (!self->after_vm_call_actions_.empty()) {
auto f = std::move(self->after_vm_call_actions_.front());
self->after_vm_call_actions_.pop_front();
f(); // if f() re-adds an action, loop never terminates
}
```
Several Envoy-side callbacks re-defer themselves when `current_context_ != nullptr` (meaning: "VM is still on the stack, defer until it unwinds"):
- `Context::onRedisCallFailure` (envoy `context.cc:1273-1277`)
- `Context::onRedisCallSuccess` (envoy `context.cc:1243-1249`)
- `Context::onHttpCallSuccess` (envoy `context.cc:2352-2358`)
- `Context::onHttpCallFailure` (envoy `context.cc:2382-2386`)
Under a synchronous reentry path — `injectEncodedDataToFilterChain` (envoy `context.cc:2085-2091`) is called from inside an HTTP/Redis callback — the outer `SaveRestoreContext` keeps `current_context_` non-null even after the inner `~SaveRestoreContext` unwinds. So the drain loop pops a self-reenqueueing callback, fires it, it sees `current_context_ != nullptr`, re-adds itself, loop iterates again → infinite → CPU 100%.
Full call chain (10 hops, each with file:line) is in [the #4034 proposal issue](https://github.com/higress-group/higress/issues/4064) body and SPEC-001.
## Fix
Swap the member queue to a local deque before iterating, so any action re-added during the drain lands in the (now-empty) member queue and is picked up by the next-outer `DeferAfterCallActions` frame — not re-fired within the same drain.
```cpp
void doAfterVmCallActions() {
if (!after_vm_call_actions_.empty()) {
auto self = shared_from_this();
std::deque<std::function<void()>> local;
local.swap(self->after_vm_call_actions_);
for (auto& f : local) f();
}
}
```
`std::deque::swap` is O(1) whole-content exchange. After swap: `local` has the pre-existing queue, `self->after_vm_call_actions_` is empty. The for-loop iterates `local`; any `addAfterVmCallAction` call during iteration appends to the empty member, invisible to this drain.
## Coverage
The same primitive covers BOTH:
- **#4034 (CPU-spin)**: re-added actions go to the empty member queue → picked up by next outer frame → loop terminates.
- **#326 (stage-mismatch)**: pre-existing actions stay in `local` → inner drain (from a nested `~DeferAfterCallActions`) sees empty member → does nothing → pre-existing actions fire in the outer drain's correct phase.
The coverage of #326 is non-obvious — see Meta-lessons below.
## Evidence
| Claim | Citation |
| --- | --- |
| Unguarded drain loop | `higress-group/proxy-wasm-cpp-host` `include/proxy-wasm/wasm.h:158-169` |
| Self-reenqueue guard `if (current_context_ != nullptr)` | `higress-group/envoy` `source/extensions/common/wasm/context.cc:1273-1277` (onRedisCallFailure); also lines 1243, 2352, 2382 |
| Sync `injectEncodedDataToFilterChain` (the trigger) | `higress-group/envoy` `source/extensions/common/wasm/context.cc:2085-2091` |
| Async sibling `injectEncodedDataToFilterChainOnHeader` (the contrast) | `higress-group/envoy` `source/extensions/common/wasm/context.cc:2093-2103` |
| `sendLocalReply` is deferred via `addAfterVmCallAction` | `higress-group/envoy` `source/extensions/common/wasm/context.cc:1638-1657` |
| `continueDecoding` is deferred via `addAfterVmCallAction` | `higress-group/envoy` `source/extensions/common/wasm/context.cc:1508-1514` |
| `~DeferAfterCallActions` drains unconditionally | `higress-group/proxy-wasm-cpp-host` `src/context.cc` (top, `~DeferAfterCallActions` calls `wasm_->doAfterVmCallActions()`) |
| `SaveRestoreContext` save/restore semantics | `higress-group/proxy-wasm-cpp-host` `include/proxy-wasm/wasm_vm.h:392-405` |
| `current_context_` is thread_local | `higress-group/proxy-wasm-cpp-host` `src/exports.cc:26` |
## Meta-lessons
This investigation had three rounds of analysis. The meta-lessons are the most valuable part for future investigators.
### Lesson 1: a single primitive can cover two bug classes — check coverage explicitly
The first analysis round concluded "Layer A fixes #4034 but NOT #326" and proposed them as separate work items. That was wrong. The `swap-to-local` primitive covers BOTH because:
- For #4034: re-added actions go to the empty member.
- For #326: pre-existing actions stay in `local`, invisible to the nested drain.
**Generalization**: whenever a fix changes how a queue/collection is iterated (swap-to-local, snapshot-then-iterate, copy-on-iterate), check whether it ALSO changes cross-frame or cross-stage visibility. These primitives often cover more than the primary bug.
### Lesson 2: model the PATCHED code, not the unpatched code
When analyzing whether a fix covers a related bug, it's easy to accidentally reason about the current (unpatched) behavior instead of the patched behavior. The first analysis round said "the inner drain sees `[continueDecoding]`" — which is true for the CURRENT code but FALSE for the patched code, because the patch's swap empties the member before the inner drain runs.
**Generalization**: when someone proposes a fix and asks "does this also cover X?", re-read the PATCHED code, not just the diff. The patch changes program state at moments you might not be tracking.
### Lesson 3: "X is deferred" vs "X is direct" is a load-bearing distinction
Several rounds of analysis hinged on whether `sendLocalReply` and `continueDecoding` are called directly or deferred via `addAfterVmCallAction`. Both are deferred — and that determines the queue order that produces the bug.
**Generalization**: when a function's name suggests "does X immediately" (like `sendLocalReply`), don't trust the name. Read the function body and check whether it actually calls X directly or wraps it in `addAfterVmCallAction` / `dispatcher().post(...)` / similar defer primitives. The defer-vs-direct distinction changes everything about reentrancy analysis.
### Lesson 4: maintainer pushback is a signal to re-verify, not restate
When the maintainer challenged the "Layer A doesn't fix #326" conclusion, the right move was to re-read the code with their specific objection in mind — not to restate the conclusion with different words. The re-read found that the swap-to-local empties the member queue, which the first analysis had missed.
**Generalization**: maintainer pushback is high-signal information. They know the codebase. If they say "I think X", dispatch a fresh worker to verify X with code evidence. Don't confirm your own previous answer; that's confirmation bias.
### Lesson 5: the #326 reproducer's plugin call order matters
The #326 reproducer calls `SendHttpResponse` THEN `ResumeHttpRequest`. If the order were reversed, the queue would be `[continueDecoding, sendLocalReply]`, FIFO drain would pop `continueDecoding` first (in the correct phase), then `sendLocalReply` (triggering encoder reentry with empty queue) — and there would be no stage mismatch even WITHOUT Layer A.
**Generalization**: when analyzing a reproducer, note the exact order of plugin calls. Stage-mismatch bugs often depend on call order, and a "harmless" reorder can mask the bug.

View File

@@ -0,0 +1,71 @@
# Patterns — How to Add a New Investigation Case-Study
This directory accumulates case-studies from past deep investigations. Each pattern captures both the technical root cause AND the meta-lessons (what tripped up the analysis, what the maintainer caught, what the fix primitive generalizes to). The library grows by one entry per major investigation.
## When to add a pattern
Add a pattern after an investigation that:
- Required cross-repo code-trace evidence (envoy + proxy-wasm-cpp-host, or similar).
- Resulted in a non-trivial fix (not just a one-line config change).
- Has a meta-lesson worth carrying forward (a gotcha, a verification failure, a primitive that generalizes).
If the investigation was routine (single-file bug, obvious fix, no pushback), skip the pattern — the diff IS the documentation.
## File naming
`YYYY-MM-short-slug.md` where the date is when the investigation concluded. Examples:
- `2026-07-wasm-doaftervmcallactions-reentry.md`
- `2026-09-filter-chain-headers-double-dispatch.md`
- `2026-11-redis-pool-exhaustion-under-failover.md`
## Pattern structure
Use this template. Sections in `[brackets]` are required; sections in `<angle brackets>` are optional but encouraged.
```markdown
# <Bug class one-line summary>
> Source: #ISSUE_NUMBER — <issue title>
> Date: YYYY-MM
> Status: <fixed / investigating / deferred>
## Symptom
<What observable behavior was wrong? Include perf data / repro steps if relevant.>
## Root cause
<One-paragraph explanation. Cite file:line for key claims.>
<Numbered call-chain trace, one hop per line, with file:line.>
## Fix
<The primitive. Code snippet if non-trivial.>
<Why it fixes the bug — the causal chain.>
## Coverage
<What bug classes does this fix cover? What does it NOT cover? Why?>
## Evidence
<File:line citations grouped by claim.>
## Meta-lessons
<What tripped up the first analysis round? What did the maintainer catch? What generalizes to other bugs in this area?>
<If the analysis was wrong initially and then corrected, document BOTH the wrong reasoning AND the correction — this is the most valuable part for future investigators.>
```
## Worked example
See [`2026-07-wasm-doaftervmcallactions-reentry.md`](2026-07-wasm-doaftervmcallactions-reentry.md) for a complete pattern that follows this structure, including a meta-lesson section documenting an initial wrong conclusion that was corrected after maintainer pushback.
## Linking from new investigations
When a new investigation reuses a primitive or hits a similar gotcha to a past pattern, link to the pattern in the new investigation's report. Patterns are most valuable when they're cross-referenced — the second time you see a `std::deque::swap`-style fix, you should be able to find the #4034 pattern and reason by analogy.

View File

@@ -0,0 +1,73 @@
# Code-Trace Verification
The format and rules for verifying claims with code evidence during a deep investigation. Maintainers trust analysis they can audit; file:line citations are the audit trail.
## Core rule
**Every claim must cite `file:line`.** If you can't cite it, you haven't read it; if you haven't read it, you don't know.
## Acceptable evidence
| Claim type | What to cite |
| --- | --- |
| "Function X does Y" | `repo/path/file.cc:NN` + ≤5-line excerpt |
| "Function X is synchronous" | The call site that lacks `dispatcher().post(...)` (or equivalent) + the sibling function that DOES use it, as contrast |
| "The queue state is `[a, b]` at this point" | The sequence of `push_back` / `pop_front` calls leading to that state, each with file:line |
| "The maintainer's claim is wrong" | Re-state their claim, then cite the specific code that contradicts it |
| "This fix covers bug class Y too" | The mechanism by which it covers Y, with the call chain showing why |
## Unacceptable evidence
- "I think" / "probably" / "should be" — read the code.
- "Similar to upstream" — check the fork commit pin in `bazel/repository_locations.bzl`.
- "The function name implies X" — names lie. `injectEncodedDataToFilterChain` sounds innocuous; it's the trigger for a CPU-spin bug.
- Reasoning from the bug report without code verification — the report describes symptoms.
## State-trace table format
The most effective format for communicating queue/state transitions. Use it whenever the bug involves a sequence of state changes (queue operations, `current_context_` saves/restores, reference-counting transitions).
Template:
| Step | Action | `<state variable 1>` | `<state variable 2>` |
| --- | --- | --- | --- |
| 1 | Enter context | `[]` | `nullptr` |
| 2 | Operation X | `[x]` | `nullptr` |
| 3 | Operation Y | `[x, y]` | `ctx` |
| ... | ... | ... | ... |
Rules:
- One row per discrete state change.
- State columns should be the variables that matter for the bug (queue contents, `current_context_`, depth counter, etc.).
- Mark the critical transition with ★ and explain it below the table.
- If a state is "unknown" or "depends on plugin behavior", say so — don't fake precision.
## Worked example (from #4034)
The #4034 investigation hinged on the queue state at the moment of a nested drain. The state-trace table that settled the disagreement:
| Step | Action | `self->after_vm_call_actions_` | outer drain's `local` |
| --- | --- | --- | --- |
| 1 | Enter `onHttpCallResponse` | `[]` | — |
| 2 | Plugin calls `SendHttpResponse` | `[sendLocalReply]` | — |
| 3 | Plugin calls `ResumeHttpRequest` | `[sendLocalReply, continueDecoding]` | — |
| 4 | Outer `~DeferAfterCallActions` fires | same | — |
| 5 | **Layer A: `local.swap(after_vm_call_actions_)`** ★ | `[]` | `[sendLocalReply, continueDecoding]` |
| 6 | iter 1: fire `sendLocalReply``encodeHeaders``onResponseHeaders` | `[]` | same |
| 7 | Inner `~DeferAfterCallActions` fires | `[]` (empty → no-op) | same |
| 8 | iter 2: fire `continueDecoding` in outer context | `[]` | same |
The ★ at step 5 is the key insight: `std::deque::swap` empties the member, so the inner drain at step 7 sees an empty queue and does nothing. The earlier (wrong) analysis confused "queue before swap" with "queue at inner-drain time".
## Re-verification protocol (when the maintainer pushes back)
When a maintainer disagrees with your conclusion:
1. **Don't restate the conclusion.** Restating without new evidence wastes their time.
2. **Identify the specific point of disagreement.** What exactly do they claim?
3. **Dispatch a fresh worker** with the specific objection. The worker brief should say: "Maintainer claims X. Verify whether X is true. Cite file:line."
4. **If the worker confirms the maintainer**: acknowledge the error publicly, explain what you missed, update artifacts.
5. **If the worker refutes the maintainer**: respond with the specific code evidence, walk through the state-trace table, be respectful.
The third verification (yours) doesn't count if it just restates the second (also yours). You need to actually re-read the code with the maintainer's objection in mind, not just re-derive your previous answer.

View File

@@ -0,0 +1,118 @@
# Investigation Methodology
The step-by-step process for deep-diving a Higress bug. This is the canonical workflow; adapt it to the bug at hand but don't skip the evidence-collection steps.
## 0. Before you start: scope the investigation
Confirm the bug is real and worth a deep dive. Signs it's worth it:
- Reproducer exists (even if intermittent).
- perf data or stack trace points at specific functions.
- The fix likely requires code changes (not config/docs).
- The blast radius is significant (production-affecting, multiple users).
If the bug is a one-off without a reproducer, or clearly a user-config issue, don't burn a deep investigation on it.
## 1. Symptom → code surface
Read the bug report and extract:
- **Symptom**: what observable behavior is wrong? (CPU spin, crash, wrong response, hang.)
- **Trigger conditions**: what's the minimal repro path?
- **perf / stack data**: which functions dominate?
Then map symptoms to code surfaces. For Wasm-related bugs, the surface is almost always split across:
- `higress-group/envoy` `source/extensions/common/wasm/context.cc` — Higress-specific `Context` subclass, foreign functions like `injectEncodedDataToFilterChain`.
- `higress-group/proxy-wasm-cpp-host` `include/proxy-wasm/wasm.h` + `src/context.cc` — VM host runtime, `WasmBase`, `ContextBase`, deferred-action queue.
**Always check the envoy fork's `bazel/repository_locations.bzl` for the pinned `proxy-wasm-cpp-host` commit.** The fork diverges from upstream; reasoning about upstream code without checking the pin leads to wrong conclusions.
## 2. Parallel investigation (worker dispatch)
For non-trivial bugs, dispatch focused worker agents IN PARALLEL for independent aspects. Each worker gets a self-contained brief (it has no shared context with you). Example dispatch for a Wasm reentrancy bug:
- **Worker A (envoy side)**: verify the trigger path in `higress-group/envoy`. Find the synchronous call, the self-reenqueueing callbacks, the foreign-function registration. Return file:line evidence.
- **Worker B (proxy-wasm-cpp-host side)**: verify the engine. Find the queue, the drain loop, the `SaveRestoreContext` semantics. Return file:line evidence.
- **Worker C (upstream references)**: fetch any linked upstream issues/PRs, determine if they're related/resolved/portable.
Worker rules:
- Each worker is **research-only** unless you explicitly authorize code changes. Don't let workers edit code on their own initiative.
- Each worker must **cite file:line for every claim**. Reject worker reports that say "the function does X" without a citation.
- Cap report length (e.g., 600-800 words) to keep synthesis tractable.
- Use `general-purpose` agent type for investigations that need to clone repos and reason across files; use `Explore` for narrow single-repo lookups.
## 3. Code-trace verification
Every claim in the final analysis must reduce to a file:line citation. The output format that works best for maintainer communication is the **step-by-step state-trace table**:
| Step | Action | Member queue state | Local queue state |
| --- | --- | --- | --- |
| 1 | Enter callback | `[]` | — |
| 2 | Plugin calls X | `[x]` | — |
| ... | ... | ... | ... |
See [`code-trace-verification.md`](code-trace-verification.md) for the full format and a worked example.
Anti-patterns to avoid:
- **"I think the function does X"** — unacceptable. Read the code and cite the line.
- **"This is probably similar to upstream"** — check the fork commit pin.
- **"The drain loop terminates because of course it does"** — verify the actual loop condition.
- **Reasoning from the bug report without reading the code** — the report describes symptoms, not causes.
## 4. Synthesize root cause
Combine worker findings into a numbered call-chain trace. For each hop:
- The function name.
- The file:line.
- What state it reads/writes (e.g., `current_context_` value, queue contents).
End with a one-sentence root cause: "X happens because Y, which is unguarded because Z."
## 5. Fix design + side effects
Propose a fix primitive. For each candidate:
- **What it changes**: the specific code transformation.
- **Why it fixes the bug**: the causal chain.
- **Side effects**: what else it touches. Be honest — even a 5-line fix can have ABI or lifecycle implications.
- **Coverage**: does it also cover related bug classes? (E.g., drain-to-local covers both CPU-spin and stage-mismatch.)
If a fix has unacceptable side effects, document it as "deferred" and explain why. Don't sneak a behavior-change through as a "bug fix".
## 6. Cross-check related bug classes
Before declaring done, check: does the fix also cover bug classes that share the same machinery?
- Look at upstream issues in `proxy-wasm/proxy-wasm-cpp-host` that reference the same functions.
- Look at the Higress fork's divergence from upstream — are there Higress-specific code paths that have the same bug?
- Look at the `addAfterVmCallAction` callsites in envoy — are there other self-reenqueueing or cross-stage patterns?
Document the coverage explicitly: "Fixes X. Does NOT fix Y because Z. Y tracked separately as ..."
## 7. Verify when challenged
If the maintainer pushes back on a conclusion, **do not just restate the conclusion**. Re-read the code with their specific objection in mind.
Common failure modes in the first analysis round:
- **Confusing "state before operation" with "state after operation"**. (E.g., "the queue is `[X]`" — but is that before or after the swap?)
- **Assuming a function is sync/async without checking**. (E.g., assuming `sendLocalReply` is direct when it's actually deferred via `addAfterVmCallAction`.)
- **Reasoning about the patched code as if it were the unpatched code**. (E.g., analyzing Layer A's `doAfterVmCallActions` as if it still iterates the member queue in place, when actually it swaps to a local first.)
- **Assuming plugin call order**. (E.g., assuming `ResumeHttpRequest` is called before `SendHttpResponse` — verify from the reproducer.)
When you re-verify, dispatch a fresh worker with the specific objection. The worker should be told: "the maintainer claims X; verify whether X is true with code evidence." This avoids confirmation bias.
## 8. Document the pattern
After the investigation concludes, add a pattern file to `patterns/` capturing:
- The bug class (one-line summary).
- The root cause (with file:line evidence).
- The fix primitive (with the code snippet).
- The meta-lesson (what tripped up the analysis, what the maintainer caught, what generalizes).
This is what makes the skill accumulate value over time. See [`patterns/README.md`](../patterns/README.md) for the format.

View File

@@ -0,0 +1,64 @@
# Code-Trace Table Template
Use this template for the state-trace table that communicates queue/state transitions to maintainers. This format was developed during the #4034 investigation and proved effective for settling disagreements about program state at specific moments.
## When to use
- The bug involves a sequence of state changes (queue operations, `current_context_` saves/restores, reference-count transitions).
- You need to communicate "what the program state is at moment X" to someone who wasn't in your head.
- A maintainer is challenging a conclusion about state at a specific point in the call chain.
## Template
```markdown
| Step | Action | `<state variable 1>` | `<state variable 2>` | ... |
| --- | --- | --- | --- | ... |
| 1 | <entry point> | `<initial value>` | `<initial value>` | ... |
| 2 | <operation> | `<value after>` | `<value after>` | ... |
| 3 | <operation> | `<value after>` | `<value after>` | ... |
| ... | ... | ... | ... | ... |
| N ★ | <critical transition> | `<value>` | `<value>` | ... |
```
## Rules
1. **One row per discrete state change.** Don't combine multiple operations into one row.
2. **State columns are the variables that matter for the bug.** Common ones for Wasm bugs:
- `self->after_vm_call_actions_` (the deferred-action queue)
- `current_context_` (the thread_local ContextBase*)
- `SaveRestoreContext` depth (how many nested VM frames)
- `local` (the outer drain's swap target, post-Layer-A)
3. **Mark the critical transition with ★** and explain it below the table. This is the moment where the bug happens (or doesn't).
4. **Show the queue contents explicitly**`[sendLocalReply, continueDecoding]`, not "non-empty". Order matters for FIFO drains.
5. **If state is unknown / depends on plugin behavior, say so.** Don't fake precision. Better: "depends on whether the plugin called X before Y; both branches shown below".
6. **Pair the table with a prose explanation of the ★ transition.** The table shows WHAT; the prose explains WHY.
## Worked example
From #4034 — the table that settled whether Layer A's swap-to-local covers #326:
```markdown
| Step | Action | `self->after_vm_call_actions_` | outer drain's `local` |
| --- | --- | --- | --- |
| 1 | Enter `onHttpCallResponse` | `[]` | — |
| 2 | Plugin calls `SendHttpResponse` | `[sendLocalReply]` | — |
| 3 | Plugin calls `ResumeHttpRequest` | `[sendLocalReply, continueDecoding]` | — |
| 4 | Outer `~DeferAfterCallActions` fires | `[sendLocalReply, continueDecoding]` | — |
| 5 ★ | **Layer A: `local.swap(after_vm_call_actions_)`** | `[]` | `[sendLocalReply, continueDecoding]` |
| 6 | iter 1: fire `sendLocalReply``encodeHeaders``onResponseHeaders` | `[]` | `[sendLocalReply, continueDecoding]` |
| 7 | Inner `~DeferAfterCallActions` fires | `[]` (empty → no-op) | `[sendLocalReply, continueDecoding]` |
| 8 | iter 2: fire `continueDecoding` in outer context | `[]` | (iterating) |
```
**Why the ★ matters**: `std::deque::swap` is O(1) whole-content exchange. After step 5, the member queue is empty — it's NOT `[continueDecoding]` anymore. So when step 7's inner drain checks `if (!after_vm_call_actions_.empty())`, it sees `false` and returns. `continueDecoding` stays in the outer drain's `local` and fires in step 8, in the outer drain's context (correct phase).
The earlier (wrong) analysis had assumed the member queue still contained `continueDecoding` at step 7 — confusing "queue before swap" with "queue after swap".
## Anti-patterns
Don't do these:
- **Combining steps**: "steps 5-8: drain happens, continueDecoding fires" — too coarse; the reader can't see the state at each moment.
- **Hiding the critical transition**: burying the ★ in a paragraph of prose. The table IS the argument; make it sharp.
- **Vague state**: "queue has some items" — which items? In what order? FIFO drains care about order.
- **No prose explanation**: the table alone isn't enough; pair it with 2-3 sentences explaining the ★.

View File

@@ -0,0 +1,109 @@
# Investigation Report Template
Use this structure when starting a new deep investigation. Fill in the brackets; remove sections that don't apply. The goal is a report that a maintainer can audit end-to-end: every claim has a citation, every fix has a side-effect analysis.
---
# Investigation: <one-line bug summary>
> Source: #ISSUE_NUMBER
> Investigator: <name / agent session>
> Date: YYYY-MM-DD
> Status: <in-progress / root-caused / fix-designed / fixed>
## Symptom
<What observable behavior is wrong? Include:>
- **Observable**: <what the user sees — CPU spin, crash, wrong response, hang>
- **Trigger conditions**: <minimal repro path>
- **perf / stack data**: <paste perf sampling, stack trace, or relevant telemetry>
- **Blast radius**: <which versions, which components, how many users>
## Code surfaces
<Which repos and files are involved? Use a table:>
| Concern | Repo | File | Why it matters |
| --- | --- | --- | --- |
| <concern> | <repo URL> | <path:line> | <one-line role> |
<For Wasm bugs, always check the envoy fork's `bazel/repository_locations.bzl` for the pinned proxy-wasm-cpp-host commit. Note it here.>
## Worker dispatch
<If you dispatched parallel workers, list them with their brief and key finding:>
- **Worker A (<repo>) — <one-line brief>**: <VERDICT — one-line finding>. Key evidence: <file:line>.
- **Worker B (<repo>) — <one-line brief>**: ...
<If the investigation was single-threaded (no workers), say so and explain why.>
## Root cause
<One-paragraph explanation. Be specific about the mechanism.>
<Numbered call-chain trace, one hop per line:>
1. <Function> (`repo file.cc:NN`) — <what it does>
2. <Function> (`repo file.cc:NN`) — <what it does>
3. ...
<For queue/state bugs, include a state-trace table here. See `code-trace-verification.md` for the format.>
**Root cause (one sentence)**: <X happens because Y, which is unguarded because Z>.
## Fix
<The primitive. Code snippet if non-trivial.>
```cpp
// Before
<current code>
// After
<fixed code>
```
**Why it fixes the bug**: <causal chain, 2-3 sentences.>
**Side effects**: <what else the change touches. Be honest — even a 5-line fix can have ABI or lifecycle implications. If there are none, say "none identified" and explain the verification.>
## Coverage analysis
<Does the fix cover related bug classes? Check upstream issues, related callbacks, similar patterns.>
| Bug class | Covered? | Why |
| --- | --- | --- |
| <primary bug> | yes | <mechanism> |
| <related class 1> | yes/no | <mechanism or reason> |
| <related class 2> | yes/no | ... |
<For each "no", note where it's tracked (separate issue, deferred SPEC, etc.).>
## Evidence index
<All file:line citations grouped by claim. This is the audit trail.>
| Claim | Citation |
| --- | --- |
| <claim> | `repo/path/file.cc:NN` |
| ... | ... |
## Open questions
<Any unresolved questions for the maintainer. Each should be specific and actionable.>
- **Q1**: <question>
- **Q2**: <question>
## Next steps
<What happens next. Be concrete:>
- [ ] <action 1>
- [ ] <action 2>
---
<After the investigation concludes, distill the meta-lessons and add a pattern file to `patterns/`. See `patterns/README.md` for the format.>

View File

@@ -0,0 +1,35 @@
---
name: issue-spec-apply
description: Implement PROCESS comments for an issue-spec change and keep PR traceability synchronized.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Apply
Use when the user asks for /issue-spec:apply, issue-spec apply, or implementing PROCESS/TASK scopes from an issue-spec change.
## Steps
1. Read proposal/design/implement issue context and list typed comments with issue-spec comment list --json.
2. Confirm issue-spec auth status --json includes the expected GitHub backend. Local gh-authenticated sessions can use the native gh backend; keep ISSUE_SPEC_TOKEN="$(gh auth token)" only as an older-version or forced-rest compatibility path.
3. Create or update PROCESS comments with owner agent, scope, dependencies, write ownership, and status.
4. Split non-trivial work into independent worker PROCESS nodes when file/module ownership does not overlap; execute independent workers in parallel when available.
5. Add dedicated review PROCESS nodes for non-trivial changes. Review PROCESS nodes should own review scopes such as CLI/API behavior, workflow docs, tests, compatibility, or security-sensitive surfaces.
6. Link each PROCESS to its TASK comments with issue-spec link.
7. Implement the code changes for one PROCESS scope at a time, or integrate completed worker outputs by dependency order.
8. Link every worker and review PROCESS to the PR with issue-spec pr link-process.
9. Add PR rationale comments on key changed lines with issue-spec pr rationale, each linked to a SPEC comment.
10. Mark PROCESS comments done only after implementation/review work and focused verification evidence exist.
## Coordinator DAG Execution
1. Build the ready set from PROCESS nodes whose dependencies are done.
2. Keep immediate blocking work local when the next step depends on it.
3. Spawn or assign independent worker agents only when their write ownership is disjoint.
4. Spawn or assign independent review agents only when their review scopes are disjoint.
5. Integrate completed outputs by dependency order and update PROCESS evidence before marking done.

View File

@@ -0,0 +1,24 @@
---
name: issue-spec-archive
description: Create the post-merge durable spec archive PR for an issue-spec change.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Archive
Use when the user asks for /issue-spec:archive, issue-spec archive, or creating the post-merge durable spec PR.
## Steps
1. Confirm the implementation PR is merged.
2. Create the durable spec PR:
issue-spec archive durable-spec --repo higress-group/higress --proposal <issue> --capability <capability> --create-pr --branch issue-spec/durable-spec-<capability> --json
3. Review the durable spec PR for long-lived behavior only. Do not copy process records, review findings, or verification logs into durable specs.
4. After durable spec PR merge, keep proposal/design/implement issues as audit history unless the project policy says to close them.

View File

@@ -0,0 +1,58 @@
---
name: issue-spec-github
description: Use GitHub CLI for GitHub issues, pull requests, CI runs, and API queries that issue-spec does not wrap.
license: MIT
compatibility: Requires GitHub CLI (gh).
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# GitHub CLI
Use the `gh` CLI to interact with GitHub repositories, issues, pull requests, CI, and API endpoints.
## When To Use
- Checking PR status, reviews, mergeability, or CI checks.
- Creating, viewing, updating, closing, or commenting on GitHub issues.
- Listing or inspecting pull requests, workflow runs, releases, labels, or repository metadata.
- Calling GitHub API endpoints with `gh api` when issue-spec does not provide a dedicated command.
## When Not To Use
- Local git operations such as commit, branch, fetch, merge, or push. Use `git` directly.
- Non-GitHub repositories. Use the matching provider CLI instead.
- Complex code review across local diffs. Read the repository files directly and use issue-spec review commands for traceable findings.
## Setup
```bash
gh auth login
gh auth status
```
## Common Commands
```bash
gh issue list --repo owner/repo --state open
gh issue view 42 --repo owner/repo --json number,title,state,url,body
gh issue comment 42 --repo owner/repo --body "Comment body"
gh pr list --repo owner/repo
gh pr view 17 --repo owner/repo --json number,title,state,headRefName,baseRefName,url
gh pr checks 17 --repo owner/repo
gh run list --repo owner/repo --limit 10
gh run view <run-id> --repo owner/repo --log-failed
gh api repos/owner/repo/labels --jq '.[].name'
```
## Notes
- Always pass `--repo owner/repo` when the current directory is not definitely inside the target repository.
- Use GitHub URLs directly when convenient, for example `gh pr view https://github.com/owner/repo/pull/17`.
- Prefer structured output with `--json` and `--jq` when another command or agent step consumes the result.
- issue-spec owns the proposal, design, implement, typed comment, review, verify, and archive workflow state. Use `gh` for adjacent GitHub operations that are outside issue-spec's command surface.

View File

@@ -0,0 +1,37 @@
---
name: issue-spec-propose
description: Create or continue proposal, SPEC, QUESTION, design, and TASK artifacts for an issue-spec change.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Propose
Use when the user asks for /issue-spec:propose, issue-spec propose, creating a change proposal, drafting SPEC comments, or preparing design/tasks after questions converge.
## Steps
1. Create the proposal issue:
issue-spec issue create proposal --repo higress-group/higress --change <change-name> --body-file <proposal.md>
2. If the proposal body needs revision after discussion, update it in place:
issue-spec issue update --repo higress-group/higress --issue <proposal-issue> --body-file <proposal.md> --summary "<what changed>"
3. Add SPEC comments with issue-spec comment upsert --type SPEC. SPEC comments must use MUST/SHALL and WHEN/THEN scenarios.
4. Add QUESTION comments for unresolved behavior with issue-spec question create and resolve blocking questions before design.
5. Create the design issue after SPEC/QUESTION convergence:
issue-spec issue create design --repo higress-group/higress --change <change-name> --proposal <proposal-issue-or-url> --body-file <design.md>
6. Add TASK comments with issue-spec comment upsert --type TASK and link every TASK to covered SPEC comments with issue-spec link.
7. Create the implement issue once tasks are ready:
issue-spec issue create implement --repo higress-group/higress --change <change-name> --proposal <proposal-issue-or-url> --design <design-issue-or-url> --body-file <implement.md>
8. Run issue-spec verify-links and fix missing backlinks before implementation.

View File

@@ -0,0 +1,32 @@
---
name: issue-spec-review
description: Review an issue-spec implementation PR, create PR line findings, reply after fixes, and sync REVIEW comments.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Review
Use when the user asks for /issue-spec:review, issue-spec review, or a PR review gate for an issue-spec implementation.
## Steps
1. Run issue-spec review sync --repo higress-group/higress --pr <number> --implement <issue> --id REVIEW-<n> --json to capture current rationale comments, findings, and checks.
2. For non-trivial PRs, spawn or assign dedicated review agents as review PROCESS owners. Multiple review agents can run in parallel when their review scopes are independent.
3. Give each review agent a concrete scope and expected output: actionable findings only, severity, file/line, linked SPEC, owner PROCESS, and suggested fix.
4. Create actionable PR line findings with issue-spec review finding. Use P0/P1 for blockers and P2 for non-blocking follow-up.
5. Assign every finding to a PROCESS owner. If no findings are found, record that result in REVIEW or VERIFY evidence.
6. After the worker fixes a finding, reply to the original thread with issue-spec review reply --status resolved.
7. Re-run review sync. P0/P1 findings must be resolved before final verify/archive.
## Review DAG Policy
1. Every non-trivial PR should have at least one dedicated review PROCESS node before final verify.
2. Use multiple review agents in parallel when scopes are independent, for example CLI/API behavior, workflow docs, tests, compatibility, or security-sensitive surfaces.
3. A review agent reports findings only; the coordinator converts actionable line findings into issue-spec review finding comments.
4. P0/P1 findings block final verify until the owner PROCESS fixes them and issue-spec review reply records the resolution on the original thread.
5. If a review agent finds no issues, record that result in REVIEW or VERIFY evidence before marking the review PROCESS done.

View File

@@ -0,0 +1,28 @@
---
name: issue-spec-verify
description: Run final issue-spec verification across traceability, questions, review findings, PR rationale, PR checks, and durable spec draft.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Verify
Use when the user asks for /issue-spec:verify, issue-spec verify, or final readiness evidence before merge/archive.
## Steps
1. Run focused project tests and record evidence in VERIFY comments.
2. Run issue-spec verify-links --repo higress-group/higress --proposal <issue> --design <issue> --implement <issue> --json.
3. Render a durable spec draft:
issue-spec archive durable-spec --repo higress-group/higress --proposal <issue> --capability <capability> --output /tmp/<capability>-spec.md --json
4. Run final verify:
issue-spec verify --repo higress-group/higress --proposal <issue> --design <issue> --implement <issue> --pr <pr> --durable-spec /tmp/<capability>-spec.md --json
5. Final verify must fail if blocking questions, missing links, missing PROCESS rationale, open P0/P1 findings, failed or pending PR checks, or durable spec omissions exist.

View File

@@ -0,0 +1,53 @@
---
name: issue-spec-workflow
description: Use issue-spec to run an issue-native OpenSpec-style workflow with GitHub issues, typed comments, PR review comments, final verification, and durable spec archive PRs.
license: MIT
compatibility: Requires issue-spec CLI.
metadata:
author: issue-spec
version: "1.0"
generatedBy: "issue-spec"
---
# Issue Spec Workflow
Use this skill for issue-native OpenSpec work. Active change artifacts live in GitHub issues and issue comments; durable specs are repository files created after implementation merge.
## Start
1. Run issue-spec auth status --json and confirm the active auth source and GitHub backend.
2. Run issue-spec status --repo higress-group/higress --proposal <issue> --design <issue> --implement <issue> --json when issues already exist.
3. For new work, create proposal, design, and implement issues with issue-spec issue create and pass --body-file with concrete markdown content.
4. When an issue body changes, update it in place with issue-spec issue update --body-file and include --summary for the human-readable audit trail.
5. Store requirements, tasks, process ownership, review, and verify evidence as typed comments.
## GitHub Backend
- Local agents may rely on native GitHub CLI support: when no ISSUE_SPEC_TOKEN, GH_TOKEN, GITHUB_TOKEN, keyring token, or issue-spec config token is present and gh auth status --active succeeds for the target host, issue-spec auto-selects the gh backend.
- Explicit env or stored issue-spec tokens keep the rest backend under auto selection. Set ISSUE_SPEC_GITHUB_BACKEND=rest or ISSUE_SPEC_GITHUB_BACKEND=gh only when a workflow needs deterministic backend selection.
- The gh backend proxies GitHub API operations through gh api and uses gh --hostname for Enterprise hosts. It does not replace local git commands.
- ISSUE_SPEC_API_URL applies to the rest backend. Forced gh mode should be used only with hosts that gh can address.
- Use ISSUE_SPEC_TOKEN="$(gh auth token)" only for older issue-spec versions or when deliberately forcing rest while sourcing the token from gh.
## Rules
- Create SPEC comments before design; each SPEC must be testable and include WHEN/THEN scenarios.
- Do not leave active proposal/design/implement issue bodies as TBD placeholders.
- Resolve blocking QUESTION comments before design/tasks, or explicitly record accepted assumptions.
- Link SPEC <-> TASK and TASK <-> PROCESS with issue-spec link.
- Link every PROCESS to the implementation PR with issue-spec pr link-process.
- For non-trivial changes, include review PROCESS nodes in the DAG; review agents are scheduled like worker agents and can run in parallel when their review scopes are independent.
- Small changes may stay coordinator-only, but record the serial execution decision in the implement or VERIFY evidence.
- Before human review, add PR rationale comments with issue-spec pr rationale for every active PROCESS.
- Use issue-spec review finding for PR line findings and issue-spec review reply to close the original thread.
- Run issue-spec review sync and issue-spec verify before declaring ready.
- After the implementation PR merges, create the separate durable spec PR with issue-spec archive durable-spec --create-pr.
## Coordinator DAG Execution
1. Treat PROCESS comments as DAG nodes with explicit owner, dependencies, write or review scope, PR link, and evidence.
2. Select ready PROCESS nodes whose dependencies are done and whose scopes do not overlap.
3. Dispatch independent worker PROCESS nodes in parallel when their file/module ownership is disjoint.
4. Dispatch independent review PROCESS nodes in parallel for non-trivial PRs after PR rationale exists.
5. Integrate completed worker outputs by dependency order; route P0/P1 review findings back to the owner PROCESS.
6. Mark PROCESS nodes done only after their implementation or review evidence is recorded and blocking findings are resolved.

4
.issue-spec/config.json Normal file
View File

@@ -0,0 +1,4 @@
{
"hostname": "github.com",
"repo": "higress-group/higress"
}

View File

@@ -36,7 +36,9 @@ header:
- 'pkg/ingress/kube/gateway/istio/testdata'
- 'release-notes/**'
- '.cursor/**'
- '.claude/**'
- '.claude/**'
- '.agents/**'
- '.issue-spec/**'
comment: on-failure
dependency: