Multi-Vendor Adversarial Review
Using a different AI model (different vendor or different model tier) to review the work produced by the implementing model. Catches single-model blind spots, hallucination patterns, and style biases that same-model review misses.The Problem It Solves
A model reviewing its own work (or work from the same model family) shares the same:- Training biases
- Hallucination tendencies
- Pattern preferences
- Blind spots for certain logic errors
Implementations
Dangeresque (per-task)
Dangeresque runs a mandatory adversarial reviewer after every worker agent. The reviewer is a different model from the implementer. Worker → verify → adversarial review → human-merge gate.Karpathy llm-council (reference implementation)
Karpathy LLM Council is a local web app implementing the full 3-stage Council Pattern: parallel dispatch → anonymized peer review → Chairman LLM synthesis. Key addition to existing implementations: anonymization (model identities hidden during review to prevent provider favoritism) and Chairman synthesis (designated model produces the final answer rather than a human). OpenRouter-based. Q&A focused, not production-grade.AgentOps /council (design decisions)
AgentOps (boshu2) formalizes multi-vendor consensus as a CLI command. Multiple models (Claude, Codex, Cursor) analyze a question simultaneously; disagreements are surfaced as explicit artifacts.
BMAD workflow (community)
One r/ClaudeCode commenter’s workflow: Opus 4.7 plans → Codex 5.5 adversarially reviews the plan → iterate until agreement → Claude implements → Codex code-reviews the implementation.Pocock (implicit)
Matt Pocock recommends Sonnet for implementation, Opus for review. Same vendor, different model tier — shares some biases but provides genuine judgment upgrade. Cheaper than cross-vendor; misses some blind spots.When It Matters Most
- Architecture decisions: where framing effects dominate — model A’s preferred solution may not be B’s
- Security review: different training data = different threat pattern coverage
- Long-horizon plans: model planning and execution biases differ
- High-stakes merges: when you can’t afford single-model blind spots
Council with GitHub Copilot Models
For users with a GitHub Copilot subscription: the GitHub Models API (https://models.inference.ai.azure.com, authenticated with a GitHub PAT) provides cross-vendor council without separate API keys.
Use Pi Agent (pi-mono)‘s
@mariozechner/pi-ai as the unified API layer to call these without vendor-specific client code.
Cross-Vendor vs Same-Family Tiering
For most workflows: same-vendor tiering (Sonnet implements, Opus reviews) is the right default. Cross-vendor review for architecture and security gates.
/council for design decisions where consensus matters.
LLM-as-Judge as the Evaluation Mechanism
The cross-vendor reviewer in multi-vendor adversarial review is typically implemented as an LLM-as-judge — an LLM with a structured evaluation prompt that scores the work against defined dimensions. Key evaluation modes:- Pairwise: judge compares two responses, picks winner (more reliable for ranking)
- Direct scoring: judge rates against rubric dimensions (Correctness, Conciseness, Actionability, Relevance)
- Chain-of-thought (G-Eval): judge reasons step-by-step before scoring — more accurate, provides audit trail
Relation to Existing Wiki
- Verification Pipeline — multi-vendor review as an implementation of the reviewer tier
- Agent Primitive Selection — model tier routing; cross-vendor as an extension
- Dangeresque — adversarial reviewer built into the pipeline
- AgentOps (boshu2) —
/councilas formalized multi-vendor consensus - LLM-as-Judge — the evaluation mechanism used in the cross-vendor reviewer role