CouncilAsAJudge
Overview
CouncilAsAJudge evaluates a candidate response rather than producing one. It spins up six specialized judge agents — one per evaluation dimension — that each score and critique the response in parallel, then an aggregator agent combines their rationales into a single, comprehensive verdict. Use it when you need structured, multi-criteria quality assessment of an LLM output: grading, review, or QA gates in a larger pipeline.
The six evaluation dimensions are fixed and always run together:
- Accuracy — factual correctness, up-to-date claims, internal consistency
- Helpfulness — practical value, feasibility, completeness for the user’s actual question
- Harmlessness — safety, bias, appropriateness
- Coherence — structure, logical flow, clarity of argument
- Conciseness — redundancy, verbosity, information density
- Instruction Adherence — whether the response followed the prompt’s explicit requirements
CouncilAsAJudge builds its own judge agents and aggregator internally — it does not run the agents you supply. The request must still include at least one entry in agents (every swarm type except HeavySwarm requires a non-empty agents list), but that entry is not executed. A single lightweight placeholder agent is sufficient.Architecture
All six judges evaluate the same input in parallel, each producing a detailed rationale for its dimension. The aggregator reads every rationale and writes the final verdict.Task Format
Because the council evaluates a response, not a request for new work,task must contain both the original task/prompt and the response you want scored — not just a question. A typical shape is:
CouncilAsAJudge-Specific Parameters
Use Cases
- Automated grading and scoring of LLM outputs against a rubric
- Quality assurance gates before publishing generated content
- Comparing candidate responses from different models or prompts
- Red-teaming and safety review of model outputs
- Building an evaluation harness for prompt or model regression testing
API Usage
- Shell (curl)
- Python (requests)
- JavaScript (fetch)
Best Practices
- Always include both the original task and the candidate response inside
task— evaluating a bare question with no response to grade produces a meaningless ruling - Use a strong
council_judge_model_name— it drives the six dimension evaluations the final synthesis is built from (the aggregator itself always runs ongpt-5.4) - Pair
CouncilAsAJudgewith another swarm call: generate a response first, then post it (plus the original task) throughCouncilAsAJudgeas a quality gate - Keep the required placeholder
agentsentry minimal and cheap — it is billed but never executed - Best for scoring and review workflows, not for generating original content