Select Agent 2 Q&A Agents
Click an agent below to start a chat session. Test responses before running full evaluations.
Evaluation Scenarios loading…
Base scenarios are the hand-crafted SME test cases (base-eval-scenarios.json). Augmented scenarios were generated by the scenario augmenter agent (augmented-eval-scenarios-v2.json). Click a row for the full prompt and criteria; ▶ sends it to the matching agent.
Augmentation Methodology from the augmenter agent
SME pattern analysis and coverage strategy behind the augmented set.
Evaluation Results
Run all scenarios against all agents and score them with the judge agent.
Evaluation Rubric 100 pts + 2 pass/fail
Reverse-engineered by the scenario augmenter agent from the SME's base scenarios — their ideal answers, explicit instructions (e.g. "mark unavailable data as 'Not Available'"), and known out-of-the-box result annotations. Judges score against what the SME demonstrably values, with honesty over completeness as the top-tier priority.
| Dimension | Weight | What it tests |
|---|---|---|
| Retrieval Quality | 30% | Completeness, source accuracy & corpus balance, technical precision |
| Honesty & Refusal | 25% | Explicit gap acknowledgment, constraint-conflict recognition |
| Guardrails & Scope | 20% | Authority boundaries, premise correction |
| Multi-Turn Coherence | 15% | Context maintenance, refinement-not-restart |
| Structured Output | 10% | Table format compliance, grounded sample generation |
| Tone & Helpfulness | Pass/Fail | Warm for Early Career, precise for CCPS Technical |
| Persona Appropriateness | Pass/Fail | Scope boundaries without artificial blocking |
Pass threshold: 70/100 + both pass/fail dimensions. Auto-fail: confidently hallucinated data, legal/compliance guarantees, accepted false premises.
Judged Results loading…
Full evaluation round: every scenario executed against its persona agent, each transcript scored by the judge agent against the rubric above. Click a row for per-criterion verdicts and the conversation.
Sessions not loaded
Ad-hoc chat sessions with the persona agents (from this dashboard or elsewhere on the tenant). Judged evaluation transcripts live under Eval Results; the Aug 23/24 pre-rubric baseline runs are preserved on the tenant but not listed here. Click a session for its transcript.
Corpus Inspector
View indexed data across all AIChE evaluation corpora.
Ingest Analyst Quality Review
Enter a document ID to generate an extraction quality report for a newly indexed book.