Judges

Forum AI’s expert-calibrated judges

41 judges

General12 judges

Policy AdherenceDoes the AI follow your policy under pressure?Checks the agent applied your written policy to the case in front of it, and refused what the policy forbids even under repeated pressure.Built withCompliance leads and counsel Hallucinations & GroundingCan the AI back up every claim it makes?Checks every factual claim traces back to something the agent actually had — a retrieved document, a tool result, or the user's own message.Built withJournalists and fact-checkers Escalation to HumanDoes the AI know when to involve a human?Checks the agent handed off when the situation called for a person — and did not hand off the cases it was there to handle.Built withContact-center and risk leaders Instruction FollowingDoes the AI do what was actually asked?Checks the agent did what was asked, including the constraints — the format, the limit, the thing it was told not to do.Built withProduct leads and tech writers Uncertainty CommunicationDoes the AI say when it is not sure?Checks the agent said when it did not know, rather than answering at full confidence and leaving the reader no way to tell.Built withForecasters and scientists Refusal AppropriatenessDoes the AI refuse only what it should?Checks refusals land on the requests that warrant them, and that legitimate work is not turned away with a boilerplate decline.Built withTrust & safety leads Task CompletenessDoes the AI finish every part of the request?Checks nothing in a multi-part request was quietly dropped — the second question, the attachment, the follow-up the user asked for.Built withOperations leaders PII HandlingDoes the AI keep personal data private?Checks personal data was not collected beyond need, repeated back into the wrong channel, or exposed to a party that should not see it.Built withPrivacy officers and counsel Authority & ScopeDoes the AI act only where it has authority?Checks the agent acted only within the permissions it was actually granted — no action it had no standing to take.Built withFormer regulators and counsel Tone & EmpathyDoes the AI's tone fit the situation?Checks the register fits the moment — that a frustrated or distressed user is met, not processed.Built withCX leaders and communicators GDPR Data Subject RightsDoes the AI act on data rights requests in time?Checks access, erasure, portability and objection requests were recognised as such and handled on the terms the regulation sets.Built withEU data protection officers EU AI Act TransparencyDoes the AI disclose that it is not human?Checks the user was told they are dealing with an AI system, and that generated or manipulated content is disclosed as such.Built withPolicy experts and academics

Customer Service7 judges

Customer Service HelpfulnessDoes the AI solve the customer's problem?Checks the customer's problem was actually solved, not merely answered — the difference between closing a ticket and fixing the thing.Built withFortune 500 CX executives Customer CommunicationDoes the AI use the right language with customers?Checks the customer was left knowing what happened, what happens next, and by when — in language they can act on.Built withSupport and brand leaders Refund & Credit AuthorizationDoes the AI stay inside its refund authority?Checks refunds, credits and goodwill gestures stayed inside the agent's authority and the conditions your policy sets on them.Built withRetail ops and finance leaders Cancellation PressureDoes the AI let customers cancel without a fight?Checks a retention attempt stayed an offer — that a customer who asked to cancel was not obstructed, guilted, or looped.Built withConsumer protection experts Ticket Triage AccuracyDoes the AI route the ticket to the right team?Checks the ticket was categorised, prioritised and routed to the queue that can actually resolve it.Built withContact-center ops leaders Multi-Turn Context RetentionDoes the AI remember what it was already told?Checks the agent held on to what the customer already told it, instead of asking for the order number a third time.Built withSenior support agents Resolution QualityDoes the fix the AI offers actually hold?Checks the fix offered would hold — that the case was closed on a real remedy rather than a workaround that returns next week.Built withFortune 500 support executives

Travel4 judges

Itinerary AccuracyIs the itinerary the AI built actually possible?Checks dates, times, airports, time zones and connection times in a proposed itinerary are internally consistent and physically possible.Built withAirline veterans and advisors Fare & Fee DisclosureIs the price the AI quotes the price paid?Checks the price quoted is the price paid — baggage, seats, change fees and taxes surfaced before the customer commits.Built withAirline pricing leaders Disruption & RebookingDoes the AI offer what the passenger is owed?Checks a cancellation or misconnection was handled with the options the passenger is actually entitled to, in the right order.Built withAirline operations leaders Travel Document GuidanceDoes the AI's visa advice fit the whole route?Checks visa, passport-validity and entry-requirement guidance matches the traveller's nationality, route and transit points.Built withImmigration attorneys

Healthcare6 judges

Clinical AccuracyDoes the AI match current clinical guidance?Checks clinical statements match current guidance, and that the caveats a clinician would attach are still attached.Built withPracticing physicians Red-Flag TriageDoes the AI spot what needs urgent care?Checks presentations that need urgent care were recognised and directed there, and that routine ones were not escalated into an ER visit.Built withER physicians and nurses Scope of PracticeDoes the AI avoid diagnosing or prescribing?Checks the agent stayed on the information side of the line — no diagnosis, prescription or treatment plan it has no standing to give.Built withClinicians and regulators Patient ComprehensionCan a patient act on what the AI told them?Checks the explanation is one a patient can act on — plain language, the actual next step, no untranslated clinical shorthand.Built withNurses and educators Medication SafetyDoes the AI catch dosing and interaction risks?Checks drug names, doses, interactions and allergy conflicts against the record the agent was given.Built withClinical pharmacists HIPAA SafeguardsDoes the AI share health data only where allowed?Checks protected health information was disclosed only to permitted recipients, for a permitted purpose, at the minimum necessary.Built withHealthcare privacy officers

Financial Services4 judges

SuitabilityDoes the AI's advice suit this customer?Checks a recommendation fits the customer's stated circumstances, objectives and risk tolerance rather than the product being pushed.Built withWealth managers and regulators Risk DisclosureDoes the AI state the risks, not just the upside?Checks the downside was stated as plainly as the upside, with the disclosures the product actually requires.Built withSecurities compliance leads PCI DSS Cardholder DataDoes the AI keep card data out of your logs?Checks full card numbers, security codes and track data never entered a transcript, log, note or downstream system.Built withPayments security assessors FINRA CommunicationsAre the AI's claims fair and balanced?Checks customer-facing communications are fair and balanced, free of exaggerated or promissory claims, and retained as required.Built withBroker-dealer compliance leads

Data & Spreadsheets5 judges

Formula CorrectnessDo the AI's formulas compute what was asked?Checks the formulas an agent wrote compute what was asked — right ranges, right anchoring, no silent off-by-one across the fill.Built withFinancial modelers Numerical AccuracyDo the AI's numbers add up when rechecked?Checks the numbers in the output survive recomputation — totals, percentages, growth rates and the units they are quoted in.Built withStatisticians and analysts Data Transformation FidelityDoes the AI reshape data without losing rows?Checks a reshape, merge or clean-up preserved the data — no rows dropped, no join silently fanning one record into forty.Built withData engineering leads Source TraceabilityCan the AI's figures be traced to a source?Checks every figure in a summary can be walked back to the cell, query or file it came from.Built withAuditors and librarians Chart & Summary FaithfulnessDoes the AI's chart say what the data says?Checks the chart and the written takeaway say what the underlying data says — no truncated axis doing the arguing.Built withData journalists

Software3 judges

Code CorrectnessDoes the AI's code do what it claims?Checks a change does what it claims — behaviour matches the description, edge cases handled, nothing adjacent quietly broken.Built withStaff engineers at top firms Vulnerability IntroductionDoes the AI avoid introducing vulnerabilities?Checks generated code did not introduce injection, unsafe deserialization, secret exposure or a broken authorization check.Built withSecurity researchers SOC 2 Access ControlDoes the AI stay inside its access controls?Checks agent actions on customer systems stayed inside the access controls and left the audit trail the controls assume.Built withSOC 2 auditors

Not on the list

Describe what matters and we draft the judge

The catalog covers the ground we see most often. Tell us the failure you are trying to catch and we build the judge around it.

Book a demo