Why Banking Assistants Must Be Tested Beyond Their Final Answers

Introduction

A polished answer is not enough to show that a banking assistant handled a customer request correctly. An assistant may choose the wrong account, rely on stale information, ask for details it already has, or write an invalid value through a tool after stating the correct one. In financial operations, these are substantive failures rather than minor conversational flaws.

IndicBankBench was created to evaluate those failure modes in a more realistic Indian retail-banking setting. Instead of judging only the final text, it evaluates the sequence of decisions and actions that leads to the response.

A four-stage evaluation pipeline

The benchmark contains 799 synthetic cases. They cover five operational domains, a capability and refusal domain, and 20 primary axes. Cases run inside a mock banking environment, where an assistant may need to retrieve account-specific information, call tools, or perform a write operation.

Evaluation proceeds through four stages:

• Safety: whether the request is allowed and whether required safeguards are satisfied;
• Action and tool use: whether the assistant identifies the right account, uses fresh tool evidence, and obtains confirmation before writing;
• Response adequacy: whether the final answer fully and accurately addresses the request;
• Advisory quality: whether explanations or recommendations are appropriate for the case.

Tool behavior and most safety checks are deterministic, reducing dependence on subjective grading. A narrow resolver is used only for ambiguous confirmation-before-write cases, while a separate language-model judge assesses semantic response adequacy.

Reliability is not the same as one successful run

Every case is executed three times. IndicBankBench reports strict pass^3, meaning that a case counts as successful only when all three trials pass. Across the 11 evaluated models, strict reliability ranges from 43.7% to 58.2%. By contrast, at-least-once success ranges from 60% to 74%.

That gap is the central lesson. A system can appear capable if the evaluation records whether it succeeded on any one attempt, while still behaving inconsistently in a banking workflow. Repeated trials therefore provide a more demanding view of operational dependability.

The case-level diagnostics also separate different failure patterns. Some systems ask unnecessary questions even when the necessary information is available. Others take action but fail to reconcile customer context, select the appropriate account, or fully resolve the original request. These distinctions are more useful for engineering than a single aggregate score because they point to different fixes in prompting, orchestration, tool design, or safeguards.

Why the benchmark matters

IndicBankBench reframes banking-agent evaluation as an end-to-end process: interpret the request, verify context, gather current evidence, choose an action, obtain confirmation where needed, execute safely, and explain the result. A fluent final message can hide failures earlier in that chain.

The released cases, mock environment, and evaluation harness should make it easier to reproduce results and examine specific weaknesses. More broadly, the benchmark argues that high-stakes assistants should be measured for consistency and traceability, not merely for whether they can produce a plausible answer once.

Source: Hugging Face Daily Papers