/

Ideas

Money, Money, Money: Don't Judge a Small Model Until You've Trained It

We fine-tuned a 0.8B open model to route banking support messages across 77 intents. On a MacBook, in under four hours, test accuracy went from 38.9% to 90.6%. We think most routing decisions in financial services should be built this way.

We fine-tuned a 0.8B open model to route banking support messages across 77 intents. On a MacBook, in under four hours, test accuracy went from 38.9% to 90.6%. We think most routing decisions in financial services should be built this way.

Ronnit Roy Burman

Ronnit Roy Burman

AI Architect

Message slips routed through a small model into a grid of 77 intents, six of them lit



We fine-tuned a 0.8B open model to route banking support messages across 77 intents. On a MacBook, in under four hours, test accuracy went from 38.9% to 90.6%. We think most routing decisions in financial services should be built this way.

Most of the AI in a bank’s support queue isn’t conversation. It’s a decision: which of a few dozen places should this message go?

That decision runs thousands of times a day, it has to be fast, and the data behind it is some of the most sensitive a bank holds. Yet a common pattern is to send every message to a frontier model on someone else’s servers and ask it to pick from a list.

We wanted to know how far a small model could get instead. So we took a 0.8B-parameter open model, trained it on a laptop for an afternoon, and scored it on a public banking benchmark. It went from wrong most of the time to right nine times in ten.

Routing is a multiple-choice question

In earlier Labs posts, we argued that a lot of “AI work” is really a bounded decision: classify this, route that, approve or escalate. Those decisions have a fixed set of answers. They don’t need a model that can draft a memo or reason across a 200-page contract. They need a model that picks the right label, every time, quickly.

That’s a different job, and it rewards a different kind of model: small, specialised, and cheap enough to run on every message instead of a sample.

What we did

The test was deliberately simple.

  • The data. Banking77, a public set of 13,083 customer support messages released by PolyAI in 2020. Each is labelled with one of 77 intents, from card_arrival to direct_debit_payment_not_recognised.

  • The model. Qwen3.5-0.8B, an open-weights model with 752 million parameters.

  • The method. LoRA fine-tuning, which trains a small set of added weights (here, 0.5% of the model) rather than the whole thing. Two passes over 10,003 training messages.

  • The machine. One MacBook with an Apple M5 Pro chip. 3 hours 56 minutes, with 6 GB peak memory.

  • The scoring. Both models got the same prompt, listing all 77 intents and asking for exactly one, on all 3,080 test messages. An answer counted only if it matched a real label.

A printed ATM receipt listing the model, training setup and results of the fine-tuning run

The whole run on one slip.

Out of the box, the small model wasn’t good enough

Untuned, the model got 38.9% of messages right. Worse, about one answer in fourteen wasn’t a label at all: near-misses like top_up that don’t exist in the list.

This is where most teams stop. They try a small model, see a result like this, and conclude small models can’t do the job. We think that’s the wrong conclusion. It’s like judging a new hire on their first morning, before anyone has shown them how the queue works.

Four hours later, it was

An ATM screen showing available accuracy of 90.6%, up from a previous balance of 38.9%

The results as a balance inquiry. The screen is an illustration; the numbers are from the run.

Measure

Base model

Fine-tuned

Test accuracy (3,080 messages)

38.9%

90.6%

Macro-F1 (balance across all 77 intents)

36.6%

90.5%

Answers that are a real label

93%

100%

Validation accuracy (200 held-out messages)

35%

86.5%

One training run. With 3,080 test messages, each test figure is good to within about ±2 points at 95% confidence.

Three things in this table matter more than the headline.

Macro-F1 moved with accuracy, so the gains are spread across the 77 intents rather than concentrated in a few common ones.

Validation accuracy climbed alongside training. A model memorising its training data shows training loss falling while held-out loss stalls or turns back up. That didn’t happen.

Line chart of training and validation loss by step. Both fall from the start and level off together, validation at 0.08 and training at 0.05.

Loss on 500 held-out messages tracked training loss down and flattened with it.

And every answer became a real label. A routing system can’t do anything with top_up. A model that answers inside the list every time is one you can wire straight into a workflow.

At 97 milliseconds per message on the same laptop, it’s also fast enough to sit in the path of every request.

The first hour did most of the work

“Four hours” undersells it. We checked accuracy on the 200 held-out messages every 100 steps. Fifteen minutes in, the model had doubled its score, from 35% to 71%. By 39 minutes it was at 80.5%. The remaining three hours added about six points.

Line chart of held-out accuracy against time on the laptop. It jumps from 35% to 71% in 15 minutes, reaches 80.5% at 39 minutes, and levels off around 86.5% by four hours.

Held-out accuracy against wall-clock time. The long stretch after the first hour is the laptop running other work alongside training.

That changes the cost of finding out. You don’t need to commit an afternoon to learn whether a small model can pick up your task. A held-out set and twenty minutes will tell you.

The fixes are about meaning

The improvements weren’t cosmetic. Early in training, “suspicious direct debit” moved from declined_card_payment to direct_debit_payment_not_recognised. “Post ATM withdrawal” moved from atm_support to pending_cash_withdrawal.

Customer message

Base model

Fine-tuned

“There is a suspicious direct debit payment.”

declined_card_payment ✗

direct_debit_payment_not_recognised ✓

“how long does it take to post atm with drawl”

atm_support ✗

pending_cash_withdrawal ✓

“Is there a verification code for the top-up card?”

top_up ✗ not a label

verify_top_up ✓

Three test messages, routed by the base model and by the fine-tuned model at step 250 of 1,188.

Those are exactly the distinctions that matter in a bank. A possible unauthorised debit routed to a card-decline queue isn’t just slow service. It’s a potential fraud or dispute case sitting with the wrong team.

Why this matters in financial services

Accuracy is the easy part to talk about. The stronger argument for small tuned models in regulated work is everything around it.

  • The data stays where it is. A model that runs in 6 GB of memory can run inside your own perimeter. Where a model runs is often a harder constraint than how smart it is.

  • You own the behaviour. The weights are yours. A vendor’s model update can’t quietly change how your messages get routed.

  • The answers are auditable. The model picks from a fixed list you defined. Every decision is a label you can log, count and review.

  • Every message, not a sample. When a decision takes milliseconds on hardware you already own, checking everything stops being a budget conversation.

None of this means retiring large models. The pattern we like is a small tuned model handling the bounded decisions, with anything ambiguous or out of scope escalated to a larger model or a person.

What this doesn’t prove

We’d rather say this ourselves than have a reader say it for us.

  • Banking77 is clean. Short, single-intent English messages. Real queues have typos, several requests in one message, other languages and personal data.

  • We picked a hard starting point. We chose Banking77 partly because the base model struggled with it, so the jump is bigger than it would be on an easier task.

  • It isn’t a leaderboard result. The paper that introduced Banking77 reported 93.7% with a fine-tuned BERT classifier back in 2020, above our 90.6%. Our point isn’t that this is a record. It’s that a general-purpose model you can prompt, tune on a laptop and run locally gets within about three points of a purpose-built classifier in an afternoon.

  • We didn’t run a frontier model on the same test. That comparison, on accuracy, latency and cost per message, is the obvious next step.

  • One run, one seed. We haven’t yet measured how much results vary between runs.

What we’re testing next

  • The same prompt and test set against a frontier model and a classic fine-tuned classifier

  • Messier, client-like data with multi-intent and multilingual messages

  • Confidence thresholds that decide when to escalate

  • How accuracy holds up as inputs get longer

The takeaway

Fine-tuning used to sound like a research project: GPU clusters, an ML team, months of work. This one ran on a laptop in an afternoon.

If your workflows are full of decisions with a fixed set of answers, test a small model. Just don’t judge it until you’ve trained it.

Have an AI priority that has to work?

Have an AI priority that has to work?

Tell us what you're working on.

Senior AI expertise for work that has to perform.

Senior AI expertise for work that has to perform.

Senior product and engineering judgment

for AI that has to work.