Training a model: the three cases where it is worth it

"Could you train it on our data?" That question comes up in every second first meeting, and it asks for something different from what the person asking imagines. It is asked to get correct answers; training mainly acts on the manner of answering.

The confusion is expensive because it points the budget at the heaviest tool in the box, when the hoped-for result is almost always obtained another way, faster and for a fraction of the price.

This article says what training really changes, what you have to supply it with — examples written by you, not documents — and the three situations where it is the right instrument. It also says what it freezes, which is the part nobody raises before signature.

You do not train a model with your PDFs. You train it with pairs: a request on one side, the answer you would have wanted to see on the other. Hundreds of pairs, written or approved by somebody who knows what a good answer looks like at your company.

That is where most projects stop, and it is better that they stop there than after the invoice. A business with ten years of archives has plenty of documents and almost no pairs, because nobody has ever had a reason to write the bad version next to the good one.

There is one exception worth looking for before giving up: your customer-service conversations, if they are kept, are already pairs. A customer’s question, an employee’s answer. They need serious sorting — the bad answers are in there too — but they exist.

A pair also has to carry what the employee had in front of them at the moment of answering. A good answer that assumes an open file, a stock level consulted or yesterday’s conversation teaches the model to produce that answer without that context — that is, to guess. It is the most tedious sort in the collection and the one that cannot be delegated, because it requires remembering what was known that day.

The practical consequence is that the main cost of training is not the computation, which is counted in machine hours. It is your people’s time, and that is counted in weeks.

Available models write excellent French and English, correct standard Arabic, and a darija that immediately betrays that it comes from elsewhere. For a business whose customers write in darija, that gap is not a stylistic detail: it decides whether the answer is read.

It is the case where training does work no instruction does, because a register is not described — it is shown. It shares that property with the first case: what is missing is a form, and a form is learned by imitation.

Collection here is heavier than elsewhere and what darija adds to the work is covered in full elsewhere on this blog. The useful summary: the examples have to come from your real conversations, and the spelling has to stay the one your customers use, including where it is irregular.

There is a trap peculiar to this case and it concerns spelling. Darija is written several ways depending on region, age and keyboard, and an example set built from a single source teaches one of those ways. The model becomes excellent for some of your customers and foreign to others, which is harder to spot than uniformly bad darija.

The honest limit is that training improves the register and does not create understanding. A model that writes better darija does not understand an ambiguous darija request better, and that is a frequent disappointment because the two look alike from outside.

If you were to take away one thing from a training project, it would be the set of pairs. It is reusable with any model, at any supplier, in five years, and it represents the entirety of the work your teams put in.

The model, for its part, expires. It is tied to a base, to a version, sometimes to a platform, and its value declines as the base ages. Confusing the two leads to protecting the wrong object.

That has a simple contractual consequence, often absent from quotes: the example set has to be delivered to you, in an open format, and it has to be yours without restriction. A supplier who keeps the pairs keeps the project’s only durable asset.

There is a further reason to maintain them, and it is one of verification. A trained model cannot be questioned about what it learned: there is no way to ask it for the list. The pairs are therefore the only readable record of what the system was set to learn, and the only place a teaching error can be found before it is observed in an answer.

It also has an internal consequence. That set is maintained the way documentation is maintained: when a standard answer changes at your end, the corresponding pair changes, or the next training will relearn the old one.

We start by measuring what a carefully written instruction already achieves, on your cases, before talking about training. That measurement takes two days and it has ruled the project out more often than it has confirmed it, which is the result it is supposed to produce.

When training is justified, we build the pairs with your teams rather than in their place, we set the evaluation set aside before starting, and we deliver both — the pairs and the scores — in a format you can take elsewhere.

We refuse to manufacture the examples with a model. It would give a clean, fast set, and it would teach your system to imitate a machine, which is the exact opposite of what you are buying.

We also refuse to announce a gain before the first measurement. What you can do without us is precisely that measurement: a hundred real cases, the current output, and your judgement on each. It is one day, and it decides the rest.