Measuring darija transcription: the protocol, and the figure we do not publish

The standard measure is word error rate. Between the automatic transcription and the human reference you count substituted words, deleted words and inserted words, and divide that total by the number of words in the reference.

It has two properties worth holding in mind. It can exceed a hundred percent, because a talkative system inserts more words than were there. And it treats all errors as equivalent: confusing two function words and getting an order number wrong count the same.

It is that second property that makes it misleading for professional use. A system that transcribes everything correctly except names and references will have a flattering rate and be unusable, because the words it misses are exactly the ones that trigger an action.

From that follows a way of reading rates published by others, where they exist. The first question is not "how much" but "on what": which sample, which language exactly, which reference, which date. A rate missing those four answers does not compare to yours, and quoting it means borrowing somebody’s conclusion without their experience.

The rate remains useful for comparing two systems on the same audio. It does not say whether a system is good enough for your use — those are two different questions, and the second is measured differently.

The human reference is the expensive part and it decides the value of everything else. It requires somebody whose darija is your customers’ — not merely somebody who speaks Arabic.

The instruction given to the transcriber has to be written and short: transcribe what was said and not what should have been, keep repetitions and hesitations, correct nothing, mark inaudible passages rather than guessing them. A transcriber who improves the text builds a reference even a perfect system cannot reach.

The pace deserves announcing so that the budget surprises nobody: transcribing spontaneous speech takes several times the duration of the audio, more still when the line is bad and passages have to be replayed. Thirty conversations are not transcribed in a morning, and a team promised otherwise produces a rushed reference, which ruins the measurement more surely than a bad model does.

Double entry is the precaution worth its cost on part of the sample: two people transcribe the same five calls independently, and their disagreement is measured. That figure is the floor of your measurement — no system can do better than the agreement between two humans on the same audio.

That floor is also what makes a comparison honest. A gap between two models smaller than the disagreement between your two transcribers proves nothing, and it is the result most published comparisons never report.

A comparison only means something under identical conditions, and the list of what has to be identical is longer than people think: the same audio, the same normalisation, the same period, and the same absence of a custom dictionary.

That last point deserves insisting on. A system that has been given the list of your products and neighbourhoods is no longer comparable to a raw one, and the measured difference is the list’s, not the model’s. That reconciliation layer is often the right solution — but then it is measured separately, as an improvement applied to the best system.

The honest result of a comparison is therefore rarely "system A is better". It is rather: on this kind of call, in these conditions, on this date, A loses fewer decisive elements than B, and the gap exceeds the disagreement between our transcribers.

There is finally one request to put to the supplier, and only one is worth making: the exact version of the system used, with its date. Many will answer that they do not disclose it, which is already information — it means your measurement will not be reproducible at their end, and that degradation from one day to the next can never be held against them.

Write the date out in the table. Transcription models change without notice and without changing name, and an undated table describes a state of the world that no longer exists.

The full protocol takes a day’s work and a legal agreement. There is a one-hour version that gives no error rate but answers the question "is it worth going further".

Take twenty recent calls. Listen to them, noting only, for each, the decisive elements spoken — the name, the number, the quantity. Have the same twenty transcribed by the system you are considering. Count how many of those elements are correct.

You get no measurement comparable to anything, and that is not the point. You get the one piece of information that decides at this stage: the proportion of the information that triggers an action and arrives intact.

If it is bad, no tuning will save it and the full protocol is a wasted expense. If it is good, you know the serious measurement is worth the day it costs, and you already know which calls to put into it.