Two systems that never quite agree

An integration is declared finished the day data passes from one system to the other. That is a real milestone and it is not the end: two connected systems begin to diverge, slowly, and nobody notices until somebody compares two screens.

Divergence is not a bug in the usual sense. Nothing breaks, no error is logged, both pieces of software display figures consistent with their own contents. They simply no longer tell the same story, and the day the difference is found, nobody knows how long it has been going on.

This article is about the life of a running integration: who is right when two systems contradict each other, how often to reconcile, what to do with a duplicate, how to replay without duplicating, what happens when a field is renamed, and how far back into the past you can go. The companion article covers feasibility — the inventory before the quotation, what comes out of a piece of software, the till that exposes nothing — and none of that is repeated here.

It carries no chart, and the reason is more interesting than usual: what it is about has never been counted by anybody, because neither of the two systems records it as an error.

Divergence always starts small and with a special case. An order cancelled while the synchronisation was running. A record changed twice in the same minute. A forty-second network outage in the middle of a transfer. Each of those produces a one-line gap, and one line is not noticed.

What makes the phenomenon dangerous is that it accumulates without announcing itself. After three weeks the gap is a few dozen lines, after a year it is structural, and it is discovered by a physical stocktake or by a customer insisting they paid for something the system has never heard of.

At that point the question asked is “since when”, and it is the most expensive question to answer after the fact. With no regular count kept, there is no way to date the start of the drift, and the only option is to redo the entire period.

We cannot say how often this happens, and the explanation is worth giving rather than working around. There is no published statistic on divergence rates between systems, not because it is confidential, but because nobody has ever been able to measure it: as far as the two pieces of software are concerned, the operation succeeded. The send succeeded, the write succeeded. The event is recorded nowhere as a failure, so there is no population of errors to sample. That is the only figure on this page, and nobody has ever counted it.

The system at the other end will be unavailable, and not rarely: maintenance, network outage, quota reached, a server saturated on a busy day. The question is not how to avoid it but what your side does meanwhile.

There are three possible behaviours and only one is acceptable. Losing the send, which produces the divergence of section 2 and is invisible. Blocking and waiting, which turns an outage at their end into an outage at yours. Or setting aside and retrying, which is the only response that preserves both service and accuracy.

The holding queue must have a limit and a voice. A queue that grows indefinitely is a deferred loss: at some point somebody empties it to unblock the situation, and nobody will ever know what was in it. A queue that speaks up when it passes a threshold turns an outage into information.

Retries also have to space themselves out. Retrying every second against a system already in difficulty adds load to an outage, and some suppliers treat that insistence as abuse and cut access. Spacing retries is a technical courtesy that mostly protects whoever applies it.

When an integration is put in, the request that follows immediately is to recover the past: bring across the last two years of orders, fill the new system with the old one’s history. It is legitimate and it has a limit best stated straight away.

What can be brought across is what was recorded. What was never entered exists nowhere and no migration will make it appear: if the old system did not note the acquisition channel, the new one will not have it for past years, however good the migration work.

It is a distinction requests rarely make and it changes the estimate completely. “Bring across the history” may mean copying fields that exist, which is mechanical and feasible work, or reconstituting information that was never collected, which is invention and has to be refused.

Good practice is to separate the two in the quotation, in plain words: here are the fields brought across, here are the fields that will stay empty for the earlier period. The client then decides knowingly, and nobody discovers six months later that the column meant to drive everything is empty on three quarters of the rows.

What we take on is bounded: writing the source of truth field by field with the people who use both systems, building the integration with its correspondence table and its stable reference, and installing the reconciliation that gets kept — the least spectacular part and the one that makes everything else repairable.

We do not build an integration without the source of truth being written for every field that moves, even when the client finds the question theoretical. It is not: without that page, the first conflict is settled by the execution order of the programs, which is leaving a management decision to an accident of scheduling.

We do not reconstitute a history that was never recorded. We copy what exists and we say which columns will stay empty for the earlier period. Filling those columns by estimation would produce apparently complete data on which somebody will base a decision, which is more damaging than an empty cell.

Finally, we do not promise an error rate. This page explains why no published one exists — the event is recorded as a failure by neither system — and it would be incoherent to announce one after writing that. What we can promise is that the gap will be measured at your end, every week, and kept, so that the question “since when” always has an answer.