Two Billing Rails, One Field: The Bug That Hides Half Your Customers

Your data team is probably fine. The model is probably right. The wiring underneath it is where I would look first.

Here is the specific shape of the problem, because the general version is useless.

The mechanism

A company sells through more than one billing system. In e-commerce this is almost everyone: app-store billing plus direct billing, or Shopify checkout plus a subscription platform, or a marketplace alongside your own store.

The reporting gets built when the first rail is the only rail. A churn metric keys on the cancellation signal that rail emits. Someone adds the second rail two years later, and it emits a different signal, in a different place, with a different shape.

Nothing breaks. No error appears. The metric keeps returning a number, and the number keeps looking plausible, because a plausible-looking churn rate is exactly what an understated churn rate looks like.

Why it survives review

Three reasons, and none of them are anyone being careless.

The number moves. It goes up some weeks and down others, so it passes the sniff test that a hardcoded value would fail. A freshness check does not catch it either, because the data is arriving on time. It is simply arriving from one place instead of two.

And the first thing built on top of it inherits it. If someone ran an analysis on that series and reported a lift, the lift is real and the magnitude is wrong. By the time anyone checks, the wrong magnitude has been repeated in three documents and nobody remembers which one was the source.

The check

This takes about an hour and you do not need a data team to run it.

1. ▸Write down every system that can take money from a customer. Not the systems you think of first. All of them, including the one that came with an acquisition.
2. ▸Open the definition of your churn metric and find the field it keys on. If it is a single field, name which system writes it.
3. ▸Pick one week. Count cancellations by hand in every billing system for that week.
4. ▸Compare that count to what the dashboard said for the same week.

If the two agree, you have learned something worth knowing and it cost you an hour. If they do not, the ratio between them is the size of the problem, and you now have a number to take to whoever owns the pipeline.

What to do about the answer

Fix the pipeline, not the chart. The temptation is to add a second tile for the second rail, which leaves two numbers on the screen and a person deciding which to trust. Consolidate upstream so there is one series, then make it break loudly when a rail stops reporting.

Then go back and find every conclusion that was drawn from the old series. That part is not technical work and it is the part people skip.

The wider version

This is not really about billing. It is about the gap between the model and the wiring. Anywhere a metric keys on one field that one system happens to write, the same failure is available, and it fails silently because a partial answer and a complete answer look identical on a chart.

So: which rail can your churn metric see?