Backfill and Late-Arriving Event Handling in Metering Systems

Metering systems fail in the same handful of ways, over and over, and almost none of those failures come from bad math. They come from timing. A distributed system cannot guarantee that events arrive in order or on schedule, and when the metering pipeline assumes otherwise, the corruption doesn't show up immediately. It shows up three weeks later, in a customer's inbox, attached to an invoice that doesn't match their own usage logs.

What breaks when late events go unhandled

There are three distinct failure modes here, and they surface at different times to different people.

The first is aggregate corruption. Usage totals are wrong at the moment billing runs, either understated because events got missed or overstated because retries got double-counted. The trouble is that nobody notices at the time it happens. The error sits quietly until invoice generation, and by then, fixing it means reopening a billing period that finance already considers closed.

The second is enforcement reading stale state. In any real-time credit or quota system, something has to check a balance before approving a request. If metering has fallen behind, that check is working off numbers that are already wrong, and a customer can blow through a hard limit with no warning at all because the system simply doesn't know yet. This is the exact mechanism behind the kind of bill-shock incident that has become shorthand in AI billing circles: an architecture that can't catch up fast enough lets usage exhaust an entire plan before anyone, including the vendor, sees it coming.

The third is trust erosion, and it's the one that compounds. A customer who gets a corrected invoice after they've already paid, or an invoice that contradicts their own logs, stops trusting the billing system, full stop. For AI products, where a meaningful share of customers are already anxious about cost predictability in the first place, a billing discrepancy lands much harder than it would for, say, a flat-fee SaaS product. Hybrid plans make this worse: a late event that pushes a customer just over a seat-included allowance can trigger an overage charge that shouldn't exist, and now someone on the billing team is manually reconciling a dispute that never needed to happen.

There's a fourth failure that's quieter than all of these, and arguably more dangerous because nobody complains about it. Dropped events are revenue nobody collected. No customer calls to say "you undercharged me." The leak stays invisible until an audit, or a pricing model change, forces someone to recount. For AI products running on thin gross margins, uncollected overages don't just look bad. They compress margin directly, dollar for dollar.

How the metering pipeline is structured and where timing failures enter it

Most metering systems break down into three stages: ingestion, aggregation, and rating or billing. Ingestion collects raw telemetry, whether that's API calls, tokens consumed, storage written, or agent actions completed, arriving through REST endpoints, webhooks, SFTP drops, or streaming protocols. Aggregation takes that raw stream and turns it into measurable metrics, grouped by customer, time window, and billable dimension. Rating applies pricing rules to those aggregated metrics and produces the actual charges.

The architectural fact that makes timing a first-class problem, not an afterthought, is this: pricing gets applied at reconciliation, not at the moment an event is captured. An event that shows up after its aggregation window has already run has to go somewhere. It either gets folded into a recalculation or it gets discarded. There's no neutral third option, no way to just ignore the question.

Late events enter at each stage differently. At ingestion, the problem is out-of-order arrival, duplicate retries fired by client-side logic, and burst traffic that exceeds whatever capacity the window was sized for. At aggregation, the problem is boundary logic: an event that arrives after the window closes but before the invoice actually gets generated either gets included or it doesn't, and that decision needs to be made deliberately, not by accident. At rating, backdated plan changes force a re-rating of events that already passed through aggregation under a pricing rule that no longer applies.

Most pipelines handle the low-volume, everything-arrives-on-time case just fine. Failure shows up at scale and at the seams between stages, never in the happy path. That's part of why these bugs are so hard to catch in testing: the test data behaves.

It's worth noting that the industry has already converged on one common pattern for this, even if implementations vary. A fixed window during which late events are still accepted, after which they're not, trades a guarantee of completeness for operational simplicity. That's a design choice, not a universal answer, and teams need to decide for themselves whether that trade makes sense for their own risk tolerance.

Idempotency and deduplication at the ingestion layer

Client-side retry logic isn't a bug. It's correct behavior in a distributed system where a request can time out even after it succeeded on the server side. The ingestion layer's job is to absorb that retry without turning one event into two.

That means every event needs a stable, globally unique identifier, generated by the producer, not assigned later by the ingestion layer itself. The ingestion layer keeps track of identifiers it has already seen and rejects or silently drops anything that repeats. The key requirement here is stability: the same logical event, whether it arrives once or ten times because of retries, needs to produce the exact same identifier every time. If the producer regenerates a new ID on each retry attempt, deduplication is dead on arrival.

Batches need atomic handling too. If a batch contains one invalid or duplicate event, the right move is to reject the whole batch rather than write the good records and drop the bad one. A partial write creates a partial aggregate, and partial aggregates are much harder to audit later than a clean rejection that the producer can just retry with the bad event fixed or removed.

Where deduplication records get stored is a real trade-off, not a solved problem. In-memory options, something like a Redis set with a TTL or a Counting Bloom Filter, are fast, but they have a finite horizon; identifiers age out once they pass the retention window. Persistent, database-backed deduplication is authoritative and doesn't forget, but it adds latency on every write, and it only earns its cost when the backdating window the system commits to stretches into weeks rather than hours. There's no universally correct answer here. The choice follows directly from how long the system promises to accept late arrivals.

Asynchronous architecture helps carry this load. Isolating the workers that handle event ingestion from the workers that handle billing, invoicing, and webhooks means a deduplication check doesn't sit in the critical path of downstream processing. The check happens, and everything else keeps moving.

Watermark strategies and windowing models for late-event acceptance

Here's the tension at the center of all of this: billing windows have to close eventually so invoices can go out, but closing a window before every event has arrived guarantees that some events get left behind. There's no version of a metering system that avoids this trade-off entirely.

Watermarks are the standard answer from stream-processing theory, and they're worth taking seriously. A watermark is a threshold in event time, a point where the system declares that everything before this timestamp has now arrived, which lets the window close based on event time instead of wall-clock time. Heuristic watermarks estimate the maximum latency the system expects and advance on that basis. They're wrong sometimes. That's not a flaw in the design; it's the entire reason a late-event handling policy needs to exist in the first place.

Three windowing strategies show up most often, and each carries different billing consequences. Tumbling windows, fixed and non-overlapping (a calendar month, for instance), are simple to reason about but offer zero tolerance for anything that arrives after the period closes. Sliding or session windows follow actual usage patterns rather than the calendar, which fits agent workflows where activity clusters unevenly, but they're harder to reconcile against a billing date a finance team can point to. And then there's a fixed backdating window: accept events up to some number of days after their event timestamp, and after that, either drop them or route them to a corrections queue for manual handling.

None of these choices should be left as an unstated default. What's the maximum late-arrival window the system will actually honor? What happens to an event that shows up after that window closes, silent drop, manual queue, or a corrected invoice? And does every event type get the same grace period, or does a large token batch or a premium feature call earn a longer one because the dollar amount at stake is bigger? These are policy decisions, and somebody on the team needs to own the answer explicitly rather than discover it by accident during an incident review.

One more thing that trips up systems that weren't built for this from day one: aggregation that keys off wall-clock time instead of the event's own timestamp will misattribute late arrivals to the wrong billing period. It's a quiet error, and it's exactly the kind that doesn't announce itself until someone reconciles two invoices and finds they don't add up.

Backfill pipelines: reprocessing historical events without corrupting live data

Backfills get triggered for reasons that have nothing to do with a single event arriving late. A pricing model shifts from per-call to per-token, and every historical event needs re-aggregation under the new metric. An upstream system produced wrong values and someone has to correct the record. A new billable dimension gets added to a product, and historical data needs to populate it retroactively. A customer disputes a past invoice and demands an authoritative re-run of that billing period.

The core requirement, and the one teams get wrong most often, is isolation. A backfill job cannot write to the same aggregation tables that live ingestion is writing to at the same time, or it will corrupt data that's actively in flight. The standard pattern is to write backfill output to a staging partition, validate it there, and only then swap it atomically into the live partition. Idempotency matters here just as much as it does at ingestion: a backfill job that can be safely re-run without double-counting is also a job that can be interrupted midway and retried without anyone losing sleep over it.

Versioned aggregates help make all of this auditable. Storing aggregates with a version or computation-run identifier lets the system tell an "original billing run" apart from a "correction run," which means for any given invoice, someone can trace exactly which version of the aggregate produced it. It also opens the door to incremental correction: applying a delta when only a subset of events changed, instead of recomputing an entire period from scratch every time.

Backfill jobs compete for the same storage and processing capacity that live events depend on, so rate limiting the backfill writes matters; a backfill running at full throttle can starve the live ingestion path without anyone noticing until latency spikes on current traffic. And before any of this touches production, the backfill should run against a production-like dataset in staging, with its output compared line by line to the original billing run. The diff between the two is the correction. It should get reviewed by a human before any invoice actually gets regenerated.

Re-rating on backdated plan and price changes

A customer upgrades mid-cycle. Or downgrades. Or gets moved onto a new contract that's supposed to take effect from a specific date, not from the next billing cycle boundary. The new pricing has to apply from that effective date forward, which means the system needs to know not just how many events happened in the period, but precisely when within the period each one occurred.

Without event-level timestamps preserved all the way through aggregation, splitting a billing cycle at an arbitrary mid-point is simply impossible; the information needed to do it correctly has already been thrown away.

The re-rating pattern itself, which has roots in telecom billing and is now standard practice across enterprise SaaS and AI contracts, follows a consistent sequence. Detect the backdated change and identify its effective date. Pull all raw events for that customer in the current period with timestamps after that date. Re-run the rating step against those events using the new pricing rule. Then generate either a corrected invoice or a credit and debit adjustment for the difference.

Proration and re-rating get treated as synonyms sometimes, and they shouldn't be. Proration splits a fixed fee by time, which works fine for seat-based charges. Re-rating recomputes actual usage charges under a new rule entirely, which is what metered components require. A hybrid plan, seats plus usage, needs both operations applied correctly to the right dimension, and conflating them is a fast way to generate an invoice nobody can explain.

The GitHub Copilot migration to AI Credits is a useful real-world illustration of how complicated this gets at scale. Monthly plan users migrated on a fixed date, while annual plan holders stayed on the original pricing until their term expired, which meant pricing rules had to be applied differently depending on each customer's plan type and timing throughout the migration. That's not a hypothetical edge case. That's what a real migration looks like once the customer base is large enough.

There's an accounting dimension too, and it's not optional. Under ASC 606, variable revenue recognized during a re-rated period may require adjusting amounts that were already recognized in a prior close. That means the billing system needs to produce an audit trail clean enough that finance can hand it to auditors without manually reconstructing what happened after the fact.

Keeping enforcement consistent with metering state during lag

Metering is asynchronous by design. Enforcement, meaning quota checks and credit balance gates, has to be synchronous because a request either gets approved right now or it doesn't. Those two facts are in direct tension, and no amount of clever engineering makes the tension disappear entirely; it can only be managed.

The usual fix is a local cache for entitlement state, so a balance check resolves against something fast and nearby instead of a live database query on every single request. That cache needs a configurable TTL, and the right value depends entirely on how much staleness the product can tolerate. For AI products where a single request can carry real per-token cost, a longer TTL means the enforcement layer can be wrong for longer, which makes the cache window a business decision about acceptable risk, not just a tuning knob an ops engineer sets and forgets.

Then there's the question of what happens when the metering layer itself goes down or falls behind. Failing open, meaning the system allows the request through and catches up on the accounting later, makes sense when blocking a legitimate customer costs more than an occasional overage does. Failing closed, blocking the request outright, makes sense when an uncapped overage is the more expensive outcome, which is typically true for prepaid credit models where the customer's balance is a hard, finite number.

Whichever way a team goes, the choice needs to be explicit and written down somewhere. Defaulting to fail-open without anyone actually deciding that's the policy is exactly how a company ends up explaining a five-figure overage to a furious customer, and discovering only then that nobody had actually chosen that outcome. They'd just never chosen anything else.