Real-Time Doesn't Mean Right: Designing Streaming Pipelines That Handle Late and Duplicate Data

Real-time data has become part of how energy businesses run. Operators watch generation and demand as it happens, and decisions about the grid are made on numbers that are only seconds old.

But receiving data quickly does not make it accurate.

Imagine an energy provider monitoring thousands of rooftop solar inverters across a region. Each inverter continuously reports its power output, voltage and operating status, and the expectation is simple: the dashboard shows what is happening across the network right now.

In practice, the data behind that dashboard is messier than it looks:

• A home's internet drops, and its readings arrive an hour late.
• A device doesn't hear back that its message was received, so it sends the same reading again.
• Readings arrive in a different order from the one in which they happened.
• Some devices report the wrong time altogether.

A pipeline that ignores these realities can still run perfectly, with no errors and no alerts, while showing figures that are quietly wrong. And because the figures look normal, people act on them.

The challenge is not simply to process data faster. It is to make sure the numbers reflect what actually happened. This article walks through eight design decisions that make the difference.

1. Keep the original data before changing it

A common design mistake is to process incoming data immediately, keep only the calculated results, and throw away what the source actually sent. It saves storage, but it means that when something turns out to be wrong, there is nothing to go back to.

A more reliable approach is to keep every original event first, exactly as received, and only then start transforming it. If an inverter reading arrives late, or a calculation needs to change, the pipeline can revisit the original events rather than trying to unpick totals that have already been produced.

A layered design keeps these responsibilities separate:

The benefit is more than tidiness. If the business changes how a metric is defined, or a problem is found in last month's figures, the numbers can be rebuilt from the original data rather than patched by hand.

2. Events don't always arrive in the order they happened

Take three readings from one inverter, one minute apart:

Reading C arrived before Reading B, even though B happened first. A pipeline that assumes arrival order is the true order would report C's status as "latest", then overwrite it with B's, leaving the dashboard showing an older state.

The fix is to assign each reading to a time interval based on when it happened, not when it arrived. Here are the same three readings placed into one-minute intervals:

By arrival time, B and C swap places and both land in the wrong interval. By the time each reading happened, every reading lands in the interval it belongs to, however late it arrives, and the latest interval correctly holds C as the most recent state. The same applies at any interval length, such as the five-minute intervals typically used for operational reporting.

One caution worth stating plainly: the business metric must be defined carefully too. Power readings are snapshots in kilowatts; they can't simply be added up and called energy generated in kilowatt-hours. Getting the time order right only helps if the calculation itself matches what the business means.

3. The same reading can arrive twice

An inverter sends a reading of 4.2 kW. The network is slow, the inverter doesn't receive confirmation that it arrived, so it sends the same reading again. The platform now holds two messages describing one event. Count both, and the inverter appears to have produced twice what it did.

Across thousands of devices, these repeats are not rare; they happen every day, and they always push figures in the same direction: up.

The key question is: how do we know two messages describe the same event? The answer should be based on the event itself, not the message carrying it. Typically that means a combination of:

• the device that took the reading,
• the time the reading was taken, and
• a sequence number or event ID from the device, where one is provided.

The exact rule depends on what each source provides, which is why it needs to be agreed for each data source rather than assumed.

There's one important exception. If two messages describe the same event but carry different values, they are not duplicates. That could be a correction from the source, or a fault. Either way, the pipeline shouldn't silently keep one and discard the other; it should flag the conflict for someone to look at.

4. Deciding how long to wait for late data is a business decision

A streaming pipeline can't wait forever before reporting a five-minute interval. At some point it has to treat the interval as complete enough and move on. In streaming technology this cut-off is called a watermark: the amount of lateness the pipeline will accept during its normal processing.

It is usually set once, by an engineer, as a number that sounds reasonable. But it is really a trade-off with business consequences:

The right setting should come from evidence, not guesswork. Because the original events are kept (section 1), the team can measure how late readings really arrive. Most may arrive within seconds, while a small share arrive much later after internet outages. Those late readings often come from the same homes every time, so excluding them doesn't just make figures slightly lower; it can make some areas consistently under-represented.

Two things should be decided explicitly, with the business involved:

• How long the live figure waits, based on how late data actually arrives and how quickly decisions need it.
• What happens to data that arrives after that, so it's handled by a correction process (section 7) instead of being quietly left out.

5. Not every timestamp can be trusted

Organising data by when events happened (section 2) depends on the time each device reports, and devices get it wrong. A clock may be set incorrectly, drift over time, or reset after a power cut.

Imagine a reading taken at 10:05 that carries a timestamp of 10:45. If the pipeline trusts it, it believes it has already seen data from 10:45. That can make it stop waiting for genuine readings from 10:05 to 10:45, treating them as "too late", so one faulty clock can affect the figures for every other inverter.

Timestamp checks should therefore happen before the data is used in any calculation:

• Flag readings stamped unreasonably far in the future.
• Flag readings with missing, malformed or impossibly old timestamps.
• Compare when a reading says it happened with when it actually arrived, to spot unusual gaps.
• Track devices whose clocks are repeatedly wrong, so they can be fixed at the source.

The aim isn't to throw suspicious readings away. It is to set them aside, so they can be checked or corrected, before they distort the figures everyone else relies on.

6. Show how complete each number is

A dashboard showing "3,850 kW" says nothing about whether every inverter has reported. That distinction matters more than it seems.

If 100 of 1,000 inverters haven't reported yet, the total will be lower than reality. Without any other context, a user will reasonably read the drop as a real fall in generation, and may act on it.

A more useful dashboard shows the figure and how much data stands behind it:

Two details make this trustworthy:

• The expected number must come from an authoritative source, such as the register of active inverters, not from the data stream. An inverter that has gone silent can't report its own absence
• Don't quietly fill the gap. It can be tempting to scale up the reported total to estimate the full fleet. If that's useful, show it as a separate, clearly labelled estimate, never in place of the measured figure. 

The principle is simple: users should always be able to tell whether a figure is complete, provisional or still awaiting correction.

7. Plan for corrections from the start

Even a well-designed pipeline can't make every reading arrive on time. Late data and corrections are a normal part of real-world operations, so they should be designed for, not treated as failures.

The most reliable approach is to run two paths that feed the same result:  

Fast path and correction path · one trusted result

The fast path gives operators figures within seconds, clearly marked as provisional. The correction path runs on a schedule, picks up readings that arrived late, recalculates the affected intervals and marks them final. For example, if a reading taken at 10:02 arrives at 10:08, the 10:00 to 10:05 interval is recalculated and its status updated.

A few design rules keep this safe:

• Keep enough original datato recalculate any interval that might need correcting.
• Recalculate only what changed, rather than rebuilding everything every time.
• Make processing repeatable, so handling the same reading twice never counts it twice.
• Record each revision, so anyone can see when and why a figure changed.
  
  
• Agree when a figure becomes final for each purpose. Regulatory or financial reporting usually needs a stricter and later cut-off than an operations dashboard.

The goal is for corrections to be controlled, traceable and expected, rather than surprises that undermine trust in the data.

8. Monitor the data, not just the pipeline

A pipeline can be technically healthy while producing poor-quality figures. The platform reports that messages are flowing and every job has succeeded, yet a large share of inverter readings are arriving late or carrying the wrong time.

Monitoring should therefore cover the data itself, not only the systems moving it:

These measures also give data teams and business users a shared, concrete definition of what "reliable data" means.

Questions to ask about your real-time data

Whether you build streaming pipelines or rely on their output, these questions reveal how trustworthy the figures are:

• Is the original data kept, so figures can be rebuilt if something is wrong?
• Are figures organised by when events happened, not when they arrived?
• Is there an agreed rule for what counts as a duplicate, and are conflicting values flagged?
• Was the waiting period for late data chosen from evidence, with the business involved?
• Are device timestamps checked before they affect calculations?
• Does each figure show how complete it is, and whether it is provisional or final?
• Is there a correction process for data that arrives after the cut-off?
• Is data quality monitored alongside technical health?

If the answer to several of these is "no" or "we're not sure", the dashboard may be fast, but it isn't yet reliable.

Final thoughts

Real-time data is not just about shortening the time between an event and a figure on a screen. It's about making sure that figure still means something when the underlying data is late, repeated, incomplete or out of order.

Technologies such as Apache Spark Structured Streaming and Delta Lake provide strong building blocks for this. But technology alone doesn't make a pipeline reliable. That comes from deliberate decisions about how events are identified, how time is handled, how completeness is shown and how corrections are made, and from agreeing those decisions with the people who use the data.

In a real-time environment, getting the data quickly is useful. Getting it right is what makes it valuable. If your organisation needs help with this, please contact us.