Turning ad hoc business processes into automated pipelines for time series forecasting.
Conversations about forecasting almost always start with the forecast itself — the model, the vendor platform, the accuracy of the output. That is the visible end of the process, and it is the part that gets evaluated.
In our experience, though, the quality of a forecast is largely settled well before the model sees anything. It is settled by how the inputs were assembled: where they came from, what was done to them on the way through, and whether any of that can be repeated next quarter. By the time data reaches a forecasting engine, most of the decisions that determine whether the answer can be trusted have already been made.
How forecasting processes actually evolve
Time series forecasting in asset-intensive organisations has usually been done for years before anyone sets out to industrialise it. The process works. It produces numbers the business uses. It has simply never been designed as a whole, because it was never built as a whole.
What we typically find is a process assembled in layers over time. Transformation logic distributed across pipelines, notebooks and stored procedures, each written to solve the problem in front of someone on the day. Some flows in one tool, some in another, occasionally something in a legacy database that has quietly outlived several reorganisations. Manual inputs versioned by filename in a document library and moved by hand into storage for collection. Extracts pulled directly from operational systems at run time.
None of that is a failure of engineering. Every one of those choices was a reasonable answer to an immediate need, made by people who knew the data well and had a deadline. The pattern is close to universal, and it is what a process looks like before anyone has had the time or the mandate to engineer it end to end.
It does, however, have three consequences that compound.
It becomes slow, because work that was designed to run once now runs in long manual batches. It becomes difficult to verify, not because the numbers are wrong but because with logic spread across several technologies there is no straightforward way to trace an output back through the steps that produced it. And it becomes difficult to repeat, particularly where extracts are pulled live from source at run time and no copy of those inputs is retained.
That third consequence is the one worth designing against. In regulated and asset-intensive environments, the question that eventually arrives is not how quickly a forecast can be produced. It is how a particular number was arrived at, and whether the same result can be produced again.
Start by mapping what you already have
The temptation on this kind of engagement is to begin with the target architecture, because the destination is reasonably well understood and the technology decisions feel like the substance of the work.
We consistently find the opposite. The first phase is tracing the existing process in full: every input back to its origin, every manual step, every handoff between systems, drawn out as complete dataflow diagrams.
It is unglamorous, and it is always tempting to compress. It is also where a surprising amount of the value is created, because the people who understand each part of a process rarely have cause to hold the whole of it in view at once. Mapping surfaces the things that only become visible from above — the same data arriving by two paths, interventions that were never written down, feeds still running whose purpose has moved on.
You cannot automate a process you cannot yet see in full. Until it is visible, any architecture is an assumption with a diagram attached.
Designing for the second run
The rebuild we delivered most recently was a metadata-driven medallion architecture on Databricks. Metadata-driven matters here more than it sounds: the platform's behaviour is defined in configuration rather than hard-coded per source, so onboarding a new input becomes a configuration change rather than a development project. That is what keeps a platform extensible after the delivery team has moved on.
Alongside that, the things that are easy to defer and expensive to retrofit. Logging and error capture as first-class tables rather than an afterthought. Ingestion and error dashboards, so a failure is something you see rather than something you discover downstream. Proper modelling in each layer — conformed dimensions, aggregates, and slowly changing dimensions so that history is retained rather than overwritten.
That last one is the direct answer to repeatability. Once the state of every input is captured and versioned, a previous forecast can be reconstructed, explained, and tested against a proposed change.
Manual data deserves particular attention, because it is usually treated as the exception and it rarely is. Some inputs legitimately originate with a person, and they need a controlled way in rather than a workaround. We built a validated file drop: uploads schema-checked on arrival and rejected at the door with a clear reason, rather than accepted and failing in a downstream job hours later. Every submission attributed and timestamped, landed into the raw layer, versioned into history automatically. It became the single point of entry for material that had previously travelled by email and hand, and then the place the whole forecast is run from.
The shape we are aiming for is a chain where every link is visible:
Source → validated ingestion → versioned history → modelled layer → forecast input → traceable output
Reuse beats rebuild
Two decisions did more for the long-term value of this work than any individual pipeline.
The first was consuming what already existed. A wider team was already producing interval time series data on the platform, so we aligned to read from that rather than recomputing it. That avoided duplicating substantial compute, and more importantly avoided creating a second version of a number other parts of the business already relied on.
The second was conformance. Notebooks, jobs, table structures and deployment were aligned to the standards and CI/CD process being established across the client's platform, in close collaboration with their team and ours. Work built to shared standards compounds. Work built beside them becomes another thing to reconcile later.
The compounding showed up quickly. An asset master hierarchy built to structure the forecast is now used elsewhere in the business, as well as other key dimensions built as a result of this project. We were engaged to industrialise a forecasting process and produced reference data the organisation turned out to need for other things.
The code was the fast part
Much of the pipeline code in this delivery was AI-generated. That is worth saying plainly, because it changed the shape of the engagement in a way that is often misunderstood.
The assumption is that AI-assisted delivery means the same work, finished sooner. What we found is that the time did not disappear — it moved. Writing transformation logic stopped being the constraint, and the hours that freed up went into testing, validating and reconciling outputs against source, and into interrogating the edge cases buried in years of historical data.
That is where the improvement actually came from. A correction routine that standardises a large volume of historical time series data is only as good as the testing behind it, and the output quality at each level of the asset hierarchy improved because there was time to interrogate it properly — not because the code was written faster.
The broader point is that AI changes the cost of building. It does not change the cost of being wrong. When building gets cheaper, the discipline that pays is verification, and the teams that benefit most from AI-assisted delivery are the ones that reinvest the time it saves into proving their data rather than into shipping more of it.
The handover is the deliverable
The completed process and its master execution jobs were handed to the forecasting and data science team, who run it themselves.
That is the real measure of this kind of work. A pipeline only its builders can operate hasn't removed a dependency, it has relocated one. The aim is a process the organisation owns, extends and runs without us — and can explain to anyone who asks.
A forecast is a claim about the future. What makes it credible is not the sophistication of the model, but whether the path to it can be walked again.
If your forecasting or reporting depends on a process that is difficult to repeat, contact the One51 team to talk through the options.
