One of our energy clients, a large Australian electricity distribution business, has spent the last two years consolidation the majority of its analytical data estate onto Databricks as the foundation of its Enterprise Data Platform (EDP).
Up until now SAS has sat alongside this new data platform, running statistical and reporting workloads for several teams. It is a mature, widely used platform in its own right. With all the dependent datasets now available in Databricks, we are migrating a large number of SAS programs across, one at a time, with the intention of fully decommissioning SAS once the programme is complete.
Why consolidate onto one platform
The case for the migration is not the organization has any issues with the capabilities of SAS. It is simply that running two analytical platforms against a copy of the same data creates duplication that a modern lakehouse architecture no longer needs to carry:
- Platform maintenance: Two software stacks need to be kept patched, upgraded and monitored, each with their own release cadence and support. Jobs, connectivity, monitoring and incident response all must be built once for each platform, even when the analysis they run is similar.
- Access and governance: Access policies and permissions have to be defined and maintained on both platforms in parallel. Any change to who can see what has to land in two places, and auditing access over time becomes a reconciliation exercise across systems rather than a single source of truth.
- Team skills: Analysts and engineers need genuine depth in both SAS and Databricks, not one consistent skill set. Hiring and knowledge transfer both get harder when improvements built on one side cross over to the other.
- Licensing cost: SAS is priced per seat and per platform tier, payable regardless of usage. Some of the significant processing in SAS is only run annually. That cost is harder to justify once the same analysis can run directly against data already held in Databricks – which is costed on actual usage.
- Data locality: The data lives in the lakehouse. Every SAS job runs an inbound step to pull a copy of its inputs out of the lakehouse before it can start, and then an outbound step to push its outputs back in when it finishes. These steps can be slow and costly for large data volumes. Databricks reads and writes those same lakehouse tables natively – within the platform- faster.
- Performance: Databricks plans the whole set of transformations up front rather than executing step by step and works directly against the lakehouse tables. Less data movement plus a smarter execution plan, on the same architecture, is often enough on its own to make a job meaningfully faster.
Approach to Migration
Not every SAS program is migrated the same way. The work boils down to two building blocks that combine differently depending on what a given program actually does – firstly bulk-loading the dataset used by SAS into and translating the SAS program into a Databricks notebook.
Bulk-loading happens through our reusable migration framework. Under the hood, a specialised driver takes advantage of the SAS Dataset’s internal page structure to read it in parallel across many workers at once. Where a program writes to a table with years of history behind it, that history is loaded first so the translated program can cut over cleanly from an agreed date and simply add new data to a table that is already complete. Where there is no program at all, just a dataset, the same framework loads it straight into Databricks and the work stops there.
Translating the program is its own piece of work, each SAS step is translated into a reviewable block of code and checked side by side against the source. Where the same logic recurs across programs, it is built once as a shared, reusable utility, reaching for tried and tested Python libraries, rather than reinventing routines from scratch.
Translation is not line-by-line porting. The logic is re-written the way Databricks runs it best: Databricks' built-in functions are used in preference to custom user-defined functions (UDFs). Delta features such as partitioning, liquid clustering and predicate pushdown are applied where they earn their keep. Migrated programs run as native Databricks workloads that inherit the platform's optimisations by design. Fidelity of output is non-negotiable. The output table that the migrated program writes to is created through a validation framework that enforces alignment with the SAS output schema at write time, so any mismatch in columns, types or ordering surfaces as the program runs rather than later. The way we get there is where the platform's strengths are put to work.
Reconciliation and Sign-Off
Reconciliation is a parallel run. The translated notebook runs in Databricks and writes its output to a Delta table in databricks. In parallel, the equivalent output from the original SAS program (the SAS dataset itself) is loaded into a second Delta table. This side of the comparison runs on a separate reconciliation framework, which uses the same specialised driver under the hood to move SAS data into Delta at scale.
The two tables are then checked on structure first. Column counts, column names and data types must all line up between the SAS side and the Databricks side. Next, the comparison moves to content; hashing every remaining column and comparing record for record. Decimals are rounded to a consistent precision before hashing, since SAS and Databricks can round floating-point numbers slightly differently even with an identical calculation. An exact hash match is the goal. Any residual differences must be explainable precision effects rather than genuine differences in the calculation. The reconciliation framework produces a reconciliation report for stakeholder sign-off, and the result is logged to a central table for auditing: an electronic paper trail, not a one-time assurance.
Lessons Learned
Two systems can quietly disagree about what "the same" data means. A common example is how each treats a blank or missing value when combining datasets. It is a small definitional difference, but one that produces a subtly wrong answer without ever throwing an error. That kind of failure is far more important to catch early than one that is loud and obvious.
Fidelity comes before improvement. A program running reliably for years usually encodes real decisions, some intentional, some accidental, most forgotten. The instinct is to tidy things up while rewriting. The more disciplined approach is to preserve behaviour first, and to raise any genuine issue as a separate conversation with the end stakeholder rather than making a call unilaterally and mid-migration.
The last mile is often where trust is won or lost. Getting the calculation right is not enough on its own. A migrated report also must look and behave how people expect it, or it can erode confidence even when the numbers are correct. Comparing against the original output is not a final QA step. It is part of the work.
Looking Ahead
The programme of work is organised to move through the SAS estate one program at a time. Each migration refines the translation, validation and reconciliation patterns underneath, so the next program is faster and safer to move than the one before.
Consolidation also opens the door to work that was awkward while workloads were split across two platforms. With the data now in one place, it is easier to join datasets across the organisation, enrich existing reporting with context that used to live in a different platform, and support new use cases without another round of data movement first.
If your organisation is planning a similar consolidation, contact the One51 team to talk through the approach.
