16 October 2023
Published by One51

Microsoft Fabric is Improving our Data Quality!

Data Quality is a constant need for organisations, especially those with extensive regulatory reporting responsibilities. For our client (a utilities provider in NSW) One51 was asked to develop a report to detect and report on current data quality exceptions. The report targeted Data Stewards, and those responsible for regulatory reporting. Additionally, any identified data quality issues will be managed and actioned by this group.

One51’s reporting solution effectively streamlined the team’s operations by eliminating the need for manual issue identification, responsibility tracking, issue resolution, and verification of successful resolutions. This saved the team valuable time and resources. Impressed by the time-saving achievements among the data stewards, Senior Management decided to elevate data quality within the system. They opted to do this by enhancing and refining the data entry process, offering additional training, and extending access to the reporting system for the source system users responsible for data entry.

With the investments made into these data quality initiatives (the extra training for data entry & increased visibility on data quality), Senior Management needed to understand further the impact these strategies had on Data Quality over time.

Inspired by all the incredible new features of Microsoft Fabric, One51 Consultants (with the support of the Client Senior Manager) utilised Microsoft Fabric components to take a snapshot of the operational system daily, record the exception counts, write back the results to a Warehouse and create trending Reports from the Warehouse tables. This has now allowed improvements in Data Quality to be tracked over time, day to day, at a granular level.

The solution created (shown above) involved using a gateway to transfer data from on-premises via a gen2 dataflow. A Fabric Data Pipeline orchestrates the process.

  1. Data Quality exceptions are landed into a Fabric Dataflow, which is in turn used to produce the current Data Quality Report.
  2. Data is copied into a Fabric Warehouse, wherein a stored procedure copies daily data into a historical table as a snapshot view.
  3. The Power BI Report (in DirectQuery mode) will always show the latest values using the current and historical data.
  4. The Trend Report patterns can be analysed in the trend report for upward or downward change.

Further inspired by the release of Public Preview for Data Activator, One51 also created a few reflexes to alert team leaders and data stewards when there were spikes in Data Quality issues to be investigated or a shift in trend either positive or negative.

The outcome for the client is a simple, low-code, configurable data workflow to capture point-in-time metrics and be able to carry out trend analysis over time.

About One51

Drawing on a wealth of expertise and a deep understanding of the Energy, Supply Chain, FMCG and other industry sectors, One51 is dedicated to helping businesses navigate the complexities of their operations by harnessing the power of data-driven insights.
With a customer-centric approach, we collaborate closely with our clients to uncover hidden patterns, mitigate risks, and realise new avenues for innovation, all while bolstering their bottom line.
Our tailored solutions, aligned with best-of-breed cloud technologies and delivered using comprehensive analytics frameworks, enable companies to optimise their processes, identify growth opportunities, and make informed strategic decisions.
As a leader in helping companies harness the power of data, the One51 team will work with you to transform complex data into actionable intelligence, helping your business gain a competitive edge in a rapidly evolving landscape.
One51 offers a comprehensive range of services encompassing the entire data and analytics lifecycle, from strategy to successful implementation and ongoing support. Our expertise covers various areas, including Data Assessments, Data Strategy, Data Governance, Data Architecture, Data Management, Data Visualisation, Advanced Analytics and Managed Services.
For more information, please visit one51.consulting