Composer
One51's Metadata-Driven Framework
Composer is our answer to building your Cloud Data Platform using a metadata-driven approach.
A metadata-driven framework improves data understanding, quality, governance, and adaptability to changing business requirements.
READY-TO-GO CONNECTORS
Build your data platform in Azure SQL Databases, Azure Synapse, Google Big Query or Snowflake with ready-to-go connectors to many source systems and source types, including databases such as Microsoft SQL Server, Oracle, Redshift, MariaDB, etc. In addition to file-based systems (CSV and JSON) from Azure Blob Containers, Amazon S3 Buckets, and Google Cloud Storage Containers.
AUTOMATION
Implement an effective and efficient development process using a fully automated solution to provision, configure, and set up the data platform components.
STANDARD PATTERNS
Ensure the use of industry best practices by adopting standard patterns and templates for the data ingestion, transformations and load processes.
REDUCE IMPLEMENTATION TIMES
Control and predict your response times to deliver data to data consumers and decision-makers. Reduce implementation times by automating the development work.
INTEGRATION WITH YOUR DEVOPS PROCESS
Full integration with Microsoft DevOps repositories, workflows and pipelines provides a collaborative environment to develop the data platform and use the pre-built deployment pipelines to deploy your code to multiple environments.
What is a metadata-driven framework?
A metadata-driven framework in data warehousing refers to an approach where a data warehouse’s structure, behaviour, and processing are defined and driven by metadata.
Metadata, in this context, refers to data that provides information about other data.
In a metadata-driven framework, the metadata acts as a set of instructions or configuration parameters that guide the data warehouse’s design, development, and maintenance. This metadata includes information about the source data, transformations, business rules, relationships between data elements, and other aspects of the data warehouse environment.
The advantages of a metadata-driven framework include:
Flexibility
Changes to the data warehouse can be easily implemented by modifying the metadata rather than making extensive changes to the underlying code.
Consistency
Since the metadata serves as a central repository of information, it helps ensure consistency in data definitions, transformations, and business rules across the entire data warehouse.
Documentation
The metadata serves as documentation for the data warehouse, making it easier for developers, administrators, and other stakeholders to understand and maintain the system.
Automation
Automation tools can leverage metadata to generate code, execute processes, and perform other tasks, reducing the need for manual intervention.
By adopting a metadata-driven approach, organisations can enhance their data warehouse implementations’ agility, maintainability, and overall effectiveness.
What problem does a metadata-driven framework solve in a data warehouse project?
A metadata-driven framework in a data warehouse project addresses several key challenges and problems, enhancing the project’s efficiency, flexibility, and overall success.
Challenge
Solution
Data Complexity and Diversity
Data in a data warehouse often comes from diverse sources, each with its own structure, format, and semantics.
Metadata provides a standardised way to define and understand data elements, ensuring consistency and coherence across diverse data sources.
Data Integration and Transformation
Integrating and transforming data from multiple sources into a cohesive and usable format can take time and effort.
Metadata-driven frameworks automate the design and execution of ETL (Extract, Transform, Load) processes by capturing and utilising transformation logic, reducing manual effort and errors.
Data Quality and Consistency
Inconsistencies and errors in data quality can arise due to variations in data definitions and transformations.
Metadata facilitates data quality management by providing a clear understanding of data lineage, definitions, and transformations, enabling better control and validation.
Documentation and Understanding
Without proper documentation, understanding the data model, transformations, and business rules can be challenging for developers and stakeholders.
Metadata acts as comprehensive documentation, offering insights into the structure, meaning, and usage of data, fostering better understanding and collaboration.
Change Management
Changes to data structures, business rules, or source systems can have widespread implications that are not always immediately apparent.
Metadata-driven frameworks support impact analysis, helping teams understand and manage the effects of changes before implementation.
Data Governance and Compliance
Ensuring compliance with data governance policies and standards is crucial but can be difficult without a clear understanding of data lineage and usage.
Metadata provides visibility into how data is used, facilitating data governance efforts and ensuring adherence to compliance requirements.
Agility and Adaptability
Business requirements evolve, and the data warehouse needs to adapt quickly to these changes.
A metadata-driven approach enhances agility by allowing for more rapid modifications to the data warehouse structure and processes through updates to the metadata.
Operational Efficiency
Manual coding of ETL processes and other data integration tasks can be time-consuming and error-prone.
Automation enabled by metadata-driven frameworks reduces manual effort, streamlining operations and improving overall efficiency.
How to get started?
Contact us today! We are looking forward to walking you through the process.