Declarative Data-Quality Contracts with dbt and Great Expectations in Data-Mesh Architectures
Keywords:
data quality, data mesh, dbt, Great Expectations, YAML contracts, data validation, referential integrityAbstract
Decentralised data architectures like data mesh fail to enforce data quality across domain-specific pipelines. YAML-based "quality-as-code" criteria simplify declarative data-quality contracts for dbt models. These contracts provide row-level completeness, nullability restrictions, and referential integrity across interdependent data domains. Great Expectations validation generates contract compliance DataDocs automatically. This helps us identify data issues early in pipeline construction and execution. We enable product-focused teams and federated data governance and observability. Retail promotional analytics case studies demonstrate our method. It cuts quality regressions by 80% and boosts accountability and SLA compliance. Large, federated analytical ecosystems may facilitate contract-driven quality enforcement.
Downloads
References
Z. D. Dehghani, Data Mesh: Delivering Data-Driven Value at Scale, O’Reilly Media, 2022.
T. N. Tran, D. S. Wang, and Y. K. Lee, “A survey of data quality: Definitions, dimensions, and management,” ACM Journal of Data and Information Quality (JDIQ), vol. 10, no. 2, pp. 1–29, Jan. 2019.
E. Monash, “The Modern Data Stack: Architecture, Trends, and Principles,” DBMS2 Blog
dbt Labs, “dbt Documentation,”
Superconductive, “Great Expectations Documentation,”
D. Loshin, The Practitioner’s Guide to Data Quality Improvement, Morgan Kaufmann, 2010.
T. Griffin and G. Nicholson, “Data Testing in the Modern Data Stack,” in 2020 Proceedings of the Data Council, Austin, TX, 2020.
P. Groves, “Schema Drift in Evolving Data Pipelines: Detection and Governance,” in Proc. IEEE Int. Conf. on Big Data, Dec. 2021, pp. 3175–3182.
M. Stonebraker and J. Hellerstein, “What Comes After the Data Lake?” Communications of the ACM, vol. 62, no. 6, pp. 28–33, Jun. 2019.
L. A. Marinos and N. Kampis, “Data Governance in Federated Architectures,” IEEE Access, vol. 9, pp. 19217–19228, 2021.
J. Kreps, “The Log: What Every Software Engineer Should Know About Real-Time Data’s Unifying Abstraction,” LinkedIn Engineering Blog, 2014.
T. N. Thakkar and A. Patel, “OpenLineage: Tracking Data Lineage in the Modern Data Stack,” in Proc. ACM Data Engineering Conf. (DEC), 2021.
M. Batini, C. Cappiello, C. Francalanci, and A. Maurino, “Methodologies for Data Quality Assessment and Improvement,” ACM Computing Surveys, vol. 41, no. 3, pp. 1–52, 2009.
R. G. Cong, Y. Zeng, and F. W. Li, “Metadata Management for Distributed Data Platforms,” IEEE Trans. on Services Computing, vol. 14, no. 3, pp. 565–578, May 2021.
A. McKinney, “Contract Testing in Data Pipelines,” Thoughtworks Technology Radar, vol. 23, pp. 7–11, 2021.
A. Ghosh, J. Clifford, and R. Elmasri, “Semantic Integrity Constraints for Federated Databases,” IEEE Trans. on Knowledge and Data Engineering, vol. 13, no. 6, pp. 1128–1138, Nov. 2001.
H. V. Jagadish et al., “Big Data and Its Technical Challenges,” Communications of the ACM, vol. 57, no. 7, pp. 86–94, Jul. 2014.
D. Martens and B. Baesens, Data Mining for Business Applications, Springer, 2014.
S. Borkin et al., “Visualization of Validation Results Using DataDocs in Industrial Data Pipelines,” in Proc. IEEE VIS, 2020.
E. Samadi, A. Abouzahra, and H. Singh, “CI/CD for Data Pipelines: Bridging Software and Data Engineering,” in Proc. IEEE Int. Conf. on DevOps, 2022, pp. 145–152.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
