A Scalable Data Governance Architecture for Establishing Reliable Data Lineage, Metadata Management, and Quality Controls in Hybrid Enterprise Data Platforms
DOI:
https://doi.org/10.63282/3050-9246.IJETCSIT-V4I2P126Keywords:
Data Governance, Data Lineage, Metadata Management, Data Quality, Hybrid Data Platforms, Data ProvenanceAbstract
Hybrid enterprise data platforms share a single business fact across on-premises databases, enterprise data warehouses, data lakes, cloud storage, distributed processing engines and a range of streaming channels. Governance designed for one warehouse does not scale: metadata is fragmented; lineage exists only where instrumentation happens to exist, quality controls attach to pipelines rather than to the assets they protect, and policy enforcement differs by platform. In this article, scalable data governance architecture is proposed, which incorporates metadata standardization, end-to-end lineage and provenance tracking, embedded quality controls and consistent policy enforcement across heterogeneous platforms. The architecture stems from a well-structured analysis of governance frameworks, metadata and provenance studies, quality standards and documented enterprise systems and carves out a separation of a central governance control plane from distributed data ownership. No implementation was developed, and no dataset or measurement instrument was available; therefore, no experimental performance results are reported. The article contributes a twelve-component architecture traceable to functional and nonfunctional requirements, defined behavior under six operational stress cases, ten reproducible metric definitions, and an evidence-based analysis separating findings reported by prior sources from untested proposals. Empirical validation remains future work.
References
[1] V. Khatri and C. V. Brown, "Designing data governance," Commun. ACM, vol. 53, no. 1, pp. 148–152, Jan. 2010, doi: 10.1145/1629175.1629210.
[2] K. Weber, B. Otto, and H. Österle, "One size does not fit all A contingency approach to data governance," ACM J. Data Inf. Qual., vol. 1, no. 1, art. 4, Jun. 2009, doi: 10.1145/1515693.1515696.
[3] R. Abraham, J. Schneider, and J. vom Brocke, "Data governance: A conceptual framework, structured review, and research agenda," Int. J. Inf. Manage., vol. 49, pp. 424–438, Jul. 2019, doi: 10.1016/j.ijinfomgt.2019.07.008.
[4] DAMA International, DAMA-DMBOK: Data Management Body of Knowledge, 2nd ed. Basking Ridge, NJ, USA: Technics Publications, 2017, ISBN 978-1-63462-234-9.
[5] Y. Cui, J. Widom, and J. L. Wiener, "Tracing the lineage of view data in a warehousing environment," ACM Trans. Database Syst., vol. 25, no. 2, pp. 179–227, Jun. 2000, doi: 10.1145/357775.357777.
[6] P. Buneman, S. Khanna, and W.-C. Tan, "Why and where: A characterization of data provenance," in Database Theory ICDT 2001, LNCS 1973. Berlin, Germany: Springer, 2001, pp. 316–330, doi: 10.1007/3-540-44503-X_20.
[7] J. Cheney, L. Chiticariu, and W.-C. Tan, "Provenance in databases: Why, how, and where," Found. Trends Databases, vol. 1, no. 4, pp. 379–474, 2009, doi: 10.1561/1900000006.
[8] M. Herschel, R. Diestelkämper, and H. Ben Lahmar, "A survey on provenance: What for? What form? What from?" VLDB J., vol. 26, no. 6, pp. 881–906, Dec. 2017, doi: 10.1007/s00778-017-0486-1.
[9] B. Pérez, J. Rubio, and C. Sáenz-Adán, "A systematic review of provenance systems," Knowl. Inf. Syst., vol. 57, no. 3, pp. 495–543, Dec. 2018, doi: 10.1007/s10115-018-1164-3.
[10] L. Moreau and P. Missier, Eds., "PROV-DM: The PROV data model," W3C Recommendation, Apr. 30, 2013. [Online]. Available: https://www.w3.org/TR/prov-dm/
[11] Thusoo, J. Sen Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy, "Hive—A warehousing solution over a map-reduce framework," Proc. VLDB Endowment, vol. 2, no. 2, pp. 1626–1629, Aug. 2009, doi: 10.14778/1687553.1687609.
[12] Halevy, F. Korn, N. F. Noy, C. Olston, N. Polyzotis, S. Roy, and S. E. Whang, "Goods: Organizing Google's datasets," in Proc. ACM SIGMOD Int. Conf. Management of Data, San Francisco, CA, USA, 2016, pp. 795–806, doi: 10.1145/2882903.2903730.
[13] J. M. Hellerstein et al., "Ground: A data context service," in Proc. 8th Biennial Conf. Innovative Data Systems Research (CIDR), Chaminade, CA, USA, Jan. 2017.
[14] M. Stonebraker, G. Beskales, D. Bruckner, M. Cherniack, I. F. Ilyas, S. Zdonik, A. Pagan, and S. Xu, "Data curation at scale: The Data Tamer system," in Proc. 6th Biennial Conf. Innovative Data Systems Research (CIDR), Asilomar, CA, USA, Jan. 2013.
[15] Apache Software Foundation, "Apache Atlas 1.2.0 Data governance and metadata framework for Hadoop," documentation, released Jun. 6, 2019. [Online]. Available: https://atlas.apache.org/1.2.0/
[16] Marquez Project, "Marquez: An open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata," GitHub repository; OpenLineage, "An open standard for metadata and lineage collection." [Online]. Available: https://github.com/MarquezProject/marquez and https://github.com/OpenLineage/openlineage
[17] M. Armbrust et al., "Delta Lake: High-performance ACID table storage over cloud object stores," Proc. VLDB Endowment, vol. 13, no. 12, pp. 3411–3424, Aug. 2020, doi: 10.14778/3415478.3415560.
[18] Apache Software Foundation, "Apache Griffin: Big data quality solution for batch and streaming." [Online]. Available: https://griffin.apache.org/
[19] R. Y. Wang and D. M. Strong, "Beyond accuracy: What data quality means to data consumers," J. Manage. Inf. Syst., vol. 12, no. 4, pp. 5–33, 1996, doi: 10.1080/07421222.1996.11518099.
[20] Batini, C. Cappiello, C. Francalanci, and A. Maurino, "Methodologies for data quality assessment and improvement," ACM Comput. Surv., vol. 41, no. 3, art. 16, Jul. 2009, doi: 10.1145/1541880.1541883.
[21] ISO/IEC 25012:2008, Software Engineering Software Product Quality Requirements and Evaluation (SQuaRE) Data Quality Model. Geneva, Switzerland: ISO/IEC, Dec. 2008. [Online]. Available: https://www.iso.org/standard/35736.html
[22] ISO 8000-61:2016, Data Quality Part 61: Data Quality Management: Process Reference Model. Geneva, Switzerland: ISO, Nov. 2016. [Online]. Available: https://www.iso.org/standard/63086.html
[23] Z. Dehghani, "How to move beyond a monolithic data lake to a distributed data mesh," martinfowler.com, May 20, 2019. [Online]. Available: https://martinfowler.com/articles/data-monolith-to-mesh.html
[24] M. Tang, S. Shao et al., "SAC: A system for big data lineage tracking," in Proc. IEEE 35th Int. Conf. Data Engineering (ICDE), 2019, pp. 1964–1967, doi: 10.1109/ICDE.2019.00215.
[25] L. L. Pipino, Y. W. Lee, and R. Y. Wang, "Data quality assessment," Commun. ACM, vol. 45, no. 4, pp. 211–218, Apr. 2002, doi: 10.1145/505248.506010.
