Measure Data Architecture
Published in Governance, 2024
Consistency on what we measure and how we measure data domains. An method with an example scenario
the art and science of building data products
In the foundation section, I defined a high-level aspirational vision of the data analytics capabilities required to deliver business value. Now, we need frameworks and solution concepts that can consistently deliver these capabilities. It is important to consider various factors that influence how data analytics solutions are designed and implemented.
The architecture needs to consider each of these elements
- assumptions - These are conditions believed to be true for the architecture but have not yet been validated.
- constraints - These are limitations that must be adhered to, such as budget, time, technology, or legal restrictions.
- domain-specific principles - These are guiding principles specific to the domain that influence how the architecture is developed.
- policies & standards - These are established norms, specifications, and rules that the architecture must comply with.
MDM Patterns. All are relevant and can coexist
Published in Master Data, 2024
Leverage Business Capability Maps for Data Domains
Federated Data Management through Domain-Oriented Decentralized Data Ownership
Published in , 1900
Leverage Business Capability Maps for Data Domains
Collect: Data Capture
Published in Processing, 2024
Capture data from source system for processing in an Analytics System
Data Store: Raw Layer
Published in Processing, 2024
Keep the Raw Layer “Raw”
Data Platform - Enteprise Semantic Layer Requirements
Published in Data Platform, 2024
Deliver Data as an Organized, Unified and Consistent Product
Data Products: Adopting Microservice Architecture Principles
Published in Product, 2024
By applying microservice principles, data products can be designed to be modular, scalable, and maintainable, providing greater flexibility and agility in data-driven environments
Curate: Data Cleansing
Published in Processing, 2024
Deliver quality data
Collect: Data Profiling
Published in Processing, 2024
Data profiling is essential for understanding the quality, structure, and consistency of data
Collect: Pre-ingest vs Post-ingest Processing
Published in Processing, 2024
I do not tend to draw hard lines between applying processing logic directly on the source system before extracting the data or performing transformations post-ingestion in an analytics platform. Both approaches are valid, depending on factors such as data volume, complexity, real-time requirements, and system architecture. However, most modern data scale needs require processing to be done post-ingestion.
Collect - Extract and Load Patterns
Published in Processing, 2024
Transfer data from source to taget
Collect - Extraction: Transfer Compressed Data
Published in Processing, 2024
Move Compresssed data
Collect - Extraction: Batch Transfer
Published in Processing, 2024
Batch Transfer: One of the most common scenarios of extraction. For now atleast