Collect - Extraction: Batch Transfer
Published in Processing, 2024
Batch Transfer: One of the most common scenarios of extraction. For now atleast
Posts by Tags
Collect
Collect - Extraction: Transfer Compressed Data
Published in Processing, 2024
Move Compresssed data
Collect - Extract and Load Patterns
Published in Processing, 2024
Transfer data from source to taget
Collect: Pre-ingest vs Post-ingest Processing
Published in Processing, 2024
I do not tend to draw hard lines between applying processing logic directly on the source system before extracting the data or performing transformations post-ingestion in an analytics platform. Both approaches are valid, depending on factors such as data volume, complexity, real-time requirements, and system architecture. However, most modern data scale needs require processing to be done post-ingestion.
Collect: Data Profiling
Published in Processing, 2024
Data profiling is essential for understanding the quality, structure, and consistency of data
Collect: Data Capture
Published in Processing, 2024
Capture data from source system for processing in an Analytics System
Curate
Curate: Data Cleansing
Published in Processing, 2024
Deliver quality data
Governance
Measure Data Architecture
Published in Governance, 2024
Consistency on what we measure and how we measure data domains. An method with an example scenario
Integration
MDM Patterns. All are relevant and can coexist
Published in Master Data, 2024
Leverage Business Capability Maps for Data Domains
Master Data
MDM Patterns. All are relevant and can coexist
Published in Master Data, 2024
Leverage Business Capability Maps for Data Domains
Product
Data Products: Adopting Microservice Architecture Principles
Published in Product, 2024
By applying microservice principles, data products can be designed to be modular, scalable, and maintainable, providing greater flexibility and agility in data-driven environments
Store
Data Store: Raw Layer
Published in Processing, 2024
Keep the Raw Layer “Raw”