Latest Articles: page 16
692 posts from Alex's blogs and newsletters, newest first. Thoughts on tech, data, policy, and philosophy.
The Endgame – Building an Autonomous Optimization Pipeline for Apache Iceberg
Learn how to automate compaction, snapshot expiration, and layout optimization in Apache Iceberg using metadata-driven triggers and orchestration tool...
Managing Large-Scale Optimizations – Parallelism, Checkpointing, and Fail Recovery
Learn how to scale Apache Iceberg table optimizations across large datasets using parallelism, checkpointing, and fail recovery to ensure reliability ...
Unlocking the Power of Agentic AI with Apache Iceberg and Dremio
Unlocking the Power of Agentic AI with Apache Iceberg and Dremio...
Hidden Pitfalls – Compaction and Partition Evolution in Apache Iceberg
Partition evolution in Apache Iceberg is a powerful feature, but if not managed carefully, it can introduce fragmentation and impact compaction perfor...
Using Iceberg Metadata Tables to Determine When Compaction Is Needed
Discover how to use Apache Iceberg's metadata tables to proactively detect small files, bloated manifests, and table fragmentation - so you can trigge...
Designing the Ideal Cadence for Compaction and Snapshot Expiration
Learn how to design an effective schedule for compaction and snapshot expiration in Apache Iceberg to balance cost, performance, and data freshness....
Avoiding Metadata Bloat with Snapshot Expiration and Rewriting Manifests
Learn how to prevent and clean up metadata bloat in Apache Iceberg by expiring snapshots and rewriting manifests for better performance and manageabil...
Smarter Data Layout – Sorting and Clustering Iceberg Tables
Improve query performance in Apache Iceberg by organizing your data layout with sorting and Z-order clustering. Learn how to reduce scan cost and impr...
Optimizing Compaction for Streaming Workloads in Apache Iceberg
Learn how to design fast, incremental compaction strategies in Apache Iceberg to support high-throughput streaming pipelines without disrupting freshn...
The Basics of Compaction – Bin Packing Your Data for Efficiency
Learn how standard compaction works in Apache Iceberg and why bin packing your data files is essential for maintaining query performance and cost effi...
The Cost of Neglect – How Apache Iceberg Tables Degrade Without Optimization
Learn how Apache Iceberg tables can degrade over time without optimization and what issues this causes for performance, cost, and governance....
How to Discover or Organize Lakehouse & Apache Iceberg Meetups
Guide on How to Be Part of the Lakehouse Community...
Introduction to Data Engineering Concepts | What is Data Engineering?
Introduction to the terms in data engineering...
Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion
Introduction to the terms in data engineering...
Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines
Introduction to the terms in data engineering...
Introduction to Data Engineering Concepts | Batch Processing Fundamentals
Introduction to the terms in data engineering...
Introduction to Data Engineering Concepts | Streaming Data Fundamentals
Introduction to the terms in data engineering...
Introduction to Data Engineering Concepts | Data Modeling Basics
Introduction to the terms in data engineering...
Introduction to Data Engineering Concepts | Data Warehousing Fundamentals
Introduction to the terms in data engineering...
Introduction to Data Engineering Concepts | Data Lakes Explained
Introduction to the terms in data engineering...
Introduction to Data Engineering Concepts | Storage Formats and Compression
Introduction to the terms in data engineering...
Introduction to Data Engineering Concepts | Data Quality and Validation
Introduction to the terms in data engineering...
Introduction to Data Engineering Concepts | Metadata, Lineage, and Governance
Introduction to the terms in data engineering...
Introduction to Data Engineering Concepts | Scheduling and Workflow Orchestration
Introduction to the terms in data engineering...