本文探讨了Flink与Iceberg在流式数据入湖中的配置与优化，分析了checkpoint间隔、提交频率与小文件数量的关系。通过调整checkpoint参数和并行度，优化提交过程，减少背压对数据可见性的影响。同时介绍了预聚合和bucket分区策略，以提高写入效率并降低小文件生成。最后提供了CDC入湖作业的检查清单，确保数据一致性与性能。

本文探讨了数据湖与开放表格式的关系，分析了Hive表的局限性及其在对象存储中的应用问题。Hive表依赖目录重命名，缺乏原子提交，导致部分提交和并发写入问题。开放表格式（如Iceberg、Delta、Hudi）通过将表拆分为不可变数据文件、可变元数据和原子切换的catalog指针，解决了这些问题，实现了在对象存储上支持ACID和时间旅行的能力。

Snowflake's pg_lake and Databricks' Lakebase both wrap PostgreSQL for lakehouse workloads, but they're nearly opposite architectures.

HTAP（混合事务/分析处理）旨在将OLTP和OLAP系统合并，实现实时数据更新和高效分析。通过行列双引擎、行列一体存储和Lakehouse等方案，HTAP解决了传统架构中的数据一致性和性能问题。文章探讨了不同HTAP系统的工作负载隔离、新鲜度边界及评估方法，并提供了选型决策树，帮助企业选择合适的HTAP解决方案。

Lakehouse architectures enable multiple engines to operate on shared data using open table formats such as Apache Iceberg. However, differences in SQL identifier resolution and catalog naming...

电商平台的订单数据从500万条增长到5亿条，推动了数据架构的演进。从MapReduce、Lambda架构到Kappa架构，最终实现了批流一体和湖仓一体。Lambda架构虽然兼顾批流，但开发维护成本高，数据一致性难以保证；Kappa架构简化为单一流处理，但历史数据处理复杂。现代架构趋向于用同一引擎处理批流数据，Flink和Spark各有优势。Lakehouse融合了数据湖与数据仓库，解决了存储与查询性能问题。

MySQL HeatWave is a fully-managed MySQL database service that combines transactions, analytics, machine learning, and GenAI services, without ETL duplication. Also included is HeatWave Lakehouse,...

In traditional MySQL, analytics require data to be ingested into InnoDB tables and processed on a single primary instance using row-based storage engine and limited parallelism. Scaling analytical...

In traditional MySQL, analytics require data to be ingested into InnoDB tables and processed on a single primary instance using row-based storage engine and limited parallelism. Scaling analytical...

MySQL HeatWave Lakehouse enables queries on file formats like Parquet, Avro, ND-JSON and CSV in an object store. This blog introduces support for the Delta Lake table format, thereby enabling...

With automatic refresh of Lakehouse tables, MySQL HeatWave brings near real-time analytics and simplified data pipelines to data from object storage also. In this blog post we take a look at this...