BigQuery Lakehouse & Apache Iceberg

Open table formats have changed the data-warehouse conversation. Organisations that once argued about warehouse versus lake now expect both: BigQuery's performance and governance, and Apache Iceberg's open format readable by Spark, Trino, Flink and anything else that speaks the Iceberg specification. BigQuery now supports this natively, and getting the architecture right up front avoids the most common failure mode we see — a lakehouse that is open in theory and a tangle of copies in practice.

Managed Iceberg Tables in BigQuery

BigQuery tables for Apache Iceberg store data as Parquet in your Cloud Storage bucket, with Iceberg metadata that external engines can read, while BigQuery manages the table: partitioning and clustering, automatic storage optimisation, DML, streaming ingestion through the Storage Write API, and multi-statement transactions. You keep BigQuery's query engine, row- and column-level security, and INFORMATION_SCHEMA observability, and your data remains in an open format you own. We design the bucket layout, partition specs and write patterns so both sides of that bargain hold.

One Catalog, Many Engines

The BigLake metastore (now positioned by Google as the lakehouse runtime catalog) exposes a single Iceberg REST catalog over BigQuery datasets. That means a table written by BigQuery can be read by Spark on Dataproc or Google Cloud Serverless for Apache Spark, by a Trino cluster, or by a third-party engine without copying data or maintaining a second metadata store. We implement the catalog, wire up each engine with the right credential model, and test the cross-engine read and write paths before anyone depends on them.

When Iceberg Wins — and When It Does Not

We are candid about this, because the wrong choice is expensive to reverse:

• Choose Iceberg when more than one engine must read the same data, when an open-format mandate or exit strategy is a real requirement, when you are consolidating an existing Parquet data lake, or when partners outside Google Cloud need direct file access
• Stay on native BigQuery tables when BigQuery is the only consumer, when you rely on features that are native-only (time travel beyond Iceberg snapshots, certain materialised-view and BI Engine optimisations, the full change-history feature set), or when the team has no appetite to manage a storage bucket's lifecycle

Most estates end up with a mix: native tables for curated analytics marts, Iceberg for shared raw and conformed layers. We help you draw that line deliberately.

Migration Paths

We have migration playbooks for the common starting points:

• Hive-style Parquet data lakes — register existing files as Iceberg tables via the catalog, then incrementally adopt BigQuery-managed writes
• Delta Lake — read in place with BigLake external tables during transition, then convert to Iceberg with validated row counts and checksums
• Native BigQuery tables that now need external consumers — CREATE TABLE ... AS SELECT into a managed Iceberg table with partitioning preserved, cut-over behind a view so downstream SQL never changes

Every migration includes a reconciliation step that compares source and target by partition before the old path is retired.

Governance Stays Intact

Opening data to more engines must not mean losing control of it. We apply Dataplex Universal Catalog policies, IAM conditions on the storage bucket, and BigQuery's row-level and column-level security consistently, and we document which controls apply when data is read through BigQuery versus directly from Cloud Storage — a distinction security teams rightly care about.

Talk to Us About Your Lakehouse

Whether you are consolidating a data lake into BigQuery or opening a BigQuery estate to Spark for the first time, we can scope the work in a single architecture workshop. Contact us to arrange one.

Back to services