• Documentation
    • Search documentation
    karat

    +

    K

    API Reference ↗
    Data connectivity & integrationIceberg tablesIceberg tables & Foundry datasets

    Iceberg tables & Foundry datasets

    Iceberg tables and datasets are both resource types which allow you to represent tabular data in Foundry. Iceberg tables offer interoperability with the broader Iceberg ecosystem along with capabilities such as row-level edits and changelogs, while datasets currently support a wider set of Foundry features.

    Benefits of Iceberg tables

    The following benefits of Iceberg tables are currently available in Foundry:

    • Interoperability: Open Iceberg format means third-party tools can more easily read and write Palantir Iceberg tables.
    • Compaction: Support for compaction without affecting incremental reads.
    • Row edits: Support for DELETE, UPDATE, and MERGE INTO statements, which allow you to conditionally modify rows without the need to re-snapshot.
    • Changelogs: Incrementally consume row deletions and updates.
    • Enriched table history: Enhanced history view surfacing Iceberg table metadata ↗.

    The interface for viewing Iceberg table history can be seen in the following screenshot:

    The Iceberg table history view in the dataset application.

    Foundry functionality not available for Iceberg tables

    Note that certain Foundry features are not currently supported for Iceberg tables, including:

    Notable differences between Iceberg tables and Foundry datasets

    The following differences in behavior between Iceberg tables and Foundry datasets are important to note.

    • Default branches: Iceberg's main branch is called main, whereas Foundry's main branch is called master. In Foundry's integration with Iceberg, main and master are treated as the same, which means a Foundry job running on master will write to an Iceberg table's main. See Branching for details on how Foundry isolates schema changes per branch.
    • Automatic schema evolution: Iceberg table schema evolution is not always automatic. Certain workflows, such as table replaces, evolve the table's schema automatically; however, others require explicit schema evolution commands or settings, such as ALTER TABLE. When writing through Foundry's transforms APIs and Pipeline Builder, Foundry uses table replaces when running non-incrementally to handle this for you. Other workflows, such as incremental schema evolution, may fail and require manual resolution. See the Apache Iceberg documentation on schema evolution ↗ for details.

    Iceberg terminology disambiguation

    Iceberg introduces terms that do not always have direct equivalents in Foundry:

    Iceberg termMeaningFoundry disambiguation
    Table metadata ↗Metadata describing the structure, schema, and state of an Iceberg table.Foundry also tracks metadata for tables and datasets across all formats, but this metadata is managed internally by Foundry services. For Iceberg tables, Foundry exposes the native Iceberg metadata explicitly as a JSON file via the Iceberg catalog.
    Snapshots ↗A record of the table's state at a specific point in time. Iceberg creates a new snapshot with every data modification.An Iceberg snapshot is roughly equivalent to a Foundry transaction. However, Foundry also uses the term "snapshot" differently: in Foundry, a "snapshot" refers to a transaction that fully replaces all data in a dataset, while an Iceberg snapshot captures all types of data operations, including incremental changes. See Iceberg snapshot types for a complete disambiguation of Foundry catalog dataset transaction types and Iceberg snapshot types.