feat(hive): add Iceberg-to-HMS schema and record conversion helpers - #904
Open
MisterRaindrop wants to merge 2 commits into
Open
feat(hive): add Iceberg-to-HMS schema and record conversion helpers#904MisterRaindrop wants to merge 2 commits into
MisterRaindrop wants to merge 2 commits into
Conversation
Creating an Iceberg table through HMS means rendering its schema as Hive DDL. Add `TypeToHiveString` for one `Type` and `SchemaToHiveColumns` for a whole `Schema`; nested struct / list / map render recursively. `timestamptz` follows iceberg-java and emits `timestamp with local time zone`, since engines reading through HMS take their column types from this DDL. Java's sub-Hive-3 fallback is not reproduced: there is no Hive-version probe here and the vendored IDL is Hive 4.0.1. Iceberg v3 types that no reference maps to a real Hive type (timestamp_ns, variant, geometry, geography, unknown) return kNotImplemented rather than guessing a spelling.
HMS speaks `Database` and `Table`; Iceberg speaks `Namespace`, `TableIdentifier` and `Schema`. Add the conversion between them, plus the parameter keys (`metadata_location`, `table_type`, `EXTERNAL`) that iceberg-java and iceberg-rust use to recognise an Iceberg table, and the LazySimpleSerDe descriptor that keeps Spark and Trino reading it. The helpers produce plain structs rather than Thrift objects, so HmsClient stays the only translation unit that includes the generated bindings. A database owner is carried by `hive.metastore.database.owner` and `hive.metastore.database.owner-type`, the keys both references use; a bare `owner` is the table owner key. `owner-type` requires an owner and must name an HMS PrincipalType -- USER, GROUP or ROLE, compared case-insensitively.
MisterRaindrop
force-pushed
the
feat/hive-conversion-layer
branch
from
September 2, 2026 06:34
69b9934 to
8a4041c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Next step of the HiveCatalog split (follows #796): two conversion layers between Iceberg's catalog model and the HMS record shape. No Thrift, no I/O, fully unit tested.
hive_schema— renders an IcebergSchemaas the Hive DDL column list HMS stores, nested types included.hive_utils—Namespace/TableIdentifier↔ HMSDatabase/Table, the Iceberg-on-HMS parameter keys (metadata_location,table_type,EXTERNAL), and the LazySimpleSerDe descriptor Spark and Trino expect. Plain structs rather than Thrift objects, soHmsClientstays the only translation unit that includes the generated bindings.Where the two reference implementations disagree:
timestamptz→timestamp with local time zone, following iceberg-java; iceberg-rust rejects zoned timestamps outright, which would make such tables uncreatable. Java's sub-Hive-3 fallback is not reproduced — no Hive-version probe here, and the vendored IDL is Hive 4.0.1.hive.metastore.database.owner/hive.metastore.database.owner-type, the keys both references use; a bareowneris the table owner key.Iceberg v3 types that neither reference maps to a real Hive type return
kNotImplementedrather than guessing a spelling. Nothing calls any of this yet —HiveCatalogremains a stub;HmsClient's Thrift wrappers and the catalog CRUD paths come next.33 new unit tests;
hive_catalog_testgoes from 17 to 50 cases.