GH-3708: Inline parquet.thrift

[divjotarora]

Rationale for this change

It is currently not possible to build working POCs for unreleased parquet-format changes in parquet-java because the parquet-format dependency must be updated using a released version. This blocks reference implementations from merging and will become a bigger issue if the current versioning proposal goes through and we start having more "preview" features where writes are expected to support unreleased spec changes behind feature flags.

What changes are included in this PR?

This PR removes the dependency on github.com/apache/parquet-format and instead adds an inlined copy of parquet.thrift (parquet-format-structures/src/main/thrift/parquet.thrift) as well as a "sidecar" metadata file (parquet-format-structures/src/main/thrift/parquet-format.version) to indicate the parquet-format commit that's being inlined. There is a new dev/update-parquet-thrift.sh [$REF] script that takes in a parquet-format ref (commit SHA or tag) and updates the inlined parquet.thrift file to match that version. The script downloads from github.com/apache/parquet-format by default but can be pointed at a fork by setting the PARQUET_FORMAT_REPO environment variable (e.g. for POC implementations of unmerged format changes).

Are these changes tested?

Edit inlined parquet.thrift with a meaningless change and then validate it gets overwritten:

-- Validate diff
> git diff
--- a/parquet-format-structures/src/main/thrift/parquet.thrift
+++ b/parquet-format-structures/src/main/thrift/parquet.thrift
...

-- Force an update
> ./dev/update-parquet-thrift.sh apache-parquet-format-2.13.0
Resolving apache-parquet-format-2.13.0 ...
  -> commit: c47e2a66e88943fc46fde1b028a9432f14fdf5c0
Fetching parquet.thrift at c47e2a66e88943fc46fde1b028a9432f14fdf5c0 ...
  -> written to /home/div.arora/parquet-java-inline-thrift/parquet-java/parquet-format-structures/src/main/thrift/parquet.thrift
  -> sidecar updated: /home/div.arora/parquet-java-inline-thrift/parquet-java/parquet-format-structures/src/main/thrift/parquet-format.version

Update complete:
  old commit: c47e2a66e88943fc46fde1b028a9432f14fdf5c0
  new commit: c47e2a66e88943fc46fde1b028a9432f14fdf5c0

-- Validate diff
> git diff
-- empty

-- Update to parquet-format commit containing FILE type
> ./dev/update-parquet-thrift.sh e94a5d090b324a0c0ee1adbb8ea6b099852dc3cc
Fetching parquet.thrift at e94a5d090b324a0c0ee1adbb8ea6b099852dc3cc ...
  -> written to /home/div.arora/parquet-java-inline-thrift/parquet-java/parquet-format-structures/src/main/thrift/parquet.thrift
  -> sidecar updated: /home/div.arora/parquet-java-inline-thrift/parquet-java/parquet-format-structures/src/main/thrift/parquet-format.version

Update complete:
  old commit: c47e2a66e88943fc46fde1b028a9432f14fdf5c0
  new commit: e94a5d090b324a0c0ee1adbb8ea6b099852dc3cc

-- Validate diff
> git diff
--- a/parquet-format-structures/src/main/thrift/parquet-format.version
+++ b/parquet-format-structures/src/main/thrift/parquet-format.version
@@ -1,2 +1,2 @@
 # Provenance of the inlined parquet.thrift. Maintained by dev/update-parquet-thrift.sh.
-parquet-format.commit=c47e2a66e88943fc46fde1b028a9432f14fdf5c0
+parquet-format.commit=e94a5d090b324a0c0ee1adbb8ea6b099852dc3cc
...
diff --git a/parquet-format-structures/src/main/thrift/parquet.thrift b/parquet-format-structures/src/main/thrift/parquet.thrift
...
+struct FileType {
+}
+

-- Download from a fork, commit hash is before IEEE sort order + nan_count changes
> PARQUET_FORMAT_REPO="https://github.com/divjotarora/parquet-format" ./dev/update-parquet-thrift.sh f4288e602a41deba78c58e8bd116b36076a03588
Update complete:
  old commit: c47e2a66e88943fc46fde1b028a9432f14fdf5c0
  new commit: f4288e602a41deba78c58e8bd116b36076a03588

> git diff
--- a/parquet-format-structures/src/main/thrift/parquet.thrift
+++ b/parquet-format-structures/src/main/thrift/parquet.thrift
...
-   9: optional i64 nan_count;
...

Are there any user-facing changes?

No

Closes #3708

[divjotarora]

@rdblue Thanks for the review! You raised a few big points, I'll recap them all here rather than replying to individual comments:

1. Release validation: the release check to ensure parquet-java tracks a released version of parquet-format was a suggestion from the mailing list thread. I audited the C++ and Rust implementations and didn't find a matching check there, so I've removed it.
2. Update script: I've removed the --version argument and now the update script just takes a single ref (either a full commit SHA or a tag) and it just downloads the corresponding parquet.thrift. I've also removed the retry logic.
3. Sidecar file: the parquet-format.version sidecar file now only stores a commit SHA, not a semantic version. In the case that the update script is invoked with a tag ref, it's first resolved to the corresponding SHA and that's stored in the sidecar.

[Fokko]

Thanks @divjotarora for working on this. I'm wondering if it wouldn't be easier to pull the structures from the snapshots repository: https://repository.apache.org/content/groups/snapshots/org/apache/parquet/parquet-format-structures/. It looks like it is missing some versions, but I think it would make sense to just take the snapshot from there and avoid a shell script.

[divjotarora]

Thanks @divjotarora for working on this. I'm wondering if it wouldn't be easier to pull the structures from the snapshots repository: https://repository.apache.org/content/groups/snapshots/org/apache/parquet/parquet-format-structures/. It looks like it is missing some versions, but I think it would make sense to just take the snapshot from there and avoid a shell script.

@Fokko thanks for taking a look. The suggestion to pull in parquet-format via Maven solves the issue for local development and unblocks building reference implementations against a local copy of parquet.thrift. However, it does not make CI pass because CI will still reference a commit hash of parquet-format.

FWIW the motivation for this change was that the Rust and C++ Parquet implementations both vendor this file (Rust technically has hand-rolled Thrift structs and a custom parser, so they don't vendor parquet.thrift but they do make custom changes to those structs for reference implementations and it's conceptually the same thing), so there is precedent for this. Based on my investigation, vendoring the file is the most flexible option.

[rdblue]

Using snapshot builds is an interesting idea, but I think it's simpler to just vendor the file. I like having the file here so that we can see when it was updated, what version it came from, and whether there have been local changes. With a copy from a snapshot build, I think we'd end up not really knowing exactly what is being used (at least not easily). It would also be harder to test out changes before committing them.

I was also surprised by the size of this PR, though. I think that the reason is we are adding requirements that we didn't in the other implementations, like an easy way to check the version or update it automatically. I think it's nice to have those, but the main goal is just to vendor the file and point to the local one. I wouldn't want to split this PR, but if the scope of the util scripts is too much we could to focus on not being tied to a parquet-format release and, separately, the management scripts in a follow up.

[divjotarora]

@rdblue Pretty much all of the management script code has been removed, there is just one script now to pull in a provided ref of parquet-format. This could be removed in favor of copy/pasting in the file, but its < 150 LoC and quite straightforward. The bulk of the PR size comes from the parquet.thrift addition (+1,443 out of the total +1,574 LoC).

[Fokko]

We can do this separately, but I think it would also make sense to run this against a fork.

[divjotarora]

Done, the script honors PARQUET_FORMAT_REPO if set and defaults to apache/parquet-format if not. Tested via

> PARQUET_FORMAT_REPO="https://github.com/divjotarora/parquet-format" ./dev/update-parquet-thrift.sh f4288e602a41deba78c58e8bd116b36076a03588

[Fokko]

I'd rather avoid having two copies of parquet.thrift living in separate places, but it is not a hill I'm willing to die on. I do agree that it speeds up the development cycle, at the same time, I don't expect anyone to iterate much on the thrift definitions.

The only thing I'm missing here is adding some docs on how to use the script.

[divjotarora]

I'd rather avoid having two copies of parquet.thrift living in separate places, but it is not a hill I'm willing to die on. I do agree that it speeds up the development cycle, at the same time, I don't expect anyone to iterate much on the thrift definitions.

The only thing I'm missing here is adding some docs on how to use the script.

@Fokko Can you give me a pointer on where to add such docs? There is no README in the dev folder.