Skip to content

Fix JavaReservoirSampleOperator returning the whole input when the dataset size is unknown - #885

Open
sasmith-14 wants to merge 1 commit into
apache:mainfrom
sasmith-14:fix-java-reservoir-sample-unknown-size
Open

sasmith-14 wants to merge 1 commit into
apache:mainfrom
sasmith-14:fix-java-reservoir-sample-unknown-size

Conversation

@sasmith-14

Copy link
Copy Markdown
Contributor

Fixes #884

What changes were proposed in this pull request?

JavaReservoirSampleOperator returned the entire input instead of a sample whenever the dataset size was unknown. The "return all" shortcut

if (sampleSize >= datasetSize) { //return all

is always taken when datasetSize is UNKNOWN_DATASET_SIZE (0, the default), so sampleSize >= 0 always holds.

This guards the shortcut with isDataSetSizeKnown() — the same check JavaRandomSampleOperator already uses:

if (this.isDataSetSizeKnown() && sampleSize >= datasetSize) { //return all

When the size is unknown the operator now runs the reservoir algorithm, which already handles inputs smaller than the sample size.

How was this patch tested?

Added testExecutionWithUnknownDatasetSize (no setDatasetSize call): it fails on main (expected: <5> but was: <10>) and passes with this change. Full wayang-java suite: 56 run, 0 failures, 0 errors, 5 skipped.

…taset size is unknown apache#884

The reservoir operator short-circuited to "return all" whenever sampleSize >= datasetSize, but datasetSize defaults to UNKNOWN_DATASET_SIZE (0) when the size is not known, so the comparison was always true and the operator returned the entire input instead of a sample. Guard the shortcut with isDataSetSizeKnown(), matching JavaRandomSampleOperator; the streaming reservoir algorithm already handles inputs smaller than the sample size.

Fixes apache#884
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JavaReservoirSampleOperator returns the whole input when the dataset size is unknown

1 participant