Skip to content

Skip NULL values in SQL aggregates - #893

Open
AarushShetty346 wants to merge 1 commit into
apache:mainfrom
AarushShetty346:fix-sql-aggregate-nulls
Open

AarushShetty346 wants to merge 1 commit into
apache:mainfrom
AarushShetty346:fix-sql-aggregate-nulls

Conversation

@AarushShetty346

Copy link
Copy Markdown

Fixes #891

What changes were proposed in this pull request?

SQL aggregates now ignore NULL values, as in standard SQL:

  • AggregateAddCols: COUNT(col) adds 0 for a row where its argument is null. COUNT(*) has no arguments, so it still adds 1. For AVG, the column now holds {value, 1}, or {null, 0} when the value is null, so the number of non-null values is tracked separately from the row count.
  • AggregateFunction: castAndMap returns the other side when one side is null. SUM, MIN and MAX therefore skip nulls, and if all values are null the result stays null. AVG merges the sum and the count separately. This also removes the countDone handling, which updated the shared row count from inside the AVG case.
  • AggregateGetResult: AVG divides the sum by the number of non-null values. If there are none, it returns null.

How was this patch tested?

I added three tests to SqlToWayangRelTest, using a new exampleNullGroups.csv (six rows in groups a, b, c; group c has only a NULL):

  • javaAggregatesSkipNulls runs COUNT(*), COUNT(col1), SUM, AVG, MIN and MAX without GROUP BY. Each aggregate runs as its own query because of a separate problem with several aggregates and no GROUP BY, which I can open an issue for.
  • javaGroupedAggregatesSkipNulls and sparkGroupedAggregatesSkipNulls run COUNT(col1), SUM and AVG per group. The expected rows are (a, 1, 1, 1.0), (b, 2, 8, 4.0) and (c, 0, null, null).

On main, COUNT(col1) returns 6 instead of 3, the grouped Java query throws an NPE in JavaReduceBy, and the Spark test fails as well. With this change, all wayang-api-sql tests pass (34 in total, Spark included).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQL API: aggregates don't skip NULL values (COUNT(col) counts them, SUM and AVG throw NPE)

1 participant