Skip to content

[Bug] Contradictory partition-key equals are treated as a partition drop #9631

Description

@LuciferYang

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version

master, 475be566f (2.1-SNAPSHOT).

Compute Engine

Flink. SupportsRowLevelOperationFlinkTableSink is the only production user of this visitor.

Minimal reproduce step

This one is a contract problem rather than something I could trigger from SQL, and I want to be upfront about that.

OnlyPartitionKeyEqualVisitor decides whether a DELETE can be executed by dropping partitions. It collects the equality literals into a map, one entry per partition key:

public Boolean visitEqual(FieldRef fieldRef, Object literal) {
    boolean contains = partitionKeys.contains(fieldRef.name());
    if (contains) {
        partitions.put(fieldRef.name(), literal.toString());
        return true;
    }
    return false;
}

Two equality predicates on the same key therefore overwrite each other, and the conjunction still reports "droppable". Feeding it pt = 'a' AND pt = 'b':

OnlyPartitionKeyEqualVisitor visitor = new OnlyPartitionKeyEqualVisitor(Arrays.asList("pt", "dt"));
Predicate contradiction = PredicateBuilder.and(equal(pt, "a"), equal(pt, "b"));
contradiction.visit(visitor);   // true
visitor.partitions();           // {pt=b}

The Flink sink reads that as a partition drop of pt = 'b' and deletes every row in that partition, while the predicate itself matches nothing.

What I could not do is get such a predicate to the sink from SQL. DELETE FROM t WHERE pt = 'a' AND pt = 'b' is folded to FALSE by Calcite's simplification before applyDeleteFilters runs, and PredicateBuilder.and does not merge equality predicates itself, so the visitor only sees two Equal leaves if something hands them over unfolded. So today the guarantee that this cannot happen lives in the planner, not in Paimon.

What doesn't meet your expectations?

The visitor reduces a conjunction of constraints to a last-write-wins map, which loses the information that the constraints conflict. For a decision whose consequence is dropping a whole partition, that seems worth deciding inside Paimon rather than relying on an external optimizer to never pass a contradiction through. A future engine version, another engine binding, or a caller that builds the predicate directly would not have that protection.

Anything else?

The same class already returns false for everything it does not understand (visitIsNull, visitIn, ranges, OR), so a conservative answer for a contradiction fits how it is written.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions