Skip to content

config.splitting is accepted and never applied #88

Description

@jzonthemtn

Description

config.splitting is part of the redaction policy schema and this port never reads it. There is no
split service, and splitting appears nowhere in phileas/. A policy configuring it is accepted,
round-trips intact, and is filtered as though the setting were absent.

The round-trip is what makes it quiet. Policy keeps the config object whole
(phileas/policy/policy.py:94, "Raw config object, kept whole so nothing in it is dropped"), so
a policy with splitting enabled serializes back byte for byte. Nothing in the round-trip, the model,
or the filtering result indicates the setting had no effect.

What a policy author loses

  • Documents are never split, whatever threshold says, so a large document is filtered whole.
  • overlap does nothing, including the boundary-straddling entity recovery it exists for
    (philterd/phisql schema 1.2.0).
  • method does nothing.

The Java and .NET ports both implement splitting, so a policy moved between ports behaves
differently in a way the policy itself gives no hint of.

Scope note

This is a larger gap than the split-method defects filed for the other two ports
(philterd/phileas#392, philterd/phileas-dotnet#105), which are about an unknown method name
silently falling back. Those do not apply here, because there is nothing to fall back from. They are
worth reading first anyway: both ports resolve the method name loudly now, and this port should not
reintroduce a silent fallback when it gains one.

Two things learned implementing splitting in .NET that are worth carrying over rather than
rediscovering:

  • Locate each piece in the input and apply the replacements once to the original text. Filtering
    pieces separately and concatenating them makes span offsets index the output rather than the input,
    and loses the input's whitespace (philterd/phileas-dotnet#92).
  • The piece locator has to skip exactly the characters the splitter trims, which is every
    character at or below U+0020, not what a general "is whitespace" test reports. Testing the wrong
    set silently drops the whole document onto the uncorrected path.

Acceptance criteria

  • config.splitting is bound on the policy model: enabled, threshold, method, overlap.
  • A document at or over threshold is split before filtering when enabled is true, and is not
    when it is false.
  • method selects the splitter, and an unrecognised name raises rather than silently selecting
    another.
  • overlap gives each piece after the first the trailing characters of the previous one, and
    spans duplicated at a seam are de-duplicated.
  • Span offsets index into the original input with splitting enabled, for every method and at
    every overlap, including a policy whose replacements change the text length.
  • Whitespace in the input is preserved in the filtered text, and a document with nothing to
    redact is returned exactly as it was.
  • spec/v1.2.0/examples/splitting-overlap.json both round-trips and takes effect.
  • The documentation states which split methods are supported.

Related

  • philterd/phileas-dotnet#92 and #82: the located-split mechanism and overlap, with a working
    implementation and tests.
  • philterd/phileas-dotnet#105 and philterd/phileas#392: the unknown-method defect in the ports
    that do implement splitting.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdocumentationImprovements or additions to documentationphisqlCloses a gap against the PhiSQL specification

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions