feat(mp_filtercheck): accept SAS temporal literals in RAW_VALUE

[sasjs-dev]

What

mp_filtercheck rejects SAS date, datetime and time literals in RAW_VALUE, even though a temporal literal is a valid operand for a numeric column - both in a WHERE clause and in the comparison the macro itself generates. The macro header carried this as an open item:

@todo Support date / hex / name literals and exponents in RAW_VALUE field

The check strips quoted strings and then flags any non-digit remainder, so the trailing d / dt / t fails the test:

RAW_VALUE = '20APR2042'd   ->  strip quotes  ->  d  ->  notdigit  ->  bad record

Change

Temporal literals are now accepted for numeric columns:

• a scalar value with the comparison operators (=, NE, >, <, >=, <=, GE, LE);
• every entry of an IN / NOT IN list;
• either side of a BETWEEN.

They remain invalid for character columns, and other literal suffixes (hex '...'x, name '...'n) are still rejected - the pattern is anchored to the temporal suffixes only.

Also fixed

While adding the list cases, the value loop was corrected. The return sat inside the per-value loop, so only the first value of a numeric IN / NOT IN / BETWEEN list was ever validated - a bad second value passed silently. Every value is now checked, and a bad one produces a bad record.

Why

A client can now send the readable form of a filter. Data Controller shows the resolved WHERE clause above the grid, so a date filter read SOME_DATE = 34322 (the internal day count) rather than a date. Sending '20APR2042'd gives a clause that is both valid and legible, and the rows returned are unchanged.

Tests

tests/base/mp_filtercheck.test.sas gains four cases:

• temporal literals (scalar, IN, BETWEEN) against numeric columns produce no bad records;
• a temporal literal against a character column is still rejected;
• a non-temporal literal suffix ('20APR2042'x) is still rejected;
• an invalid value after the first in an IN list is now caught.

sasjs lint reports no errors for the changed files.

[sasjs-dev]

Reviewed sasjs/core #460 (feat(mp_filtercheck): accept SAS temporal literals in RAW_VALUE) at head 6314dd1 - the full diff, base/mp_filtercheck.sas, the generated all.sas bundle, the new tests, and CI status.

• .github/workflows/run-tests.yml did not execute for this head: the Node.js CI run (id 38881470675) concluded action_required with no jobs started, so the new cases in tests/base/mp_filtercheck.test.sas are unverified and mergeable_state is unstable. Not a green pipeline.

• base/mp_filtercheck.sas:275: the added return; at the end of the IN/NOT IN/BETWEEN block is unconditional, but the per-value check above it only runs when vtype='N'. Previously the return sat inside the if vtype='N' loop, so character-column IN/BETWEEN lists fell through to the generic string check at lines 280-324. The comment ("checked one by one above") does not hold for vtype='C' - char lists are now accepted without that validation (a bare-name char list such as (a,b) used to be name/type checked). mp_filtervalidate still runs afterwards, so impact is limited; either gate the return on vtype='N' or confirm char IN/BETWEEN needs no check.

• base/mp_filtercheck.sas:98-102: re_temporal is added to the length statement but is not in the drop (only tmp is), so it is emitted as an extra column in the &outds bad-records dataset. Add it to the drop list if the output shape matters.

• base/mp_filtercheck.sas:248-254: removing goto endstep from the list type-mismatch branch is a behaviour change - a compared-type mismatch inside an IN/BETWEEN list no longer skips the rest of that record's checks. Consistent with the check-every-value fix, but not called out in the PR description.

• base/mp_filtercheck.sas:233-235: empty if prxmatch(...) then do; end; block - if prxmatch(...) then ; (or inverting the condition) reads better.

The temporal-literal regex (^'[^']*'(dt|d|t)$, case-insensitive) and the placement of the scalar early-return after the GROUP_LOGIC/OPERATOR_NM checks look correct, and the four new test cases cover the behaviour the PR claims. all.sas is the generated bundle, so the line references above are against base/.

5 findings above for review.

[sasjs-dev]

Thanks - all four code findings are addressed in 072fdc9.

• char IN / BETWEEN (finding 2): you are right, the unconditional return changed that path. It is now gated: if vtype='N' then return;. The value-by-value loop was only ever numeric, so a character list still falls through to the generic check exactly as before - no claim is being made that a character list needs no check. Pinned with a new test: Name IN (Alfred,Alice) against a character column is still flagged (with the unconditional return it would have been accepted).

• re_temporal in the output (finding 3): added to the drop, so the working regex id no longer becomes a column of the bad-records table. Pinned with a new test that opens &outds and asserts varnum(...,'RE_TEMPORAL') is 0.

• goto endstep (finding 4): restored. With it back, the only behaviour changes in this PR are the two intended ones - temporal literals are accepted for numeric columns, and every value of a numeric list is checked (the return that used to sit inside the loop meant a bad second value passed silently). A compared-type mismatch again ends that record's checks rather than re-checking the remaining values.

• empty then do; end; (finding 5): the temporal test is inverted - if prxmatch(re_temporal,cats(tmp))=0 then do; ... end; - so there is no empty block.

• CI (finding 1): correct, and I cannot clear it from here. The run concluded action_required because the branch was pushed by the app installation rather than a user, so GitHub is holding it for maintainer approval. sasjs lint reports no errors on the changed files locally, and the tests are in the PR for when the run is approved.

sasjs lint is clean for base/mp_filtercheck.sas and tests/base/mp_filtercheck.test.sas after the change.

[sasjs-dev]

Reviewed the current diff at head 093258f, scoped to the delta since the previous review at 6314dd1 (base/mp_filtercheck.sas, the generated all.sas, and tests/base/mp_filtercheck.test.sas).

• The Node.js CI test workflow still has not run: the run for this head (id 38885589519) concluded action_required with no jobs started, so the SAS test suite - including the two new cases - remains unverified and mergeable_state is unstable.

1 finding above for review.

[sasjs-dev]

On the CI point (finding 1) - I cannot clear the approval gate from here, but I did run the test file against a real Viya engine so the new cases are not unverified.

Method: built the branch with a Viya target (sasjs cb), then executed the compiled test program directly (sasjs run on sasjsbuild/tests/macros/mp_filtercheck.test.sas) against a Viya estate using a reusable compute context. The whole file ran, not just the new cases.

Result: 15 assertions, all PASS, and no SAS errors in the log.

• pre-existing cases, all still passing: Valid filter query, Invalid column name, Invalid raw value, Invalid IN value, Code injection - column name, Code injection - raw value abort, Throw error if RAW_VALUE is incorrect, Invalid IN syntax;
• new cases, all passing: Temporal literals in RAW_VALUE, Temporal literal against a character column, Invalid value after the first in an IN list, Non temporal literal suffix, Character IN list is still validated, re_temporal is not in the bad records table.

The two cases added in the review fix are what proves the gate: Character IN list is still validated fails if the early return is not gated on vtype='N', and re_temporal is not in the bad records table fails without the drop.

I have not attached the log - it carries estate-identifying detail - but I can share the relevant excerpt if useful.

[github-actions]

🎉 This PR is included in version 5.4.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀