diff --git a/CLAUDE.md b/CLAUDE.md index 9edeb86..c2ff56c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,16 +72,18 @@ corpus of linked documents. Crate `tread`, binary `tread`. | `src/md/` | `mod.rs` (`parse`), `block.rs`, `inline.rs`, `ast.rs`, `sanitize.rs` | | `src/csv/` | the CSV foundation: RFC 4180 `parse.rs`, lazy `index.rs` with `index_store.rs` (`RowStore`, re-exported so it stays `csv::index::RowStore`), windowed `read.rs`, `delim.rs`. `RowStore::row` is **bounded**: it spends at most one `MAX_ROW_BYTES` of scanning to find where a row ends and otherwise serves it clipped, because proving where the *last* known row ends is a scan to end-of-file on a file whose tail holds no terminator — which on the paint path is a first frame that never flushes and a `q` that is never read. The index is grammar-agnostic: `RowStore::lines` builds it over the quoting-free `Scanner::lines`, and is the **one** constructor every record-per-line format uses (`.jsonl` and plain text). There is exactly one line indexer in the crate; a second copy would drift, and running the CSV grammar over a log would let one `"` swallow the rest of the file | | `src/json/` | the JSON foundation: RFC 8259 `parse.rs`, the source-faithful `value.rs` tree, `write.rs`, `error.rs`, and the lazy structural `index.rs` — all iterative, nothing recurses on nesting | -| `src/source/` | the format seam: `Source`, `markdown.rs`, `csv/`, `json/`, `jsonl/`, `jsonrow.rs`, `detect.rs`. Formats are compiled in, never loaded | +| `src/source/` | the format seam: `Source`, `markdown.rs`, `csv/`, `json/`, `jsonl/`, `jsonarray/`, `jsonrow.rs`, `record/`, `detect.rs`. Formats are compiled in, never loaded | | `src/source/jsonrow.rs` | the **one** JSON tree-row grammar: indent, fold marker, key spelling, collapsed summary, scalar colours, path steps. Both JSON sources build every row through it, and `tests/json_differential.rs` reads the same content both ways and compares — two renderers would drift and the same object would look different depending on which file it came from | | `src/source/json/` | JSON behind the seam: the lazily indexed `tree.rs` with `ident.rs` (a member's bytes, key and path — **derived** from the parent chain, never stored, or the paths alone cost O(depth²)), the fold state and flatten in `flat.rs`, `render.rs`, the `Source` impl in `view.rs`, and `export.rs` (`--to-jsonl`) | -| `src/source/jsonl/` | `.jsonl` / `.ndjson` behind the seam: a record per line over the *CSV* lazy line index, `rowmap.rs` (which rows an open record owns), `tree.rs` (one iterative walker: count, rows, path, over `jsonrow`'s grammar), `rows.rs`, `view.rs`, and the lens half — `plan.rs` (items: which records share a row, and the two-level row arithmetic) and `lensrow.rs` (the lens state on the source, and the rows it paints) | +| `src/source/record/` | records in general, above any one record *format* (SPEC.md §Lenses). There is **one** record source — `source.rs` (`RecordSource`: state, cache, search sweep), `view.rs` (the `Source` impl), `rows.rs` (summary rows, expansion, outline, `--toc`), `ladder.rs` (`Enter`/`za`: the three levels a record has and what descending one costs), `lensstate.rs` (the lens state and its forwarders), `tree.rs` (one record laid out as tree rows, over `jsonrow`'s grammar) — and what a record *format* supplies is `store.rs`'s `Store`: how many records the index has found, how to push it along inside a byte budget, and record `i` as bytes or as a value. Nothing here opens a file, names a format or mentions a line. The lens machinery under it is `plan.rs` (items: which records share a row, and the row arithmetic — a record's own rows depend on the **width**, because its text is wrapped, so `set_width` re-lays every one without reclassifying anything) with `plan_rows.rs` beside it (the level a record is at, its measured heights, and which of its calls is open — all keyed by *record*, because a step inside an open run is as much a thing a reader opens as a message is) and `plan_block.rs` (where a block starts — a member of an open run is a block, and *everything under that member's row* is in it, never in the run's), `rowmap.rs` (which rows an open record owns; `Plan::extra` is a second one of these, holding the rows a member of an open run shows under itself, so `own`, `inside` and `blocks_of_item` keep the shape they had), `lensrow.rs` (the rows a lens paints, the row-to-record translation under them, and the measurement), `body.rs` (one wrap and one clip for every stretch of text this seam shows — a message, a step's reasoning, a call's argument, its output — with the `⋯ +N lines` row that says what the clip left out; a lens never hides anything), `parts.rs` (the **open level**: a record's tool calls as one row apiece, each opening into its arguments and its output under a row that names it. It decides rows — the columns, the glyph, the inset a member of an open run carries — and nothing about what a record contains, which is the dialect's `detail()`), `ops.rs` (what `zR`, `zM`, `Tab`, `Y` and a fold id off the outline do to a plan), and the shared gutter and fold vocabulary: `marker`/`leaf`, `/4` for a record beside `plan::group_id`'s `g4` for a group, the two spellings that must not collide. `Records` is the trait `lensrow`/`ops` reach records through — `with_value` hands the record to a closure, so it is generic rather than `dyn`: a record can be tens of megabytes and lives behind a `RefCell`, and handing out a clone to read one field would undo the cache. A change all lenses need belongs here, not in a dialect | +| `src/source/jsonl/` | `.jsonl` / `.ndjson`: the `Store` that says a record is a *line*, over the **CSV** lazy line index (`RowStore::lines` — the one line indexer in the crate, called and not copied), plus the parse of one line and the two constructors. Everything else it used to hold is `record/`, shared | +| `src/source/jsonarray/` | records that are the elements of an **array inside a JSON document** (SPEC.md §Lenses, ATIF trajectories): the `Store` over the *existing* lazy structural index (`json::tree::Doc` — a member's byte range, no value built), plus the synthesised record 0 that keeps the document's other top-level keys so a lens hides nothing. Finding the array is a structural scan of the top level — the index knows a member only once it has walked past it — so the first row waits on a byte walk of the file and says so; no record is parsed until it is painted. `At::Root` / `At::Key` is the source's half of `lens::RecordsAt`, and nothing here names a dialect | | `src/source/text/` | plain text behind the seam (SPEC.md §Plain text): the file's lines, verbatim, over the *CSV* lazy line index — `mod.rs` (state, file access, the tab/control-character painting rule) and `view.rs` (the `Source` impl). No outline, no folds, no links: the honest empty answers. Anything whose extension names no parser lands here, and so does a corpus link to a `.sh` | | `src/code/` | the language grammars, pure like `md`/`csv`/`json`: a lexer and a declaration recogniser per language (`rust`, `ts`, `py`, `java`), over the shared line arithmetic in `decl.rs`. `docs/code.md` is the contract | | `src/source/code/` | code behind the seam: rows and colouring (`render.rs`, `paint.rs`), import resolution (`jump.rs`, `tsconfig.rs`, `workspace.rs`), the `Source` impl | | `src/source/fold.rs` | what folds and what folding hides. Prose infers a section from the next heading; code *states* where a region ends, because a block that ended at the next heading would swallow the statements after it | -| `src/lens/` | the `--lens` seam (SPEC.md §Lenses): `mod.rs` holds `Lens`, `Summary` and the registry; one module per dialect (`agent.rs` — Claude Code session logs). A lens is a **transform over records**, never a `Source`: it says what one record *is* and nothing about rows, folding or search. A record it does not recognise falls back to the generic tree and is never hidden. Adding a dialect is a module plus one line in `LENSES` — `docs/lenses.md` is the contract | -| `src/open.rs` | resolving the input and building the `Box` behind it; `open/lens.rs` is the one place a `--lens` meets a format | +| `src/lens/` | the `--lens` seam (SPEC.md §Lenses): `mod.rs` holds `Lens`, `Summary`, `RecordsAt` and the registry; one module per dialect (`agent.rs` — Claude Code session logs; `atif.rs` — ATIF trajectories, whose records live inside a document). A lens is a **transform over records**, never a `Source`: it says what one record *is* — including its `Summary::body`, which is a bounded head plus the path back to the text inside the record, never a copy of the message, because a summary is kept for every record — plus the one declaration `records_at()` — where its records live, which is what `src/open/lens.rs` turns into a format and a refusal — and nothing about rows, folding or search. A record it does not recognise falls back to the generic tree and is never hidden. `part.rs` is the row-free vocabulary a dialect answers `detail()` in — `Part::Text` and `Part::Call`, every stretch of text a bounded `Body`, plus `args_of`, the **one** reading of a call's argument list, which both dialects use so an `arguments` that is not the object the schema promises becomes one named row rather than nothing; `agent_parts.rs` is that dialect's half of it, split so both stay under the size limit. Adding a dialect is a module plus one line in `LENSES` — `docs/lenses.md` is the contract | +| `src/open.rs` | building the `Box` behind an input; `open/input.rs` resolves *what* to open (the path or the pipe, the first bytes, the encoding refusal), and `open/lens.rs` is the one place a `--lens` meets a format — it turns a dialect's `RecordsAt` into a reader and into the refusal for pointing a record-per-line lens at a document, or a document lens at a `.jsonl` | | `src/render/` | AST + width to styled wrapped lines | | `src/theme.rs` | palette, heading styles, banner glyphs | | `src/pager/` | viewport, scrolling, collapse tree, search | @@ -155,6 +157,8 @@ TREAD_JS_CORPUS=~/some/node_project cargo test --bin tread a_real_javascript_c TREAD_PY_CORPUS=/usr/lib/python3.11 cargo test --bin tread a_real_python_corpus -- --nocapture TREAD_JAVA_CORPUS=~/some/java_project cargo test --bin tread a_real_java_corpus -- --nocapture TREAD_TS_PROJECT=~/some/ts_project cargo test --bin tread a_real_project_resolves -- --nocapture +TREAD_ATIF_TRAJECTORY=~/some/trajectory.json \ + cargo test --bin tread a_real_atif_trajectory -- --nocapture ``` The last reports what fraction of a project's own imports resolve; run it after diff --git a/README.md b/README.md index a8bb9f8..61b5cf9 100644 --- a/README.md +++ b/README.md @@ -210,8 +210,8 @@ Exit codes: `0` ok, `1` runtime error, `2` usage error. | Key | Action | | --- | --- | -| `j / ↓` | line down | -| `k / ↑` | line up | +| `j / ↓` | one row down | +| `k / ↑` | one row up | | `d` | half page down | | `u` | half page up | | `space / f` | page down | @@ -225,13 +225,14 @@ Exit codes: `0` ok, `1` runtime error, `2` usage error. | `w` | widen the column under the cursor to fit the screen | | `a` | show or hide what this view hides (dotfiles, code bodies) | | `za` | toggle the section at the cursor | -| `Enter` | follow the focused link, open the row, else fold | +| `Enter` | follow the focused link, else one level into the row | +| `r` | show the raw record under the cursor, from either level | | `zo` | open the section at the cursor | | `zc` | close the section at the cursor | | `zM` | collapse every section | | `zR` | expand every section | -| `Tab` | next heading | -| `S-Tab` | previous heading | +| `Tab` | next heading (next block under a lens) | +| `S-Tab` | previous heading (previous block under a lens) | | `o` | outline overlay | | `/` | search forward | | `?` | search backward | @@ -407,6 +408,18 @@ bookkeeping — collapse into one row that opens with `Enter`. `Tab` steps between messages and runs rather than through what a run folded away, and a search hit inside a folded run opens it. +**A record has two levels, and `Enter` toggles them.** The row is a headline +with what was said — or, on a step, what it was thinking — clipped underneath. +`Enter` opens that whole, and lists the record's tool calls as tool calls: +`▸ bash cargo test -q → 32 lines`, one to a line. `Enter` on *that* line shows +the arguments the call was made with and the output it returned, clipped the +same way. `Enter` again is back to the clip. The raw record is `r`, from either +level and back: one key reads the record, the other shows its bytes, and +neither undoes the other. `r` is about records, so in a format that has none — +markdown, CSV, code, plain text, a JSON document — it says `nothing to open +here`; on a `.jsonl` with no lens it opens the record's tree, which is all a row +there is. + A lens only ever *adds* interpretation. A record it does not recognise renders exactly as it would without one, and every summary row still opens into the whole record: `Y` on a run copies every record in it as JSON. Reading a real diff --git a/SPEC.md b/SPEC.md index dd184f3..e7f8199 100644 --- a/SPEC.md +++ b/SPEC.md @@ -139,17 +139,42 @@ Wrapping must use display width, not byte or char count: handle wide CJK Implement a compact `char_width()` in `render.rs` from Unicode ranges — no crate. Never slice a String on a non-char boundary. +## Moving through a document + +**`j` / `k` and the arrows move one visible row, in every format, always.** The +row is what the screen is made of, and the cursor's unit is the row for prose, a +CSV, a tree, code, and a trajectory under a lens alike. Nothing the reader has +just put on the screen — a message's lines, the steps a run revealed, the tree +of an opened record — may need a key other than `j` to walk. + +A coarser default cannot pay for itself. A *closed* block is exactly one row, so +where a document is folded shut "next block" and "next row" are the same motion; +they part only where a block is open, and there the difference is precisely the +rows the reader asked to see. A cursor unit larger than a row therefore skips +only what is on the screen, which makes it wrong exactly when it is doing +something. + +**`Tab` / `S-Tab` are the fast jump**: the next and previous structural landmark +— a heading in prose, a declaration in code, a **block** under a lens. Where the +landmark is a **block**, landing on it shows the whole block if it fits, or its +first row at the top of the screen if it does not (§Lenses). Where a landmark +starts and ends on its own row — a heading, a declaration — there is nothing to +frame, and the landing is scrolled into view like any other cursor move. +`S-Tab` is the exact mirror of `Tab`, over the same landmarks. `d` / `u` and +`space` / `b` count screens, `g` / `G` are the ends. + ## Keybindings ``` -j/k ↓/↑ line down/up d/u half page +j/k ↓/↑ one row down/up d/u half page space/f, b page down/up g/G top/bottom h/l horizontal scroll (code blocks, wide tables) ←/→ select a link on the row; scrolls where the row scrolls za / Enter toggle collapse at cursor heading +r show the raw record under the cursor (§Lenses) zM / zR collapse all / expand all zo / zc open / close current section -Tab / S-Tab next / previous heading +Tab / S-Tab next / previous heading (block, under a lens) n next link; Enter on a link follows it Backspace / - back in document history o document outline / table of contents overlay @@ -376,8 +401,160 @@ The first is agent trajectories, where the generic tree is close to useless: a run is a conversation, and what a reader wants is the conversation with the mechanics folded away. Messages stay visible; consecutive tool calls and their results collapse into one summary row (`⟨6 steps · 4 tool calls⟩`) that opens. -Other dialects — and the ATIF interchange format — are later work; the seam is -what this phase must get right. +Two dialects read one today — `agent` for Claude Code session logs, `atif` for +ATIF trajectories — and a third is a module and a line. + +**A row is a headline; the record is under it.** A summary row is one line — +who, when, what — and for a message that `what` **is the message's own first +line**, at the current width; the rest of what was said is wrapped under it, +indented to the same column. One wrap, split between the row and the rows below +it, so the opening words of a message are on the screen once and a message that +fits on its row has nothing under it. A message that opens with blank lines +starts on the first line that says something: a headline is what was said, and +the blank rows would otherwise be the only thing the reader got. + +**Reasoning is text, so it is shown.** A step that only thought is still one +line — what it *did* — and what it was thinking goes under that line, clipped +the same way and in the muted style. It is under the step wherever the step is, +which, since steps fold into runs, means as soon as the run is open. A folded +run is therefore as tall as what its steps have to say rather than as tall as +the count of them; the count is what the shut row is for. + +**A record has two levels, and `Enter` / `za` toggles between them:** + +```text +clipped <-> open +``` + +**Clipped** is the headline and a few rows of what was said or thought, whose +last row states what it is not showing (`⋯ +37 lines`). **Open** is a record +read as what it is: the whole of that text, and then the record's tool calls +listed *as tool calls* — one row each, naming the tool, the argument that says +what it did and the size of what came back. The second press is the first again. + +**The raw record is `r`.** `r` shows the record under the cursor as a JSON tree +— every byte of it — and shuts it again, from either level, leaving the level +where it was. One key per job: `Enter` reads the record, `r` shows the bytes, +and neither undoes the other. With a tree open, `Enter` leaves the tree alone +and still toggles the record's own rows underneath it; the way to shut a tree is +the `r` that opened it. A key that silently undid another key's work is what +this replaced — the reader who wanted the clip back had to cycle through the +JSON to reach it. + +`r` is a key about **records**, so where there are none it says so: in markdown, +CSV, code, plain text and a JSON document read as a document it reports `nothing +to open here` rather than appearing to do nothing. On a `.jsonl` read with **no +lens** there are still records — one collapsed tree per row — and `r` opens the +one under the cursor, which is the same tree the row's own fold shows. + +A record has only the levels it has content for. One whose text the clip already +showed in full and which made no calls has nothing between its headline and its +JSON, so it has no `Enter` rung at all: the key does nothing there rather than +repainting the same screen, and — this is the same rule as the one above, not an +exception to it — it does not reach for that record's raw tree instead. A record +row answers for `Enter` whether or not it has anywhere to go, at either level and +with a tree open or shut, because the alternative is that the key means one thing +on the records with something to open and `r`'s job on the records without. +`r` is still the way to that record's bytes. + +**A tool call is itself openable.** `Enter` on a call row shows the arguments +the call was made with, one to a line, and then — under a row that names it — +the output it returned, clipped like a message body, with the same row saying +what it left out. The output is named because output whose lines read +`key value` is otherwise indistinguishable from two more arguments. One call +opens at a time; leaving the level shuts it, because the rows are gone and a +call left open would come back unasked. A call with no arguments and no answer +has nothing under it, so it carries no marker and its `Enter` is the record's: +a rung that repaints the same screen is not a rung. `r` on a call row is the +**record's** tree: a call has no JSON of its own separate from the record it was +made in, so the honest raw thing under that row is the record. + +An argument list that is not the object the schema promises — an array, a +`arguments` string the wire cut in half — is one argument holding what the file +said, under that name. It may be clipped; it may not go missing. + +Every one of those levels is a *reading*. A clip may never be silent about what +it left out, the sizes a level states are the true ones, and the record itself +is never further than `r`, which opens the raw tree of the record under the +cursor from either level and leaves the level where it was — the bytes in one +press, and one press back. `zR` puts every record the viewport has +reached at the open level and opens its tree — which is what a batch (`--plain`, +`--toc`) is — and `zM` puts them all back to their clip. A pipe therefore gets +what was said, what was thought, what was called, and the whole record under it. + +A message therefore makes a block as tall as the width allows, and a resize +re-wraps it. That is the one place in a record document where rows move without +a fold changing, and it is why a mark into a file read *through a lens* is the +**record**: the cursor comes back to what it was reading, on that record's own +row. With no lens nothing wraps, and a mark there is the row it always was. + +**A document read in blocks is *framed* in blocks.** A trajectory under a lens +is a list of **blocks** — a message with what was said under it, or a folded run +of mechanics. `j` / `k` are rows here as everywhere (§"Moving through a +document"), so the lines of a message, the steps an opened run revealed and the +rows of an opened record are all walked one press at a time. What the blocks are +for is `Tab` / `S-Tab`, the fast jump: they step from one block to the next, and +landing on a block shows the block — one that fits is scrolled fully into view, +one taller than the viewport puts its first row at the top. There is one +definition of where a block starts, and `Tab` has always stepped between exactly +those. + +`Tab` is the block and not the conversation turn. A message *is* a block, so the +block jump reaches every message a message jump would; the reverse is false, and +a jump that skipped the runs would put the mechanics — most of what a trajectory +is — out of reach of everything but `j`. It is also the general answer: one +boundary for every format, rather than a second one that exists only under a +lens. A record no dialect recognised is a block like any other, which is how +"nothing is hidden" applies to motion, and why `Tab` keeps moving through a file +whose dialect nothing reads. + +**A boundary descends into a run that is open.** A shut run is one block — that +is what makes it a summary — but opening one is the reader saying "show me what +is in here", so the steps inside it become the blocks: the run's own row, then +one block per step, a step and whatever tree it has open counting as one the +same way a message and its body do. `S-Tab` mirrors that exactly, out of the run +and on to the block above it, and the block the status bar counts is the block +`Tab` jumps by, so opening a run makes the total grow rather than drift. At the +tail of a document the jump keeps going through the mechanics rather than +dead-ending, and never past the end. + +Where a block cannot be placed — past the classified prefix of a big trajectory, +where grouping is not decided yet — `Tab` says so rather than moving to a row the +next keystroke would disagree with, and `j` is unaffected: it was never asking. + +Every other format keeps its landmarks as they were: prose, a CSV row, a source +line, a tree node and a record file with no lens are each already the thing they +look like, and their `Tab` is untouched. + +**Records inside a document.** A record file is usually one record per line, but +a trajectory in the ATIF interchange format is a single JSON document whose +records are the elements of a named array, alongside top-level keys describing +the run. A dialect therefore *declares* where its records live, and `--lens` +reads whichever file that names: a `.jsonl` for a record-per-line dialect, one +`.json` document for a records-in-a-document one. Pointing either at the other +is refused by name rather than rendered wrongly — a document read as records is +one enormous record, which is not an error a reader can see. + +Two rules follow from a document holding more than its records: + +- **The keys that are not the records are record 0.** `schema_version`, + `session_id`, `agent` — whatever the envelope holds — get a summary row above + the first record, which opens into their generic tree. A lens adds + interpretation and never hides data, and that applies to the document around + the records as much as to the records. The cost is that record numbering is + shifted by one against the array's own indices, which the status bar and `#n` + say plainly. +- **Finding the records costs one structural scan, and no parse.** The + structural index knows a member only once it has walked past that member's + last byte, and the array is one member — so the first row of a document lens + waits on a byte walk of the file, reported as `≥N (indexing P%)` like every + other scan, with nothing parsed and nothing held in memory. After that it is + the ordinary contract: a record is parsed when it is painted and not before, + at any file size. A record per line has no such wait, which is the reason to + prefer that envelope when a format is being designed rather than read. + A **batch** — `--toc`, `--plain` — waits for that scan rather than printing + what one slice happened to reach: a short list and a zero exit status is the + one answer a script cannot tell from "this file has no records". ## Plain text diff --git a/docs/lenses.md b/docs/lenses.md index a33c59a..a3b3f0d 100644 --- a/docs/lenses.md +++ b/docs/lenses.md @@ -1,6 +1,6 @@ --- status: Active -updated: 2026-08-06 +updated: 2026-08-12 related: - layout.md - testing.md @@ -15,7 +15,8 @@ scan, and consecutive mechanical records fold into one row that opens. ```sh tread --lens agent ~/.claude/projects//.jsonl -tread --lens list # what there is; exits 2 +tread --lens atif trajectory.json # records inside a document +tread --lens list # what there is; exits 2 ``` ```text @@ -32,21 +33,212 @@ Two rules the seam is built around, and neither is negotiable: renders exactly as it would with no lens: the generic collapsed-record row, openable into the whole record. Nothing is hidden and nothing is dropped. * **Every row still opens into the raw record.** A summary is a headline. - `Enter` / `za` on a message row shows the record as a tree; on a folded run it - shows the records inside it, each of which opens in turn. `Y` on a run copies - every record it holds, as JSON. + `Enter` / `za` on a record descends one level — what was said in full and its + tool calls listed, and back again; `r` shows that record as a tree + from any level. On a folded run, `Enter` shows the records inside it, each of + which opens in turn. `Y` on a run copies every record it holds, as JSON. + +## The message under a row + +A summary row is one line, and for a message **that line is the first line of +what was said**. The rest of it is under the row, wrapped to the view width and +indented to the `what` column: + +```text +▾ assistant 10:55 Reading the failing test first. The suite names a + fixture that no longer exists, so that is where this + starts. + ⋯ +37 lines +``` + +It is one wrap, split in two: row 1 is the summary row's `what` column, rows +2..N are the body. The message's opening words are therefore on the screen +**once** — the row is not an excerpt with the same words repeated under it. A +message that fits on the summary row has no rows under it at all. A message that +opens with blank lines starts on its first line with words on it, rather than +spending the headline and the rows under it on the newlines someone happened to +type first. + +`Summary::what` is unchanged by this: it is still the one-line excerpt, and it +is what `--toc` prints, which is the right answer for a list rather than for a +screen. + +## The two levels a record has, and the key that shows its bytes + +`Enter` / `za` toggles the two (SPEC.md §Lenses); `r` is the record itself, from +either of them: + +```text +clipped <-> open r: the raw JSON tree, over whichever level is showing +``` + +| Level | What is on the screen | +| --- | --- | +| **clipped** | the headline, and the text under it cut to six rows in all — whose last row says what it is not showing, in the text's own lines, or in bytes when what is left is the tail of one long line | +| **open** | the whole of that text, then one row per tool call: `▸ bash cargo test -q → 32 lines` | + +A record has only the levels it has content for. A message the clip already +shows whole that made no calls has nothing between its headline and its JSON, so +it has no `Enter` rung at all and the key does nothing there rather than +repainting the same rows. It does **not** fall through to the record's own tree: +a record row claims `Enter` whether or not it has a rung, at either level, with +that tree open or shut — the absolute below has no exception for the records that +happen to have nothing to open. `r` is still the way to its bytes. + +**`Enter` never opens a tree and never shuts one.** It used to: the tree was a +third rung, and getting from it back to the clip meant walking round the whole +ladder. Now `r` owns the tree and `Enter` owns the reading, so with a tree open +`Enter` leaves it alone and toggles the record's own rows *underneath* it — a +key that silently undid another key's work is the thing that was removed. Two +keys, two jobs, and each one is its own way back. + +**A call row opens too.** `Enter` on one shows the arguments the call was made +with, one to a line, and then the output it returned under a row that **names** +it — clipped like a message body, with the same `⋯ +N lines` tail. One call at a +time, wherever the other one was; leaving the level shuts it. + +```text +▾ assistant 10:55 Reading the failing test first, since the suite names a + fixture that no longer exists. + thinking + The fixture was renamed two commits ago. + ▾ bash cargo test -q parse → 32 lines + command cargo test -q parse + timeout 120 + output + test parse::empty_input … ok + ⋯ +26 lines + ▸ read src/parse.rs → 40 lines +``` + +The `output` row is not decoration: without it, output whose lines happen to +read `key value` sits at the argument-name column with nothing between it and +the arguments, and a reader sees three arguments two of which are output. + +An argument's name is written **into** the pad its value's wrap left for it, so +the value's first row starts in the same column as its own continuation rows. +That is one column of arithmetic (`KEY_COL`, the name field *and* the space +after it) and getting it wrong pushed every argument's first row off the side of +the view while its wrapped remainder sat at the correct indent. + +A call with **nothing** under it — no arguments and no answer — carries no +marker, and `Enter` there is the record's rather than the call's. It is the same +rule the rung above obeys: a key that repaints the same screen has not descended. +Everything under a member of an open run is inset with that member's own row, so +a step's reasoning and its calls line up with the step's words rather than +sitting two columns left of them. + +`r` is orthogonal to all of it: it opens the record's own tree from either +level and leaves the level alone, which makes it the way to the bytes in one +press and the same press back. On a **call row** `r` is the row's *record*: a +call has no JSON of its own separate from the record it was made in, so that is +the honest raw thing under it. On a **run's row** there is no one record, so `r` +says "nothing to open here" rather than appearing to do nothing — `Enter` is the +key that opens a run. + +`zR` puts every record the viewport has reached at the **open** level and opens +its tree; `zM` puts every one back to its clip and drops any call that was open. +A batch (`--plain`, `--toc`) is `zR`: a pipe gets what was said, what was +thought, what was called, and the whole record under it. `--toc` is unchanged — +it is a list, one line per record, and `Summary::what` is that line. + +`zR` deliberately does **not** expand every call in the file. A run of steps +would become thousands of rows behind one keystroke, and every argument and +every output is already in the record's own tree, which `zR` opened. + +## Reasoning is text, so it is shown + +A step that only thought is one line — what it *did* — and the thought goes +under that line, clipped, muted. It appears wherever the step is, which, since +steps fold into runs, means as soon as the run is open. + +That cost a change to the row arithmetic worth knowing about. A group's own rows +are still `1 + count`, and a member's own rows — its body and its parts — are a +**second prefix sum** beside the tree one (`Plan::extra`, a `RowMap` of exactly +the shape the trees use). So `own`, `inside`, `row_of_record` and +`blocks_of_item` keep the shape they had, and the invariant they rest on is the +one they always rested on: *a hidden record owns no rows*, which is why closing +a run closes its members' rows as well as their trees. + +**What this costs.** A block's rows now depend on the **width**, so a resize +that changes it re-lays every body — without reclassifying anything: a record is +read once, in file order, and only the wrap is redone. It is also why a `Mark` +into a record document **read through a lens** is the *record* rather than the +row, so the cursor comes back to what it was reading; the offset inside a record +does not survive, and a cursor on line four of a message lands on that message's +own row. With no lens nothing wraps, so a mark stays the row it always was and +the cursor keeps its place inside an open record. ## Keys, under a lens +A trajectory read through a lens is a list of **blocks** — a message with what +was said under it, or a folded run of mechanics. `j` and `k` do **not** move by +those: they move one visible row, here as in every other format +(SPEC.md §"Moving through a document"). A closed block is one row, so a block +cursor differed from a row cursor only where a block was open — over exactly the +message body, the opened run and the opened record the reader had just asked to +see. That is what it skipped, and it is why it is gone. + +Blocks are what `Tab` / `S-Tab` jump between, and there is exactly one +definition of where one starts — `src/source/record/plan_block.rs`, which the +jump, the framing of a landing and the status bar's counter are all read off. +(`plan.rs` calls a run of records that share a row an `Item`. An item is one +block while it is shut, and its own row plus one block per step — `1 + count` — +while it is open, which is the whole of the descent rule; *block* is the only +word above that module and the only one a reader ever sees.) + | Key | What it does | | --- | --- | -| `Enter` / `za` | open the run under the cursor, or the record's raw tree | -| `zR` / `zM` | open the runs the viewport has reached / shut every run | -| `Tab` / `S-Tab` | next / previous **item** — a message or a run, skipping what a run folded | +| `j` / `k` | next / previous **row** — a message's next line, a step, a row of an opened record | +| `Tab` / `S-Tab` | next / previous **block** — a message, a shut run, or a step inside a run that is open | +| `Enter` / `za` | the record's two levels: clipped ↔ open. On a run's row, open the run; on a call row, that call's arguments and output. Never opens or shuts a tree | +| `r` | show the raw record under the cursor, from either level, leaving the level where it was; on a call row, that row's record | +| `zR` / `zM` | every record the viewport has reached at the open level, with its tree / every one back to its clip | | `/` `n` `N` | search the record source text; a hit inside a folded run **opens that run** | -| `y` | the value under the cursor · `Y` the record (or the whole run) · `c` the source line verbatim | +| `y` | the value under the cursor · `Y` the record (or the whole run) · `c` the record's own source text verbatim | + +Landing on a block with `Tab` shows the block: one that fits is scrolled fully +on screen, one taller than the viewport puts its first row at the top. `d` / `u`, +`space` / `b` and `g` / `G` still count screens and rows, and `v` then `j` +extends a visual selection a row at a time, which is what a selection is made +of. + +**Why the jump is the block and not the message.** `Tab` used to be the +conversation turn. A message is a block, so the block jump reaches every message +the message jump did; the reverse is false, and a jump that stepped over the +runs would leave the mechanics — most of what a trajectory is — reachable only +by `j`. One boundary for every format also beats a second one that exists only +under a lens. `S-Tab` is the exact mirror: the same table, walked backwards, +framed the same way. + +**A block boundary descends into an open run.** A shut run is one block — that +is what makes it a summary — and `Tab` steps over it. Once it is open, the +blocks inside it are the run's own row and then each step: `Enter` is the reader +asking for what is in there, so that is what the jump walks. A step and the tree +it may have open are **one** block, exactly as a message and its body are, so +`Tab` clears an opened record in one press while `j` reads it a row at a time. +`S-Tab` mirrors the sequence step for step, including stepping back out of the +run to the block above it. At the tail of a document — a trailing run of +mechanics with nothing said after it — `Tab` keeps going through the run rather +than dead-ending, and never moves past the end. + +Where a block cannot be placed — past the classified prefix, the tail of a big +trajectory where grouping is not decided yet — the jump has no answer and says +so, rather than landing on a row the next keystroke would renumber. `j` is +unaffected: it never asked what block it was in. + +The status bar reads +`agent · record 412/2354 · block 96/≥181 · .message.content[0].text`. The block +clause is lens-only, and it carries `≥` for the same reason the record count +does and then one more: the lens has read a prefix, and grouping makes the block +total *shrink* as classification catches up. Opening a run moves it the other +way — its steps are blocks while it is open — and the counter says so on the +keystroke that opens it, because the index and the total are read off the same +table `Tab` jumps by. Past that prefix there is no block clause at all, rather +than a number the next keystroke changes. The record count keeps saying +**record** — it is the file's own unit, which `#n` and `--toc` also mean, and a +`.jsonl` line is not a step. -The status bar reads `agent · record 412/2354 · .message.content[0].text`. `--toc --lens agent` prints the same reading as a list — `line, actor, time, what`, one record per line, tab-separated — with the records the lens does not recognise keeping their generic summary. @@ -89,24 +281,124 @@ the API calls it a user turn. That is precisely why the generic tree is close to useless on a trajectory — half the "user messages" are machine output — and why a `user` record whose only content is a `tool_result` is mechanics here. +## The `atif` dialect + +Agent trajectories in the ATIF interchange format (`ATIF-v1.7`, as written by +`opencode`). One JSON **document**, not a record per line: `schema_version`, +`session_id` and `agent` describe the run, and the records are the elements of +`steps`. That is all the lens says about the envelope — +`records_at() -> RecordsAt::Member("steps")` — and `src/source/jsonarray/` does +the rest, knowing nothing about ATIF. + +```text +▾ session ATIF-v1.7 · opencode 1.2.3 · vendor/model · sxs_… +▾ user make the parser handle empty input rather than + raising, and say so in the docstring. +▾ assistant 10:55 Reading the failing test first. The suite names a + fixture that no longer exists. +▾ tool 10:55 thinking · bash(pytest -q tests/) → 32 lines + ▸ ⟨4 steps · 6 tool calls⟩ 10:56 +``` + +A message row starts with the message's first line and continues under itself; +a step row has no body and is the one-line description of what it did. + +**Record 0 is the session**: the document's top-level keys that are not `steps`, +as one row that opens into their tree. Nothing the document says is lost, and +the price is that record numbering is shifted by one — `steps[0]` is record 2, +which is what `record n/N` and `#n` mean. + +**Top-level keys of a step.** `step_id` (an integer, and the recogniser), +`source` (`user` / `agent`), `timestamp` (ISO-8601 with a numeric `+00:00` +offset — the row shows `HH:MM`, as recorded), `message`, `reasoning_content`, +`tool_calls[]`, `observation.results[]`, plus `model_name`, `metrics` and +`llm_call_count`, which the summary leaves to the opened tree. + +| A step whose… | Rows as | +| --- | --- | +| `message` says something | **message** — never folded away. It starts on its own summary row and continues under it. Its tool calls collapse to a count on `Summary::what` (`· 3 tool calls`) rather than becoming rows — which is what `--toc` prints; the painted row gives that column to what was said, because the message is what the reader came for | +| `message` is empty | **step** — folds into a run with its neighbours | + +A step's row is what it did, and what it was **thinking** is under that row, +clipped and muted — `reasoning_content` is text, and a run of steps that only +thought used to say `thinking` five times and nothing else. Opening the step +(`Enter`) shows the thought whole and lists its calls; opening a call shows the +arguments it was given and the output it returned, matched by `source_call_id` +within the step. A result no call claimed gets a part of its own at that level +rather than the bare `· 1 result` the row can spare. + +`arguments` is read as an object *or* as the JSON-encoded string the wire format +emits, and in the second case the decoded object is a temporary — so an argument +is always a head with no path back, which is why a long `command` or a +`patchText` shows its opening and says how much more there is. A string that +does **not** parse is kept as the string it is, under the name `arguments`: a +truncated `arguments` is exactly the thing a reader opened the level to look at. + +A result's size is stated once and measured once. A `content` that is not a +string is measured as the JSON it is — the same bytes the open level shows one +rung down — rather than as the word `ok`, which is not a size; and a `content` +of `null` reads as absent, which is what `null` means everywhere else here. + +A step row is what it did: `thinking`, then each call as +`bash(make -j8) → 42 lines`. The argument is named by `command`, `filePath`, +`pattern`, `query`, `url`, in that order — a `glob` carrying both `path` and +`pattern` shows the pattern — and a tool whose arguments name none of them +shows `todowrite()`. A result is matched to its call by `source_call_id` +**within the step**, which is where every answer sits; an orphan result is +counted (`· 1 result`) rather than dropped. Adjacent entries that read +identically collapse: `bash(pkg-config --exists onig) ×2`. + +**The first step of a real trajectory carries `message`, `source` and `step_id` +and nothing else** — no timestamp. The row leaves the clock empty rather than +inventing one, and that is the first row on the first screen. Absent, `null` and +`[]` all mean the same thing, `arguments` is read as an object *or* as the +JSON-encoded string the wire format this schema descends from emits, and a step +this does not recognise keeps its generic row. + +Measured on a real 200KB, 49-step trajectory (`TREAD_ATIF_TRAJECTORY`, 140 +columns): 50 records, 50 read by the lens, 91 rows shut and 1231 open. Of the +records visible with every run shut, 26 have an open level, contributing 103 +part rows in all, and every one of those carries a call that opens further. +Opening every one of those calls in turn — 38 of them — paints every row at 80 +columns and at 140, with no row *under* an opened call wider than the view: the +arguments and the output are a wrap, and the call row above them is the only +thing that scrolls sideways. +`--toc --lens atif` prints the whole run as 50 tab-separated lines, headlines +only — a list is a list, and the levels are for a screen. + ## Adding a dialect -opencode and OpenAI Codex logs are wanted and **not implemented**. Adding one is -a module and a line, and nothing else: +opencode's own logs and OpenAI Codex logs are wanted and **not implemented**. +Adding one is a module and a line, and nothing else: 1. **A module under `src/lens/`** with a struct implementing `lens::Lens`: * `name()` — the `--lens` word. * `about()` — one line for `--lens list`. + * `records_at()` — where the records are: `RecordsAt::Lines` (the default, a + `.jsonl`), `RecordsAt::Root` (a document whose root array is the records) + or `RecordsAt::Member("steps")` (a document whose records are one of its + keys, every *other* key becoming record 0). This is the only thing a + dialect says about files, and the routing in `src/open/lens.rs` turns it + into a format and a refusal. * `read(&mut self, &Value) -> Option` — called **once per record, in file order**, so a dialect may carry state (the agent lens keeps a bounded ring of `tool_use` ids so a result can name its call). Return `None` for anything it does not recognise; that record then renders generically. + * `detail(&self, &Value) -> Vec` — optional, and what the **open + level** shows: what this record's parts *are*. Called for the record the + reader opened, when they open it, and thrown away when they close it — + never once per record and never stored. `&self` rather than `&mut self` is + the contract: `read` runs far ahead of the viewport, so whatever state a + dialect carried across records is long past by the time a key is pressed. + A dialect that cannot answer from *this record alone* returns what it can + and leaves the rest `None`; the raw tree is one `r` away, and a gap is + better than a guess. 2. **One entry in `lens::LENSES`** — `(NAME, || Box::new(Mine::default()))`. 3. **Tests beside it** (`src/lens/_tests.rs`), with **hand-written fixtures**. Real session logs are private; read one to learn the shape, never copy it into the repository. -A `Summary` is five fields and no styling: +A `Summary` is seven fields and no styling: | Field | Meaning | | --- | --- | @@ -114,14 +406,70 @@ A `Summary` is five fields and no styling: | `who` | `User` / `Assistant` / `Tool` / `System` — colour only | | `actor` | the text in the speaker column: `user`, `tool`, `↳ assistant` | | `time` | `HH:MM`, or `None`; `lens::clock` does ISO-8601 | -| `what` | one line: `lens::excerpt` collapses whitespace and cuts to width | +| `what` | one line: `lens::excerpt` collapses whitespace and cuts to width. What `--toc` prints, and what the row paints for a record with no `body` | | `calls` | tool calls in this record, for a run's `· 4 tool calls` | +| `body` | the record's own text, for the rows under the summary: what was said on a message, what it was thinking on a step. `None` when it has none. `class` decides how it is painted — a message's body is one wrap split between its row and the rows under it; a step's row keeps saying what it *did* and its text goes wholly underneath | + +A `Part` is two variants and no styling either: + +| Variant | Fields | What it is | +| --- | --- | --- | +| `Text` | `label: &'static str`, `body: Body` | a named stretch of text the row's own body is not already showing: a thought beside a message, a second text block. Shown **whole** — parts are the open level, and the open level is the whole of that text | +| `Call` | `tool`, `arg`, `args: Vec<(String, Body)>`, `result: Option` | a call to a tool: what was called, the one argument a headline shows, every argument, and what came back | + +`Part::opens` is what the call row's marker is painted from, and it is false for +a call with no arguments *and* no result — the row is still there, saying what +was called; it just does not advertise a fold it has not got. -What a dialect **never** touches: rows, folding, row arithmetic, search, -yanking, the outline or the status bar. Grouping is -`src/source/jsonl/plan.rs`, painting is `src/source/jsonl/lensrow.rs`, and both -are dialect-agnostic. If a new dialect needs a change there, that change is -about *all* lenses and belongs in the seam. +There is **one** reading of an argument list, `lens::part::args_of`, and both +dialects use it. An object is its members; anything else — an array, a bare or +half-written `arguments` string — is one entry named `arguments` holding what +the file said. Dropping those was the seam's one silent clip: the open level +said the call had no arguments while the command sat in the tree one `r` away. +Absent, `null` and `[]` still mean the same thing, and mean it quietly. + +`result` is an `Option` because of the `agent` dialect and not in spite of it: +its calls are answered by a **later record**, and a `Body`'s path starts at the +record it belongs to. The call part carries `None` and the result record +contributes a part of its own. A dialect whose answers are in the same record — +`atif` — fills it in. + +Every stretch of text in a `Part` is a `Body`, on purpose. One record can carry +five calls whose outputs are fifteen kilobytes apiece, and a level that read +them into memory to count its rows would make opening one step cost what the +step cost. Two of those paths are real and one is not, and the code says which: +a **result** is one string node of the record (`observation.results[3].content`) +and opens whole; an **argument** sits under a key that is its own name, and a +`Step::Key` is `&'static str`, so it is a head with no path — clipped, with the +row under it stating the true remainder, and whole in the tree one `r` away. + +A `Summary` is kept for **every** classified record, and nothing here may +allocate per document — so a `Body` is *not* the message. It is the first +`lens::BODY_KEEP` bytes of it, the whole message's byte and line counts, and the +path back to the text inside the record (`message`, or +`message.content[1].text`). The clip and the row arithmetic are answered from +the head alone, so a resize reads no file; opening a message longer than the +head reads it back out of the record that is being painted anyway. Holding whole +messages instead would make a long log's summaries as big as the log. + + +What a dialect **never** touches: rows, folding, row arithmetic, the levels, the +key that walks them, search, yanking, the outline or the status bar — including +how tall a message is, which is `src/source/record/body.rs`'s answer and depends +on the width. A dialect says what a record *is* and what its parts *are*; where +those land on a screen is not its business. Grouping is +`src/source/record/plan.rs` with the levels and heights in +`src/source/record/plan_rows.rs`, painting is `src/source/record/lensrow.rs` and +`src/source/record/parts.rs`, the ladder is `src/source/record/ladder.rs`, the +fold keys are `src/source/record/ops.rs`, and all of them are dialect-agnostic — +and format-agnostic with it. There is one record source +(`src/source/record/source.rs`), and what a record *format* provides is the +`Store` trait: how many records the index has found, how to push it along +inside a byte budget, and record `i` as bytes or as a value. +`src/source/jsonl/` is a record per line over the CSV line index; +`src/source/jsonarray/` is an array inside a document over the JSON structural +index. Neither holds a row number. If a new dialect needs a change there, that +change is about *all* lenses and belongs in the seam. ## What a lens costs @@ -131,7 +479,46 @@ the viewport never move; rows below shift as grouping catches up, exactly as `len()` grows while the index scans. `G` waits for the lens the way it waits for the index, reporting progress rather than jumping to a wrong end. +What a **body** costs is bounded per record and stated in the code: a `Summary` +keeps at most `lens::BODY_KEEP` (1 KB) of a message, whatever its size, plus the +path back to the rest. Since a step's reasoning is a body too, a trajectory that +records thinking pays that ceiling on those records as well: on a synthetic +45 MB, 20 000-step ATIF file with a 420-byte thought on every other step, that +is +4 MB of resident memory against the same file with the thoughts removed — +the cost tracks the *number* of records that thought, not what they thought. +Open time is unmoved (2 ms either way), because none of it is read until the +record is classified. + +What the **open level** costs is nothing at all until it is asked for. +`Lens::detail` runs for the record being measured or painted and its answer is +dropped; a document holds one `Under` — two `usize` — per classified record and +not one byte of a tool's output. The one keystroke that pays is `zR`, which puts +every record it reached at that level and so asks every dialect about every +record: 215 ms on that 45 MB file, against 24 ms before there was a level to +open. `zR` was already the keystroke that scales with the file, and this is the +same trade in the same place. On a synthetic 4 MB / 8 MB ATIF trajectory that is ++1.6 MB / +3.2 MB of resident memory against the same file before bodies, with +open time unchanged (≈2 ms) — the cost tracks the number of *messages*, not +what was said in them. `zR` is the one keystroke that now scales with the file: +it shows every message the viewport has reached in full, which means wrapping +them (58 ms on 4 MB, 131 ms on 8 MB). A resize that *changes the width* wraps +too — the clip, which is six rows out of the head — one of them the summary row — +with no file read and no reclassification, or the whole of every message once `zR` has opened them, which +costs what `zR` costs. A resize that does not change the width — a height-only +one, which is most of what dragging a window's edge produces — costs nothing: +`RecordSource::remeasure` re-lays bodies only when `Plan::set_width` says the +width moved, and the pager calls `set_width` on every terminal size change. + Measured on a real 4.0 MB, 2354-record session (release build, 140 columns): open plus the first screen **25–30 ms**, reading the whole file through the lens ~65 ms, painting the last screen under 1 ms. 2354 of 2354 records reachable, folded into 633 rows. + +A **document** lens pays one thing more, and SPEC.md §Lenses states it: the +records are one member of the document, and the structural index only knows a +member once it has walked past its last byte, so the first row waits on a byte +walk of the file. No record is parsed for it and nothing is held in memory — +the wait is a scan, reported as `≥N (indexing P%)` — but it is O(file) where a +record per line is O(screen). `--toc` and `--plain` are batches and wait for it: +they spend slice after slice until the index has what they asked for, because a +truncated list that exits 0 reads as an empty file. diff --git a/src/cli.rs b/src/cli.rs index aa8dc46..d446dc6 100644 --- a/src/cli.rs +++ b/src/cli.rs @@ -118,8 +118,9 @@ OPTIONS -V, --version Show the version and exit. KEYS - j/k d/u space/b g/G scroll h/l scroll horizontally - za Enter zM zR collapse Tab/S-Tab next/prev heading + j/k one row h/l scroll horizontally + d/u space/b g/G scroll Tab/S-Tab next/prev heading, or + za Enter zM zR collapse block under a lens n \u{2190}/\u{2192} pick a link Enter follow / open Backspace back o i outline/index / ? search @@ -142,8 +143,14 @@ LENSES --lens turns a .jsonl trajectory back into the conversation it recorded: messages stay on screen, and runs of tool calls and their results fold into one row — \u{27e8}6 steps \u{b7} 4 tool calls\u{27e9} — that opens with Enter or za. - A record the lens does not recognise renders as the generic tree, whole: - a lens adds interpretation and never hides data. + Under a row is what was said, or what the step was thinking, clipped to + six lines that say how many they hid. Enter or za toggles the record's two + levels: the whole of that text with the record's tool calls listed as + calls, and back to the clip. Enter on a call row shows the arguments it + was made with and the output it returned. r shows the raw record from + either level, and shuts it again. A record the lens does not recognise + renders as the generic tree, whole: a lens adds interpretation and never + hides data. The mouse is never captured, so terminal-native drag-select always works. " diff --git a/src/lens/agent.rs b/src/lens/agent.rs index 0661cf6..7080e59 100644 --- a/src/lens/agent.rs +++ b/src/lens/agent.rs @@ -35,7 +35,7 @@ //! renders as the generic tree with nothing hidden. #![deny(unsafe_code)] -use super::{excerpt, record_clock, record_type, Class, Lens, Summary, Who, ARG, EXCERPT}; +use super::{excerpt, record_clock, record_type, Body, Class, Lens, Part, Step, Summary, Who, ARG, EXCERPT}; use crate::json::Value; pub const NAME: &str = "agent"; @@ -52,6 +52,11 @@ const RECENT_CALLS: usize = 64; struct Blocks { /// Text blocks — the only thing that makes a record a message. spoken: Vec, + /// What goes **under** the row: the first text block that says something, + /// or, on a record that only thought, the thought. [`body_block`] is the + /// one definition of which, so [`Lens::detail`] can skip exactly that block + /// rather than showing the reader a paragraph twice. + body: Option, calls: Vec, results: Vec, thoughts: usize, @@ -82,6 +87,48 @@ impl Lens for Agent { }; Some(sum.at(time)) } + + /// A record's content blocks, read as what they are. + /// + /// # What this dialect can say, and what it cannot + /// + /// A `tool_use` block carries its whole `input`, so a call's arguments are + /// here in full. **Its result is not**: the API answers a call in a *later* + /// `user` record, and a [`Body`]'s path starts at the record it belongs to + /// — so the call part carries `result: None` and the result record + /// contributes a part of its own. That is the honest shape for a transcript + /// where the two halves are two records, and it is why `Part::Call::result` + /// is an `Option` rather than a `Body`. + /// + /// The tool a result belongs to is looked up in the same ring `read` used, + /// and that ring is [`RECENT_CALLS`] deep while classification runs far + /// ahead of the viewport — so by the time a reader opens an old record the + /// call is long evicted and the part is labelled `result`. Widening the ring + /// would be a per-document allocation, which this seam does not do; the + /// tool's name is on the row above either way, where `read` put it. + fn detail(&self, v: &Value) -> Vec { + let Some(blocks) = v.get("message").and_then(|m| m.get("content")).and_then(|c| c.as_array()) + else { + return Vec::new(); + }; + let mut out: Vec = Vec::new(); + // The block the row's own body is already showing, skipped here so the + // reader is not shown the same paragraph twice. + let said = body_block(blocks); + for (i, b) in blocks.iter().enumerate() { + if said == Some(i) { + continue; + } + match b.get("type").and_then(|t| t.as_str()).unwrap_or("") { + "text" => out.extend(text_part(b, "text", "message", text_at(i))), + "thinking" => out.extend(text_part(b, "thinking", "thinking", thinking_at(i))), + "tool_use" => out.push(call_part(b)), + "tool_result" => out.push(self.result_part(b, i)), + _ => {} + } + } + out + } } impl Agent { @@ -96,7 +143,9 @@ impl Agent { }; // A prompt typed by a person: `content` is a bare string. if let Some(text) = msg.get("content").and_then(|c| c.as_str()) { - return Some(said(who, actor, excerpt(text, EXCERPT))); + let at = vec![Step::Key("message"), Step::Key("content")]; + let body = Body::new(text, at); + return Some(said(who, actor, excerpt(text, EXCERPT), Some(body))); } let blocks = msg.get("content")?.as_array()?; Some(self.blocks_row(blocks, who, actor)) @@ -104,7 +153,8 @@ impl Agent { /// The block array of one message, read into one row. fn blocks_row(&mut self, blocks: &[Value], who: Who, actor: String) -> Summary { - let seen = self.scan(blocks); + let mut seen = self.scan(blocks); + let body = seen.body.take(); // Anything a person would read keeps the record on screen; everything // else is a step, however many blocks it took. if !seen.spoken.is_empty() { @@ -113,7 +163,7 @@ impl Agent { what.push_str(" \u{b7} "); what.push_str(&seen.calls.join(" \u{b7} ")); } - return Summary { calls: seen.calls.len(), ..said(who, actor, what) }; + return Summary { calls: seen.calls.len(), ..said(who, actor, what, body) }; } let what = match (seen.calls.is_empty(), seen.results.is_empty()) { (false, _) => seen.calls.join(" \u{b7} "), @@ -127,7 +177,9 @@ impl Agent { true => (who, actor), false => (Who::Tool, "tool".to_string()), }; - Summary { class: Class::Step, who, actor, time: None, what, calls: seen.calls.len() } + // A step's row stays one line — what it *did* — and whatever it was + // thinking goes under it. Reasoning is text, so it is shown. + Summary { class: Class::Step, who, actor, time: None, what, calls: seen.calls.len(), body } } /// Read the blocks of one message into the four things a row can say. @@ -148,6 +200,9 @@ impl Agent { _ => {} } } + // One rule for what goes under the row, read off the blocks themselves, + // so [`Lens::detail`] can skip exactly the block this took. + seen.body = body_block(blocks).and_then(|i| block_body(blocks.get(i)?, i)); seen } @@ -174,6 +229,27 @@ impl Agent { .map(|(_, n)| n.as_str()) } + /// A `tool_result` block as the half of a call this record *does* hold: the + /// answer, named after the call it answers when the ring still remembers it + /// and `result` when it does not. + fn result_part(&self, block: &Value, i: usize) -> Part { + let tool = block + .get("tool_use_id") + .and_then(|i| i.as_str()) + .and_then(|id| self.tool_of(id)) + .unwrap_or("result"); + let failed = block.get("is_error").and_then(|e| e.as_bool()).unwrap_or(false); + Part::Call { + tool: tool.to_string(), + arg: match failed { + true => "error".to_string(), + false => String::new(), + }, + args: Vec::new(), + result: result_body(block.get("content"), i), + } + } + /// `Bash → 42 lines`, or `Bash → error`. fn result_text(&self, block: &Value) -> String { let name = block @@ -193,7 +269,7 @@ impl Agent { } /// A record someone would read. -fn said(who: Who, actor: String, what: String) -> Summary { +fn said(who: Who, actor: String, what: String, body: Option) -> Summary { Summary { class: Class::Message, who, @@ -201,6 +277,7 @@ fn said(who: Who, actor: String, what: String) -> Summary { time: None, what, calls: 0, + body, } } @@ -250,7 +327,7 @@ const ARG_KEYS: [&str; 10] = [ "skill", ]; -fn tool_arg(input: &Value) -> Option { +pub(super) fn tool_arg(input: &Value) -> Option { for key in ARG_KEYS { if let Some(text) = input.get(key).and_then(|v| v.as_str()) { let cut = excerpt(text, ARG); @@ -336,6 +413,14 @@ fn field(v: &Value, key: &str) -> Option { Some(excerpt(v.get(key)?.as_str()?, ARG)) } +/// The blocks of one record read as [`Part`]s — a child module so the two +/// halves of this dialect stay under the size limit and share one set of +/// imports. +#[path = "agent_parts.rs"] +mod parts; + +use parts::{block_body, body_block, call_part, result_body, text_at, text_part, thinking_at}; + #[cfg(test)] #[path = "agent_tests.rs"] mod tests; diff --git a/src/lens/agent_parts.rs b/src/lens/agent_parts.rs new file mode 100644 index 0000000..b6b307c --- /dev/null +++ b/src/lens/agent_parts.rs @@ -0,0 +1,141 @@ +//! The content blocks of one `agent` record, read as [`Part`]s. +//! +//! The half of this dialect that answers [`Lens::detail`](super::Lens::detail): +//! what a record's blocks *are*, once you stop reading them as JSON. Split from +//! `agent.rs` so both stay under the size limit; nothing here carries state, and +//! every path it builds starts at the record it was given. +#![deny(unsafe_code)] + +use super::super::{part, Body, Part, Step}; +use super::tool_arg; +use crate::json::Value; + +/// Which content block goes **under** the record's row as its [`Body`]. +/// +/// What someone would read first: the first text block that says something, +/// and failing that the first thought that does. A record whose only content is +/// thinking is a step, and its row says `thinking` — reasoning is text, so the +/// text goes under the row rather than being summarised out of existence. +/// +/// One definition, used by [`Agent::scan`] to build the body and by +/// [`Lens::detail`] to skip it. Two would paint a paragraph twice. +pub(super) fn body_block(blocks: &[Value]) -> Option { + let says = |b: &Value, key: &str| { + b.get("type").and_then(|t| t.as_str()) == Some(key) + && b.get(key).and_then(|t| t.as_str()).is_some_and(|t| !t.trim().is_empty()) + }; + if let Some(i) = blocks.iter().position(|b| says(b, "text")) { + return Some(i); + } + blocks.iter().position(|b| says(b, "thinking")) +} + +/// Block `i` as a body, whichever of the two text-bearing kinds it is. +pub(super) fn block_body(block: &Value, i: usize) -> Option { + let kind = block.get("type").and_then(|t| t.as_str())?; + let text = block.get(kind)?.as_str()?; + let at = match kind { + "text" => text_at(i), + _ => thinking_at(i), + }; + Some(Body::new(text, at)) +} + +/// A text-bearing block as a [`Part`], when it says anything at all. +pub(super) fn text_part(block: &Value, key: &str, label: &'static str, at: Vec) -> Option { + let text = block.get(key)?.as_str()?; + match text.trim().is_empty() { + true => None, + false => Some(Part::Text { label, body: Body::new(text, at) }), + } +} + +/// A `tool_use` block: the call, with every argument it was given. +/// +/// `result` is `None` and stays `None` — the answer is a later record, which is +/// this dialect's shape and not something a `Body` can address. +pub(super) fn call_part(block: &Value) -> Part { + let tool = block.get("name").and_then(|n| n.as_str()).unwrap_or("tool"); + let input = block.get("input"); + Part::Call { + tool: tool.to_string(), + arg: input.and_then(tool_arg).unwrap_or_default(), + args: input.map(input_args).unwrap_or_default(), + result: None, + } +} + +/// Every argument of a call — [`part::args_of`], exactly as `atif` reads them, +/// so the two dialects cannot disagree about what an argument list is. An +/// `input` that is not an object keeps what it said under one row rather than +/// being dropped. +pub(super) fn input_args(input: &Value) -> Vec<(String, Body)> { + part::args_of(input) +} + +/// Where block `i`'s text sits inside its record, so the whole of it can be +/// read back when the reader opens the body. +pub(super) fn text_at(i: usize) -> Vec { + vec![ + Step::Key("message"), + Step::Key("content"), + Step::At(i), + Step::Key("text"), + ] +} + +/// What a `tool_result` block returned, as a body. +/// +/// Two shapes, and only one of them has a path. A `content` that is a string is +/// one node of this record (`message.content[i].content`) and opens whole. The +/// block-array form is several nodes joined with a newline, and no path +/// addresses a join — so that one is a bounded head that states its true size, +/// which is the same promise a clipped message makes. +pub(super) fn result_body(content: Option<&Value>, i: usize) -> Option { + let content = content?; + if let Some(text) = content.as_str() { + let at = vec![ + Step::Key("message"), + Step::Key("content"), + Step::At(i), + Step::Key("content"), + ]; + return Some(Body::new(text, at)); + } + let text = joined_text(content)?; + Some(Body::new(&text, Vec::new())) +} + +/// The block-array form of a result, as one string. +/// +/// **Every** block, not only the ones with a `text` key. A result may hold an +/// image beside its text, and keeping the text alone made the join — and so the +/// `bytes` and `lines` the row states — the size of what survived rather than +/// of what came back: the clip then had nothing to report and the image was +/// gone unannounced, which is the one thing a lens may not do (SPEC.md +/// §Lenses). A block that is not text is its own JSON, which is what the +/// record's tree shows. +pub(super) fn joined_text(content: &Value) -> Option { + let items = content.as_array()?; + Some( + items + .iter() + .map(|b| match b.get("text").and_then(|t| t.as_str()) { + Some(text) => text.to_string(), + None => part::as_text(b), + }) + .collect::>() + .join("\n"), + ) +} + + +/// Where block `i`'s thinking sits inside its record. +pub(super) fn thinking_at(i: usize) -> Vec { + vec![ + Step::Key("message"), + Step::Key("content"), + Step::At(i), + Step::Key("thinking"), + ] +} diff --git a/src/lens/agent_tests.rs b/src/lens/agent_tests.rs index e5bfc4f..a5b029a 100644 --- a/src/lens/agent_tests.rs +++ b/src/lens/agent_tests.rs @@ -269,3 +269,226 @@ fn remembered_calls_are_bounded() { assert_eq!(lens.tool_of(&format!("t{last}")), Some("Bash")); assert_eq!(lens.tool_of("t0"), None); } + +/// A typed prompt and a text block both put what was said under the row, and +/// the path names where it came from so the whole of it can be read back. +#[test] +fn a_message_carries_its_text_as_a_body() { + use crate::lens::Step; + let sum = read( + r#"{"type":"user","message":{"role":"user","content":"one\ntwo\nthree"}}"#, + ) + .expect("recognised"); + let body = sum.body.expect("a body"); + assert_eq!(body.head, "one\ntwo\nthree"); + assert_eq!(body.lines, 3); + assert_eq!(body.at, vec![Step::Key("message"), Step::Key("content")]); + + // A block array: the body is the first text block, wherever it sits. + let json = r#"{"type":"assistant","message":{"role":"assistant","content":[ + {"type":"thinking","thinking":"hm"}, + {"type":"text","text":"Here is the plan."}]}}"#; + let sum = read(json).expect("recognised"); + let body = sum.body.expect("a body"); + assert_eq!(body.head, "Here is the plan."); + assert_eq!( + body.at, + vec![Step::Key("message"), Step::Key("content"), Step::At(1), Step::Key("text")] + ); + let record = crate::json::parse(json.as_bytes()).expect("fixture"); + assert_eq!(body.text_in(Some(&record)), "Here is the plan."); +} + +/// Mechanics stay one line, which is also what keeps a folded run exactly as +/// tall as the records it holds. +#[test] +fn a_step_never_carries_a_body() { + let mechanics = [ + r#"{"type":"assistant","message":{"role":"assistant","content":[ + {"type":"tool_use","id":"t1","name":"Bash","input":{"command":"ls"}}]}}"#, + r#"{"type":"user","message":{"role":"user","content":[ + {"type":"tool_result","tool_use_id":"t1","content":"ok"}]}}"#, + r#"{"type":"system","subtype":"turn_duration","durationMs":1200}"#, + r#"{"type":"mode","mode":"plan"}"#, + ]; + for json in mechanics { + let sum = read(json).expect("recognised"); + assert_eq!(sum.class, Class::Step, "{json}"); + assert!(sum.body.is_none(), "{json}"); + } +} + +// -- the open level, and what this dialect can and cannot say about it ----------- + +fn detail(json: &str) -> Vec { + Agent::default().detail(&crate::json::parse(json.as_bytes()).expect("fixture parses")) +} + +/// The seam is not an ATIF special case: a Claude Code record's blocks are the +/// same two things — text, and calls to tools. +#[test] +fn a_call_block_becomes_a_part_with_every_argument_it_was_given() { + let json = r#"{"type":"assistant","message":{"role":"assistant","content":[ + {"type":"tool_use","id":"t1","name":"Bash", + "input":{"command":"cargo test","description":"run the suite","timeout":120}}]}}"#; + match detail(json).first().expect("a part") { + Part::Call { tool, arg, args, result } => { + assert_eq!(tool, "Bash"); + assert_eq!(arg, "cargo test"); + let names: Vec<&str> = args.iter().map(|(k, _)| k.as_str()).collect(); + assert_eq!(names, vec!["command", "description", "timeout"]); + assert_eq!(args[2].1.head, "120"); + assert!( + result.is_none(), + "the answer is a later record, and a path starts at the record it is in" + ); + } + other => panic!("{other:?}"), + } +} + +/// The other half of that pair: the result record carries the answer, and names +/// the call when the ring still remembers it. +#[test] +fn a_result_record_carries_the_answer_and_names_its_call_while_it_can() { + let mut lens = Agent::default(); + let call = r#"{"type":"assistant","message":{"role":"assistant","content":[ + {"type":"tool_use","id":"t1","name":"Bash","input":{"command":"ls"}}]}}"#; + let result = r#"{"type":"user","message":{"role":"user","content":[ + {"type":"tool_result","tool_use_id":"t1","content":"a\nb\nc"}]}}"#; + lens.read(&crate::json::parse(call.as_bytes()).expect("fixture")); + let v = crate::json::parse(result.as_bytes()).expect("fixture"); + match lens.detail(&v).first().expect("a part") { + Part::Call { tool, result, .. } => { + assert_eq!(tool, "Bash", "named after the call it answers"); + let body = result.as_ref().expect("the output"); + assert_eq!(body.lines, 3); + assert_eq!( + body.at, + vec![ + Step::Key("message"), + Step::Key("content"), + Step::At(0), + Step::Key("content") + ], + "a string result is one node of this record" + ); + } + other => panic!("{other:?}"), + } + // A lens that has forgotten the call still shows the answer, and says + // `result` rather than guessing a tool. + match detail(result).first().expect("a part") { + Part::Call { tool, result, .. } => { + assert_eq!(tool, "result"); + assert!(result.is_some(), "the output is never lost with the name"); + } + other => panic!("{other:?}"), + } +} + +/// The block-array form of a result is several nodes joined, and no path +/// addresses a join — so it is a bounded head that states its true size. Honest +/// rather than whole, and never silent. +#[test] +fn a_block_array_result_is_a_head_that_knows_its_size() { + let json = r#"{"type":"user","message":{"role":"user","content":[ + {"type":"tool_result","tool_use_id":"t9","content":[ + {"type":"text","text":"one"},{"type":"text","text":"two"}]}]}}"#; + match detail(json).first().expect("a part") { + Part::Call { result, .. } => { + let body = result.as_ref().expect("the output"); + assert_eq!(body.head, "one\ntwo"); + assert!(body.at.is_empty(), "there is no single node to point at"); + assert!(body.whole(), "and this one fits, so nothing is claimed to be missing"); + } + other => panic!("{other:?}"), + } +} + +/// **Every** block of that array, not only the ones with a `text` key. +/// +/// Keeping the text alone made the join — and so the `bytes` and `lines` the +/// call row states — the size of what survived rather than of what came back: +/// the clip then had nothing to report and an image beside the text was gone, +/// unannounced. That is the silent case SPEC.md §Lenses forbids. +#[test] +fn a_block_that_is_not_text_is_still_in_the_result() { + let json = r#"{"type":"user","message":{"role":"user","content":[ + {"type":"tool_result","tool_use_id":"t1","content":[ + {"type":"text","text":"line one"}, + {"type":"image","source":{"type":"base64","data":"AAAABBBBCCCC"}}, + {"type":"text","text":"line two"}]}]}}"#; + match detail(json).first().expect("a part") { + Part::Call { result, .. } => { + let body = result.as_ref().expect("the output"); + assert_eq!(body.lines, 3, "three blocks, three lines: {:?}", body.head); + assert!(body.head.contains("AAAABBBBCCCC"), "the image is there: {:?}", body.head); + assert_eq!(body.bytes, body.head.len(), "and the size is of all of it"); + } + other => panic!("{other:?}"), + } +} + +/// An `input` that is not an object keeps what it said, exactly as `atif`'s +/// `arguments` does — one list-reading, so the two dialects cannot disagree. +#[test] +fn an_input_that_is_not_an_object_is_shown_rather_than_dropped() { + let json = r#"{"type":"assistant","message":{"role":"assistant","content":[ + {"type":"tool_use","id":"t1","name":"Bash","input":["cargo","test"]}]}}"#; + match detail(json).first().expect("a part") { + Part::Call { args, .. } => { + assert_eq!(args.len(), 1, "{args:?}"); + assert_eq!(args[0].0, crate::lens::part::RAW_ARGS); + assert_eq!(args[0].1.head, r#"["cargo","test"]"#); + } + other => panic!("{other:?}"), + } +} + +/// Thinking is text. A record whose only content is a thought puts it under the +/// row, exactly as a message goes under a message's row. +#[test] +fn a_thinking_only_record_shows_what_it_was_thinking() { + let json = r#"{"type":"assistant","message":{"role":"assistant","content":[ + {"type":"thinking","thinking":"the fixture moved\nso the path is what fails","signature":"x"}]}}"#; + let sum = read(json).expect("recognised"); + assert_eq!(sum.class, Class::Step); + assert_eq!(sum.what, "thinking"); + let body = sum.body.expect("the thought is under the row"); + assert_eq!(body.lines, 2); + assert!(!detail(json).iter().any(|p| matches!(p, Part::Text { .. })), "not twice"); +} + +/// Beside a message the thought has nowhere else to go, and becomes a part. +/// So does a second text block, which the row could only ever excerpt. +#[test] +fn text_a_row_could_not_hold_becomes_a_part() { + let json = r#"{"type":"assistant","message":{"role":"assistant","content":[ + {"type":"text","text":"first"}, + {"type":"thinking","thinking":"a thought","signature":"x"}, + {"type":"text","text":"second"}]}}"#; + let sum = read(json).expect("recognised"); + assert_eq!(sum.body.expect("the first text block").head, "first"); + let parts = detail(json); + let labels: Vec<&str> = parts + .iter() + .filter_map(|p| match p { + Part::Text { label, .. } => Some(*label), + _ => None, + }) + .collect(); + assert_eq!(labels, vec!["thinking", "message"], "in the record's own order"); + match &parts[1] { + Part::Text { body, .. } => assert_eq!(body.head, "second"), + other => panic!("{other:?}"), + } +} + +/// A record with no blocks at all — a typed prompt, a bookkeeping row — has no +/// parts, and the ladder there is one rung shorter. +#[test] +fn a_record_with_no_blocks_has_no_parts() { + assert!(detail(r#"{"type":"user","message":{"role":"user","content":"just typing"}}"#).is_empty()); + assert!(detail(r#"{"type":"mode","mode":"plan"}"#).is_empty()); +} diff --git a/src/lens/atif.rs b/src/lens/atif.rs new file mode 100644 index 0000000..5f5c413 --- /dev/null +++ b/src/lens/atif.rs @@ -0,0 +1,495 @@ +//! The `atif` lens: agent trajectories in the ATIF interchange format. +//! +//! # The dialect, as found in a real ATIF-v1.7 trajectory +//! +//! One JSON *document*, not a record per line: `schema_version`, `session_id` +//! and `agent` describe the run, and the records are the elements of `steps`. +//! That is what [`Lens::records_at`] declares, and it is the only thing about +//! this dialect that is not a `read` of one record — the document half is +//! [`crate::source::jsonarray`], which knows nothing about ATIF. +//! +//! A step: +//! +//! | key | what it is | +//! | --- | --- | +//! | `step_id` | an integer, ascending. Present on every step; the recogniser | +//! | `source` | `user` or `agent` | +//! | `timestamp` | ISO-8601 with a numeric offset (`+00:00`), and **absent on the opening step** | +//! | `message` | what was said. Present on every step and *empty* on most of them | +//! | `reasoning_content` | the model's thinking, when it recorded any | +//! | `tool_calls[]` | `function_name`, `arguments` (an object), `tool_call_id` | +//! | `observation.results[]` | `source_call_id`, `content` — matched back to the call it answers | +//! | `model_name`, `metrics`, `llm_call_count` | bookkeeping, left to the opened tree | +//! +//! # The one decision that shapes the document +//! +//! **A step whose `message` says something is conversation**; everything else +//! is mechanics. So a step with a message is a [`Class::Message`] and stays on +//! screen with its tool calls collapsed to a count on its row, and a step with +//! an empty message is a [`Class::Step`] that folds into a run with its +//! neighbours. Nothing is hidden either way: every row opens into the step's +//! own tree, whole. +//! +//! Nothing here assumes any field is present. `null`, `[]` and absent all mean +//! the same thing, `arguments` is tolerated as an object *or* as the +//! JSON-encoded string the OpenAI-shaped wire format emits, and a step this +//! does not recognise returns `None` and renders as the generic tree. +#![deny(unsafe_code)] + +use super::{ + excerpt, part, record_clock, Body, Class, Lens, Part, RecordsAt, Step, Summary, Who, ARG, + EXCERPT, +}; +use crate::json::Value; + +pub const NAME: &str = "atif"; +pub const ABOUT: &str = "ATIF agent trajectories: the conversation, mechanics folded away"; + +/// The top-level key whose array holds the records. +pub const STEPS: &str = "steps"; + +/// The argument that says what a call *did*, by the name ATIF tools give it. +/// +/// Ordered, not guessed: a `glob` call carries both `path` and `pattern`, and +/// the pattern is what a reader scans for. A tool whose arguments name none of +/// these shows its own name alone rather than an arbitrary field. +const ARG_KEYS: [&str; 5] = ["command", "filePath", "pattern", "query", "url"]; + +#[derive(Default)] +pub struct Atif; + +impl Lens for Atif { + fn name(&self) -> &'static str { + NAME + } + + fn about(&self) -> &'static str { + ABOUT + } + + /// The records are `steps` inside one document — everything else at the top + /// level is record 0 and is never hidden (SPEC.md §Lenses). + fn records_at(&self) -> RecordsAt { + RecordsAt::Member(STEPS) + } + + fn read(&mut self, v: &Value) -> Option { + match v.get("step_id") { + Some(_) => step(v), + None => session(v), + } + } + + /// A step's parts: its thinking when the row's body is the message rather + /// than the thought, then one part per tool call with the result it was + /// answered by. + /// + /// Everything here is **inside this record** — ATIF matches a result to its + /// call by `source_call_id` within the step — so every path resolves and + /// nothing is guessed. The session record (record 0) has no parts: it is an + /// envelope, and `Enter` on it opens the envelope. + fn detail(&self, v: &Value) -> Vec { + if v.get("step_id").is_none() { + return Vec::new(); + } + let mut out: Vec = Vec::new(); + // The thought is a part only when it is not already the body under the + // row: a step that said nothing shows its thinking there, and painting + // it twice would be the one thing a reader cannot un-see. + if !thought_is_body(v) { + if let Some(body) = thought_body(v) { + out.push(Part::Text { label: "thinking", body }); + } + } + let mut results = results_of(v); + for call in calls_of(v) { + let result = take_result(&mut results, call.id.as_deref()).and_then(|at| result_body(v, at)); + let args = args_of(v, &call); + out.push(Part::Call { + tool: call.name, + arg: call.arg.unwrap_or_default(), + args, + result, + }); + } + // A result no call claimed is still a result. It gets a part of its own + // rather than being counted and dropped, because at this level there is + // room to show it (SPEC.md §Lenses: every byte stays reachable). + for (_, at) in results { + out.push(Part::Call { + tool: "result".to_string(), + arg: String::new(), + args: Vec::new(), + result: result_body(v, at), + }); + } + out + } +} + +/// Record 0: the document's own keys, which the source keeps precisely so this +/// row can exist. `None` when they are not an ATIF envelope, which sends the +/// record to the generic tree with nothing lost. +fn session(v: &Value) -> Option { + let schema = text(v, "schema_version"); + let id = text(v, "session_id"); + if schema.is_none() && id.is_none() { + return None; + } + let agent = v.get("agent"); + let mut parts: Vec = Vec::new(); + parts.extend(schema); + if let Some(a) = agent { + let name = text(a, "name").unwrap_or_else(|| "agent".to_string()); + parts.push(match text(a, "version") { + Some(ver) => format!("{name} {ver}"), + None => name, + }); + parts.extend(text(a, "model_name")); + } + parts.extend(id); + Some(Summary { + class: Class::Message, + who: Who::System, + actor: "session".to_string(), + time: None, + what: parts.join(" \u{b7} "), + calls: 0, + // The envelope is a headline over a tree, not something anyone said: + // its keys are the row, and `Enter` opens them. + body: None, + }) +} + +/// One step of the trajectory. +fn step(v: &Value) -> Option { + let source = v.get("source").and_then(|s| s.as_str()).unwrap_or("agent"); + let time = record_clock(v); + let said = v.get("message").and_then(|m| m.as_str()).unwrap_or(""); + let calls = calls_of(v); + let thinking = v + .get("reasoning_content") + .and_then(|r| r.as_str()) + .is_some_and(|r| !r.trim().is_empty()); + let sum = match said.trim().is_empty() { + false => spoken(source, said, thinking, calls.len()), + true => worked(source, v, thinking, calls), + }; + Some(sum.at(time)) +} + +/// A step that says something: the message, and a count of what it did +/// alongside. The calls are deliberately *not* rows — a message is the thing a +/// reader came for, and its mechanics are one clause on the end of it. +/// +/// The message itself goes under the row as the [`Body`], read straight back +/// out of `message` when the reader opens it. +fn spoken(source: &str, said: &str, thinking: bool, calls: usize) -> Summary { + let (who, actor) = speaker(source); + let mut what = excerpt(said, EXCERPT); + if thinking { + what.push_str(" \u{b7} thinking"); + } + if calls > 0 { + what.push_str(&format!(" \u{b7} {}", call_count(calls))); + } + let body = Body::new(said, vec![Step::Key("message")]); + Summary { class: Class::Message, who, actor, time: None, what, calls, body: Some(body) } +} + +/// A step that only did things: the thought and the calls it made, each with +/// what came back, and adjacent repeats collapsed. +fn worked(source: &str, v: &Value, thinking: bool, calls: Vec) -> Summary { + let n = calls.len(); + let mut results = results_of(v); + let mut parts: Vec = Vec::new(); + if thinking { + parts.push("thinking".to_string()); + } + for call in &calls { + let answer = match take_result(&mut results, call.id.as_deref()) { + Some(at) => format!(" \u{2192} {}", size_of(result_content(v, at))), + None => String::new(), + }; + parts.push(format!("{}{answer}", call.text())); + } + // Results the calls in this step did not claim: an orphan `source_call_id`, + // or a step that carries answers and no calls at all. Counted rather than + // dropped — the row is a headline, but it may not lie about what is there. + if !results.is_empty() { + parts.push(result_count(results.len())); + } + if parts.is_empty() { + parts.push("(no content)".to_string()); + } + let (who, actor) = match n { + 0 => speaker(source), + _ => (Who::Tool, "tool".to_string()), + }; + Summary { + class: Class::Step, + who, + actor, + time: None, + what: collapse(parts).join(" \u{b7} "), + calls: n, + // Reasoning is text, so it is shown: a step that only thought puts the + // thought under its row, clipped, the way a message puts what was said + // there. The row keeps saying what the step *did* — the body goes + // wholly underneath rather than splitting across the two. + body: thought_body(v), + } +} + +/// The model's thinking on this step, as a body. +fn thought_body(v: &Value) -> Option { + let text = v.get("reasoning_content")?.as_str()?; + match text.trim().is_empty() { + true => None, + false => Some(Body::new(text, vec![Step::Key("reasoning_content")])), + } +} + +/// Is the thought the body under this step's row? True exactly when the step +/// said nothing else — which is the same test [`step`] makes when it chooses +/// between [`spoken`] and [`worked`], and it is made in one place because two +/// answers would paint the thinking twice or not at all. +fn thought_is_body(v: &Value) -> bool { + let said = v.get("message").and_then(|m| m.as_str()).unwrap_or(""); + said.trim().is_empty() +} + +/// Who a step is attributed to. An unknown `source` reads as the assistant +/// rather than as nothing: dropping a step to the generic tree in the middle of +/// a conversation would be worse than naming it approximately. +fn speaker(source: &str) -> (Who, String) { + match source { + "user" => (Who::User, "user".to_string()), + _ => (Who::Assistant, "assistant".to_string()), + } +} + +/// One tool call, as much of it as a row can say. +struct Call { + name: String, + arg: Option, + id: Option, + /// Which element of `tool_calls` this is — the way back to its arguments + /// when the reader opens the call. + at: usize, +} + +impl Call { + /// `bash(cargo test)`, or the bare name for a tool whose arguments name + /// nothing on [`ARG_KEYS`]. + fn text(&self) -> String { + match &self.arg { + Some(arg) => format!("{}({arg})", self.name), + None => format!("{}()", self.name), + } + } +} + +fn calls_of(v: &Value) -> Vec { + let Some(items) = v.get("tool_calls").and_then(|c| c.as_array()) else { + return Vec::new(); + }; + items + .iter() + .enumerate() + .map(|(at, c)| Call { + name: text(c, "function_name").unwrap_or_else(|| "tool".to_string()), + arg: arg_of(c), + id: text(c, "tool_call_id"), + at, + }) + .collect() +} + +/// The argument worth showing, from `arguments` — an object, or the +/// JSON-encoded string the wire format this schema descends from emits. A +/// value that is not a string (a `todos` list, a numeric `limit`) names +/// nothing, which is why the tool then shows its name alone. +fn arg_of(call: &Value) -> Option { + let raw = call.get("arguments")?; + let decoded; + let args = match raw.as_str() { + Some(text) => { + decoded = crate::json::parse(text.as_bytes()).ok()?; + &decoded + } + None => raw, + }; + for key in ARG_KEYS { + if let Some(text) = args.get(key).and_then(|v| v.as_str()) { + let cut = excerpt(text, ARG); + if !cut.is_empty() { + return Some(cut); + } + } + } + None +} + +/// `(source_call_id, which result it is)` for every result on this step. +/// +/// The *index* rather than the text, because both readings of a result start +/// from it: the row wants its size, and the open level wants a [`Body`] whose +/// path is `observation.results[at].content`. +fn results_of(v: &Value) -> Vec<(Option, usize)> { + let Some(items) = results_array(v) else { + return Vec::new(); + }; + items + .iter() + .enumerate() + .map(|(at, r)| (text(r, "source_call_id"), at)) + .collect() +} + +fn results_array(v: &Value) -> Option<&[Value]> { + v.get("observation")?.get("results")?.as_array() +} + +/// The content of result `at`, when the step has one there. +/// +/// An explicit `null` reads as absent, which is what this dialect means by it +/// everywhere else: the call was answered and the answer said nothing. Both +/// readings of a result come through here, so the row and the open level cannot +/// disagree about whether there is one. +fn result_content(v: &Value, at: usize) -> Option<&Value> { + match results_array(v)?.get(at)?.get("content")? { + Value::Null => None, + content => Some(content), + } +} + +/// The result a call is owed, removed from the list so a second call with the +/// same id cannot claim it twice. Matching is *within the step*, which is where +/// every answer sat in the trajectory this was written against; an id that +/// matches nothing is left in the list and counted rather than attached. +fn take_result(results: &mut Vec<(Option, usize)>, id: Option<&str>) -> Option { + let id = id?; + let at = results.iter().position(|(k, _)| k.as_deref() == Some(id))?; + Some(results.remove(at).1) +} + +/// Every argument of one call, in the record's own order, each value as text. +/// +/// [`part::args_of`] does the reading, so `atif` and `agent` cannot disagree +/// about what an argument list is — including the shapes the schema does not +/// promise: an `arguments` that is an array, or a JSON-encoded string streaming +/// cut in half, becomes one `arguments` row holding what the file said rather +/// than vanishing. +fn args_of(v: &Value, call: &Call) -> Vec<(String, Body)> { + match call_arguments(v, call.at) { + Some(args) => part::args_of(&args), + None => Vec::new(), + } +} + +/// The `arguments` of call `at`, decoded when the wire format wrote them as a +/// JSON-encoded string. +/// +/// Returns an owned value in that case, which is why this cannot hand back a +/// borrow into the record and why an argument's `Body` has no path: the object +/// an argument would be addressed through may not exist in the record at all. +/// +/// A string that does **not** parse is handed back as the string it is. It is +/// still what the call was made with, and a truncated `arguments` is exactly +/// the case a reader opened the level to look at. +fn call_arguments(v: &Value, at: usize) -> Option { + let raw = v.get("tool_calls")?.as_array()?.get(at)?.get("arguments")?; + match raw.as_str() { + Some(text) => Some(crate::json::parse(text.as_bytes()).unwrap_or_else(|_| raw.clone())), + None => Some(raw.clone()), + } +} + +/// What result `at` returned, as a body — path and all, because a result *is* +/// one string node of this record and opening the call reads the whole of it +/// back out of the record being painted. +fn result_body(v: &Value, at: usize) -> Option { + let content = result_content(v, at)?; + let text = match content.as_str() { + Some(s) => s.to_string(), + // A result that is not a string is shown as the JSON it is, and there + // is then no single string node to path back to. + None => return Some(Body::new(&content.to_json(), Vec::new())), + }; + Some(Body::new( + &text, + vec![ + Step::Key("observation"), + Step::Key("results"), + Step::At(at), + Step::Key("content"), + ], + )) +} + +/// How much a tool returned: `42 lines`, or the size when it is one long line. +/// +/// A `content` that is not a string is measured as the JSON it is, because that +/// is exactly what [`result_body`] shows one rung down. Saying `ok` here while +/// the call row under it said `→ 55 bytes` put two different sizes for one +/// result on the screen at once, and SPEC.md §Lenses asks for the true one. +fn size_of(content: Option<&Value>) -> String { + let Some(content) = content else { + return "ok".to_string(); + }; + let text = part::as_text(content); + if text.is_empty() { + return "empty".to_string(); + } + match text.lines().count() { + 0 | 1 => crate::source::jsonrow::size(text.len() as u64), + n => format!("{n} lines"), + } +} + +fn call_count(n: usize) -> String { + match n { + 1 => "1 tool call".to_string(), + n => format!("{n} tool calls"), + } +} + +fn result_count(n: usize) -> String { + match n { + 1 => "1 result".to_string(), + n => format!("{n} results"), + } +} + +/// Adjacent entries that read the same collapse to `bash(make) ×3`. A run of +/// identical calls is one fact, and spelling it three times pushes what +/// happened *next* off the row. +fn collapse(parts: Vec) -> Vec { + let mut out: Vec<(String, usize)> = Vec::new(); + for p in parts { + match out.last_mut() { + Some((last, n)) if *last == p => *n += 1, + _ => out.push((p, 1)), + } + } + out.into_iter() + .map(|(text, n)| match n { + 1 => text, + n => format!("{text} \u{d7}{n}"), + }) + .collect() +} + +/// A string field, when it is there and is a string. +fn text(v: &Value, key: &str) -> Option { + Some(v.get(key)?.as_str()?.to_string()) +} + +#[cfg(test)] +#[path = "atif_tests.rs"] +mod tests; + +#[cfg(test)] +#[path = "atif_parts_tests.rs"] +mod parts_tests; diff --git a/src/lens/atif_parts_tests.rs b/src/lens/atif_parts_tests.rs new file mode 100644 index 0000000..572d24a --- /dev/null +++ b/src/lens/atif_parts_tests.rs @@ -0,0 +1,277 @@ +//! The `atif` dialect's **open level**: what [`Atif::detail`] says a step's +//! parts are (SPEC.md §Lenses). +//! +//! Split from `atif_tests.rs` for the reason `agent_parts.rs` is split from +//! `agent.rs`: two halves of one dialect, each under the size limit. Every +//! fixture is written by hand — the trajectory this dialect was derived from is +//! private, and is read to learn the shape, never copied. +#![deny(unsafe_code)] + +use super::*; + +fn read(json: &str) -> Option { + Atif.read(&crate::json::parse(json.as_bytes()).expect("fixture parses")) +} + +/// A step with a message, its tool calls, and whatever else it carries. +fn step(id: u32, source: &str, message: &str, rest: &str) -> String { + format!( + r#"{{"step_id":{id},"source":"{source}","timestamp":"2026-04-23T10:55:31.123456+00:00", + "message":"{message}"{}{rest}}}"#, + if rest.is_empty() { "" } else { "," } + ) +} + +fn detail(json: &str) -> Vec { + Atif.detail(&crate::json::parse(json.as_bytes()).expect("fixture parses")) +} + +/// A step that only thought puts the thought **under its row**, the way a +/// message puts what was said there. Its row keeps saying what it did. +#[test] +fn a_step_that_only_thought_carries_the_thought_as_its_body() { + let sum = read(&step( + 7, + "agent", + "", + r#""reasoning_content":"the fixture was renamed\nso the path is what fails""#, + )) + .expect("recognised"); + assert_eq!(sum.class, Class::Step, "still mechanics"); + assert!(sum.what.starts_with("thinking"), "{:?}", sum.what); + let body = sum.body.expect("a step's reasoning is under its row"); + assert_eq!(body.lines, 2); + assert!(body.head.starts_with("the fixture was renamed"), "{:?}", body.head); + assert_eq!(body.at, vec![Step::Key("reasoning_content")], "the way back to the whole of it"); +} + +/// And it is not *also* a part, or the reader would be shown it twice. +#[test] +fn the_thought_under_a_step_is_not_repeated_as_a_part() { + let json = step(7, "agent", "", r#""reasoning_content":"one thought""#); + assert!( + !detail(&json).iter().any(|p| matches!(p, Part::Text { .. })), + "the body already says it" + ); +} + +/// On a step that *said* something the body is the message, so the thought has +/// nowhere else to go and becomes a part of the open level. +#[test] +fn a_message_that_also_thought_keeps_the_thought_as_a_part() { + let json = step(7, "agent", "here is what I found", r#""reasoning_content":"a thought""#); + let sum = read(&json).expect("recognised"); + assert_eq!(sum.class, Class::Message); + assert!(sum.body.expect("the message").head.starts_with("here is what")); + match detail(&json).first().expect("a part") { + Part::Text { label, body } => { + assert_eq!(*label, "thinking"); + assert_eq!(body.head, "a thought"); + } + other => panic!("{other:?}"), + } +} + +/// An empty or whitespace-only `reasoning_content` is not a body and not a +/// part: absent, `null` and `""` all mean the same thing here. +#[test] +fn an_empty_thought_is_no_body_at_all() { + for raw in [r#""reasoning_content":"""#, r#""reasoning_content":" ""#, r#""reasoning_content":null"#, ""] { + let json = step(7, "agent", "", raw); + assert!(read(&json).expect("recognised").body.is_none(), "{raw}"); + assert!(!detail(&json).iter().any(|p| matches!(p, Part::Text { .. })), "{raw}"); + } +} + +// -- the parts of a step --------------------------------------------------------- + +/// A call is a part: what was called, with what, and what came back — the +/// result matched to its call by `source_call_id`, within the step. +#[test] +fn a_call_becomes_a_part_with_its_arguments_and_its_result() { + let json = step( + 7, + "agent", + "", + r#""tool_calls":[{"tool_call_id":"c1","function_name":"bash", + "arguments":{"command":"cargo test","timeout":120}}], + "observation":{"results":[{"source_call_id":"c1","content":"a\nb\nc"}]}"#, + ); + match detail(&json).first().expect("a part") { + Part::Call { tool, arg, args, result } => { + assert_eq!(tool, "bash"); + assert_eq!(arg, "cargo test", "the one argument the row names"); + let names: Vec<&str> = args.iter().map(|(k, _)| k.as_str()).collect(); + assert_eq!(names, vec!["command", "timeout"], "every argument, in the record's order"); + // A number is its own JSON, which is what the tree would show. + assert_eq!(args[1].1.head, "120"); + let result = result.as_ref().expect("the answer it was given"); + assert_eq!(result.lines, 3); + assert_eq!( + result.at, + vec![ + Step::Key("observation"), + Step::Key("results"), + Step::At(0), + Step::Key("content") + ], + "a result is one node of this record, so opening it reads the whole of it" + ); + } + other => panic!("{other:?}"), + } +} + +/// An argument's value has no path — its key is its own name and a `Step::Key` +/// is static — so it is a bounded head that still states its true size. Never +/// silent (SPEC.md §Lenses). +#[test] +fn an_argument_is_a_bounded_head_that_knows_how_big_it_really_is() { + let long = "x".repeat(4000); + let json = step( + 7, + "agent", + "", + &format!(r#""tool_calls":[{{"function_name":"bash","arguments":{{"command":"{long}"}}}}]"#), + ); + match detail(&json).first().expect("a part") { + Part::Call { args, .. } => { + let body = &args[0].1; + assert_eq!(body.bytes, 4000, "it knows what it is short of"); + assert_eq!(body.head.len(), crate::lens::BODY_KEEP); + assert!(!body.whole(), "so the row under it says what it left out"); + assert!(body.at.is_empty(), "and it is honest about having no way back"); + } + other => panic!("{other:?}"), + } +} + +/// `arguments` as the JSON-encoded string the wire format emits: the same +/// parts, read out of the decoded object. +#[test] +fn arguments_written_as_a_string_are_still_arguments() { + let json = step( + 7, + "agent", + "", + r#""tool_calls":[{"function_name":"bash","arguments":"{\"command\":\"make\"}"}]"#, + ); + match detail(&json).first().expect("a part") { + Part::Call { tool, args, .. } => { + assert_eq!(tool, "bash"); + assert_eq!(args.len(), 1); + assert_eq!(args[0].0, "command"); + assert_eq!(args[0].1.head, "make"); + } + other => panic!("{other:?}"), + } +} + +/// `arguments` that is not an object at all — an array, or a JSON-encoded +/// string streaming cut in half — keeps what the file said under one row. +/// +/// Dropping it made the open level say the call had **no** arguments, which is +/// the one thing a clip may never do (SPEC.md §Lenses): the command existed and +/// nothing on the screen admitted to leaving it out. +#[test] +fn arguments_that_are_not_an_object_are_shown_rather_than_dropped() { + let one = |args: &str| { + let json = step( + 7, + "agent", + "", + &format!(r#""tool_calls":[{{"tool_call_id":"c1","function_name":"bash","arguments":{args}}}]"#), + ); + match detail(&json).into_iter().next().expect("a part") { + Part::Call { args, .. } => args, + other => panic!("{other:?}"), + } + }; + // A JSON-encoded string that does not parse is still what was sent. + let cut = one(r#""cargo test --all-features -q""#); + assert_eq!(cut.len(), 1, "{cut:?}"); + assert_eq!(cut[0].0, crate::lens::part::RAW_ARGS); + assert_eq!(cut[0].1.head, "cargo test --all-features -q"); + // An array is its own JSON, which is what the record's tree shows. + let list = one(r#"["cargo","test"]"#); + assert_eq!(list.len(), 1, "{list:?}"); + assert_eq!(list[0].1.head, r#"["cargo","test"]"#); + // And the genuinely empty shapes stay genuinely empty, so the row that + // carries them advertises no fold. + for empty in ["{}", "null", "[]"] { + assert!(one(empty).is_empty(), "{empty}"); + } +} + +/// A call with no arguments and no answer has nothing under it, and says so — +/// `Part::opens` is what the row's fold marker is painted from. +#[test] +fn a_call_with_nothing_under_it_does_not_claim_to_open() { + let bare = step(7, "agent", "", r#""tool_calls":[{"function_name":"ls","arguments":{}}]"#); + assert!(!detail(&bare).first().expect("a part").opens(), "nothing under it"); + let armed = step( + 7, + "agent", + "", + r#""tool_calls":[{"function_name":"ls","arguments":{"path":"/"}}]"#, + ); + assert!(detail(&armed).first().expect("a part").opens(), "an argument to show"); +} + +/// A result whose `content` is not a string is measured the same way on both +/// rows that state its size: the shut row's `→ …` and the call row's. +#[test] +fn a_structured_result_states_one_size_on_both_rows() { + let json = step( + 7, + "agent", + "", + r#""tool_calls":[{"tool_call_id":"c1","function_name":"read","arguments":{"filePath":"a.rs"}}], + "observation":{"results":[{"source_call_id":"c1","content":{"blocks":["one"]}}]}"#, + ); + let body = match detail(&json).into_iter().next().expect("a part") { + Part::Call { result, .. } => result.expect("the answer"), + other => panic!("{other:?}"), + }; + let what = read(&json).expect("recognised").what; + assert_eq!(what, format!("read(a.rs) \u{2192} {} bytes", body.bytes), "{what:?}"); + assert_eq!(body.head, r#"{"blocks":["one"]}"#); +} + +/// A result no call claimed gets a part of its own rather than a count: at this +/// level there is room to show it, and every byte stays reachable. +#[test] +fn an_orphan_result_is_a_part_rather_than_a_number() { + let json = step( + 7, + "agent", + "", + r#""tool_calls":[{"tool_call_id":"c1","function_name":"bash","arguments":{"command":"ls"}}], + "observation":{"results":[{"source_call_id":"nobody","content":"orphaned"}]}"#, + ); + let parts = detail(&json); + assert_eq!(parts.len(), 2, "the call, and the answer nothing asked for: {parts:?}"); + match &parts[1] { + Part::Call { tool, result, .. } => { + assert_eq!(tool, "result"); + assert_eq!(result.as_ref().expect("the content").head, "orphaned"); + } + other => panic!("{other:?}"), + } + // And the row still counts it, as it always did. + assert!(read(&json).expect("recognised").what.contains("1 result")); +} + +/// The envelope is not a step: `Enter` on record 0 opens the envelope, and +/// there is nothing between its row and that. +#[test] +fn the_session_row_has_no_parts() { + assert!(detail(r#"{"schema_version":"ATIF-v1.7","session_id":"s"}"#).is_empty()); +} + +/// A step with no calls and no thought has no parts, which is what gives it one +/// rung fewer on the ladder rather than a keystroke that does nothing. +#[test] +fn a_step_with_nothing_in_it_has_no_parts() { + assert!(detail(&step(7, "user", "just words", "")).is_empty()); +} diff --git a/src/lens/atif_tests.rs b/src/lens/atif_tests.rs new file mode 100644 index 0000000..b004e62 --- /dev/null +++ b/src/lens/atif_tests.rs @@ -0,0 +1,377 @@ +//! The `atif` dialect, against the shapes a real ATIF-v1.7 trajectory has. +//! +//! Every fixture here is written by hand. The trajectory this dialect was +//! derived from is a private file: it is read to learn the shape — which keys +//! exist, which are missing, which are empty — and never copied into the +//! repository. +#![deny(unsafe_code)] + +use super::*; + +fn read(json: &str) -> Option { + Atif.read(&crate::json::parse(json.as_bytes()).expect("fixture parses")) +} + +/// A step with a message, its tool calls, and whatever else it carries. +fn step(id: u32, source: &str, message: &str, rest: &str) -> String { + format!( + r#"{{"step_id":{id},"source":"{source}","timestamp":"2026-04-23T10:55:31.123456+00:00", + "message":"{message}"{}{rest}}}"#, + if rest.is_empty() { "" } else { "," } + ) +} + +// -- the envelope --------------------------------------------------------------- + +/// Record 0 is the document's own keys, which is the whole reason they are kept +/// (SPEC.md §Lenses: a lens never hides data). +#[test] +fn the_documents_other_keys_are_a_session_row() { + let sum = read( + r#"{"schema_version":"ATIF-v1.7","session_id":"sxs_abc", + "agent":{"name":"opencode","version":"1.2.3","model_name":"vendor/model"}}"#, + ) + .expect("recognised"); + assert_eq!((sum.class, sum.who), (Class::Message, Who::System)); + assert_eq!(sum.actor, "session"); + assert_eq!(sum.time, None); + assert_eq!(sum.what, "ATIF-v1.7 \u{b7} opencode 1.2.3 \u{b7} vendor/model \u{b7} sxs_abc"); +} + +/// A document that is not an ATIF envelope is not summarised at all — it keeps +/// its generic collapsed row rather than being described wrongly. +#[test] +fn a_stranger_at_the_top_level_is_left_to_the_generic_tree() { + assert!(read(r#"{"title":"notes","items":[1,2]}"#).is_none()); + assert!(read("[1,2,3]").is_none()); + assert!(read("42").is_none()); +} + +/// Half an envelope is still this dialect's: a trajectory written by a tool +/// that omits `agent` must not fall off the lens. +#[test] +fn a_partial_envelope_still_reads() { + let sum = read(r#"{"session_id":"sxs_1"}"#).expect("recognised"); + assert_eq!(sum.what, "sxs_1"); + let sum = read(r#"{"schema_version":"ATIF-v1.7","agent":{}}"#).expect("recognised"); + assert_eq!(sum.what, "ATIF-v1.7 \u{b7} agent"); +} + +// -- conversation --------------------------------------------------------------- + +#[test] +fn a_step_that_says_something_is_a_message() { + let sum = read(&step(2, "user", "build me a reader\\nplease", "")).expect("recognised"); + assert_eq!((sum.class, sum.who), (Class::Message, Who::User)); + assert_eq!(sum.actor, "user"); + assert_eq!(sum.time.as_deref(), Some("10:55")); + assert_eq!(sum.what, "build me a reader please"); + assert_eq!(sum.calls, 0); +} + +#[test] +fn an_agent_step_is_the_assistant() { + let sum = read(&step(3, "agent", "Here is the plan.", "")).expect("recognised"); + assert_eq!((sum.class, sum.who), (Class::Message, Who::Assistant)); + assert_eq!(sum.actor, "assistant"); +} + +/// The settled shape: a message keeps its tool calls as a **count** on its own +/// row. They are not rows and they are not a run — the message is what the +/// reader came for, and the calls are one clause on the end of it. +#[test] +fn a_message_collapses_its_calls_to_a_count() { + let sum = read(&step( + 4, + "agent", + "Checking the tree.", + r#""reasoning_content":"weighing it up", + "tool_calls":[ + {"function_name":"bash","arguments":{"command":"git status"},"tool_call_id":"c1"}, + {"function_name":"read","arguments":{"filePath":"/a/b.rs"},"tool_call_id":"c2"}]"#, + )) + .expect("recognised"); + assert_eq!(sum.class, Class::Message); + assert_eq!(sum.what, "Checking the tree. \u{b7} thinking \u{b7} 2 tool calls"); + assert_eq!(sum.calls, 2); + // One call is singular, and a message with none says nothing about them. + let one = read(&step( + 5, + "agent", + "Looking.", + r#""tool_calls":[{"function_name":"bash","arguments":{"command":"ls"}}]"#, + )) + .expect("recognised"); + assert_eq!(one.what, "Looking. \u{b7} 1 tool call"); +} + +/// A whitespace-only message is not something anyone said. +#[test] +fn a_blank_message_is_mechanics_not_speech() { + let sum = read(&step(6, "agent", " ", r#""reasoning_content":"hm""#)).expect("recognised"); + assert_eq!(sum.class, Class::Step); + assert_eq!(sum.what, "thinking"); +} + +// -- mechanics ------------------------------------------------------------------ + +/// A step with no message is a step: the call, and what came back, matched by +/// `source_call_id` within the step. +#[test] +fn a_call_names_its_argument_and_its_answer() { + let sum = read(&step( + 7, + "agent", + "", + r#""tool_calls":[{"function_name":"bash","arguments":{"command":"make -j8"}, + "tool_call_id":"c1"}], + "observation":{"results":[{"source_call_id":"c1","content":"a\nb\nc"}]}"#, + )) + .expect("recognised"); + assert_eq!((sum.class, sum.who), (Class::Step, Who::Tool)); + assert_eq!(sum.actor, "tool"); + assert_eq!(sum.what, "bash(make -j8) \u{2192} 3 lines"); + assert_eq!(sum.calls, 1); + assert_eq!(sum.time.as_deref(), Some("10:55")); +} + +/// The argument is named in order, so a call carrying both `path` and +/// `pattern` shows the pattern — the thing a reader scans for. +#[test] +fn the_argument_shown_is_the_one_that_says_what_happened() { + let one = |args: &str| { + read(&step( + 8, + "agent", + "", + &format!(r#""tool_calls":[{{"function_name":"t","arguments":{args}}}]"#), + )) + .expect("recognised") + .what + }; + assert_eq!(one(r#"{"path":"/x","pattern":"*.rs"}"#), "t(*.rs)"); + assert_eq!(one(r#"{"filePath":"/x/y","limit":40}"#), "t(/x/y)"); + assert_eq!(one(r#"{"url":"https://x/y"}"#), "t(https://x/y)"); + assert_eq!(one(r#"{"query":"who"}"#), "t(who)"); + // A tool whose arguments name none of them keeps its name, and a + // non-string value is not an argument a row can show. + assert_eq!(one(r#"{"todos":[{"text":"a"}]}"#), "t()"); + assert_eq!(one(r#"{"command":41}"#), "t()"); + assert_eq!(one("{}"), "t()"); + // The wire format this schema descends from encodes the arguments as a + // string; both spellings mean the same call. + assert_eq!(one(r#""{\"command\":\"ls -l\"}""#), "t(ls -l)"); + assert_eq!(one(r#""not json at all""#), "t()"); +} + +/// Adjacent repeats collapse. A run of the same call is one fact, and spelling +/// it three times pushes what happened next off the row. +#[test] +fn adjacent_identical_entries_collapse_to_a_count() { + let sum = read(&step( + 9, + "agent", + "", + r#""tool_calls":[ + {"function_name":"bash","arguments":{"command":"pkg-config --exists onig"}}, + {"function_name":"bash","arguments":{"command":"pkg-config --exists onig"}}, + {"function_name":"read","arguments":{"filePath":"/a"}}]"#, + )) + .expect("recognised"); + assert_eq!( + sum.what, + "bash(pkg-config --exists onig) \u{d7}2 \u{b7} read(/a)" + ); + assert_eq!(sum.calls, 3); +} + +/// Thinking rides in front of the calls it led to, on the same row. +#[test] +fn a_thought_and_the_call_it_led_to_share_a_row() { + let sum = read(&step( + 10, + "agent", + "", + r#""reasoning_content":"try pkg-config", + "tool_calls":[{"function_name":"bash","arguments":{"command":"ls"}}]"#, + )) + .expect("recognised"); + assert_eq!(sum.what, "thinking \u{b7} bash(ls)"); +} + +/// An answer nothing asked for, and a call nothing answered: both are visible +/// rather than dropped, because a headline may not lie about what is there. +#[test] +fn an_orphan_result_is_counted_and_an_unanswered_call_is_bare() { + let sum = read(&step( + 11, + "agent", + "", + r#""tool_calls":[{"function_name":"bash","arguments":{"command":"ls"}, + "tool_call_id":"c1"}], + "observation":{"results":[{"source_call_id":"zz","content":"x"}]}"#, + )) + .expect("recognised"); + assert_eq!(sum.what, "bash(ls) \u{b7} 1 result"); + // Results and no calls at all. + let only = read(&step( + 12, + "agent", + "", + r#""observation":{"results":[{"source_call_id":"a","content":"x"}, + {"source_call_id":"b","content":"y"}]}"#, + )) + .expect("recognised"); + assert_eq!(only.what, "2 results"); + assert_eq!((only.who, only.actor.as_str()), (Who::Assistant, "assistant")); +} + +/// What a result says about itself: lines when it has them, a size when it is +/// one long line, and neither invented when there is no content. +#[test] +fn a_result_says_how_much_came_back() { + let one = |content: &str| { + read(&step( + 13, + "agent", + "", + &format!( + r#""tool_calls":[{{"function_name":"t","arguments":{{"command":"c"}}, + "tool_call_id":"c1"}}], + "observation":{{"results":[{{"source_call_id":"c1","content":{content}}}]}}"# + ), + )) + .expect("recognised") + .what + }; + assert_eq!(one(r#""a\nb""#), "t(c) \u{2192} 2 lines"); + assert_eq!(one(r#""hello""#), "t(c) \u{2192} 5 bytes"); + assert_eq!(one(r#""""#), "t(c) \u{2192} empty"); + assert_eq!(one("null"), "t(c) \u{2192} ok", "null reads as absent, as it does everywhere here"); + // A structured result is measured as the JSON it is, because that is what + // the open level shows one rung down. `ok` there was not a size, and the + // two rows sit on the screen together (SPEC.md §Lenses: "the sizes a level + // states are the true ones"). + assert_eq!(one(r#"{"a":1}"#), "t(c) \u{2192} 7 bytes"); +} + +// -- what a trajectory actually throws at it ------------------------------------ + +/// Step 1 of the real trajectory carries `message`, `source` and `step_id` and +/// nothing else — no timestamp. The row must not invent one, and this is the +/// first row on the first screen, so getting it wrong is the most visible bug +/// there is. +#[test] +fn the_opening_step_has_no_clock_and_none_is_invented() { + let sum = read(r#"{"step_id":1,"source":"user","message":"do the thing"}"#).expect("read"); + assert_eq!(sum.time, None); + assert_eq!((sum.class, sum.who), (Class::Message, Who::User)); + assert_eq!(sum.what, "do the thing"); +} + +/// Absent, `null` and empty are three spellings of the same thing. The sample +/// only ever omits; nothing in the format forbids the other two. +#[test] +fn null_and_empty_mean_the_same_as_absent() { + for rest in [ + r#""tool_calls":null,"observation":null,"reasoning_content":null"#, + r#""tool_calls":[],"observation":{},"reasoning_content":"""#, + r#""tool_calls":[],"observation":{"results":[]}"#, + ] { + let sum = read(&step(14, "agent", "", rest)).expect("recognised"); + assert_eq!(sum.what, "(no content)", "{rest}"); + assert_eq!(sum.calls, 0); + } +} + +/// A timestamp that is not one, or is cut off mid-character, is no clock at all +/// — never a slice of some other field and never a panic. +#[test] +fn a_broken_timestamp_is_no_clock_rather_than_a_wrong_one() { + for ts in [r#""2026-04-23T10:€z""#, r#""nope""#, r#""""#, "12345", "null"] { + let json = format!(r#"{{"step_id":1,"source":"agent","message":"x","timestamp":{ts}}}"#); + assert_eq!(read(&json).expect("recognised").time, None, "{ts}"); + } +} + +/// A `source` nobody has seen reads as the assistant rather than dropping the +/// step out of the conversation, and a step that is not an object at all is +/// left to the generic tree. +#[test] +fn an_unknown_source_still_reads_and_a_scalar_step_does_not() { + let sum = read(r#"{"step_id":9,"source":"orchestrator","message":"hi"}"#).expect("read"); + assert_eq!((sum.who, sum.actor.as_str()), (Who::Assistant, "assistant")); + // No `step_id`, no envelope keys: not this dialect's, and nothing is + // hidden — the record keeps its generic row. + assert!(read(r#"{"note":"stray"}"#).is_none()); +} + +/// A step big enough to matter is still one line. The excerpt bounds are the +/// shared ones, so a 15KB result and a 2KB message cannot blow a row open. +#[test] +fn a_huge_step_is_still_one_line() { + let long = "x".repeat(4000); + let sum = read(&step( + 15, + "agent", + &long, + &format!(r#""tool_calls":[{{"function_name":"bash","arguments":{{"command":"{long}"}}}}]"#), + )) + .expect("recognised"); + assert!(crate::render::str_width(&sum.what) <= EXCERPT + ARG + 32, "{}", sum.what); + assert!(sum.what.contains('\u{2026}')); +} + +/// The dialect declares where its records live, and that declaration is the +/// only thing routing needs to know about it. +#[test] +fn the_dialect_says_its_records_are_the_steps_of_a_document() { + assert_eq!(Atif.records_at(), RecordsAt::Member("steps")); + assert_eq!(Atif.name(), "atif"); + assert!(super::super::exists("atif")); + assert!(super::super::list_text().contains("atif")); +} + +/// The message goes under the row whole — the row is the headline, the body is +/// what was said. A step's mechanics stay one line and never get one. +#[test] +fn a_message_carries_its_text_as_a_body_and_a_step_does_not() { + // Written as JSON escapes; what the lens sees is a two-line message. + let sum = read(&step(3, "user", "first line\\nsecond line", "")).expect("recognised"); + let body = sum.body.expect("a message has a body"); + assert_eq!(body.head, "first line\nsecond line"); + assert_eq!(body.lines, 2); + assert!(body.whole()); + assert_eq!(body.at, vec![crate::lens::Step::Key("message")]); + + let mechanics = read(&step( + 4, + "agent", + "", + r#""tool_calls":[{"function_name":"bash","arguments":{"command":"ls"}}]"#, + )) + .expect("recognised"); + assert!(mechanics.body.is_none(), "mechanics stay one line"); + // And the envelope row is a headline over keys, not something anyone said. + assert!(read(r#"{"schema_version":"ATIF-v1.7"}"#).expect("session").body.is_none()); +} + +/// A message far bigger than what the seam keeps is kept as a head plus the +/// path back to the rest: a summary is held for every record, so holding whole +/// messages would make the summaries as big as the log. +#[test] +fn a_huge_message_is_kept_as_a_head_and_a_path() { + let long = "y".repeat(40_000); + let sum = read(&step(9, "user", &long, "")).expect("recognised"); + let body = sum.body.expect("a body"); + assert_eq!(body.head.len(), crate::lens::BODY_KEEP, "a head, not the message"); + assert_eq!(body.bytes, 40_000); + assert!(!body.whole(), "the head is not the whole message"); + // And the whole of it is still one lookup away, out of the record itself. + let record = crate::json::parse(step(9, "user", &long, "").as_bytes()).expect("fixture"); + assert_eq!(body.text_in(Some(&record)).len(), 40_000); + assert_eq!(body.text_in(None).len(), crate::lens::BODY_KEEP, "no record, no more than the head"); + // What it costs is bounded by the head, not by what was said. + assert!(body.head.len() + body.at.len() * 16 < 2 * crate::lens::BODY_KEEP); +} + diff --git a/src/lens/mod.rs b/src/lens/mod.rs index ea65448..4953c06 100644 --- a/src/lens/mod.rs +++ b/src/lens/mod.rs @@ -6,13 +6,16 @@ //! never owns a [`crate::source::Source`], and never decides where a row goes //! on the screen: given one parsed record it says *what that record is* — who //! spoke, when, what happened, and whether the record is conversation or -//! mechanics. The record source ([`crate::source::jsonl`]) does the rest: it +//! mechanics. The record seam ([`crate::source::record`]) does the rest: it //! turns runs of mechanics into one foldable group row and leaves the //! conversation alone. //! //! That split is the whole point. A second dialect — opencode, OpenAI Codex, //! ATIF — is a new module here and one entry in [`LENSES`]; nothing about -//! folding, row arithmetic, search or yanking is written twice. +//! folding, row arithmetic, the levels a record has, search or yanking is +//! written twice. [`part`] is the vocabulary the second half of that answer is +//! given in: what a record's parts *are*, said without saying anything about +//! rows. //! //! # The contract //! @@ -28,10 +31,20 @@ //! whose state is wrong (a record read out of order, a truncated log) must //! still return a usable summary rather than nothing. //! * Nothing here allocates per *document*: a summary is built from one record -//! and thrown away with it. +//! and thrown away with it — and a [`Body`] keeps that true for the message +//! text as well, by holding a bounded head and *where the rest is* rather +//! than a copy of it. +//! * [`Lens::detail`] is the exception that proves it: the parts of a record — +//! its tool calls, with their arguments and what came back — are read **when +//! the reader opens that record** and are never stored. A `Summary` is per +//! record and forever; a `Vec` is per keystroke and momentary. #![deny(unsafe_code)] pub mod agent; +pub mod atif; +pub mod part; + +pub use part::Part; #[cfg(test)] #[path = "tests.rs"] @@ -75,6 +88,112 @@ impl Who { } } +/// One step of the way back to a message inside its own record: a key, or an +/// index into an array. Static keys because a dialect names them in its own +/// source (`message`, `content`, `text`) — nothing is allocated to remember +/// where a message lives. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Step { + Key(&'static str), + At(usize), +} + +/// Bytes of a message the seam keeps in memory, per record. +/// +/// The ceiling on what a summary costs: a 40 KB message contributes 1 KB and a +/// path, not 40 KB. Enough to lay a clipped body out at any width up to about +/// 190 columns, and past that the clip shows fewer rows and still says what it +/// is not showing — which is the half that may not slip. +pub const BODY_KEEP: usize = 1024; + +/// The message text under a summary row (SPEC.md §Lenses). +/// +/// # Why this is a head and a path, not the text +/// +/// A summary is kept for **every** classified record, and this module's +/// contract is that nothing here allocates per document. Holding each message +/// in full would make a long log's summaries as big as the log. So a `Body` +/// keeps: +/// +/// * `head` — the first [`BODY_KEEP`] bytes, which is what a clipped row paints +/// and what the row *arithmetic* measures, so a resize re-lays every body +/// without reading a file; +/// * `bytes` and `lines` — what the whole message is, so a clipped body can say +/// what it is not showing, and so its height at a width can be *derived* +/// rather than measured; +/// * `at` — where the text sits inside the record, so painting reads the whole +/// of it back out of the record that is being painted anyway. +/// +/// The dialect supplies text and a path and makes no layout decision: how many +/// rows this becomes is [`crate::source::record::body`]'s answer, and it +/// depends on the width. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Body { + /// The first [`BODY_KEEP`] bytes of the message, on a character boundary. + pub head: String, + /// Bytes the whole message has. + pub bytes: usize, + /// Lines the whole message has, counted as the file wrote them. + pub lines: usize, + /// The path from the record's root to the text. + pub at: Vec, +} + +impl Body { + /// The body of `text`, which lives at `at` inside its record. + pub fn new(text: &str, at: Vec) -> Body { + Body { + head: head_of(text, BODY_KEEP).to_string(), + bytes: text.len(), + lines: text.lines().count().max(1), + at, + } + } + + /// Is the head the whole message? Then the record never has to be read + /// again to paint it. + pub fn whole(&self) -> bool { + self.head.len() == self.bytes + } + + /// The whole message: out of the record when the head is short of it, and + /// out of the head when it is not. `record` is `None` where the caller has + /// no record in hand, and the head is then the honest answer it can give. + pub fn text_in<'a>(&'a self, record: Option<&'a Value>) -> &'a str { + if self.whole() { + return &self.head; + } + match record.and_then(|v| self.walk(v)) { + Some(text) if text.len() == self.bytes => text, + _ => &self.head, + } + } + + fn walk<'a>(&self, root: &'a Value) -> Option<&'a str> { + let mut at = root; + for step in &self.at { + at = match step { + Step::Key(k) => at.get(k)?, + Step::At(n) => at.index(*n)?, + }; + } + at.as_str() + } +} + +/// The first `bytes` of `text`, cut on a character boundary: a message may be +/// any UTF-8 at all, and cutting mid-character would panic. +fn head_of(text: &str, bytes: usize) -> &str { + if text.len() <= bytes { + return text; + } + let mut cut = bytes; + while cut > 0 && !text.is_char_boundary(cut) { + cut -= 1; + } + &text[..cut] +} + /// One record, as a lens reads it. /// /// Everything on it is display text the source paints; no styling and no @@ -96,6 +215,15 @@ pub struct Summary { pub what: String, /// Tool calls this record makes, for the group row's `· 4 tool calls`. pub calls: usize, + /// The record's own text, under the row: what was said for a message, and + /// what the model was thinking for a step that only thought. + /// + /// A message's body is **one wrap split in two** — the summary row is its + /// first line and the rows under it are the rest. A step's is not: a step's + /// row is the description of what it did (`thinking · bash(make)`), and its + /// text goes wholly underneath. [`Class`] is what tells the two apart, and + /// `src/source/record/body.rs` is where that decision is spent. + pub body: Option, } impl Summary { @@ -109,6 +237,7 @@ impl Summary { time: None, what: what.into(), calls: 0, + body: None, } } @@ -119,6 +248,34 @@ impl Summary { } } +/// Where a dialect's records live. +/// +/// A lens is a transform over records, and this is the one thing it says about +/// where they come from — because that decides which *file* the flag can be +/// pointed at, and nothing else about the dialect does. It is a declaration, +/// not a reader: `src/open/lens.rs` turns it into a format, and +/// `src/source/record/` reaches the records the same way whichever it is. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum RecordsAt { + /// One record per line: a `.jsonl` / `.ndjson` file. The default, and what + /// every dialect meant before there was a second answer. + Lines, + /// The root of one JSON document is the array of records. + /// + /// No dialect declares this yet — `atif`, the only document dialect, names + /// a member. The source implements it all the same + /// ([`crate::source::jsonarray::At::Root`], exercised by its tests), + /// because "the records are the whole document" is the *simpler* of the two + /// shapes and leaving it out would mean the next dialect to want it had to + /// add a variant, a routing arm and a source path at once. + #[allow(dead_code)] + Root, + /// The records are the array under this top-level key of one JSON + /// document; every *other* top-level key is record 0, so nothing the + /// document says about the run is lost (SPEC.md §Lenses). + Member(&'static str), +} + /// One dialect of record file. pub trait Lens { /// The name `--lens` takes. @@ -127,11 +284,42 @@ pub trait Lens { /// One line for `--lens list`. fn about(&self) -> &'static str; + /// Where this dialect's records are. A record per line unless the dialect + /// says otherwise, which is what every dialect written before documents + /// were readable still means. + fn records_at(&self) -> RecordsAt { + RecordsAt::Lines + } + /// Read one record. `None` means "not mine" — the record renders as the /// generic tree, whole. /// /// Called once per record, in file order. fn read(&mut self, value: &Value) -> Option; + + /// What this record's parts **are**, for the level between the headline and + /// the raw JSON tree (SPEC.md §Lenses): the tool calls it made, and any text + /// it holds that the body under its row is not already showing. + /// + /// Unlike [`Lens::read`] this is **not** called once per record. It is + /// called for the record the reader opened, when they open it, and its + /// answer is thrown away when they close it — which is why a [`Part`] may + /// hold [`Body`]s at all and why nothing here is stored on a [`Summary`]. + /// + /// `&self`, not `&mut self`, and that is the contract rather than an + /// accident: `read` runs far ahead of the viewport, so any state a dialect + /// carried across records is long past by the time a reader presses `Enter`. + /// A dialect that cannot answer from **this record alone** must return the + /// parts it can and leave the rest `None` — the raw tree is still one `r` + /// away, and a guess would be worse than a gap. + /// + /// The default is no parts, which is a level with nothing in it: the ladder + /// then has one rung fewer and `Enter` goes straight on to the tree. A + /// dialect implements this when it can say something a person would say. + fn detail(&self, value: &Value) -> Vec { + let _ = value; + Vec::new() + } } /// How a lens is built. A function pointer rather than a value because a lens @@ -140,7 +328,10 @@ type Make = fn() -> Box; /// Every lens there is. **Adding a dialect is one module and one line here** /// (docs/lenses.md says what the module has to provide). -const LENSES: &[(&str, Make)] = &[(agent::NAME, || Box::new(agent::Agent::default()))]; +const LENSES: &[(&str, Make)] = &[ + (agent::NAME, || Box::new(agent::Agent::default())), + (atif::NAME, || Box::new(atif::Atif)), +]; /// The lens called `name`, or `None`. pub fn find(name: &str) -> Option> { @@ -150,6 +341,13 @@ pub fn find(name: &str) -> Option> { .map(|(_, make)| make()) } +/// Where the lens called `name` keeps its records. The routing asks before the +/// file is opened, because the answer decides which formats `--lens ` +/// will accept. +pub fn records_at(name: &str) -> Option { + find(name).map(|l| l.records_at()) +} + /// Is `name` a lens? The CLI asks before the file is opened. pub fn exists(name: &str) -> bool { LENSES.iter().any(|(n, _)| *n == name) diff --git a/src/lens/part.rs b/src/lens/part.rs new file mode 100644 index 0000000..9301023 --- /dev/null +++ b/src/lens/part.rs @@ -0,0 +1,139 @@ +//! What a record's parts **are**, said without saying anything about rows. +//! +//! A summary is a headline and a [`Body`] is what was said. Between the two and +//! the raw JSON tree there is a level that reads a record the way a person would +//! describe it — the whole message, then its tool calls listed *as tool calls* +//! (SPEC.md §Lenses, the open level). This module is the vocabulary a dialect +//! uses to say what is in there, and it is deliberately row-free: no widths, no +//! indents, no fold markers, no order on the screen. `src/source/record/parts.rs` +//! turns a `Vec` into rows, and it is the only thing that does. +//! +//! # What a part may hold +//! +//! The same ceiling a [`Body`] has, for the same reason. [`Lens::detail`] is +//! called **on demand, for the record the reader opened** — never once per +//! record and never stored on a [`Summary`] — so parts do not cost anything per +//! document. But one *record* can carry five calls whose outputs are fifteen +//! kilobytes apiece, and a level that read them into memory to count its rows +//! would make opening one step cost what the step cost. So every stretch of +//! text in a `Part` is a [`Body`]: a bounded head, the true byte and line +//! counts, and the path back into the record. +//! +//! Two of those paths are real and one is not, and the difference is stated +//! rather than hidden: +//! +//! * a **result** sits at a path of static keys and one index +//! (`observation.results[3].content`), so opening it reads the whole of it +//! back out of the record being painted; +//! * an **argument** sits under a key whose name is the argument's own +//! (`arguments.command`), and [`Step::Key`] is `&'static str` — a path there +//! would have to allocate the key. An argument is therefore a `Body` with an +//! **empty path**, which [`Body::text_in`] resolves to the head. That is not a +//! silent clip: the head carries the true `bytes` and `lines`, so the row +//! under it says `⋯ +N more` exactly as a clipped message does, and every byte +//! is still one `r` away in the record's own tree. +//! +//! A lens never hides anything (SPEC.md §Lenses). A part may be clipped; it may +//! not be quiet about it. +#![deny(unsafe_code)] + +use super::Body; +use crate::json::Value; + +/// The name an argument list gets when it is not an object at all. +/// +/// A dialect reads a call's `arguments` as the object the schema says it is, +/// but a real file writes an array, a bare string, or a JSON-encoded string +/// that streaming cut in half. Dropping those left the open level saying the +/// call had **no** arguments, which is the one thing a clip may never do +/// (SPEC.md §Lenses) — so they become one argument under this name, holding +/// exactly what the file said. +pub const RAW_ARGS: &str = "arguments"; + +/// A call's `arguments` as the list the open level lists, whatever shape they +/// arrived in. +/// +/// An object is its members, in the record's own order. Anything else is one +/// entry named [`RAW_ARGS`]: a string as itself, and any other value as the +/// JSON it is — which is what the record's tree would show, and what `r` is +/// one press away from. +/// +/// Every value is a [`Body`] with an **empty path**: an argument lives under a +/// key that is its own name and a [`Step::Key`](super::Step::Key) is +/// `&'static str`, so the value is a bounded head that states its true size +/// rather than a path back into the record. +pub fn args_of(arguments: &Value) -> Vec<(String, Body)> { + // Absent, `null` and `[]` all mean "no arguments", and a row saying + // `arguments null` would be noise rather than a fact. + if matches!(arguments, Value::Null) || arguments.as_array().is_some_and(|a| a.is_empty()) { + return Vec::new(); + } + let Some(members) = arguments.as_object() else { + return vec![(RAW_ARGS.to_string(), Body::new(&as_text(arguments), Vec::new()))]; + }; + members + .iter() + .map(|m| (m.key.clone(), Body::new(&as_text(&m.value), Vec::new()))) + .collect() +} + +/// One value as the text an argument row shows: a string as itself, anything +/// else as its own JSON. +pub fn as_text(v: &Value) -> String { + match v.as_str() { + Some(s) => s.to_string(), + None => v.to_json(), + } +} + +/// One thing a record turns out to contain, once you read it as what it is. +/// +/// Deliberately two variants and not more: text someone (or something) wrote, +/// and a call to a tool. A dialect that finds a third kind of thing in a record +/// says so here, once, and every dialect gets the rows for free. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Part { + /// A stretch of text the record holds under a name: the model's reasoning, + /// a second text block a message row could only excerpt. + /// + /// `label` is `&'static str` because a dialect names it in its own source + /// (`thinking`, `message`) — the same reason [`Step::Key`](super::Step::Key) + /// is. + Text { + label: &'static str, + body: Body, + }, + /// A call to a tool: what was called, the one argument that says what it + /// did, all of its arguments, and what came back. + /// + /// `result` is `None` where the answer is not in this record — the `agent` + /// dialect's tool results arrive as a *later* record, and a `Body`'s path + /// starts at the record it belongs to. The row then says what was called + /// and does not pretend to know the answer. + Call { + /// The tool's own name: `bash`, `Read`. + tool: String, + /// The one argument a headline shows: the command, the path, the + /// pattern. Empty when the tool's arguments name none. + arg: String, + /// Every argument, in the record's own order, each value as text. + args: Vec<(String, Body)>, + /// What the call returned. + result: Option, + }, +} + +impl Part { + /// Is this a call with something under it — arguments, or an answer? + /// + /// A [`Part::Text`] is already showing everything it has, and so is a call + /// made with no arguments that returned nothing: a rung that repaints the + /// same screen is not a rung, so that row carries no fold marker and its + /// `Enter` belongs to the record rather than to the call. + pub fn opens(&self) -> bool { + match self { + Part::Text { .. } => false, + Part::Call { args, result, .. } => !args.is_empty() || result.is_some(), + } + } +} diff --git a/src/lens/tests.rs b/src/lens/tests.rs index 8b43ef0..b0662de 100644 --- a/src/lens/tests.rs +++ b/src/lens/tests.rs @@ -27,6 +27,49 @@ fn every_lens_agrees_with_its_entry() { } } +/// A dialect declares where its records live, and the default is the one every +/// dialect written before documents were readable still means: a record per +/// line. Getting this wrong routes `--lens` at the wrong reader, so it is +/// pinned for the default *and* for a dialect that overrides it. +#[test] +fn a_dialect_says_where_its_records_are_and_lines_is_the_default() { + struct Quiet; + impl Lens for Quiet { + fn name(&self) -> &'static str { + "quiet" + } + fn about(&self) -> &'static str { + "a dialect that says nothing about where it lives" + } + fn read(&mut self, _: &Value) -> Option { + None + } + } + assert_eq!(Quiet.records_at(), RecordsAt::Lines); + + struct Rooted; + impl Lens for Rooted { + fn name(&self) -> &'static str { + "rooted" + } + fn about(&self) -> &'static str { + "a dialect whose records are a document's root array" + } + fn records_at(&self) -> RecordsAt { + RecordsAt::Root + } + fn read(&mut self, _: &Value) -> Option { + None + } + } + assert_eq!(Rooted.records_at(), RecordsAt::Root); + + // And the registered ones, which is what routing actually asks. + assert_eq!(records_at("agent"), Some(RecordsAt::Lines)); + assert_eq!(records_at("atif"), Some(RecordsAt::Member("steps"))); + assert_eq!(records_at("nope"), None); +} + /// Two lenses built from the same entry are independent: a lens carries state /// across records, so sharing one between two open files would cross them. #[test] diff --git a/src/main_tests.rs b/src/main_tests.rs index ee83683..db47537 100644 --- a/src/main_tests.rs +++ b/src/main_tests.rs @@ -4,7 +4,8 @@ #![deny(unsafe_code)] use super::*; -use crate::open::{index_path, outline, render_outline}; +use crate::open::input::index_path; +use crate::open::{outline, render_outline}; #[test] fn plain_mode_rules() { diff --git a/src/open.rs b/src/open.rs index 5e4739f..f7cf5bf 100644 --- a/src/open.rs +++ b/src/open.rs @@ -11,19 +11,21 @@ //! path so it can be indexed lazily (SPEC.md §CSV). #![deny(unsafe_code)] +pub(crate) mod input; mod lens; -use std::io::Read; use std::path::{Path, PathBuf}; use crate::csv::read::Reader; -use crate::source::detect::{self, Format}; +use crate::source::detect::Format; use crate::source::json::{self as json, JsonSource}; use crate::source::text::TextSource; use crate::source::{csv::CsvSource, jsonl::JsonlSource, markdown::MarkdownSource, Source}; use crate::plat::{path as ppath, Platform}; use crate::{cli, md, sys}; +pub use input::resolve_input; + /// A fatal error plus the process exit code it maps to. pub struct Fail { pub msg: String, @@ -141,8 +143,13 @@ pub fn corpus_root(file: &Path, cwd: &Path) -> Option { /// and nothing else's: everything above takes a `Box` /// (SPEC.md §The `Source` seam). pub fn build_source(input: &Input, args: &cli::Args) -> Result, Fail> { - if args.lens.is_some() && input.format != Format::Jsonl { - return Err(lens::needs_records(input.format)); + if let Some(name) = args.lens.as_deref() { + // A lens reads records, and each dialect says where its records live + // (SPEC.md §Lenses). Pointed at anything else it refuses by name + // rather than rendering a document as one enormous record. + if !lens::accepts(name, input.format) { + return Err(lens::needs_records(name, input.format)); + } } // A directory named on the command line is a listing, not `os error 21` // (SPEC.md §Directories). A `README.md` inside it still wins, which is @@ -154,6 +161,8 @@ pub fn build_source(input: &Input, args: &cli::Args) -> Result, } match input.format { Format::Csv => Ok(Box::new(csv_source(input, args)?)), + // A document under a lens is its records, not its tree. + Format::Json if args.lens.is_some() => Ok(Box::new(lens::array_source_with(input, args)?)), Format::Json => Ok(Box::new(json_source(input)?)), Format::Jsonl => Ok(Box::new(lens::jsonl_source_with(input, args)?)), Format::Code => Ok(Box::new(code_source(input)?)), @@ -255,6 +264,10 @@ pub fn toc_text(input: &Input, args: &cli::Args) -> Result { // million-record log would be neither a table of contents nor quick. // A JSON document's outline is its root's immediate members: the one level // that can be listed without walking the file. + if input.format == Format::Json && args.lens.is_some() { + let mut src = lens::array_source_with(input, args)?; + return Ok(src.summaries(TOC_RECORDS).iter().map(|s| format!("{s}\n")).collect()); + } if input.format == Format::Json { let mut src = json_source(input)?; return Ok(src.toc().iter().map(|s| format!("{s}\n")).collect()); @@ -278,200 +291,6 @@ pub fn toc_text(input: &Input, args: &cli::Args) -> Result { /// Records `--toc` lists for a `.jsonl`. const TOC_RECORDS: usize = 1000; -pub fn resolve_input(args: &cli::Args) -> Result { - let from_stdin = args.file.as_deref() == Some(Path::new("-")) - || (args.file.is_none() && !sys::is_tty(sys::STDIN)); - if from_stdin { - let mut raw = Vec::new(); - std::io::stdin() - .read_to_end(&mut raw) - .map_err(|e| Fail::runtime(format!("reading stdin: {e}")))?; - check_encoding("", sniff_head(&raw))?; - // Unnamed input: the content decides (SPEC.md §Multi-format reading). - let format = lens::format_for(args, None, detect::decide(args.format, None, sniff_head(&raw))); - // stdin is a pipe, so keys have to come from the controlling terminal. - return Ok(Input { - label: "".to_string(), - bytes: Some(raw), - path: None, - format, - tty: sys::tty_fd(), - }); - } - let path = match &args.file { - Some(p) => p.clone(), - None => index_path(args.index.as_deref())?, - }; - read_document(args, &path) -} - -/// Describe a file without reading it: the extension decides, and only a file -/// whose extension says nothing is sniffed — from its first bytes, never from -/// all of them. -fn read_document(args: &cli::Args, path: &Path) -> Result { - // A named pipe, a device or anything else that is not a regular file has no - // size to stat and no offset to seek to, so the lazy row index has nothing - // to work with — and opening it twice (once to peek, once to read) blocks - // forever on a fifo whose writer has already gone. It is a *stream*, so it - // takes the same path piped stdin does: read once, keep the bytes. - // A directory is read as a listing (SPEC.md §Directories), so it is settled - // before the not-a-regular-file branch below: a directory is not a stream, - // and trying to read one is where `os error 21` came from. There is no head - // to peek at and no encoding to check. - if path.is_dir() { - return Ok(Input { - label: path.display().to_string(), - bytes: None, - path: Some(path.to_path_buf()), - // Unused: `build_source` sees the directory first. A listing is not - // a file format and the detector has no name for one. - format: Format::Text, - tty: sys::tty_fd(), - }); - } - if !is_regular(path) { - return stream_document(args, path); - } - // Four bytes, whatever the name says: enough to refuse an encoding that - // would only render as mojibake, and a multi-GB file must not be read one - // byte further than that. A named file is never sniffed — its extension - // names a parser or it is plain text ([`detect::decide`]) — so there is - // nothing else the head is wanted for. - let head = head_bytes(path, BOM_BYTES)?; - check_encoding(&path.display().to_string(), &head)?; - let format = lens::format_for(args, Some(path), detect::decide(args.format, Some(path), &head)); - Ok(Input { - label: path.display().to_string(), - bytes: None, - path: Some(path.to_path_buf()), - format, - tty: None, - }) -} - -/// True when `path` is a regular file: something with a size and an offset, -/// which is what the lazy row index needs. A fifo, a character device -/// (`/dev/null`, `/dev/stdin`) or a directory is not. -/// -/// `metadata` follows symlinks, which is right: a symlink to a regular file is -/// one. A path that cannot be stat'ed at all is left to the reader below, which -/// reports the open error with the same wording it always did. -fn is_regular(path: &Path) -> bool { - match std::fs::metadata(path) { - Ok(m) => m.is_file(), - Err(_) => true, - } -} - -/// The most a non-seekable input is read into memory. -/// -/// A stream has to be held whole — the bytes exist nowhere else — so unlike a -/// file on disk its size is a memory cost, and `tread /dev/zero` must stop -/// somewhere rather than take the machine down with it. -const STREAM_CAP: usize = 256 * 1024 * 1024; - -/// Read a non-seekable path — a named pipe, a device — the way stdin is read. -fn stream_document(args: &cli::Args, path: &Path) -> Result { - let label = path.display().to_string(); - let mut f = std::fs::File::open(path).map_err(|e| Fail::runtime(format!("{label}: {e}")))?; - let mut raw = Vec::new(); - // One byte past the cap distinguishes "exactly the cap" from "more". - f.by_ref() - .take(STREAM_CAP as u64 + 1) - .read_to_end(&mut raw) - .map_err(|e| Fail::runtime(format!("{label}: {e}")))?; - if raw.len() > STREAM_CAP { - return Err(Fail::runtime(format!( - "{label}: not a regular file, and larger than {}MiB \u{2014} tread reads a \ - stream into memory; redirect it to a file first", - STREAM_CAP / (1024 * 1024) - ))); - } - check_encoding(&label, sniff_head(&raw))?; - let format = lens::format_for(args, Some(path), detect::decide(args.format, Some(path), sniff_head(&raw))); - Ok(Input { - label, - bytes: Some(raw), - // No path: there is nothing to re-open, and a device is not part of a - // corpus. Everything downstream reads the bytes we already have. - path: None, - format, - tty: None, - }) -} - -/// Bytes a content sniff looks at. -const SNIFF_BYTES: usize = 8 * 1024; - -/// Bytes an encoding check looks at: the longest byte-order mark there is. -const BOM_BYTES: usize = 4; - -fn sniff_head(raw: &[u8]) -> &[u8] { - &raw[..raw.len().min(SNIFF_BYTES)] -} - -/// Refuse a document whose byte-order mark says it is in an encoding tread -/// cannot read, and say what to do about it. A reader that renders UTF-16 as -/// half a screen of replacement characters has told the user nothing; this is -/// the one case where opening the file is worse than not (see -/// [`detect::unreadable_encoding`]). -fn check_encoding(label: &str, head: &[u8]) -> Result<(), Fail> { - match detect::unreadable_encoding(head) { - None => Ok(()), - Some(enc) => Err(Fail::runtime(format!( - "{label}: {enc} text (byte-order mark); tread reads UTF-8 \ - \u{2014} convert it first, e.g. iconv -f {enc} -t UTF-8" - ))), - } -} - -/// The first `want` bytes of a file, for the sniffer and the encoding check. A -/// file that cannot be opened is reported here rather than by whichever format -/// got it. -fn head_bytes(path: &Path, want: usize) -> Result, Fail> { - let mut f = std::fs::File::open(path) - .map_err(|e| Fail::runtime(format!("{}: {}", path.display(), e)))?; - let mut buf = vec![0u8; want]; - let mut got = 0; - while got < buf.len() { - match f.read(&mut buf[got..]) { - Ok(0) => break, - Ok(n) => got += n, - Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => continue, - Err(e) => return Err(Fail::runtime(format!("{}: {}", path.display(), e))), - } - } - buf.truncate(got); - Ok(buf) -} - -/// Where to start when no FILE was given: the `--index` target, or a README.md -/// in the working directory. -pub fn index_path(index: Option<&Path>) -> Result { - let candidate = match index { - // A directory that documents itself shows its documentation; one that - // does not is still readable as a listing (SPEC.md §Directories). - Some(p) if p.is_dir() && p.join("README.md").is_file() => p.join("README.md"), - Some(p) if p.is_dir() => p.to_path_buf(), - Some(p) => p.to_path_buf(), - None => PathBuf::from("README.md"), - }; - if candidate.exists() { - return Ok(candidate); - } - if index.is_some() { - return Err(Fail::usage(format!( - "index not found: {}", - candidate.display() - ))); - } - Err(Fail::usage(format!( - "no input: give a FILE, pipe markdown in, or use --index; try `{} --help`", - cli::BIN - ))) -} - - /// Heading outline, taken from the parsed document so `--toc` and the pager's /// outline overlay always agree with the renderer. pub fn outline(doc: &md::Document) -> Vec<(u8, String)> { @@ -567,53 +386,4 @@ mod tests { // Piped input has no location to resolve anything against. assert!(!navigable(&input(None, Format::Markdown))); } - - fn tmp_path(name: &str) -> PathBuf { - let mut p = std::env::temp_dir(); - let nanos = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map(|d| d.as_nanos()) - .unwrap_or(0); - p.push(format!("tread-open-{}-{nanos}-{name}", std::process::id())); - p - } - - #[test] - fn only_a_regular_file_can_be_read_lazily() { - let file = tmp_path("regular"); - std::fs::write(&file, b"id,name\n1,a\n").expect("write"); - assert!(is_regular(&file)); - std::fs::remove_file(&file).ok(); - - let dir = tmp_path("dir"); - std::fs::create_dir(&dir).expect("mkdir"); - assert!(!is_regular(&dir), "a directory is not a regular file"); - std::fs::remove_dir(&dir).ok(); - - // A path that cannot be stat'ed is left to the reader, which reports - // the open error with its own wording rather than calling it a stream. - assert!(is_regular(Path::new("/definitely/not/here"))); - } - - /// A character device has no size to stat and no offset to seek to, so it - /// takes the stream path — the bug this catches is a fifo being opened - /// twice (peek, then read), which blocks forever once the writer has gone. - #[test] - #[cfg(unix)] - fn a_device_is_a_stream_not_a_file() { - assert!(!is_regular(Path::new("/dev/null"))); - } - - #[test] - fn an_encoding_we_cannot_read_is_refused_with_the_fix() { - let err = check_encoding("x.csv", b"\xff\xfei\x00").expect_err("refused"); - assert_eq!(err.code, crate::EXIT_ERROR); - assert!(err.msg.contains("x.csv"), "{}", err.msg); - assert!(err.msg.contains("UTF-16"), "{}", err.msg); - assert!(err.msg.contains("iconv -f UTF-16 -t UTF-8"), "{}", err.msg); - // UTF-8, with or without a BOM, is not refused. - assert!(check_encoding("x.csv", b"\xef\xbb\xbfid,name\n").is_ok()); - assert!(check_encoding("x.csv", b"id,name\n").is_ok()); - assert!(check_encoding("x.csv", b"").is_ok()); - } } diff --git a/src/open/input.rs b/src/open/input.rs new file mode 100644 index 0000000..612f60e --- /dev/null +++ b/src/open/input.rs @@ -0,0 +1,267 @@ +//! Resolving *what* to open: the path or the pipe, and what the first bytes +//! say about it. +//! +//! Split from [`super`], which builds the [`crate::source::Source`] once this +//! has decided where the bytes are and which format they are in; the two grew +//! past one file's worth together. +//! +//! A file on disk is **not** read here. Only its first bytes are, and only when +//! the extension says nothing: a multi-GB CSV must reach its source as a path +//! so it can be indexed lazily (SPEC.md §CSV). A stream — a pipe, a fifo, a +//! device — is the one input that has to be held whole, because the bytes exist +//! nowhere else. +#![deny(unsafe_code)] + +use std::io::Read; +use std::path::{Path, PathBuf}; + +use super::{lens, Fail, Input}; +use crate::source::detect::{self, Format}; +use crate::{cli, sys}; + +pub fn resolve_input(args: &cli::Args) -> Result { + let from_stdin = args.file.as_deref() == Some(Path::new("-")) + || (args.file.is_none() && !sys::is_tty(sys::STDIN)); + if from_stdin { + let mut raw = Vec::new(); + std::io::stdin() + .read_to_end(&mut raw) + .map_err(|e| Fail::runtime(format!("reading stdin: {e}")))?; + check_encoding("", sniff_head(&raw))?; + // Unnamed input: the content decides (SPEC.md §Multi-format reading). + let format = lens::format_for(args, None, detect::decide(args.format, None, sniff_head(&raw))); + // stdin is a pipe, so keys have to come from the controlling terminal. + return Ok(Input { + label: "".to_string(), + bytes: Some(raw), + path: None, + format, + tty: sys::tty_fd(), + }); + } + let path = match &args.file { + Some(p) => p.clone(), + None => index_path(args.index.as_deref())?, + }; + read_document(args, &path) +} + +/// Describe a file without reading it: the extension decides, and only a file +/// whose extension says nothing is sniffed — from its first bytes, never from +/// all of them. +fn read_document(args: &cli::Args, path: &Path) -> Result { + // A named pipe, a device or anything else that is not a regular file has no + // size to stat and no offset to seek to, so the lazy row index has nothing + // to work with — and opening it twice (once to peek, once to read) blocks + // forever on a fifo whose writer has already gone. It is a *stream*, so it + // takes the same path piped stdin does: read once, keep the bytes. + // A directory is read as a listing (SPEC.md §Directories), so it is settled + // before the not-a-regular-file branch below: a directory is not a stream, + // and trying to read one is where `os error 21` came from. There is no head + // to peek at and no encoding to check. + if path.is_dir() { + return Ok(Input { + label: path.display().to_string(), + bytes: None, + path: Some(path.to_path_buf()), + // Unused: `build_source` sees the directory first. A listing is not + // a file format and the detector has no name for one. + format: Format::Text, + tty: sys::tty_fd(), + }); + } + if !is_regular(path) { + return stream_document(args, path); + } + // Four bytes, whatever the name says: enough to refuse an encoding that + // would only render as mojibake, and a multi-GB file must not be read one + // byte further than that. A named file is never sniffed — its extension + // names a parser or it is plain text ([`detect::decide`]) — so there is + // nothing else the head is wanted for. + let head = head_bytes(path, BOM_BYTES)?; + check_encoding(&path.display().to_string(), &head)?; + let format = lens::format_for(args, Some(path), detect::decide(args.format, Some(path), &head)); + Ok(Input { + label: path.display().to_string(), + bytes: None, + path: Some(path.to_path_buf()), + format, + tty: None, + }) +} + +/// True when `path` is a regular file: something with a size and an offset, +/// which is what the lazy row index needs. A fifo, a character device +/// (`/dev/null`, `/dev/stdin`) or a directory is not. +/// +/// `metadata` follows symlinks, which is right: a symlink to a regular file is +/// one. A path that cannot be stat'ed at all is left to the reader below, which +/// reports the open error with the same wording it always did. +fn is_regular(path: &Path) -> bool { + match std::fs::metadata(path) { + Ok(m) => m.is_file(), + Err(_) => true, + } +} + +/// The most a non-seekable input is read into memory. +/// +/// A stream has to be held whole — the bytes exist nowhere else — so unlike a +/// file on disk its size is a memory cost, and `tread /dev/zero` must stop +/// somewhere rather than take the machine down with it. +const STREAM_CAP: usize = 256 * 1024 * 1024; + +/// Read a non-seekable path — a named pipe, a device — the way stdin is read. +fn stream_document(args: &cli::Args, path: &Path) -> Result { + let label = path.display().to_string(); + let mut f = std::fs::File::open(path).map_err(|e| Fail::runtime(format!("{label}: {e}")))?; + let mut raw = Vec::new(); + // One byte past the cap distinguishes "exactly the cap" from "more". + f.by_ref() + .take(STREAM_CAP as u64 + 1) + .read_to_end(&mut raw) + .map_err(|e| Fail::runtime(format!("{label}: {e}")))?; + if raw.len() > STREAM_CAP { + return Err(Fail::runtime(format!( + "{label}: not a regular file, and larger than {}MiB \u{2014} tread reads a \ + stream into memory; redirect it to a file first", + STREAM_CAP / (1024 * 1024) + ))); + } + check_encoding(&label, sniff_head(&raw))?; + let format = lens::format_for(args, Some(path), detect::decide(args.format, Some(path), sniff_head(&raw))); + Ok(Input { + label, + bytes: Some(raw), + // No path: there is nothing to re-open, and a device is not part of a + // corpus. Everything downstream reads the bytes we already have. + path: None, + format, + tty: None, + }) +} + +/// Bytes a content sniff looks at. +const SNIFF_BYTES: usize = 8 * 1024; + +/// Bytes an encoding check looks at: the longest byte-order mark there is. +const BOM_BYTES: usize = 4; + +fn sniff_head(raw: &[u8]) -> &[u8] { + &raw[..raw.len().min(SNIFF_BYTES)] +} + +/// Refuse a document whose byte-order mark says it is in an encoding tread +/// cannot read, and say what to do about it. A reader that renders UTF-16 as +/// half a screen of replacement characters has told the user nothing; this is +/// the one case where opening the file is worse than not (see +/// [`detect::unreadable_encoding`]). +fn check_encoding(label: &str, head: &[u8]) -> Result<(), Fail> { + match detect::unreadable_encoding(head) { + None => Ok(()), + Some(enc) => Err(Fail::runtime(format!( + "{label}: {enc} text (byte-order mark); tread reads UTF-8 \ + \u{2014} convert it first, e.g. iconv -f {enc} -t UTF-8" + ))), + } +} + +/// The first `want` bytes of a file, for the sniffer and the encoding check. A +/// file that cannot be opened is reported here rather than by whichever format +/// got it. +fn head_bytes(path: &Path, want: usize) -> Result, Fail> { + let mut f = std::fs::File::open(path) + .map_err(|e| Fail::runtime(format!("{}: {}", path.display(), e)))?; + let mut buf = vec![0u8; want]; + let mut got = 0; + while got < buf.len() { + match f.read(&mut buf[got..]) { + Ok(0) => break, + Ok(n) => got += n, + Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => continue, + Err(e) => return Err(Fail::runtime(format!("{}: {}", path.display(), e))), + } + } + buf.truncate(got); + Ok(buf) +} + +/// Where to start when no FILE was given: the `--index` target, or a README.md +/// in the working directory. +pub fn index_path(index: Option<&Path>) -> Result { + let candidate = match index { + // A directory that documents itself shows its documentation; one that + // does not is still readable as a listing (SPEC.md §Directories). + Some(p) if p.is_dir() && p.join("README.md").is_file() => p.join("README.md"), + Some(p) if p.is_dir() => p.to_path_buf(), + Some(p) => p.to_path_buf(), + None => PathBuf::from("README.md"), + }; + if candidate.exists() { + return Ok(candidate); + } + if index.is_some() { + return Err(Fail::usage(format!( + "index not found: {}", + candidate.display() + ))); + } + Err(Fail::usage(format!( + "no input: give a FILE, pipe markdown in, or use --index; try `{} --help`", + cli::BIN + ))) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn tmp_path(name: &str) -> PathBuf { + let mut p = std::env::temp_dir(); + let nanos = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0); + p.push(format!("tread-open-{}-{nanos}-{name}", std::process::id())); + p + } + + #[test] + fn only_a_regular_file_can_be_read_lazily() { + let file = tmp_path("regular"); + std::fs::write(&file, b"id,name\n1,a\n").expect("write"); + assert!(is_regular(&file)); + std::fs::remove_file(&file).ok(); + + let dir = tmp_path("dir"); + std::fs::create_dir(&dir).expect("mkdir"); + assert!(!is_regular(&dir), "a directory is not a regular file"); + std::fs::remove_dir(&dir).ok(); + + // A path that cannot be stat'ed is left to the reader, which reports + // the open error with its own wording rather than calling it a stream. + assert!(is_regular(Path::new("/definitely/not/here"))); + } + + /// A character device has no size to stat and no offset to seek to, so it + /// takes the stream path — the bug this catches is a fifo being opened + /// twice (peek, then read), which blocks forever once the writer has gone. + #[test] + #[cfg(unix)] + fn a_device_is_a_stream_not_a_file() { + assert!(!is_regular(Path::new("/dev/null"))); + } + + #[test] + fn an_encoding_we_cannot_read_is_refused_with_the_fix() { + let err = check_encoding("x.csv", b"\xff\xfei\x00").expect_err("refused"); + assert_eq!(err.code, crate::EXIT_ERROR); + assert!(err.msg.contains("x.csv"), "{}", err.msg); + assert!(err.msg.contains("UTF-16"), "{}", err.msg); + assert!(err.msg.contains("iconv -f UTF-16 -t UTF-8"), "{}", err.msg); + // UTF-8, with or without a BOM, is not refused. + assert!(check_encoding("x.csv", b"\xef\xbb\xbfid,name\n").is_ok()); + assert!(check_encoding("x.csv", b"id,name\n").is_ok()); + assert!(check_encoding("x.csv", b"").is_ok()); + } +} diff --git a/src/open/lens.rs b/src/open/lens.rs index 9af4e7b..fa96eaf 100644 --- a/src/open/lens.rs +++ b/src/open/lens.rs @@ -1,64 +1,138 @@ //! Where `--lens` meets the input (SPEC.md §Lenses). //! //! Split out of `open.rs` so both stay under the size limit, and because this -//! is genuinely one decision: a lens applies to a *record file*, so this is the -//! only place that pairs a lens with a format. Nothing here knows a dialect — -//! `crate::lens` owns the registry, and the record source owns the rows. +//! is genuinely one decision: a lens applies to *records*, and this is the only +//! place that pairs a lens with the format its records come in. Nothing here +//! knows a dialect — `crate::lens` owns the registry and each dialect declares +//! [`RecordsAt`] for itself, `crate::source::record` owns the rows. +//! +//! # Two shapes of record file, and a refusal for each +//! +//! A `RecordsAt::Lines` dialect reads a `.jsonl`; a `RecordsAt::Root` or +//! `RecordsAt::Member` dialect reads one JSON *document*. Pointing either at +//! the other is a usage error rather than a silent no-op or a wrong render: a +//! `.json` read as records would be one enormous record, and a `.jsonl` read as +//! a document would find no top-level array at all. Both refusals name the +//! lens, name what the file is, and say what to do. #![deny(unsafe_code)] use std::path::Path; use super::{Fail, Input}; +use crate::lens::RecordsAt; use crate::source::detect::{self, Format}; +use crate::source::jsonarray::{ArraySource, At}; use crate::source::jsonl::JsonlSource; use crate::{cli, lens}; -/// The record source, read through `--lens` when one was asked for. +/// The `.jsonl` source, read through `--lens` when one was asked for. pub(super) fn jsonl_source_with(input: &Input, args: &cli::Args) -> Result { let mut src = super::jsonl_source(input)?; if let Some(name) = args.lens.as_deref() { - match lens::find(name) { - Some(l) => src.set_lens(l), - // Unreachable: `cli` validates the name against the same table. - None => return Err(Fail::usage(unknown_lens(name))), - } + src.set_lens(with_lens(name)?); } Ok(src) } -/// A `--lens` on something that is not a record file. +/// The document source for a lens whose records live *inside* a document. +/// +/// Only ever reached with a lens: without one a `.json` is the document tree, +/// which is what [`crate::source::json`] is for. +pub(super) fn array_source_with(input: &Input, args: &cli::Args) -> Result { + let name = match args.lens.as_deref() { + Some(name) => name, + None => return Err(needs_lens()), + }; + let at = match lens::records_at(name) { + Some(RecordsAt::Member(key)) => At::Key(key), + Some(RecordsAt::Root) => At::Root, + // Unreachable: `build_source` refuses a `Lines` lens on a document + // before it gets here, with a message that says what to do. + _ => return Err(needs_records(name, Format::Json)), + }; + let mut src = match &input.path { + Some(path) => ArraySource::open(path, at) + .map_err(|e| Fail::runtime(format!("{}: {e}", path.display())))?, + None => ArraySource::from_bytes(input.bytes.clone().unwrap_or_default(), at), + }; + src.set_lens(with_lens(name)?); + Ok(src) +} + +/// Does `--lens ` want the records-per-line reader, or the document one? +/// +/// The one question the format wiring asks about a dialect. An unknown name +/// cannot reach here — `cli` validates it against the same table — and reads as +/// the old behaviour rather than as a panic. +pub(super) fn wants_document(name: &str) -> bool { + !matches!(lens::records_at(name), None | Some(RecordsAt::Lines)) +} + +/// Whether `--lens ` can read a file of this format at all. +pub(super) fn accepts(name: &str, format: Format) -> bool { + match wants_document(name) { + true => format == Format::Json, + false => format == Format::Jsonl, + } +} + +fn with_lens(name: &str) -> Result, Fail> { + // Unreachable: `cli` validates the name against the same table. + lens::find(name).ok_or_else(|| Fail::usage(unknown_lens(name))) +} + +/// A `--lens` on something whose records it cannot find. /// /// A lens is a transform over records, so naming one for a markdown or CSV /// document is a usage error rather than a silent no-op: a flag that quietly -/// does nothing is worse than one that says why. -pub(super) fn needs_records(format: Format) -> Fail { - Fail::usage(format!( - "--lens reads a record file (.jsonl / .ndjson); this is {}. \ - Try `--format jsonl` if that is what it is.", - detect::name_of(format) - )) +/// does nothing is worse than one that says why. Which file it *would* have +/// read depends on the dialect, so the message says which one this is. +pub(super) fn needs_records(name: &str, format: Format) -> Fail { + let is = detect::name_of(format); + match wants_document(name) { + true => Fail::usage(format!( + "--lens {name} reads records inside a JSON document; this is {is}. \ + Try `--format json` if that is what it is." + )), + false => Fail::usage(format!( + "--lens {name} reads a record file (.jsonl / .ndjson); this is {is}. \ + Try `--format jsonl` if that is what it is." + )), + } +} + +fn needs_lens() -> Fail { + Fail::usage(String::from("a JSON document is only read as records under --lens")) } fn unknown_lens(name: &str) -> String { format!("unknown lens `{name}`; try `{} --lens list`", cli::BIN) } -/// A `--lens` says the input is records, which is stronger evidence than a -/// content sniff. +/// A `--lens` says what the input is, which is stronger evidence than a content +/// sniff — and *which* thing it says now depends on the dialect. /// /// The extension is evidence too, and a better one: `tread --lens agent /// notes.md` should say so rather than read prose as records. But unnamed input /// — a pipe — is only ever *guessed* at, and one JSON object on a line looks /// exactly like a JSON document to a sniffer, so `cat run.jsonl | tread --lens -/// agent` would otherwise be refused for being what it is. An explicit +/// agent` would otherwise be refused for being what it is. A document lens +/// wants the opposite answer for the same reason: `cat trajectory.json | tread +/// --lens atif` must not be forced into one enormous single record. An explicit /// `--format` always wins over both. pub(super) fn format_for(args: &cli::Args, path: Option<&Path>, sniffed: Format) -> Format { - if args.lens.is_none() || args.format.is_some() { + let Some(name) = args.lens.as_deref() else { + return sniffed; + }; + if args.format.is_some() { return sniffed; } match path.and_then(detect::from_path) { Some(_) => sniffed, - None => Format::Jsonl, + None => match wants_document(name) { + true => Format::Json, + false => Format::Jsonl, + }, } } @@ -66,39 +140,68 @@ pub(super) fn format_for(args: &cli::Args, path: Option<&Path>, sniffed: Format) mod tests { use super::*; + fn args(lens: &str) -> cli::Args { + cli::Args { lens: Some(lens.to_string()), ..cli::Args::default() } + } + /// A lens is defined over records, so it settles a *guess* about what the /// input is — and never overrules the extension or an explicit `--format`. #[test] fn a_lens_decides_what_unnamed_input_is() { - let mut args = cli::Args { lens: Some("agent".to_string()), ..cli::Args::default() }; + let mut a = args("agent"); // A pipe: one JSON object on a line sniffs as a document, and the lens // says it is a record. - assert_eq!(format_for(&args, None, Format::Json), Format::Jsonl); + assert_eq!(format_for(&a, None, Format::Json), Format::Jsonl); // A file whose extension said nothing is a guess too. - assert_eq!( - format_for(&args, Some(Path::new("run.log")), Format::Csv), - Format::Jsonl - ); + assert_eq!(format_for(&a, Some(Path::new("run.log")), Format::Csv), Format::Jsonl); // A named extension is evidence, and keeps its answer. for (name, format) in [("a.md", Format::Markdown), ("a.csv", Format::Csv)] { - assert_eq!(format_for(&args, Some(Path::new(name)), format), format); + assert_eq!(format_for(&a, Some(Path::new(name)), format), format); } // An explicit --format always wins. - args.format = Some(Format::Json); - assert_eq!(format_for(&args, None, Format::Json), Format::Json); + a.format = Some(Format::Json); + assert_eq!(format_for(&a, None, Format::Json), Format::Json); // And with no lens, nothing changes at all. let plain = cli::Args::default(); assert_eq!(format_for(&plain, None, Format::Json), Format::Json); assert_eq!(format_for(&plain, None, Format::Markdown), Format::Markdown); } - /// The two errors this seam can produce name the flag, the lens and the fix. + /// The dialect decides which way a *pipe* is read: the same bytes are a + /// record stream under `agent` and one document under `atif`. + #[test] + fn a_document_lens_reads_a_pipe_as_a_document() { + assert_eq!(format_for(&args("atif"), None, Format::Json), Format::Json); + assert_eq!(format_for(&args("atif"), None, Format::Csv), Format::Json); + assert!(wants_document("atif") && !wants_document("agent")); + // An unknown name never reaches here, and reads as the old behaviour. + assert!(!wants_document("nope")); + } + + /// Each lens accepts exactly the file its records live in, and refuses the + /// other one by name rather than rendering it wrongly. + #[test] + fn a_lens_accepts_the_shape_its_records_live_in() { + assert!(accepts("agent", Format::Jsonl) && !accepts("agent", Format::Json)); + assert!(accepts("atif", Format::Json) && !accepts("atif", Format::Jsonl)); + for f in [Format::Markdown, Format::Csv, Format::Text, Format::Code] { + assert!(!accepts("agent", f) && !accepts("atif", f)); + } + } + + /// The refusals name the flag, the lens, what the file is, and the fix. #[test] fn the_refusals_say_what_to_do() { - let f = needs_records(Format::Markdown); + let f = needs_records("agent", Format::Markdown); assert_eq!(f.code, crate::EXIT_USAGE); - assert!(f.msg.contains("--lens") && f.msg.contains("markdown"), "{}", f.msg); + assert!(f.msg.contains("--lens agent") && f.msg.contains("markdown"), "{}", f.msg); assert!(f.msg.contains("--format jsonl"), "{}", f.msg); + + let d = needs_records("atif", Format::Jsonl); + assert_eq!(d.code, crate::EXIT_USAGE); + assert!(d.msg.contains("--lens atif") && d.msg.contains("JSON document"), "{}", d.msg); + assert!(d.msg.contains("--format json"), "{}", d.msg); + assert!(unknown_lens("opencode").contains("--lens list")); } } diff --git a/src/pager/input.rs b/src/pager/input.rs index e447948..de0cd79 100644 --- a/src/pager/input.rs +++ b/src/pager/input.rs @@ -1,7 +1,8 @@ //! Input dispatch: key press -> [`Action`] -> state change. //! //! Split out of `mod.rs` to keep both files well under the size limit; these -//! are the same `Pager` methods, just living next door. +//! are the same `Pager` methods, just living next door. Where the cursor +//! actually goes once an action is picked is `motion.rs`, one more door down. #![deny(unsafe_code)] use super::keys::{self, Action}; @@ -132,6 +133,7 @@ impl Pager { fn view_action(&mut self, a: Action) { match a { Action::ToggleCollapse => self.fold(None), + Action::OpenTree => self.open_tree(), Action::OpenSection => self.fold(Some(false)), Action::CloseSection => self.fold(Some(true)), Action::CollapseAll => self.fold_all(true), @@ -159,84 +161,6 @@ impl Pager { } } - // -- movement ----------------------------------------------------------- - - fn move_cursor(&mut self, delta: isize) { - let n = self.len(); - if n == 0 { - return; - } - let max = n as isize - 1; - let next = (self.cursor as isize).saturating_add(delta).clamp(0, max); - self.cursor = next as usize; - self.clamp(); - } - - pub(super) fn goto(&mut self, row: usize) { - if self.len() == 0 { - return; - } - self.cursor = row.min(self.len() - 1); - self.clamp(); - } - - /// `h` / `l`. A format may scroll by its own unit — a CSV moves a whole - /// column, and its pinned header moves with the body because both are - /// painted at this one offset (SPEC.md §CSV). - /// - /// The format is told the *viewport* width, the same `cols` the painter - /// slices rows to and [`Pager::max_hoff`] measures against — not the layout - /// width, which `--width` can set far wider than the terminal. - pub(super) fn scroll_h(&mut self, delta: isize) { - let max = self.max_hoff(); - let hoff = self.hoff; - let view = self.cols.max(1); - let next = match self.src.hscroll(hoff, delta.signum(), view) { - Some(off) => off as isize, - None => (hoff as isize).saturating_add(delta), - }; - self.hoff = next.clamp(0, max as isize) as usize; - } - - /// `w`: let the format widen whatever is under the cursor. - fn widen(&mut self) { - match self.src.widen() { - Some(msg) => { - self.notify(msg); - self.clamp(); - } - None => self.notify("nothing to widen here"), - } - } - - /// Move the cursor to a row that is already visible, scrolling it in. - pub(super) fn jump_to_row(&mut self, row: usize) { - self.cursor = row.min(self.len().saturating_sub(1)); - let h = self.content_rows().max(1); - if self.cursor < self.top || self.cursor >= self.top + h { - self.top = self.cursor.saturating_sub(h / 3); - } - self.clamp(); - } - - /// Move the cursor to a place in the document, unfolding whatever hides it. - pub(super) fn jump_to(&mut self, anchor: Anchor) { - match self.src.reveal(anchor) { - Some(row) => self.jump_to_row(row), - None => self.clamp(), - } - } - - fn jump_heading(&mut self, forward: bool) { - match self.src.next_landmark(self.cursor, forward) { - Some(row) => self.goto(row), - None => self.notify(match forward { - true => "no further heading", - false => "no previous heading", - }), - } - } - // -- folding ------------------------------------------------------------ /// `want` = `None` toggles, `Some(true)` closes, `Some(false)` opens the diff --git a/src/pager/keys.rs b/src/pager/keys.rs index 13b6fac..ad7a54e 100644 --- a/src/pager/keys.rs +++ b/src/pager/keys.rs @@ -13,6 +13,9 @@ pub enum Action { Quit, /// Ctrl-C: leave immediately, without stepping back through the nav stack. ForceQuit, + /// `j` / `↓`: one visible row, in every format, always. A closed block is + /// one row, so a coarser default unit could only ever skip rows that are + /// on the screen. LineDown, LineUp, HalfDown, @@ -33,6 +36,9 @@ pub enum Action { /// Show or hide what the format hides by default (a listing's dotfiles). ToggleHidden, ToggleCollapse, + /// `r`: show the raw record under the cursor, where a row stands for one — + /// open its JSON tree, or shut the one that is open (SPEC.md §Lenses). + OpenTree, OpenSection, CloseSection, CollapseAll, @@ -111,13 +117,13 @@ use Action as A; pub const BINDINGS: &[Binding] = &[ Binding { keys: "j / \u{2193}", - desc: "line down", + desc: "one row down", action: A::LineDown, triggers: &[Trigger::c('j'), Trigger::k(Key::Down)], }, Binding { keys: "k / \u{2191}", - desc: "line up", + desc: "one row up", action: A::LineUp, triggers: &[Trigger::c('k'), Trigger::k(Key::Up)], }, @@ -204,10 +210,16 @@ pub const BINDINGS: &[Binding] = &[ }, Binding { keys: "Enter", - desc: "follow the focused link, open the row, else fold", + desc: "follow the focused link, else one level into the row", action: A::Follow, triggers: &[Trigger::k(Key::Enter)], }, + Binding { + keys: "r", + desc: "show the raw record under the cursor, from either level", + action: A::OpenTree, + triggers: &[Trigger::c('r')], + }, Binding { keys: "zo", desc: "open the section at the cursor", @@ -234,13 +246,13 @@ pub const BINDINGS: &[Binding] = &[ }, Binding { keys: "Tab", - desc: "next heading", + desc: "next heading (next block under a lens)", action: A::NextHeading, triggers: &[Trigger::k(Key::Tab)], }, Binding { keys: "S-Tab", - desc: "previous heading", + desc: "previous heading (previous block under a lens)", action: A::PrevHeading, triggers: &[Trigger::k(Key::BackTab)], }, @@ -354,7 +366,13 @@ pub const BINDINGS: &[Binding] = &[ /// Resolve a key press (with any pending chord prefix) to an action. pub fn lookup(prefix: Option, ev: KeyEvent) -> Option { - if ev.mods.ctrl || ev.mods.alt { + // A modifier no binding asked for means the press is not that binding: + // `Ctrl-j` is not `j`. But a key that *is* a control key already says so in + // `Key::Ctrl`, and the two decoders disagree about the modifier bits on it + // — a raw `0x03` carries none, kitty's `CSI u` sets `ctrl` — so rejecting + // on the bit alone would make `Ctrl-C` work on one terminal and be + // swallowed on another. + if (ev.mods.ctrl && !matches!(ev.key, Key::Ctrl(_))) || ev.mods.alt { return None; } BINDINGS @@ -422,6 +440,20 @@ mod tests { assert_eq!(lookup(None, ctrl_j), None); } + /// A control key from both decoders. A raw `0x03` arrives as + /// `Key::Ctrl('c')` with no modifier bits; kitty's `CSI u` sends the same + /// key *with* `ctrl` set, and both must reach the same binding — the hole + /// that swallowed `Ctrl-C` on one terminal and not the other. + #[test] + fn a_control_key_resolves_from_either_decoder() { + let raw = KeyEvent::plain(Key::Ctrl('c')); + assert_eq!(lookup(None, raw), Some(A::ForceQuit)); + let kitty = KeyEvent::with(Key::Ctrl('c'), Mods { ctrl: true, ..Mods::NONE }); + assert_eq!(lookup(None, kitty), Some(A::ForceQuit)); + // A chord prefix still gates it: `z` then Ctrl-C is nothing. + assert_eq!(lookup(Some('z'), raw), None); + } + #[test] fn help_covers_every_binding_and_never_drifts() { assert_eq!(help_rows().len(), BINDINGS.len()); diff --git a/src/pager/mod.rs b/src/pager/mod.rs index 00365cb..3b0ac44 100644 --- a/src/pager/mod.rs +++ b/src/pager/mod.rs @@ -21,6 +21,7 @@ #![deny(unsafe_code)] mod input; +mod motion; pub mod keys; mod link; pub mod navigate; @@ -391,6 +392,20 @@ impl Pager { self.clamp(); } + /// `r`: the raw thing under the cursor, where the format has one — a + /// record behind a lens row. The same shape `a` has, and for the same + /// reason: the source answers, or the pager says there was nothing there + /// rather than appearing to do nothing. + fn open_tree(&mut self) { + match self.src.toggle_tree(self.cursor) { + Some(msg) => { + self.folds_changed(); + self.notify(msg); + } + None => self.notify("nothing to open here"), + } + } + fn clamp(&mut self) { let n = self.src.len(); self.cursor = self.cursor.min(n.saturating_sub(1)); diff --git a/src/pager/motion.rs b/src/pager/motion.rs new file mode 100644 index 0000000..faa6c77 --- /dev/null +++ b/src/pager/motion.rs @@ -0,0 +1,158 @@ +//! Where the cursor goes: the motions themselves, and the framing that decides +//! what the viewport shows once it gets there. +//! +//! Split out of `input.rs` — which dispatches keys to them — to keep both files +//! under the size limit; these are the same `Pager` methods, one door down. +//! +//! Every motion here counts **rows**, including `j`/`k`: a row is the unit the +//! screen is made of, and a closed block is exactly one row, so a coarser +//! default could only ever skip rows that are on the screen. Blocks are a +//! *jump* — [`Pager::jump_heading`], `Tab`/`S-Tab` — and framing them is +//! [`Pager::frame_block`]. +#![deny(unsafe_code)] + +use super::Pager; +use crate::source::Anchor; + +impl Pager { + /// Land on a block and show the block, not just the row it starts on: the + /// minimum scroll that brings all of it on screen, or its first row at the + /// top when it is taller than the window. + /// + /// `Source::block_at` is asked for the extent rather than + /// `Source::next_landmark` a second time — the format has the arithmetic, + /// and the landmark pair cannot answer at the last block or from inside one + /// (`src/source/mod.rs` says why). `None` frames the landing row alone. + /// + /// Order matters: cursor, then `top`, then [`Pager::clamp`], which can only + /// pull `top` back to the end of the document — the cursor is inside + /// `top..top + h` in both branches, so nothing else it does can fire. + pub(super) fn frame_block(&mut self, row: usize) { + let n = self.len(); + if n == 0 { + return; + } + self.cursor = row.min(n - 1); + let h = self.content_rows().max(1); + let end = match self.src.block_at(self.cursor) { + Some(r) => r.end.max(self.cursor + 1), + None => self.cursor + 1, + }; + let last = (end - 1).min(n - 1).max(self.cursor); + match last - self.cursor >= h { + true => self.top = self.cursor, + false => { + if self.cursor < self.top { + self.top = self.cursor; + } + if last >= self.top + h { + self.top = last + 1 - h; + } + } + } + self.clamp(); + } + + /// `j` / `k`, and what every other vertical motion is counted in: rows. + /// [`Pager::clamp`] drags the window along once the cursor leaves it, so a + /// block taller than the viewport is walked a row at a time and the text + /// moves from the press that reaches the bottom of the window onward. + pub(super) fn move_cursor(&mut self, delta: isize) { + let n = self.len(); + if n == 0 { + return; + } + let max = n as isize - 1; + let next = (self.cursor as isize).saturating_add(delta).clamp(0, max); + self.cursor = next as usize; + self.clamp(); + } + + pub(super) fn goto(&mut self, row: usize) { + if self.len() == 0 { + return; + } + self.cursor = row.min(self.len() - 1); + self.clamp(); + } + + /// `h` / `l`. A format may scroll by its own unit — a CSV moves a whole + /// column, and its pinned header moves with the body because both are + /// painted at this one offset (SPEC.md §CSV). + /// + /// The format is told the *viewport* width, the same `cols` the painter + /// slices rows to and [`Pager::max_hoff`] measures against — not the layout + /// width, which `--width` can set far wider than the terminal. + pub(super) fn scroll_h(&mut self, delta: isize) { + let max = self.max_hoff(); + let hoff = self.hoff; + let view = self.cols.max(1); + let next = match self.src.hscroll(hoff, delta.signum(), view) { + Some(off) => off as isize, + None => (hoff as isize).saturating_add(delta), + }; + self.hoff = next.clamp(0, max as isize) as usize; + } + + /// `w`: let the format widen whatever is under the cursor. + pub(super) fn widen(&mut self) { + match self.src.widen() { + Some(msg) => { + self.notify(msg); + self.clamp(); + } + None => self.notify("nothing to widen here"), + } + } + + /// Move the cursor to a row that is already visible, scrolling it in. + pub(super) fn jump_to_row(&mut self, row: usize) { + self.cursor = row.min(self.len().saturating_sub(1)); + let h = self.content_rows().max(1); + if self.cursor < self.top || self.cursor >= self.top + h { + self.top = self.cursor.saturating_sub(h / 3); + } + self.clamp(); + } + + /// Move the cursor to a place in the document, unfolding whatever hides it. + pub(super) fn jump_to(&mut self, anchor: Anchor) { + match self.src.reveal(anchor) { + Some(row) => self.jump_to_row(row), + None => self.clamp(), + } + } + + /// `Tab` / `S-Tab`: the **fast jump**, one structural landmark at a time — + /// the next heading in prose, the next declaration in code, and under a + /// lens the next **block**: a message, a shut run of mechanics, or, inside + /// a run the reader has opened, one of its steps. + /// + /// Block rather than message, deliberately. A message *is* a block, so a + /// block jump reaches every message a message jump would; the reverse is + /// false, and a jump that skipped the runs would leave the mechanics — the + /// thing a trajectory is mostly made of — reachable only a row at a time. + /// It is also the general answer: one boundary, `Source::next_landmark`, + /// for every format, rather than a second one that exists under a lens. + /// + /// `S-Tab` is the exact mirror: the same landmark table walked backwards, + /// framed the same way, so `Tab` then `S-Tab` comes back to the block it + /// started on. + pub(super) fn jump_heading(&mut self, forward: bool) { + match self.src.next_landmark(self.cursor, forward) { + Some(row) => match self.src.blocks() { + true => self.frame_block(row), + false => self.goto(row), + }, + // Named after the unit this document is actually read in: a + // trajectory has no headings, and the status bar one line down is + // printing `block 96/≥181`. Same predicate as the framing above. + None => self.notify(match (self.src.blocks(), forward) { + (true, true) => "no further block", + (true, false) => "no previous block", + (false, true) => "no further heading", + (false, false) => "no previous heading", + }), + } + } +} diff --git a/src/pager/probe.rs b/src/pager/probe.rs index fcb55c3..43a4746 100644 --- a/src/pager/probe.rs +++ b/src/pager/probe.rs @@ -20,6 +20,19 @@ impl Pager { pub(crate) fn in_visual(&self) -> bool { self.select.is_some() } + /// Whether the document under the pager reads in blocks. + pub(crate) fn src_blocks(&self) -> bool { + self.src.blocks() + } + /// The rows of the block on `row`, as the source answers it — for the + /// framing tests, which have to know what they were framing. + pub(crate) fn src_block_at(&self, row: usize) -> Option> { + self.src.block_at(row) + } + /// The landmark `Tab` / `S-Tab` would jump to, as the source answers it. + pub(crate) fn src_next_landmark(&self, row: usize, forward: bool) -> Option { + self.src.next_landmark(row, forward) + } /// The text the status bar would show right now. pub(crate) fn status_line(&self) -> String { view::status_text(self) diff --git a/src/pager/tests/csv.rs b/src/pager/tests/csv.rs index 3808bcd..9645d2e 100644 --- a/src/pager/tests/csv.rs +++ b/src/pager/tests/csv.rs @@ -390,6 +390,29 @@ fn enter_opens_the_row_under_the_cursor_as_a_form() { assert_eq!(p.mode, Mode::Normal, "Esc closes it"); } +/// `r` is not one of the row detail's own keys, and taking it had to be free +/// in **every** mode rather than in the ones a lens uses. Inside the form it +/// falls through to the normal dispatcher, exactly as any unbound key there +/// does — the overlay stays up, the selected field does not move, and since a +/// CSV has no record to open the pager says so. Stated here so that binding `r` +/// inside this overlay one day is a failing test rather than a silent shadow. +#[test] +fn r_is_not_one_of_the_row_details_keys() { + let mut p = csv_pager(&csv_body(), 60, ROWS); + p.cursor = HEAD_ROWS; + key(&mut p, Key::Enter); + assert_eq!(p.mode, Mode::Detail); + let rows = p.line_count(); + press(&mut p, "j"); + let sel = p.detail_sel; + assert_eq!(sel, 1, "`j` is one of them"); + press(&mut p, "r"); + assert_eq!(p.mode, Mode::Detail, "the overlay is still up"); + assert_eq!(p.detail_sel, sel, "and `r` moved nothing in it"); + assert_eq!(p.line_count(), rows, "nor anything behind it"); + assert!(p.status_line().contains("nothing to open"), "{}", p.status_line()); +} + /// The fields a header-shaped grid cannot show are exactly what the form is /// for: they must be in it, and the grid must say so with the marker. #[test] diff --git a/src/pager/tests/lens.rs b/src/pager/tests/lens.rs new file mode 100644 index 0000000..4c0da57 --- /dev/null +++ b/src/pager/tests/lens.rs @@ -0,0 +1,579 @@ +//! The pager driving a trajectory read through `--lens`: the message under a +//! row, the two states it has, `r`, and what a resize does to the cursor. +//! +//! Pager tests, not format tests — everything below goes through key presses +//! and the rows the source hands back, which is the only way to prove that a +//! key (`za`, `r`) reaches the source at all. No terminal, no pty: the +//! reconstructed-frame trap does not apply because there is no frame to +//! reconstruct. +#![deny(unsafe_code)] + +use super::*; +use crate::source::jsonl::JsonlSource; + +/// Twelve lines of message, so a clip has something to clip. +fn long_message() -> String { + (1..=12) + .map(|n| format!("line {n} of what was said")) + .collect::>() + .join("\\n") +} + +/// A prompt with a long message, a tool call and its result, and a short +/// answer. Hand-written, in the shape a Claude Code session file has. +pub(super) fn run() -> String { + format!( + concat!( + r#"{{"type":"user","timestamp":"2026-08-05T14:01:00.000Z","#, + r#""message":{{"role":"user","content":"{}"}}}}"#, + "\n", + r#"{{"type":"assistant","timestamp":"2026-08-05T14:02:00.000Z","message":{{"role":"assistant","#, + r#""content":[{{"type":"tool_use","id":"t1","name":"Bash","input":{{"command":"git status"}}}}]}}}}"#, + "\n", + r#"{{"type":"user","timestamp":"2026-08-05T14:02:01.000Z","message":{{"role":"user","#, + r#""content":[{{"type":"tool_result","tool_use_id":"t1","content":"clean"}}]}}}}"#, + "\n", + r#"{{"type":"assistant","timestamp":"2026-08-05T14:03:00.000Z","message":{{"role":"assistant","#, + r#""content":[{{"type":"text","text":"Done."}}]}}}}"#, + "\n" + ), + long_message() + ) +} + +fn lens_pager(cols: usize, rows: usize) -> Pager { + let mut src = JsonlSource::from_bytes(run().into_bytes()); + src.set_lens(crate::lens::find("agent").expect("the agent lens")); + let mut p = Pager::new(Box::new(src), "session.jsonl".into(), cols, rows, None); + // The index and the lens are lazy; a frame is what pushes them along. + let _ = p.visible_text(); + p +} + +#[test] +fn a_message_is_read_under_its_row_and_the_clip_says_what_it_hid() { + let mut p = lens_pager(80, 40); + let rows = p.visible_text(); + // The summary row *is* the message's first line, and the rest is under it: + // six rows of message in all, then what the clip left out. + assert!(rows[0].contains("user 14:01"), "{:?}", rows[0]); + assert!(rows[0].ends_with("line 1 of what was said"), "{:?}", rows[0]); + assert_eq!(rows[1], "line 2 of what was said"); + assert_eq!(rows[5], "line 6 of what was said", "{rows:#?}"); + assert_eq!(rows[6], "\u{22ef} +6 lines", "the clip states what it left out"); + assert!( + !rows.iter().any(|r| r.contains("line 12")), + "a clipped body is clipped: {rows:#?}" + ); +} + +/// The defect: the row painted the excerpt and the body then repeated it. The +/// message's opening words are on the screen once. +#[test] +fn a_message_s_first_line_is_on_the_screen_exactly_once() { + let mut p = lens_pager(80, 40); + let rows = p.visible_text(); + let hits = rows.iter().filter(|r| r.contains("line 1 of what was said")).count(); + assert_eq!(hits, 1, "{rows:#?}"); + assert_ne!(rows[0].trim(), rows[1].trim(), "{rows:#?}"); + // And with the whole message open, still once. + press(&mut p, "za"); + let open = p.visible_text(); + let hits = open.iter().filter(|r| r.contains("line 1 of what was said")).count(); + assert_eq!(hits, 1, "{open:#?}"); +} + +/// The two levels (SPEC.md §Lenses): `za` toggles clipped and open, and the +/// raw JSON is not one of them. Three presses, and the record's own bytes are +/// on the screen at no point in the cycle — that is `r`'s job now. +#[test] +fn za_toggles_the_two_levels_and_never_reaches_the_tree() { + let mut p = lens_pager(80, 40); + let clipped = p.line_count(); + press(&mut p, "za"); + let open = p.visible_text(); + assert!(p.line_count() > clipped, "the body grew"); + assert!(open.iter().any(|r| r == "line 12 of what was said"), "{open:#?}"); + assert!(!open.iter().any(|r| r.starts_with('\u{22ef}')), "nothing left to say"); + assert!(!open.iter().any(|r| r.contains("\"type\"")), "no JSON: {open:#?}"); + + press(&mut p, "za"); + assert_eq!(p.line_count(), clipped, "and back to the clip in one press"); + let back = p.visible_text(); + assert!(back.iter().any(|r| r == "\u{22ef} +6 lines"), "{back:#?}"); + assert!(!back.iter().any(|r| r.contains("\"type\"")), "still no JSON: {back:#?}"); + + press(&mut p, "za"); + assert!(p.line_count() > clipped, "and open again"); + assert!( + !p.visible_text().iter().any(|r| r.contains("\"type\"")), + "three presses and the JSON never appeared: {:#?}", + p.visible_text() + ); +} + +/// `Enter` on a message row is the same key as `za` — it reaches the body +/// through `fold_here`, before the outline is consulted. +#[test] +fn enter_on_a_message_row_opens_its_body_too() { + let mut p = lens_pager(80, 40); + let clipped = p.line_count(); + key(&mut p, Key::Enter); + assert!(p.line_count() > clipped); +} + +/// The non-negotiable: whatever the body is doing, the record itself is one +/// keypress away, and it is the record — not a reading of it. `r` from the +/// clipped level, and `r` again to shut it. +#[test] +fn r_shows_the_raw_record_whatever_the_body_is_doing() { + let mut p = lens_pager(80, 40); + let before = p.line_count(); + press(&mut p, "r"); + let rows = p.visible_text(); + assert!(p.line_count() > before, "the tree was spliced in"); + assert!(rows.iter().any(|r| r.contains("\"type\"")), "{rows:#?}"); + assert!(rows.iter().any(|r| r.contains("timestamp")), "{rows:#?}"); + // And the body is still doing what it was doing. + assert!(rows.iter().any(|r| r == "\u{22ef} +6 lines"), "{rows:#?}"); + press(&mut p, "r"); + assert_eq!(p.line_count(), before, "and it shuts again"); + assert!(!p.visible_text().iter().any(|r| r.contains("\"type\""))); +} + +/// The same key from the **open** level, which is the other rung it has to +/// work from: the tree goes in over the whole message, and comes back out +/// leaving that message open. +#[test] +fn r_shows_the_raw_record_from_the_open_level_too() { + let mut p = lens_pager(80, 40); + press(&mut p, "za"); + let open = p.line_count(); + press(&mut p, "r"); + let with_tree = p.visible_text(); + assert!(p.line_count() > open, "the tree went in"); + assert!(with_tree.iter().any(|r| r.contains("\"type\"")), "{with_tree:#?}"); + assert!( + with_tree.iter().any(|r| r == "line 12 of what was said"), + "and the level is where it was: {with_tree:#?}" + ); + press(&mut p, "r"); + assert_eq!(p.line_count(), open, "shut, and the message still open"); + assert!(p.visible_text().iter().any(|r| r == "line 12 of what was said")); +} + +/// The documented answer to "what does `Enter` do while the tree is open": it +/// leaves the tree alone and toggles the record's own rows underneath it. `r` +/// owns the tree, and a key that silently undid another key's work is the thing +/// this change removed (SPEC.md §Lenses). +#[test] +fn enter_with_the_tree_open_leaves_the_tree_and_toggles_the_record() { + let mut p = lens_pager(80, 40); + press(&mut p, "r"); + let with_tree = p.line_count(); + press(&mut p, "za"); + let open = p.visible_text(); + assert!(p.line_count() > with_tree, "the message opened under it"); + assert!(open.iter().any(|r| r.contains("\"type\"")), "tree still there: {open:#?}"); + assert!(open.iter().any(|r| r == "line 12 of what was said"), "{open:#?}"); + press(&mut p, "za"); + assert_eq!(p.line_count(), with_tree, "back to the clip, tree untouched"); + assert!(p.visible_text().iter().any(|r| r.contains("\"type\"")), "and it is still open"); + // And `r` is what shuts it, from either level. + press(&mut p, "r"); + assert!(!p.visible_text().iter().any(|r| r.contains("\"type\""))); +} + +/// From inside a message, `r` opens the record that message came from. Two +/// presses of `j` is how the cursor gets to line 3 of the first message: `j` is +/// a row, so it walks into the body rather than over the message. +#[test] +fn r_from_a_body_row_opens_the_record_it_belongs_to() { + let mut p = lens_pager(80, 40); + let before = p.line_count(); + press(&mut p, "jj"); + assert_eq!(p.cursor_text(), "line 3 of what was said"); + press(&mut p, "r"); + assert!(p.line_count() > before, "{}", p.status_line()); + assert!(p.visible_text().iter().any(|r| r.contains("\"uuid\"") || r.contains("\"type\""))); +} + +/// On a folded run there is no one record to open, and the pager says so +/// rather than appearing to do nothing — `Enter` is the key that opens a run. +#[test] +fn r_on_a_run_says_there_is_nothing_to_open() { + let mut p = lens_pager(80, 40); + let rows = p.visible_text(); + let run = rows + .iter() + .position(|r| r.contains("\u{27e8}")) + .expect("a folded run"); + p.goto(run); + let before = p.line_count(); + press(&mut p, "r"); + assert_eq!(p.line_count(), before); + assert!(p.status_line().contains("nothing to open"), "{}", p.status_line()); +} + +/// Taking `r` had to be free everywhere, not just on a lens row. In the search +/// prompt it is a letter of the query. In the outline overlay it is none of the +/// overlay's own keys, so the selection does not move and it falls through to +/// the normal dispatcher — which is what every unbound key there has always +/// done, and what the chord it replaced did from the same place. A live +/// visual selection is +/// `Mode::Normal`, so there it opens the record and the selection follows the +/// cursor. +/// +/// The one other overlay that falls *through* is the CSV row detail, pinned in +/// `pager::tests::csv::r_is_not_one_of_the_row_details_keys`. The two that do +/// not need one swallow the key by construction rather than by binding it: +/// `Mode::Index` consumes every key it does not know (`Pager::index_key`'s +/// `_ => {}`, and its filter takes letters as query text), and `Mode::Help` +/// closes on any key at all. +#[test] +fn r_is_free_in_every_mode() { + let mut p = lens_pager(80, 40); + press(&mut p, "/"); + assert_eq!(p.mode, Mode::Search(Dir::Forward)); + let clipped = p.line_count(); + press(&mut p, "r"); + assert_eq!(p.query, "r", "the prompt takes it as a letter"); + assert_eq!(p.line_count(), clipped, "and no tree opened behind it"); + key(&mut p, Key::Esc); + assert_eq!(p.line_count(), clipped); + + p.mode = Mode::Outline; + p.outline_sel = 0; + press(&mut p, "r"); + assert_eq!(p.outline_sel, 0, "the overlay has no `r` of its own"); + assert!(p.line_count() > clipped, "it fell through, as any unbound key does"); + press(&mut p, "r"); + assert_eq!(p.line_count(), clipped, "and the second press shut it again"); + p.mode = Mode::Normal; + + press(&mut p, "v"); + assert!(p.select.is_some(), "visual mode"); + press(&mut p, "r"); + assert!(p.line_count() > clipped, "and there it is the raw record"); + assert!(p.select.is_some(), "the selection survives it"); +} + +/// A resize re-wraps every body, so rows move. The cursor stays on the record +/// it was on — which is what a `Mark` means here, and why it is a record +/// rather than a row. +#[test] +fn a_resize_keeps_the_cursor_on_the_same_record() { + let mut p = lens_pager(80, 40); + // Open the first message, so the wrap really does move what is below it: + // a *clipped* body is six rows at any width. + press(&mut p, "za"); + // Down to the answer at the end: it is one line, so `G` lands on its own + // summary row, which is the whole of it. + press(&mut p, "G"); + let _ = p.visible_text(); + let before = p.cursor_mark(); + let row = p.cursor; + let said = p.cursor_text(); + assert!(said.contains("Done."), "{said:?}"); + p.resize(40, 40); + let _ = p.visible_text(); + assert_eq!(p.cursor_mark(), before, "the mark is the record, not the row"); + assert_ne!(p.cursor, row, "the narrower body pushed the row down"); + assert_eq!(p.cursor_text(), said, "and the cursor is on the same content"); +} + +/// What a record-valued mark costs, stated: the cursor inside a message comes +/// back to that message, not to the line of it that it was on. A `Mark` is one +/// number, and it names the record. +#[test] +fn a_resize_from_inside_a_body_lands_on_its_summary_row() { + let mut p = lens_pager(80, 40); + press(&mut p, "jjj"); + let _ = p.visible_text(); + assert_eq!(p.cursor_text(), "line 4 of what was said"); + let mark = p.cursor_mark(); + p.resize(40, 40); + let _ = p.visible_text(); + assert_eq!(p.cursor_mark(), mark, "still the same record"); + assert!(p.cursor_text().contains("user 14:01"), "{:?}", p.cursor_text()); +} + +/// `zR` shows everything the viewport has reached, messages included — which +/// is also what a dump asks for — and `zM` puts every one of them back. +#[test] +fn zr_opens_the_bodies_and_zm_clips_them_again() { + let mut p = lens_pager(80, 40); + let clipped = p.line_count(); + press(&mut p, "zR"); + let open = p.visible_text(); + assert!(open.iter().any(|r| r == "line 12 of what was said"), "{open:#?}"); + press(&mut p, "zM"); + assert_eq!(p.line_count(), clipped); +} + +// -- moving ------------------------------------------------------------------ +// +// SPEC.md §"Moving through a document": `j`/`k` move one visible row, here as +// everywhere; `Tab`/`S-Tab` jump block to block and frame what they land on. +// Every assertion below is the pager's own state after a synthetic key press — +// there is no frame here to reconstruct, and none of these read one. + +/// Two long messages, so a block reached by `j` is taller than one row and +/// framing has something to do. +fn two_long_messages() -> String { + format!( + concat!( + r#"{{"type":"user","timestamp":"2026-08-05T14:01:00.000Z","#, + r#""message":{{"role":"user","content":"{}"}}}}"#, + "\n", + r#"{{"type":"assistant","timestamp":"2026-08-05T14:02:00.000Z","message":{{"role":"assistant","#, + r#""content":[{{"type":"text","text":"{}"}}]}}}}"#, + "\n" + ), + long_message(), + long_message() + ) +} + +fn pager_over(text: String, cols: usize, rows: usize) -> Pager { + let mut src = JsonlSource::from_bytes(text.into_bytes()); + src.set_lens(crate::lens::find("agent").expect("the agent lens")); + let mut p = Pager::new(Box::new(src), "session.jsonl".into(), cols, rows, None); + let _ = p.visible_text(); + p +} + +/// The user's first complaint, in a deterministic test: `j` under a lens walks +/// the message's own lines — the clipped body and the row that says what the +/// clip left out — before it ever reaches the run below. `k` mirrors it row for +/// row, and both stop at the ends rather than wrapping or stalling early. +#[test] +fn j_and_k_move_one_row_under_a_lens() { + let mut p = lens_pager(80, 40); + assert_eq!(p.cursor, 0); + assert!(p.cursor_text().contains("line 1 of what was said")); + for line in 2..=6 { + press(&mut p, "j"); + assert_eq!(p.cursor_text(), format!("line {line} of what was said")); + } + press(&mut p, "j"); + assert_eq!(p.cursor_text(), "\u{22ef} +6 lines", "then the clip's own row"); + press(&mut p, "j"); + assert!(p.cursor_text().contains("\u{27e8}"), "the folded run: {:?}", p.cursor_text()); + press(&mut p, "j"); + assert!(p.cursor_text().contains("Done."), "{:?}", p.cursor_text()); + let last = p.cursor; + assert_eq!(last, p.line_count() - 1, "which is the last row of the document"); + press(&mut p, "j"); + assert_eq!(p.cursor, last, "and `j` there stays put rather than running off"); + for row in (0..last).rev() { + press(&mut p, "k"); + assert_eq!(p.cursor, row, "`k` is the mirror, a row at a time"); + } + press(&mut p, "k"); + assert_eq!(p.cursor, 0, "and `k` on the first row stays put"); +} + +/// The invariant the reader actually depends on, over a document with a record +/// opened inside it: from the top, `j` alone reaches **every** painted row, in +/// order, and stops on the last one. Nothing on the screen needs a key that no +/// longer exists. +#[test] +fn every_painted_row_is_reachable_with_j_alone() { + let mut p = lens_pager(80, 40); + // The whole ladder open: the message in full, its parts, and the raw tree + // spliced under it — everything the reader can put on the screen at once. + press(&mut p, "zR"); + let _ = p.visible_text(); + let n = p.line_count(); + assert!(n > 20, "a full record is on the screen: {n} rows"); + let mut seen = vec![p.cursor]; + for _ in 0..n * 2 { + let before = p.cursor; + press(&mut p, "j"); + assert!(p.cursor <= before + 1, "one row at a time: {before} -> {}", p.cursor); + if p.cursor != before { + seen.push(p.cursor); + } + } + assert_eq!(seen, (0..n).collect::>(), "every row was reachable by j"); + for _ in 0..n * 2 { + press(&mut p, "k"); + } + assert_eq!(p.cursor, 0, "and k walks back to the top"); +} + +/// `j` inside a block taller than the viewport keeps the reader moving: the +/// cursor walks its rows and the window follows once the cursor reaches the +/// bottom of it, so the text is never frozen for a whole screen. +#[test] +fn j_walks_a_block_taller_than_the_viewport_and_the_window_follows() { + let mut p = pager_over(two_long_messages(), 80, 6); + key(&mut p, Key::Tab); + let h = p.content_rows(); + let block = p.src_block_at(p.cursor).expect("the second block"); + assert!(block.end - block.start > h, "the block is taller: {block:?} in {h}"); + let (start, top) = (p.cursor, p.top); + for i in 1..h { + press(&mut p, "j"); + assert_eq!(p.cursor, start + i, "a row a press"); + assert_eq!(p.top, top, "still inside the window"); + } + press(&mut p, "j"); + assert_eq!(p.top, top + 1, "and now the window follows the cursor down"); +} + +/// A block that fits the viewport is scrolled fully into view — not merely far +/// enough to show the row `Tab` landed on, which is all the cursor clamp does. +#[test] +fn landing_on_a_block_that_fits_brings_all_of_it_on_screen() { + let mut p = pager_over(two_long_messages(), 80, 9); + let h = p.content_rows(); + assert_eq!((p.cursor, p.top), (0, 0)); + key(&mut p, Key::Tab); + let block = p.src_block_at(p.cursor).expect("the second block"); + assert!(block.end - block.start <= h, "the block fits: {block:?} in {h}"); + assert!(p.top <= block.start, "the block starts on screen: {block:?}, top {}", p.top); + assert!( + block.end <= p.top + h, + "and ends on it: {block:?}, top {} + {h}", + p.top + ); + assert!(p.top > 0, "which took a scroll the cursor alone would not have"); +} + +/// A block taller than the viewport cannot be shown whole, so its first row +/// goes to the top and the reader starts at the beginning of what was said. +#[test] +fn landing_on_a_block_taller_than_the_viewport_puts_its_first_row_at_the_top() { + let mut p = pager_over(two_long_messages(), 80, 6); + let h = p.content_rows(); + key(&mut p, Key::Tab); + let block = p.src_block_at(p.cursor).expect("the second block"); + assert!(block.end - block.start > h, "the block is taller: {block:?} in {h}"); + assert_eq!(p.cursor, block.start); + assert_eq!(p.top, block.start, "its first row is the top of the screen"); +} + +/// `Tab` is the fast jump: block to block, over the message's body and over +/// the folded run alike, and `S-Tab` is its exact mirror back up the same +/// sequence. It stops at the end rather than running past it. +#[test] +fn tab_jumps_block_to_block_and_s_tab_mirrors_it() { + let mut p = lens_pager(80, 40); + let mut down = Vec::new(); + for _ in 0..2 { + key(&mut p, Key::Tab); + down.push(p.cursor); + } + assert!(p.cursor_text().contains("Done."), "{:?}", p.cursor_text()); + // Two presses crossed the whole message, body and clip row and all, and + // then the folded run: one block a press. + assert_eq!(down.len(), 2); + let last = p.cursor; + key(&mut p, Key::Tab); + assert_eq!(p.cursor, last, "and Tab at the last block stays on it"); + assert!(p.cursor < p.line_count()); + let mut up = Vec::new(); + for _ in 0..2 { + key(&mut p, Key::BackTab); + up.push(p.cursor); + } + assert_eq!(up, vec![down[0], 0], "S-Tab retraces Tab exactly: down {down:?}"); +} + +/// When the jump runs out it says so in the unit this document is read in. +/// A trajectory has no headings, and the status bar on the same screen is +/// printing `block n/N`, so the exhaustion message says **block** too. +#[test] +fn the_exhausted_jump_says_block_under_a_lens() { + let mut p = lens_pager(80, 40); + key(&mut p, Key::BackTab); + assert_eq!(p.message.as_deref(), Some("no previous block"), "at the top"); + for _ in 0..3 { + key(&mut p, Key::Tab); + } + assert_eq!(p.message.as_deref(), Some("no further block"), "at the end"); +} + +/// The status bar says which block the cursor is on, in the same shape and the +/// same vocabulary as the record it already counted. +#[test] +fn the_status_bar_names_the_block() { + let mut p = lens_pager(80, 40); + assert!(p.status_line().contains("block 1/3"), "{}", p.status_line()); + press(&mut p, "j"); + assert!(p.status_line().contains("block 1/3"), "a body row is in its block"); + key(&mut p, Key::Tab); + assert!(p.status_line().contains("block 2/3"), "{}", p.status_line()); + key(&mut p, Key::Tab); + let last = p.status_line(); + assert!(last.contains("agent"), "{last}"); + assert!(last.contains("record 4/4"), "{last}"); + assert!(last.contains("block 3/3"), "{last}"); +} + +/// The half of "`Enter` never opens a tree and never shuts one" that the +/// headline of a record *with* a rung cannot prove. `Done.` fits its row and +/// made no calls, so it has no rung at all — and a record with no rung is +/// exactly where the key used to fall through to the outline, whose entry for a +/// record is its raw tree. With the tree `r` opened, `Enter` there leaves it +/// alone; shut, `Enter` does not open one. +#[test] +fn enter_on_a_record_with_no_rung_leaves_its_tree_alone() { + let mut p = lens_pager(80, 40); + press(&mut p, "G"); + assert!(p.cursor_text().contains("Done."), "{:?}", p.cursor_text()); + let shut = p.line_count(); + key(&mut p, Key::Enter); + assert_eq!(p.line_count(), shut, "no rung, and no tree: `Enter` did nothing"); + assert!(!p.visible_text().iter().any(|r| r.contains("\"type\"")), "and no JSON"); + press(&mut p, "za"); + assert_eq!(p.line_count(), shut, "`za` is the same key and the same answer"); + + press(&mut p, "r"); + let with_tree = p.line_count(); + assert!(with_tree > shut, "`r` is what opens it"); + key(&mut p, Key::Enter); + assert_eq!(p.line_count(), with_tree, "and `Enter` leaves it exactly where it is"); + assert!(p.visible_text().iter().any(|r| r.contains("\"type\"")), "tree still there"); + press(&mut p, "r"); + assert_eq!(p.line_count(), shut, "the `r` that opened it is the way out"); +} + +/// The other row the fall-through reached: a row **inside** an open tree. It is +/// a row of the record, so the key is the record's — and this record has no +/// rung, so nothing moves. What it must not do is shut the tree the cursor is +/// standing in. +#[test] +fn enter_inside_an_open_tree_does_not_shut_it() { + let mut p = lens_pager(80, 40); + press(&mut p, "G"); + press(&mut p, "r"); + let with_tree = p.line_count(); + press(&mut p, "j"); + assert!(p.cursor_text().contains('"'), "a row of the tree: {:?}", p.cursor_text()); + key(&mut p, Key::Enter); + assert_eq!(p.line_count(), with_tree, "the tree the cursor is in survives"); + press(&mut p, "za"); + assert_eq!(p.line_count(), with_tree); + assert!(p.visible_text().iter().any(|r| r.contains("\"type\""))); +} + +/// `zR` puts every record at the open level, rungless ones included. `Enter` +/// there is still not a rung: it neither repaints the same rows and calls it a +/// descent, nor — on the press after — falls through and toggles the JSON `zR` +/// had opened. +#[test] +fn enter_on_a_record_with_no_rung_after_zr_is_inert_every_press() { + let mut p = lens_pager(80, 40); + press(&mut p, "zR"); + let rows = p.visible_text(); + let done = rows.iter().position(|r| r.contains("Done.")).expect("the answer"); + p.goto(done); + let open = p.line_count(); + for press_no in 0..3 { + key(&mut p, Key::Enter); + assert_eq!(p.line_count(), open, "press {press_no} moved rows"); + } +} diff --git a/src/pager/tests/mod.rs b/src/pager/tests/mod.rs index df77dce..28bef15 100644 --- a/src/pager/tests/mod.rs +++ b/src/pager/tests/mod.rs @@ -2,7 +2,7 @@ //! driven with synthetic key events and inspected through its state. #![deny(unsafe_code)] -use super::keys::Action; +use super::keys::{self, Action}; use super::search::Dir; use super::{Mode, Pager}; use crate::key::{Key, KeyEvent}; @@ -53,6 +53,8 @@ fn heading_ids(p: &Pager) -> Vec { } mod csv; +mod lens; +mod openrun; mod paint; // -- scrolling --------------------------------------------------------------- @@ -298,8 +300,9 @@ fn help_overlay_opens_and_closes() { assert_eq!(p.mode, Mode::Help); let mut f = Frame::new(true); p.paint(&mut f); - assert!(f.as_str().contains("line down"), "frame was: {}", f.as_str()); - for _ in 0..30 { + assert!(f.as_str().contains("one row down"), "frame was: {}", f.as_str()); + // Far enough to reach the bottom of the table, whatever it has grown to. + for _ in 0..keys::BINDINGS.len() { press(&mut p, "j"); } let mut f = Frame::new(true); @@ -474,3 +477,21 @@ fn y_on_a_metadata_row_copies_the_field() { let y = p.peek_yank().expect("a field yank"); assert_eq!(y.text, "models/A.md\n"); } + +/// Prose has no blocks, and `j`/`k` are one rendered line — which is what they +/// are in every format, blocks or none. `Tab` is the other unit, and here it is +/// what it always was: the next heading. +#[test] +fn j_and_k_move_one_row_on_a_document_with_no_blocks() { + let mut p = pager(DOC, 60, 20); + assert!(!p.src_blocks()); + for expected in 1..6 { + press(&mut p, "j"); + assert_eq!(p.cursor, expected); + } + press(&mut p, "k"); + assert_eq!(p.cursor, 4); + let heading = p.src_next_landmark(4, true).expect("a next heading"); + key(&mut p, Key::Tab); + assert_eq!(p.cursor, heading, "and Tab is the heading jump, not a row"); +} diff --git a/src/pager/tests/openrun.rs b/src/pager/tests/openrun.rs new file mode 100644 index 0000000..6c37abd --- /dev/null +++ b/src/pager/tests/openrun.rs @@ -0,0 +1,238 @@ +//! Reading **a run the reader has opened** (SPEC.md §Lenses). +//! +//! The defect these pin: `j` moved by block, so it stepped over the very steps +//! `Enter` had just revealed. `j` is a row now, in every format, so it walks +//! whatever opening a run put on the screen — and `k` mirrors it, including +//! stepping back out of the run to the row above it. `Tab` is the block jump, +//! and it is the key that clears a member whole. +//! +//! Pager tests: every assertion is the pager's own state after a synthetic key +//! press. There is no terminal and no frame here to reconstruct. +#![deny(unsafe_code)] + +use super::*; +use crate::source::jsonl::JsonlSource; + +/// A prompt, four mechanical records that fold into one run, and an answer. +/// Hand-written, in the shape a Claude Code session file has. +fn run_of_four() -> String { + concat!( + r#"{"type":"user","timestamp":"2026-08-05T14:01:00.000Z","#, + r#""message":{"role":"user","content":"add a lens"}}"#, + "\n", + r#"{"type":"assistant","timestamp":"2026-08-05T14:02:00.000Z","message":{"role":"assistant","#, + r#""content":[{"type":"tool_use","id":"t1","name":"Bash","input":{"command":"git status"}}]}}"#, + "\n", + r#"{"type":"user","timestamp":"2026-08-05T14:02:01.000Z","message":{"role":"user","#, + r#""content":[{"type":"tool_result","tool_use_id":"t1","content":"clean"}]}}"#, + "\n", + r#"{"type":"assistant","timestamp":"2026-08-05T14:02:20.000Z","message":{"role":"assistant","#, + r#""content":[{"type":"tool_use","id":"t2","name":"Read","input":{"file_path":"src/lens/mod.rs"}}]}}"#, + "\n", + r#"{"type":"user","timestamp":"2026-08-05T14:02:21.000Z","message":{"role":"user","#, + r#""content":[{"type":"tool_result","tool_use_id":"t2","content":"a\nb"}]}}"#, + "\n", + r#"{"type":"assistant","timestamp":"2026-08-05T14:03:00.000Z","message":{"role":"assistant","#, + r#""content":[{"type":"text","text":"Done."}]}}"#, + "\n" + ) + .to_string() +} + +/// The pager, with the run under the cursor already open: `j` onto it, `Enter`. +fn with_open_run(rows: usize) -> Pager { + let mut src = JsonlSource::from_bytes(run_of_four().into_bytes()); + src.set_lens(crate::lens::find("agent").expect("the agent lens")); + let mut p = Pager::new(Box::new(src), "session.jsonl".into(), 80, rows, None); + let _ = p.visible_text(); + press(&mut p, "j"); + assert!(p.cursor_text().contains('\u{27e8}'), "on the run: {:?}", p.cursor_text()); + key(&mut p, Key::Enter); + p +} + +/// The user's own sequence: `j` onto the run, `Enter`, and the next `j` lands +/// on the **first step inside it** rather than on the block after the run. +#[test] +fn j_after_enter_lands_inside_the_run() { + let mut p = with_open_run(40); + let run = p.cursor; + press(&mut p, "j"); + assert_eq!(p.cursor, run + 1, "the first step, not the block after the run"); + assert!(p.cursor_text().contains("Bash(git status)"), "{:?}", p.cursor_text()); +} + +/// And it keeps going: every member in order, then out of the run onto the +/// block after it. +#[test] +fn j_visits_every_member_then_leaves_the_run() { + let mut p = with_open_run(40); + let mut seen = Vec::new(); + for _ in 0..5 { + press(&mut p, "j"); + seen.push(p.cursor_text()); + } + assert!(seen[0].contains("Bash(git status)"), "{seen:#?}"); + assert!(seen[1].contains("Bash \u{2192}"), "{seen:#?}"); + assert!(seen[2].contains("Read(src/lens/mod.rs)"), "{seen:#?}"); + assert!(seen[3].contains("Read \u{2192}"), "{seen:#?}"); + assert!(seen[4].contains("Done."), "out of the run: {seen:#?}"); +} + +/// `k` mirrors `j` step for step over the same rows, out of the run and on up. +#[test] +fn k_mirrors_j_over_an_open_run() { + let mut p = with_open_run(40); + let start = p.cursor; + let mut down = Vec::new(); + for _ in 0..5 { + press(&mut p, "j"); + down.push(p.cursor); + } + let mut up = Vec::new(); + for _ in 0..5 { + press(&mut p, "k"); + up.push(p.cursor); + } + let mut want: Vec = down[..down.len() - 1].to_vec(); + want.reverse(); + want.push(start); + assert_eq!(up, want, "down {down:?}"); + press(&mut p, "k"); + assert_eq!(p.cursor, 0, "and out of the run to the block above it"); +} + +/// The user's second complaint, in a deterministic test: with a step's raw JSON +/// open, `j` walks **its** rows. A member and its tree are still one block — +/// that is what `Tab` clears in a press — but the cursor unit is the row, so +/// nothing that was just revealed is skipped. +#[test] +fn j_reads_the_tree_a_member_has_open_and_tab_clears_it() { + let mut p = with_open_run(60); + press(&mut p, "j"); + let member = p.cursor; + // `r` is what opens a step's raw record: a step has no message to unfold, + // so `Enter` on one is not the key that gives it a tree. + press(&mut p, "r"); + let block = p.src_block_at(member).expect("the member's block"); + assert_eq!(block.start, member); + assert!(block.end > member + 1, "the tree is in it: {block:?}"); + assert_eq!( + p.src_block_at(block.end - 1), + Some(block.clone()), + "its last tree row too" + ); + // Every row of that tree, one press each, in order. + for row in member + 1..block.end { + press(&mut p, "j"); + assert_eq!(p.cursor, row, "j walks the tree row by row"); + } + press(&mut p, "k"); + assert_eq!(p.cursor, block.end - 2, "and k is its mirror, a row at a time"); + // The block jump is what steps over the whole of it, in either direction. + key(&mut p, Key::Tab); + assert_eq!(p.cursor, block.end, "one Tab clears the whole member"); + key(&mut p, Key::BackTab); + assert_eq!(p.cursor, member, "and one S-Tab comes back to it"); +} + +/// The status bar counts what `Tab` steps by: opening the run adds its members +/// to the block total rather than letting the counter drift. +#[test] +fn the_block_counter_follows_the_open_run() { + let mut src = JsonlSource::from_bytes(run_of_four().into_bytes()); + src.set_lens(crate::lens::find("agent").expect("the agent lens")); + let mut p = Pager::new(Box::new(src), "session.jsonl".into(), 80, 40, None); + let _ = p.visible_text(); + press(&mut p, "j"); + assert!(p.status_line().contains("block 2/3"), "{}", p.status_line()); + key(&mut p, Key::Enter); + assert!(p.status_line().contains("block 2/7"), "{}", p.status_line()); + press(&mut p, "j"); + assert!(p.status_line().contains("block 3/7"), "{}", p.status_line()); + press(&mut p, "jjjj"); + assert!(p.cursor_text().contains("Done."), "{:?}", p.cursor_text()); + assert!(p.status_line().contains("block 7/7"), "{}", p.status_line()); +} + +/// Shutting the run puts it back to one block, and the cursor is left on the +/// run's own row — the thing it just closed, not a row that no longer exists. +#[test] +fn closing_the_run_collapses_it_back_to_one_block() { + let mut p = with_open_run(40); + let run = p.cursor; + let open_rows = p.line_count(); + key(&mut p, Key::Enter); + assert_eq!(p.cursor, run); + assert!(p.cursor_text().contains('\u{27e8}'), "{:?}", p.cursor_text()); + assert_eq!(p.line_count(), open_rows - 4, "the four steps are folded again"); + assert_eq!(p.src_block_at(run), Some(run..run + 1), "one block again"); + assert!(p.status_line().contains("block 2/3"), "{}", p.status_line()); + key(&mut p, Key::Tab); + assert!(p.cursor_text().contains("Done."), "and Tab steps over it again"); +} + +/// A prompt, an answer, and then four mechanical records with nothing after +/// them — a trajectory that ends in a run, which is where `Tab` runs out of +/// messages. +fn run_at_the_tail() -> String { + concat!( + r#"{"type":"user","timestamp":"2026-08-05T14:01:00.000Z","#, + r#""message":{"role":"user","content":"add a lens"}}"#, + "\n", + r#"{"type":"assistant","timestamp":"2026-08-05T14:01:30.000Z","message":{"role":"assistant","#, + r#""content":[{"type":"text","text":"Done."}]}}"#, + "\n", + r#"{"type":"assistant","timestamp":"2026-08-05T14:02:00.000Z","message":{"role":"assistant","#, + r#""content":[{"type":"tool_use","id":"t1","name":"Bash","input":{"command":"git status"}}]}}"#, + "\n", + r#"{"type":"user","timestamp":"2026-08-05T14:02:01.000Z","message":{"role":"user","#, + r#""content":[{"type":"tool_result","tool_use_id":"t1","content":"clean"}]}}"#, + "\n", + r#"{"type":"assistant","timestamp":"2026-08-05T14:02:20.000Z","message":{"role":"assistant","#, + r#""content":[{"type":"tool_use","id":"t2","name":"Read","input":{"file_path":"src/lens/mod.rs"}}]}}"#, + "\n", + r#"{"type":"user","timestamp":"2026-08-05T14:02:21.000Z","message":{"role":"user","#, + r#""content":[{"type":"tool_result","tool_use_id":"t2","content":"a\nb"}]}}"#, + "\n" + ) + .to_string() +} + +/// `Tab` is the block jump, so at the tail of a document — past the last thing +/// anyone said — it keeps going through the mechanics rather than dead-ending, +/// and it descends into a run that is open. That is the whole argument for +/// block over message: nothing a document has is out of the jump's reach. +#[test] +fn tab_steps_through_a_trailing_run_that_is_open() { + let mut src = JsonlSource::from_bytes(run_at_the_tail().into_bytes()); + src.set_lens(crate::lens::find("agent").expect("the agent lens")); + let mut p = Pager::new(Box::new(src), "session.jsonl".into(), 80, 40, None); + let _ = p.visible_text(); + key(&mut p, Key::Tab); + assert!(p.cursor_text().contains("Done."), "the last message: {:?}", p.cursor_text()); + key(&mut p, Key::Tab); + let run = p.cursor; + assert!(p.cursor_text().contains('\u{27e8}'), "the trailing run: {:?}", p.cursor_text()); + key(&mut p, Key::Enter); + key(&mut p, Key::Tab); + assert_eq!(p.cursor, run + 1, "into the open run rather than nowhere"); + assert!(p.cursor_text().contains("Bash(git status)"), "{:?}", p.cursor_text()); +} + +/// The invariant under everything above: `j` never moves backwards and never +/// stalls while rows remain, so every row of an open run stays reachable. +#[test] +fn j_is_monotone_and_reaches_the_last_row() { + let mut p = with_open_run(40); + press(&mut p, "g"); + assert_eq!(p.cursor, 0, "back to the top"); + let n = p.line_count(); + let mut at = p.cursor; + for _ in 0..n * 2 { + press(&mut p, "j"); + assert!(p.cursor > at || p.cursor == n - 1, "stalled at {at} of {n}"); + at = p.cursor; + } + assert_eq!(at, n - 1, "the last row is reached"); +} diff --git a/src/pager/tests/paint.rs b/src/pager/tests/paint.rs index 642d4f0..54b8310 100644 --- a/src/pager/tests/paint.rs +++ b/src/pager/tests/paint.rs @@ -45,6 +45,61 @@ fn a_folded_heading_shows_a_line_count() { assert!(f.as_str().contains(crate::theme::MARKER_CLOSED)); } +/// The first painted row, with the escapes taken out — what the reader sees +/// on the top line. +fn top_row(p: &mut Pager) -> String { + let mut f = Frame::new(true); + p.paint(&mut f); + let s = f.as_str(); + let start = s.find("\u{1b}[1;1H").map(|i| i + 6).unwrap_or(0); + let rest = &s[start..]; + let end = rest.find('\u{1b}').unwrap_or(rest.len()); + rest[..end].to_string() +} + +/// The count is why the row is folded, so a headline that fills the width is +/// cut to make room for it rather than pushing it off the right edge — where +/// only a sideways scroll would have found it. +#[test] +fn a_long_folded_heading_keeps_its_line_count_on_screen() { + let long = "# ".to_string() + &"heading ".repeat(20) + "\n\nbody\nbody\nbody\n"; + let mut p = pager(&long, 40, 20); + key(&mut p, Key::Tab); + press(&mut p, "zc"); + let head = top_row(&mut p); + assert!(head.contains("lines)"), "the count is on the row: {head:?}"); + assert!( + crate::render::str_width(&head) <= 40, + "and inside the width: {head:?}" + ); +} + +/// What the cut does and does not do, at the one function that decides it — +/// a row that scrolls (a lens headline) rather than wraps (a heading). +#[test] +fn the_count_is_kept_and_the_text_gives_way() { + use crate::pager::view::with_count; + use crate::render::Span; + let row = || vec![Span::plain("x".repeat(60))]; + let note = " (49 lines)".to_string(); + + // At rest: cut to leave room, and the whole row fits the width. + let cut = with_count(row(), note.clone(), 40, 0); + let text: String = cut.iter().map(|s| s.text.as_str()).collect(); + assert!(text.ends_with("(49 lines)"), "{text:?}"); + assert!(crate::render::str_width(&text) <= 40, "{text:?}"); + + // Scrolled: the reader is reading the text, so nothing is cut for them. + let scrolled = with_count(row(), note.clone(), 40, 5); + let text: String = scrolled.iter().map(|s| s.text.as_str()).collect(); + assert_eq!(crate::render::str_width(&text), 60 + 12, "{text:?}"); + + // A row that already fits is untouched but for the count. + let short = with_count(vec![Span::plain("short")], note, 40, 0); + let text: String = short.iter().map(|s| s.text.as_str()).collect(); + assert_eq!(text, "short (49 lines)"); +} + /// The focused link keeps its own colour, so which link leaves the reader is /// visible even while it is the one under focus (SPEC.md §Navigation). #[test] @@ -91,6 +146,27 @@ fn quit_sets_the_flag() { #[test] fn every_action_is_reachable_and_survives_being_fired_blind() { let mut p = pager(DOC, 40, 6); + fire_every_action(&mut p); + assert!(!p.should_quit()); +} + +/// The same barrage against a document that reads in **blocks**, where `j`/`k` +/// and `Tab` run framing code markdown never reaches. A tiny viewport is the +/// point: the arithmetic that brings a block on screen is where an off-by-one +/// panics. +#[test] +fn every_action_survives_being_fired_blind_at_a_lensed_document() { + let mut src = crate::source::jsonl::JsonlSource::from_bytes(super::lens::run().into_bytes()); + src.set_lens(crate::lens::find("agent").expect("the agent lens")); + let mut p = Pager::new(Box::new(src), "session.jsonl".into(), 24, 3, None); + let _ = p.visible_text(); + fire_every_action(&mut p); + fire_every_action(&mut p); + assert!(!p.should_quit()); + assert!(p.cursor < p.line_count()); +} + +fn fire_every_action(p: &mut Pager) { for a in [ Action::LineDown, Action::LineUp, @@ -121,7 +197,6 @@ fn every_action_is_reachable_and_survives_being_fired_blind() { let mut f = Frame::new(true); p.paint(&mut f); } - assert!(!p.should_quit()); } #[test] diff --git a/src/pager/view.rs b/src/pager/view.rs index 5ef3a01..08d46b5 100644 --- a/src/pager/view.rs +++ b/src/pager/view.rs @@ -82,7 +82,7 @@ fn compose(p: &Pager, row: usize, line: &Line) -> Vec { let off = if scroll { p.hoff } else { 0 }; // Borrowed unless a fold summary rewrote the row: cloning the spans would // cost a copy of every cell of a 10k-column CSV row, on every frame. - let summary = fold_summary(p, row, line); + let summary = fold_summary(p, row, line, width, off); let base: &[Span] = summary.as_deref().unwrap_or(&line.spans); let full_width = base.iter().map(Span::width).sum::(); let mut cells = Cells::from_spans(&slice_spans(base, off, width)); @@ -113,7 +113,13 @@ fn selected(p: &Pager, row: usize) -> bool { } /// A folded section shows `\u{25b8} Title (N lines)`. -fn fold_summary(p: &Pager, row: usize, line: &Line) -> Option> { +fn fold_summary( + p: &Pager, + row: usize, + line: &Line, + width: usize, + off: usize, +) -> Option> { let hidden = p.src.hidden_at(row)?; let mut spans = line.spans.clone(); if let Some(first) = spans.first_mut() { @@ -127,13 +133,44 @@ fn fold_summary(p: &Pager, row: usize, line: &Line) -> Option> { return Some(spans); } // A source may describe what it hides better than a line count can. - if let Some(note) = p.src.fold_note(row) { - spans.push(Span::new(format!(" \u{b7} {note}"), theme::muted())); - return Some(spans); + let note = match p.src.fold_note(row) { + Some(note) => format!(" \u{b7} {note}"), + None => { + let unit = if hidden == 1 { "line" } else { "lines" }; + format!(" ({hidden} {unit})") + } + }; + Some(with_count(spans, note, width, off)) +} + +/// The row plus what it hides, with the text cut short when the two together +/// do not fit. +/// +/// The count is the reason the row is folded, so it is the half that must +/// survive: a headline that filled the width used to push its `(49 lines)` +/// off the right edge, where only a sideways scroll would find it. The text +/// is cut to leave room and marked with [`CUT_RIGHT`], and because the *line* +/// is still the full one, scrolling right reads the rest of it as before — +/// this trims what is painted at rest, never what the row holds. +pub(super) fn with_count(spans: Vec, note: String, width: usize, off: usize) -> Vec { + let text: usize = spans.iter().map(Span::width).sum(); + let note_width = crate::render::str_width(¬e); + // Scrolled sideways the reader is reading the text itself, and a cut that + // moved with the window would fight them for the columns. + if off > 0 || text + note_width <= width { + let mut out = spans; + out.push(Span::new(note, theme::muted())); + return out; + } + // One column for the cut mark. A width with no room for both keeps the + // count: a row that says only `(49 lines)` is still true. + let keep = width.saturating_sub(note_width + 1); + let mut out = crate::render::slice_spans(&spans, 0, keep); + if keep > 0 { + out.push(Span::new(CUT_RIGHT.to_string(), theme::muted())); } - let unit = if hidden == 1 { "line" } else { "lines" }; - spans.push(Span::new(format!(" ({hidden} {unit})"), theme::muted())); - Some(spans) + out.push(Span::new(note, theme::muted())); + out } /// The focused link is drawn reversed, so it reads differently from the other diff --git a/src/source/code/view.rs b/src/source/code/view.rs index 55ca03f..59719f4 100644 --- a/src/source/code/view.rs +++ b/src/source/code/view.rs @@ -337,3 +337,18 @@ impl CodeSource { }) } } + +#[cfg(test)] +mod tests { + use super::*; + + /// A source line is the unit; the declarations are `Tab`'s, not `j`'s. It + /// lives beside the `Source` impl rather than in `tests.rs`, which is over + /// the size limit already and must not grow. + #[test] + fn code_does_not_read_in_blocks() { + let s = CodeSource::new(PathBuf::from("t.rs"), "rust", "pub fn open() {\n let x = 1;\n}\n"); + assert!(!s.blocks()); + assert_eq!(s.block_at(0), None); + } +} diff --git a/src/source/csv/tests.rs b/src/source/csv/tests.rs index 6e25ed5..5fc5c0d 100644 --- a/src/source/csv/tests.rs +++ b/src/source/csv/tests.rs @@ -426,3 +426,14 @@ fn positions_are_stable_and_never_panic() { s.set_width(30); assert_eq!(s.locate(Mark(4)), Some(4)); } + +/// A CSV row *is* the unit and this format has no landmarks at all, so there is +/// no landing for `Tab` to frame. `j` stays one row here as in every format +/// (SPEC.md §"Moving through a document"). +#[test] +fn a_grid_does_not_read_in_blocks() { + let s = src_from("a,b\n1,2\n3,4\n"); + assert!(!s.blocks()); + assert_eq!(s.block_at(1), None); + assert_eq!(s.next_landmark(0, true), None); +} diff --git a/src/source/dir/tests.rs b/src/source/dir/tests.rs index 84e3cba..d6bfce7 100644 --- a/src/source/dir/tests.rs +++ b/src/source/dir/tests.rs @@ -257,3 +257,12 @@ fn enter_opens_a_file_from_a_listing_named_by_a_relative_parent_path() { ); } +/// A listing row is one entry, already its own unit, so a landing needs no +/// framing (SPEC.md §"Moving through a document"). +#[test] +fn a_listing_does_not_read_in_blocks() { + let t = tmp("blocks"); + let s = DirSource::open(&t.0); + assert!(!s.blocks()); + assert_eq!(s.block_at(0), None); +} diff --git a/src/source/json/tests.rs b/src/source/json/tests.rs index d054d4f..afef331 100644 --- a/src/source/json/tests.rs +++ b/src/source/json/tests.rs @@ -428,3 +428,13 @@ fn a_missing_file_is_an_error_not_a_panic() { + +/// A tree row is one node, already its own unit. Making `j` jump top-level +/// members would make a nested document unreadable — that is what `Tab` is for. +#[test] +fn a_tree_does_not_read_in_blocks() { + let mut s = src(DOC); + let _ = s.lines(0..s.len()); + assert!(!s.blocks()); + assert_eq!(s.block_at(0), None); +} diff --git a/src/source/jsonarray/level_tests.rs b/src/source/jsonarray/level_tests.rs new file mode 100644 index 0000000..0f432f9 --- /dev/null +++ b/src/source/jsonarray/level_tests.rs @@ -0,0 +1,528 @@ +//! The **ladder**, driven through the seam: `Enter` on a record, `Enter` on a +//! call inside it, `r` from either level, `zR`/`zM`, and the arithmetic under all +//! of it (SPEC.md §Lenses). +//! +//! Deterministic from end to end — a source, a row number, a key. No terminal +//! and no frame, so the composited-capture trap does not apply: every assertion +//! below reads rows the source itself handed back. +//! +//! Every fixture is written by hand, in the *shape* a real ATIF trajectory has. +#![deny(unsafe_code)] + +use super::*; +use crate::source::record::Records; +use crate::source::{Anchor, Source}; + +/// A trajectory with one of everything the level has to answer for: a message +/// long enough to clip, a thought, two calls with several arguments apiece and +/// an output longer than the clip, a step that only worked, and a message with +/// nothing under it at all. +fn run() -> Vec { + // Line 9 is long on purpose: a resize has to re-wrap it, which is the one + // thing that moves rows without a fold changing. + let said = (1..=9) + .map(|n| match n { + 9 => format!("said line 9 {}", "and more words to wrap ".repeat(12)), + n => format!("said line {n}"), + }) + .collect::>() + .join("\\n"); + let out = (1..=30).map(|n| format!("out line {n}")).collect::>().join("\\n"); + format!( + r#"{{"schema_version":"ATIF-v1.7","session_id":"sxs_1","steps":[ + {{"step_id":1,"source":"user","message":"do the thing"}}, + {{"step_id":2,"source":"agent","message":"{said}", + "reasoning_content":"a thought\nover two lines", + "tool_calls":[{{"tool_call_id":"c1","function_name":"bash", + "arguments":{{"command":"cargo test -q","timeout":120,"workdir":"/w"}}}}, + {{"tool_call_id":"c2","function_name":"read", + "arguments":{{"filePath":"src/parse.rs"}}}}], + "observation":{{"results":[{{"source_call_id":"c1","content":"{out}"}}, + {{"source_call_id":"c2","content":"fn parse() {{}}"}}]}}}}, + {{"step_id":3,"source":"agent","message":"","reasoning_content":"a short thought", + "tool_calls":[{{"tool_call_id":"c3","function_name":"glob","arguments":{{"pattern":"**/*.rs"}}}}], + "observation":{{"results":[{{"source_call_id":"c3","content":"a.rs"}}]}}}}, + {{"step_id":4,"source":"agent","message":"","tool_calls":[{{"tool_call_id":"c4", + "function_name":"bash","arguments":{{"command":"make"}}}}], + "observation":{{"results":[{{"source_call_id":"c4","content":"ok"}}]}}}}, + {{"step_id":5,"source":"agent","message":"done."}}]}}"# + ) + .into_bytes() +} + +fn lensed(cols: usize) -> ArraySource { + let mut s = ArraySource::from_bytes(run(), At::Key("steps")); + s.set_lens(crate::lens::find("atif").expect("the atif lens")); + s.set_width(cols); + let _ = s.len(); + s +} + +fn rows(s: &mut ArraySource) -> Vec { + let n = s.len(); + s.lines(0..n).iter().map(|l| l.text().trim_end().to_string()).collect() +} + +/// Records here, 0-based: 0 the envelope, then `steps[0..5]`. So the long +/// message is record 2 and the one-line `done.` is record 5. +const SAID: usize = 2; +const DONE: usize = 5; + +/// The row the long message's summary sits on. +fn message_row(s: &ArraySource) -> usize { + s.row_of_record(SAID) +} + +fn has(rows: &[String], needle: &str) -> bool { + rows.iter().any(|r| r.contains(needle)) +} + +/// Is `needle` a row **of its own** — a painted body or part row, rather than a +/// substring of the record's raw JSON on a tree row? +fn painted(rows: &[String], needle: &str) -> bool { + rows.iter().any(|r| r.trim().starts_with(needle)) +} + +// -- the ladder ------------------------------------------------------------------ + +/// The two levels, a press apiece: clipped, the whole of what was said with its +/// calls listed as calls, and back to the clip. The record's own JSON is `r` +/// and appears at neither of them (SPEC.md §Lenses). +#[test] +fn enter_toggles_the_two_levels_and_never_shows_the_json() { + let mut s = lensed(92); + let row = message_row(&s); + let clipped = s.len(); + let start = rows(&mut s); + assert!(has(&start, "said line 1"), "{start:#?}"); + assert!(has(&start, "\u{22ef} +"), "the clip says what it left out: {start:#?}"); + assert!(!painted(&start, "said line 9"), "{start:#?}"); + + // Rung two: the whole message, the thought it did not say aloud, and one + // row per call. + assert!(s.fold_here(row).is_some(), "the key is the ladder's"); + let open = rows(&mut s); + assert!(painted(&open, "said line 9"), "the whole message: {open:#?}"); + assert!(has(&open, "thinking"), "{open:#?}"); + assert!(has(&open, "a thought"), "{open:#?}"); + assert!(has(&open, "bash") && has(&open, "cargo test -q"), "{open:#?}"); + assert!(has(&open, "read") && has(&open, "src/parse.rs"), "{open:#?}"); + assert!(has(&open, "\u{2192} 30 lines"), "and what came back: {open:#?}"); + assert!(!has(&open, "out line 1"), "a call is one row until it is opened: {open:#?}"); + assert!(s.len() > clipped); + + assert!(!has(&open, "\"step_id\""), "and no JSON at this level: {open:#?}"); + + // And back, in one press rather than two. + assert!(s.fold_here(row).is_some()); + let back = rows(&mut s); + assert_eq!(s.len(), clipped, "{back:#?}"); + assert!(!has(&back, "\"step_id\""), "still no JSON: {back:#?}"); + assert!(has(&back, "\u{22ef} +"), "the clip is back: {back:#?}"); + + // A third press opens it again — two levels, not three. + assert!(s.fold_here(row).is_some()); + assert!(painted(&rows(&mut s), "said line 9")); + assert!(!has(&rows(&mut s), "\"step_id\""), "the JSON never appeared"); +} + +/// The other half of the same contract: with the tree open, `Enter` leaves it +/// open and toggles the record's own rows underneath it. `r` owns the tree, and +/// a key that silently shut what another key opened is what the third rung was +/// removed for (SPEC.md §Lenses). +#[test] +fn enter_with_the_tree_open_leaves_the_tree_alone() { + let mut s = lensed(92); + let row = message_row(&s); + s.toggle_tree(row).expect("the record's own tree"); + let with_tree = s.len(); + assert!(has(&rows(&mut s), "\"step_id\"")); + + s.fold_here(row).expect("the open level, under the tree"); + let open = rows(&mut s); + assert!(s.len() > with_tree); + assert!(has(&open, "\"step_id\""), "the tree is still open: {open:#?}"); + assert!(painted(&open, "said line 9"), "and the message is whole: {open:#?}"); + + s.fold_here(row).expect("and back to the clip"); + assert_eq!(s.len(), with_tree, "the tree is untouched by both presses"); + assert!(has(&rows(&mut s), "\"step_id\"")); + // And `r` is what shuts it. + s.toggle_tree(row).expect("shut"); + assert!(!has(&rows(&mut s), "\"step_id\"")); +} + +/// A record with nothing under its headline and no calls has **no rung at all**: +/// there is nothing between what it said and its JSON, so `Enter` has nowhere to +/// go. It still claims the key — `Some(false)`, no level changed — because the +/// caller's fall-through for `None` is the outline, and a record's outline entry +/// is the raw tree that belongs to `r`. Consuming the key for a repaint of the +/// same rows is what this states; `r` is still its bytes, which is why nothing +/// is out of reach. +#[test] +fn a_record_with_no_body_and_no_calls_has_no_rung_and_r_is_its_bytes() { + let mut s = lensed(92); + // `done.` — one line, no calls, no thought. + let row = s.row_of_record(DONE); + let clipped = s.len(); + assert_eq!(s.fold_here(row), Some(false), "claimed, and no level to descend to"); + assert_eq!(s.len(), clipped, "and nothing repainted"); + s.toggle_tree(row).expect("`r` still reaches the record"); + let tree = rows(&mut s); + assert!(has(&tree, "\"step_id\""), "the record itself: {tree:#?}"); + assert!(s.len() > clipped); + s.toggle_tree(row).expect("and shuts again"); + assert_eq!(s.len(), clipped); +} + +/// `r` is the way to the record from **either** level, and the same press back — +/// orthogonal to the level, which is the whole point of giving it its own key. +#[test] +fn r_reaches_the_tree_from_either_level_and_shuts_it_again() { + for descents in [0usize, 1] { + let mut s = lensed(92); + let row = message_row(&s); + for _ in 0..descents { + assert!(s.fold_here(row).is_some(), "descent {descents}"); + } + let before = s.len(); + assert!(s.toggle_tree(row).is_some(), "level {descents}: `r` opened it"); + let open = rows(&mut s); + assert!(has(&open, "\"step_id\""), "rung {descents}: {open:#?}"); + // And the level is untouched: what was showing is still showing. + assert_eq!( + painted(&open, "said line 9"), + descents == 1, + "level {descents}: `r` did not move the level" + ); + assert!(s.toggle_tree(row).is_some(), "and shuts again"); + assert_eq!(s.len(), before); + } +} + +// -- a call row is itself openable ------------------------------------------------- + +/// The user's words: "can we make this show the output if `Enter` on that line". +#[test] +fn enter_on_a_call_row_shows_its_arguments_and_its_output_and_shuts_again() { + let mut s = lensed(92); + let row = message_row(&s); + s.fold_here(row).expect("the open level"); + let open = rows(&mut s); + let shut = s.len(); + let call = open + .iter() + .position(|r| r.contains("cargo test -q") && r.contains('\u{25b8}')) + .expect("a shut call row"); + + assert!(s.fold_here(call).is_some(), "the key belongs to the call"); + let wide = rows(&mut s); + assert!(s.len() > shut, "it grew"); + // Every argument, one per line — including the ones the headline never had + // room for. + for (key, value) in [("command", "cargo test -q"), ("timeout", "120"), ("workdir", "/w")] { + assert!( + wide.iter().any(|r| r.contains(key) && r.contains(value)), + "{key}: {wide:#?}" + ); + } + // And the output, clipped like a message body, saying what it left out. + assert!(has(&wide, "out line 1"), "{wide:#?}"); + assert!(!has(&wide, "out line 30"), "clipped: {wide:#?}"); + assert!( + wide.iter().any(|r| r.trim().starts_with('\u{22ef}') && r.contains("lines")), + "the clip states the remainder: {wide:#?}" + ); + // The other call is untouched — one call opens, not all of them. + assert!( + wide.iter().any(|r| r.contains("src/parse.rs") && r.contains('\u{25b8}')), + "{wide:#?}" + ); + + assert!(s.fold_here(call).is_some(), "and shuts again"); + assert_eq!(s.len(), shut, "{:#?}", rows(&mut s)); +} + +/// `r` on a **call row** has a defined answer, and it is the row's *record*: a +/// call has no JSON of its own separate from the record it was made in, so the +/// honest raw thing under that row is the whole record. `Enter` there is still +/// the call's (SPEC.md §Lenses). +#[test] +fn r_on_a_call_row_is_the_records_own_tree() { + let mut s = lensed(92); + let row = message_row(&s); + s.fold_here(row).expect("the open level"); + let open = s.len(); + let call = rows(&mut s) + .iter() + .position(|r| r.contains("cargo test -q") && r.contains('\u{25b8}')) + .expect("a call row"); + assert!(!has(&rows(&mut s), "\"step_id\"")); + + s.toggle_tree(call).expect("`r` answers on a call row"); + let with_tree = rows(&mut s); + assert!(has(&with_tree, "\"step_id\""), "the record itself: {with_tree:#?}"); + assert!(has(&with_tree, "out line 30"), "every byte of it: {with_tree:#?}"); + assert!(painted(&with_tree, "said line 9"), "the level is untouched: {with_tree:#?}"); + + s.toggle_tree(call).expect("and the same press shuts it"); + assert_eq!(s.len(), open); +} + +/// Leaving the open level takes the opened call with it: the rows are gone, and +/// a call left open would reappear on the next descent unasked. +#[test] +fn an_opened_call_does_not_survive_the_level_it_was_opened_at() { + let mut s = lensed(92); + let row = message_row(&s); + s.fold_here(row).expect("open"); + let shut = s.len(); + let call = rows(&mut s) + .iter() + .position(|r| r.contains("cargo test -q") && r.contains('\u{25b8}')) + .expect("a call row"); + s.fold_here(call).expect("the call opens"); + s.fold_here(row).expect("back to the clip, and the call's rows with it"); + s.fold_here(row).expect("and open again"); + assert_eq!(s.len(), shut, "the call is shut again: {:#?}", rows(&mut s)); +} + +// -- zR and zM --------------------------------------------------------------------- + +/// `zR` is every record at the open level and every run opened; `zM` puts all of +/// it back. Both have a defined answer for the level, and it is the one a pipe +/// gets. +#[test] +fn zr_opens_every_level_and_zm_shuts_them() { + let mut s = lensed(92); + let clipped = s.len(); + s.fold_all(false); + let open = rows(&mut s); + assert!(s.len() > clipped); + assert!(painted(&open, "said line 9"), "every message whole: {open:#?}"); + assert!(has(&open, "a short thought"), "every step's reasoning: {open:#?}"); + assert!(has(&open, "**/*.rs"), "every run open, and its calls listed: {open:#?}"); + assert!(has(&open, "\"step_id\""), "and every record's own tree: {open:#?}"); + // Every byte of the longest output is in the tree, which is what makes the + // clipped part row honest rather than lossy. + assert!(has(&open, "out line 30"), "{open:#?}"); + s.fold_all(true); + assert_eq!(s.len(), clipped, "{:#?}", rows(&mut s)); +} + +/// A rungless record is rungless **at both levels**. `zR` puts every record at +/// the open level, including the ones whose two levels are the same rows; asking +/// only on the way in would then let `Enter` report a descent and flip the level +/// back while the screen stayed identical — a dead press, and the press after it +/// would be the one that fell through to the outline and toggled the JSON. +/// Whether a record opens further is a question about the record and the width, +/// never about which level it is at, so the answer is the same both ways round. +#[test] +fn a_rungless_record_has_no_rung_after_zr_either() { + let mut s = lensed(92); + s.fold_all(false); + let open = s.len(); + let row = s.row_of_record(DONE); + assert_eq!(s.fold_here(row), Some(false), "still no rung at the open level"); + assert_eq!(s.len(), open, "and no press of it moves a row"); + assert_eq!(s.fold_here(row), Some(false), "nor does the next one"); + assert_eq!(s.len(), open); +} + +// -- the arithmetic ------------------------------------------------------------------ + +/// The invariant the whole two-level arithmetic rests on, now with a third kind +/// of row in it: at every width and every level, `at(row)` inverts +/// `row_of_record`, every row paints, and the two totals agree. +#[test] +fn rows_and_records_round_trip_at_every_level_and_width() { + for width in [40usize, 92, 200] { + for descents in 0..4usize { + let mut s = lensed(width); + let row = message_row(&s); + for _ in 0..descents { + s.fold_here(row); + } + // And a call opened at the level that has them. + if descents == 1 { + if let Some(call) = rows(&mut s).iter().position(|r| r.contains('\u{25b8}')) { + s.fold_here(call); + } + } + let n = s.len(); + for r in 0..Records::known(&s) { + let at = s.row_of_record(r); + assert!(at < n, "{width}/{descents}: record {r} at {at} of {n}"); + // A record inside a shut run has no row of its own; the run's + // row stands for it, and that row is the run's first record. + match s.record_visible(r) { + true => assert_eq!(s.record_at(at), r, "{width}/{descents}: record {r}"), + false => assert!(s.record_at(at) <= r, "{width}/{descents}: record {r}"), + } + } + for row in 0..n { + assert!(s.row_line(row).is_some(), "{width}/{descents}: row {row} of {n}"); + } + assert_eq!(s.lines(0..n).len(), n, "{width}/{descents}"); + } + } +} + +/// A resize re-wraps every level, and the cursor's record survives it — the +/// same promise a body already made, now with parts under it. +#[test] +fn a_resize_re_lays_the_open_level() { + let mut s = lensed(200); + let row = message_row(&s); + s.fold_here(row).expect("open"); + let wide = s.len(); + let mark = s.mark(row).expect("a mark"); + s.set_width(40); + let narrow = s.len(); + assert!(narrow > wide, "narrower columns are more rows: {narrow} vs {wide}"); + assert_eq!(s.locate(mark), Some(s.row_of_record(SAID)), "the mark is the record"); + let got = rows(&mut s); + assert!(has(&got, "cargo test -q"), "the calls are still listed: {got:#?}"); +} + +// -- nothing is silently absent ------------------------------------------------------- + +/// The rule the whole seam is held to (SPEC.md §Lenses): what a level shows +/// plus what its clips *say* accounts for the whole of what is there. +/// +/// Read the same record at every level and check the message and the tool's +/// output against what is on the screen: either every line of them is painted, +/// or a row states exactly how many are not. +#[test] +fn every_level_accounts_for_the_whole_message_and_the_whole_output() { + let said: Vec = (1..=9).map(|n| format!("said line {n}")).collect(); + let out: Vec = (1..=30).map(|n| format!("out line {n}")).collect(); + let mut s = lensed(92); + let row = message_row(&s); + + // Every state a record can be in: its two levels, and each of them with the + // record's own tree open underneath (`r`, which is orthogonal to the level). + for level in 0..4usize { + let painted = rows(&mut s); + // Rows belonging to this record: from its own row to the next record's. + let from = s.row_of_record(SAID); + let to = s.row_of_record(SAID + 1); + let mine: Vec = painted[from..to].to_vec(); + let text = mine.join("\n"); + + let shown = said.iter().filter(|l| text.contains(l.as_str())).count(); + let missing = said.len() - shown; + if missing > 0 { + let note = mine + .iter() + .find(|r| r.trim().starts_with('\u{22ef}')) + .unwrap_or_else(|| panic!("level {level} hid {missing} lines and said nothing: {mine:#?}")); + assert!( + note.contains(&format!("+{missing} lines")), + "level {level}: {missing} lines missing but the note says {note:?}" + ); + } + + // The output is only ever *claimed* at the open level, and there its + // headline states the size even while the call is shut. + if level == 1 { + assert!( + mine.iter().any(|r| r.contains(&format!("\u{2192} {} lines", out.len()))), + "the call row states the whole size: {mine:#?}" + ); + } + match level { + 0 => drop(s.fold_here(row).expect("the open level")), + 1 => drop(s.toggle_tree(row).expect("`r`: the tree, over the open level")), + 2 => drop(s.fold_here(row).expect("back to the clip, tree untouched")), + _ => {} + } + } + + // And at the level that shows the output, what is missing from it is stated + // in its own lines: `r` shuts the tree again and `Enter` opens the record, + // which is where its calls are listed. + s.toggle_tree(row).expect("the tree shuts"); + s.fold_here(row).expect("the open level"); + let call = rows(&mut s) + .iter() + .position(|r| r.contains("cargo test -q") && r.contains('\u{25b8}')) + .expect("a call row"); + s.fold_here(call).expect("the call opens"); + let wide = rows(&mut s); + let shown = out.iter().filter(|l| wide.iter().any(|r| r.trim() == l.as_str())).count(); + let note = wide + .iter() + .filter(|r| r.trim().starts_with('\u{22ef}')) + .find(|r| r.contains("lines")) + .expect("the output's clip note"); + assert!( + note.contains(&format!("+{} lines", out.len() - shown)), + "{shown} of {} shown, note {note:?}", + out.len() + ); + + // The floor under all of it: `r` is every byte, always. + s.toggle_tree(row).expect("the record itself"); + let raw = rows(&mut s).join("\n"); + assert!(raw.contains("out line 30"), "the record holds every byte"); + assert!(raw.contains("said line 9")); +} + +/// A search hit inside a folded run still opens it, with the members' own rows +/// measured — the one path where a fold and a wrap have to agree in the same +/// keystroke. +#[test] +fn revealing_a_hit_inside_a_run_measures_what_it_revealed() { + let mut s = lensed(92); + s.set_query("**/*.rs"); + let hit = s.preview_match(Anchor(0), crate::source::Dir::Forward).expect("a hit"); + let n = s.len(); + assert!(hit.anchor.0 < n); + for row in 0..n { + assert!(s.row_line(row).is_some(), "row {row} of {n} after the reveal"); + } + assert!(has(&rows(&mut s), "a short thought"), "the member's reasoning came with it"); +} + +/// A member of an open run is inset, and everything under its row goes with it. +/// +/// SPEC.md §Lenses: the body is "indented to the same column" as the `what` it +/// belongs to. The summary row of a member is inset after its gutter, and its +/// body and part rows had no idea — so a step's reasoning sat two columns left +/// of the words it was reasoning about, while a non-member's lined up. +#[test] +fn a_members_body_and_parts_line_up_with_its_own_words() { + let mut s = lensed(92); + // The run of steps 3 and 4 — the two records with no message between the + // long message and `done.`. + let _ = rows(&mut s); // the outline is over what the last frame painted + let run = s.row_of_record(3); + let entry = s.section_at(run).expect("the run is an outline entry"); + assert!(s.set_fold(entry, false), "the run opens"); + let member = s.row_of_record(3); + let painted = rows(&mut s); + let thought = painted + .iter() + .position(|r| r.trim().starts_with("a short thought")) + .expect("the member's reasoning is on a row of its own"); + assert!(thought > member, "under the member's row: {painted:#?}"); + // Where the member's own `what` starts: past its gutter, its inset, the + // actor column and the clock. + let at = painted[member].find("thinking").expect("the member says what it did"); + let what = crate::render::str_width(&painted[member][..at]); + assert_eq!(column_of(&painted[thought]), what, "{painted:#?}"); + // Two columns right of where a record outside the run puts its own body, + // which is exactly the inset its summary row carries. + let outside = painted + .iter() + .position(|r| r.trim().starts_with("said line 2")) + .expect("a non-member's body"); + assert_eq!(column_of(&painted[thought]), column_of(&painted[outside]) + 2, "{painted:#?}"); +} + +/// The **display column** the first non-blank character of a row sits in — +/// never the byte offset: a fold glyph is three bytes and one column wide. +fn column_of(row: &str) -> usize { + crate::render::str_width(row) - crate::render::str_width(row.trim_start()) +} diff --git a/src/source/jsonarray/mod.rs b/src/source/jsonarray/mod.rs new file mode 100644 index 0000000..94e8d2f --- /dev/null +++ b/src/source/jsonarray/mod.rs @@ -0,0 +1,365 @@ +//! Records that are the elements of an **array inside a JSON document** +//! (SPEC.md §Lenses, "Records inside a document"). +//! +//! # Why this exists +//! +//! A `.jsonl` file is a record per line, and until now that was the only shape +//! `--lens` could read. A trajectory in the ATIF interchange format is one JSON +//! document whose records are the elements of a named array, alongside +//! top-level keys describing the run. Same records, different envelope — so +//! this is a [`Store`], not a second source: rows, folding, search, yanking, +//! the outline and every lens live in [`crate::source::record`], written once. +//! +//! # Nothing is parsed to find a record +//! +//! The records are found through the *existing* lazy structural index +//! ([`crate::source::json::tree`]): [`Doc`] walks bytes and hands back a +//! member's byte range, building no values. A record is read and parsed when it +//! is painted, exactly as a line is, and a 32KB step nobody scrolls to costs +//! nothing. +//! +//! # What it does cost, said plainly +//! +//! Finding the array is a **structural scan of the document's top level**: the +//! index emits a member only once it has walked past that member's last byte, +//! and the array is one member. So the first row waits on one byte walk of the +//! file — no parse, no allocation per record, constant memory — and the status +//! bar reports it as `\u{2265}N (indexing P%)` like any other scan. That is the +//! honest price of records living inside a document rather than one per line, +//! and SPEC.md §Lenses states it. It also buys the session row below: the +//! document's other top-level keys are all known by the time record 0 is shown, +//! so that row can never silently gain a key later. +//! +//! # Nothing is lost +//! +//! The keys that are *not* the record array — `schema_version`, `session_id`, +//! `agent` — are record **0**, a synthesised object that opens into the generic +//! tree like any other record. A lens adds interpretation and never hides data +//! (SPEC.md §Lenses), and a document reader that showed only the array would be +//! hiding the three keys that say what the run was. The cost is that record +//! numbering is shifted by one against the array's own indices: `steps[0]` is +//! record 1, which is what the status bar and `#n` say. +#![deny(unsafe_code)] + +#[cfg(test)] +#[path = "tests.rs"] +mod tests; + +/// The two levels a record has under a lens and the `r` that shows its bytes, +/// driven through the seam. +#[cfg(test)] +#[path = "level_tests.rs"] +mod level_tests; + +use std::cell::RefCell; +use std::io; +use std::path::Path; + +use crate::json::index::Member; +use crate::json::value::{Member as Field, Value}; +use crate::source::json::tree::{Doc, NodeId, PARSE_CAP}; +use crate::source::jsonrow; +use crate::source::record::{Record, RecordSource, Store}; + +/// Where in a document the records are. +/// +/// The source's half of [`crate::lens::RecordsAt`] — the lens declares it, the +/// routing translates it, and nothing below this line knows a lens exists. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum At { + /// The document's root *is* the array of records. No session row: there + /// are no other keys for one to hold. + Root, + /// The records are the elements of the array under this top-level key. + Key(&'static str), +} + +/// A document whose records are an array inside it. +pub type ArraySource = RecordSource; + +/// How far the search for the record array has got. +enum Found { + /// The document's top level is still being scanned. + Looking, + /// The array, and which member of the root it is (`None` when the root + /// itself is the array). + At(NodeId, Option), + /// The document holds no such array. Everything it *does* hold is still + /// record 0, so nothing is lost — there is simply nothing to fold. + Absent, +} + +pub struct Array { + /// Interior mutability because reading a member is a *file* read and every + /// [`Store`] method is `&self`. Every borrow is taken and dropped inside + /// one method; none nest. + doc: RefCell, + at: At, + found: RefCell, +} + +impl ArraySource { + /// Open `path`. Stats it and reads 64 bytes; nothing is indexed and no + /// record is parsed until one is asked for. + pub fn open(path: &Path, at: At) -> io::Result { + Ok(RecordSource::new(Array::new(Doc::open(path)?, at))) + } + + /// A source over bytes that arrived on a pipe. + pub fn from_bytes(data: Vec, at: At) -> ArraySource { + RecordSource::new(Array::new(Doc::memory(data), at)) + } +} + +impl Array { + fn new(doc: Doc, at: At) -> Array { + Array { + doc: RefCell::new(doc), + at, + found: RefCell::new(Found::Looking), + } + } + + /// Is there a session record at index 0? Only when the records sit *inside* + /// a document, because only then are there other keys to keep. + fn head(&self) -> usize { + usize::from(matches!(self.at, At::Key(_))) + } + + /// Walk the document's top level until the record array is found, spending + /// at most `budget` bytes. Returns nothing: ask [`Array::found`] after. + fn locate(&self, budget: u64) { + if !matches!(&*self.found.borrow(), Found::Looking) { + return; + } + let mut doc = self.doc.borrow_mut(); + let Some(root) = doc.root() else { + *self.found.borrow_mut() = Found::Absent; + return; + }; + // The top level is walked to its end before any record is served: a + // member is only emitted once the scan has passed it, and the session + // row must not gain keys after it has been read (see the header). + doc.index(root, usize::MAX, budget); + if !doc.node(root).complete() { + return; + } + let key = match self.at { + At::Root => { + *self.found.borrow_mut() = match doc.node(root).shape.is_container() { + true => Found::At(root, None), + false => Found::Absent, + }; + return; + } + At::Key(key) => key, + }; + *self.found.borrow_mut() = match Self::named(&mut doc, root, key) { + Some(i) => match doc.open_child(root, i) { + Some(id) => Found::At(id, Some(i)), + None => Found::Absent, + }, + None => Found::Absent, + }; + } + + /// Find the array if it is still being looked for, then index `want` of its + /// members — all inside **one** `budget`, which is what a caller is paying + /// for. The walk the search spent is deducted from the walk the members + /// get, so the tick that finds the array does not spend twice. + fn grow(&self, want: usize, budget: u64) { + let start = self.doc.borrow().walked(); + self.locate(budget); + let Some(id) = self.array() else { + return; + }; + let spent = self.doc.borrow().walked().saturating_sub(start); + let mut doc = self.doc.borrow_mut(); + doc.index(id, want, budget.saturating_sub(spent)); + } + + /// Which member of `root` carries `key`. + /// + /// Through [`Doc::key_text`], which parses the key's own bytes, rather than + /// comparing the quoted bytes: a document is free to spell `steps` as + /// `"steps"` and it is still that key. + fn named(doc: &mut Doc, root: NodeId, key: &str) -> Option { + for i in 0..doc.node(root).count() { + let m = doc.node(root).member(i)?; + if doc.key_text(m).is_some_and(|k| k == key) { + return Some(i); + } + } + None + } + + /// The array node, once it has been found. + fn array(&self) -> Option { + match &*self.found.borrow() { + Found::At(id, _) => Some(*id), + _ => None, + } + } + + /// The byte range of record `i`, or `None` for the session record and for + /// anything past the index. + fn member(&self, record: usize) -> Option { + let id = self.array()?; + let i = record.checked_sub(self.head())?; + self.doc.borrow().node(id).member(i) + } + + /// The document's other top-level keys, as one synthesised record. + /// + /// Built from byte ranges the index already has, one bounded parse per + /// member: a key too large to display says so in place of its value rather + /// than being loaded, exactly as the document reader's row does. + fn session(&self) -> Record { + let skip = match &*self.found.borrow() { + Found::At(_, i) => *i, + _ => None, + }; + let mut doc = self.doc.borrow_mut(); + let Some(root) = doc.root() else { + return Record::Bad("empty document".to_string()); + }; + let node = doc.node(root); + let (shape, start, end, count) = (node.shape, node.start, node.end, node.count()); + if !shape.is_container() { + return Record::Value(value_at(&mut doc, Member { key: None, start, end })); + } + let mut fields: Vec = Vec::new(); + for i in 0..count { + if Some(i) == skip { + continue; + } + let Some(m) = doc.node(root).member(i) else { + continue; + }; + let key = doc.key_text(m).unwrap_or_else(|| format!("{i}")); + let value = value_at(&mut doc, m); + fields.push(Field { key, value }); + } + Record::Value(Value::Object(fields)) + } +} + +/// One member as a value: parsed when it is small enough, and a note saying how +/// big it is when it is not (SPEC.md §JSON — a member past the parse cap +/// reports its size instead of being loaded). +fn value_at(doc: &mut Doc, m: Member) -> Value { + if m.len() > PARSE_CAP { + return Value::Str(jsonrow::oversize(m.len(), PARSE_CAP)); + } + let (bytes, clipped) = doc.bytes(m.start, m.end); + if clipped { + return Value::Str(jsonrow::oversize(m.len(), PARSE_CAP)); + } + match crate::json::parse(&bytes) { + Ok(v) => v, + Err(e) => Value::Str(format!( + "\u{27e8}not JSON: {} at byte {}\u{27e9}", + e.reason, + m.start + e.offset as u64 + )), + } +} + +impl Store for Array { + /// The session record, plus the array members indexed so far. + fn known(&self) -> usize { + match &*self.found.borrow() { + Found::Looking => 0, + Found::At(id, _) => self.head() + self.doc.borrow().node(*id).count(), + // A document with no root value holds nothing at all, so there is + // no session record either: an empty file is empty, not one error + // row about being empty. + Found::Absent => match self.doc.borrow().root() { + Some(_) => self.head(), + None => 0, + }, + } + } + + fn complete(&self) -> bool { + match &*self.found.borrow() { + Found::Looking => false, + Found::At(id, _) => self.doc.borrow().node(*id).complete(), + Found::Absent => true, + } + } + + /// The top-level scan while the array is still being looked for, then the + /// array's own. Both are the same honest "how far the bytes got". + fn progress(&self) -> u8 { + let doc = self.doc.borrow(); + match &*self.found.borrow() { + Found::Looking => doc.progress(), + Found::Absent => 100, + Found::At(id, _) => { + let node = doc.node(*id); + let total = node.end.saturating_sub(node.start); + match node.complete() || total == 0 { + true => 100, + false => ((node.scanned().saturating_sub(node.start) * 100) / total) as u8, + } + } + } + } + + fn index_to(&self, records: usize, budget: u64) { + self.grow(records.saturating_sub(self.head()), budget); + } + + /// An idle tick spends its whole budget on the index, and asks for every + /// member the budget will reach. + /// + /// Asking for `known + 1` instead made [`Doc::index`] return at the first + /// 4KB chunk that yielded a member and threw the rest of the slice away: + /// the index crawled forward one chunk per tick, and a 4.6MB trajectory + /// took a minute and a half to settle. [`crate::source::jsonl`] hands its + /// budget straight to the line index, which is the contract the pager's + /// idle tick is written against. + fn extend(&self, budget: u64) -> bool { + let before = self.known(); + self.grow(usize::MAX, budget); + !self.complete() || self.known() > before + } + + /// The record's own bytes out of the document. The session record has none + /// of its own — it is the keys around the array — so it is serialised. + fn raw(&self, record: usize) -> Vec { + let Some(m) = self.member(record) else { + return match record < self.head() { + true => self.session().value().map(|v| v.to_json()).unwrap_or_default().into_bytes(), + false => Vec::new(), + }; + }; + self.doc.borrow_mut().bytes(m.start, m.end).0 + } + + fn load(&self, record: usize) -> Record { + let Some(m) = self.member(record) else { + return match record < self.head() { + true => self.session(), + false => Record::Bad("no such record".to_string()), + }; + }; + if m.len() > PARSE_CAP { + let note = jsonrow::oversize(m.len(), PARSE_CAP); + return Record::Bad(format!("{note}, not parsed")); + } + let (bytes, clipped) = self.doc.borrow_mut().bytes(m.start, m.end); + if clipped { + return Record::Bad(jsonrow::oversize(m.len(), PARSE_CAP)); + } + match crate::json::parse(&bytes) { + Ok(v) => Record::Value(v), + Err(e) => Record::Bad(e.to_string()), + } + } + + fn unit(&self) -> &'static str { + "record" + } +} diff --git a/src/source/jsonarray/tests.rs b/src/source/jsonarray/tests.rs new file mode 100644 index 0000000..d6c70dd --- /dev/null +++ b/src/source/jsonarray/tests.rs @@ -0,0 +1,444 @@ +//! Records inside a document, behind the seam. +//! +//! Every fixture is written by hand, in the *shape* of a real ATIF trajectory: +//! an envelope of top-level keys, and the records under one of them. The +//! trajectory this was measured against is private and none of it is here. +#![deny(unsafe_code)] + +use super::*; +use crate::source::record::plan::Spot; +use crate::source::record::Records; +use crate::source::{End, Source}; + +/// A document with `n` steps and the three envelope keys, laid out the way a +/// real one is: pretty-printed, and the record array last. +fn doc(n: usize) -> Vec { + let steps: Vec = (0..n) + .map(|i| { + format!( + " {{\n \"step_id\": {},\n \"source\": \"agent\",\n \ + \"message\": \"\",\n \"tool_calls\": [\n {{\"function_name\": \ + \"bash\", \"arguments\": {{\"command\": \"echo {i}\"}}}}\n ]\n }}", + i + 1 + ) + }) + .collect(); + format!( + "{{\n \"schema_version\": \"ATIF-v1.7\",\n \"session_id\": \"sxs_1\",\n \ + \"agent\": {{\"name\": \"opencode\", \"version\": \"1.2.3\"}},\n \"steps\": [\n{}\n ]\n}}\n", + steps.join(",\n") + ) + .into_bytes() +} + +fn open(body: Vec, at: At) -> ArraySource { + let mut s = ArraySource::from_bytes(body, at); + s.set_width(120); + s +} + +fn rows(s: &mut ArraySource) -> Vec { + let n = s.len(); + s.lines(0..n).iter().map(|l| l.text().trim_end().to_string()).collect() +} + +/// The shape the whole commit exists for: the elements of `steps` are the +/// records, and the keys *around* them are not lost — they are record 0. +#[test] +fn the_array_is_the_records_and_the_other_keys_are_record_zero() { + let mut s = open(doc(3), At::Key("steps")); + assert_eq!(s.len(), 4, "a session row and three steps"); + let rows = rows(&mut s); + // Record 0 is the envelope, as an object of the keys that are not `steps`. + assert!(rows[0].contains("3 keys"), "{}", rows[0]); + assert!(rows[0].contains("ATIF-v1.7") && rows[0].contains("sxs_1"), "{}", rows[0]); + // And every step is there, in order, one row each. + for (i, row) in rows[1..].iter().enumerate() { + assert!(row.contains(&format!("step_id: {}", i + 1)), "{row}"); + } +} + +/// Every row still opens into the record it stands for, whole — the rule the +/// lens seam is held to, from the other side of it. +#[test] +fn a_record_opens_into_its_own_tree() { + let mut s = open(doc(2), At::Key("steps")); + assert_eq!(s.hidden_at(1), Some(s.hidden_at(1).unwrap()), "the step row folds"); + s.fold_all(false); + let text = rows(&mut s).join("\n"); + assert!(text.contains("\"command\": \"echo 0\""), "{text}"); + // The envelope's own subtree is under record 0, not thrown away. + assert!(text.contains("\"name\": \"opencode\""), "{text}"); +} + +/// A document that has no such array is not empty and is not an error: what it +/// *does* hold is still record 0, and it opens. +#[test] +fn a_document_without_the_array_keeps_everything_as_record_zero() { + let mut s = open(br#"{"title":"notes","items":[1,2,3]}"#.to_vec(), At::Key("steps")); + assert_eq!(s.len(), 1); + assert_eq!(s.end(), End::At(0), "nothing left to scan"); + s.fold_all(false); + let text = rows(&mut s).join("\n"); + assert!(text.contains("\"title\": \"notes\""), "{text}"); + assert!(text.contains("1") && text.contains("3"), "{text}"); +} + +/// A document that is one scalar, and one that is empty: an honest row and no +/// panic, which is the floor every source in this crate is held to. +#[test] +fn a_document_that_is_not_an_object_still_reads() { + let mut s = open(b"42".to_vec(), At::Key("steps")); + assert_eq!(rows(&mut s), vec![" 42".to_string()]); + let mut empty = open(Vec::new(), At::Key("steps")); + assert_eq!(empty.len(), 0); + assert!(empty.lines(0..10).is_empty()); +} + +/// [`At::Root`]: the document's root array *is* the records, and there is no +/// session row because there are no other keys for one to hold. +#[test] +fn the_root_array_can_be_the_records() { + let mut s = open(br#"[{"step_id":1},{"step_id":2}]"#.to_vec(), At::Root); + assert_eq!(s.len(), 2); + let rows = rows(&mut s); + assert!(rows[0].contains("step_id: 1"), "{}", rows[0]); + assert!(rows[1].contains("step_id: 2"), "{}", rows[1]); + // A root that is not a container has no records at all. + let scalar = open(b"\"hello\"".to_vec(), At::Root); + assert_eq!(Source::len(&scalar), 0); +} + +/// A record that is not JSON says so and the document keeps rendering — half a +/// trajectory is still worth reading. +#[test] +fn a_record_that_does_not_parse_is_an_error_row() { + let body = br#"{"steps":[{"a":1},{"b":@},{"c":3}]}"#.to_vec(); + let mut s = open(body, At::Key("steps")); + let rows = rows(&mut s); + assert_eq!(rows.len(), 4); + assert!(rows[2].contains("record 3:"), "{}", rows[2]); + assert!(rows[3].contains("c: 3"), "{}", rows[3]); +} + +/// `c` copies the record's own bytes out of the document — pretty-printing and +/// all — rather than this reader's re-serialisation of them, and says which +/// record it was. A member of a document is a `record`, not a `line`. +#[test] +fn the_verbatim_yank_is_the_documents_own_bytes() { + let mut s = open(doc(2), At::Key("steps")); + let _ = s.lines(0..s.len()); + let y = s.yank_block(1).expect("a yank"); + assert_eq!(y.what, "record 2 verbatim"); + assert!(y.text.starts_with('{') && y.text.contains("\n \"step_id\": 1"), "{}", y.text); +} + +// -- laziness ------------------------------------------------------------------ + +/// The claim SPEC.md makes, from the inside: finding the array is a byte walk +/// of the document's top level, and after that a screen costs a screen. No +/// record is parsed to *find* another one, and the row count is honestly a +/// lower bound until the array's own scan finishes. +#[test] +fn painting_a_screen_indexes_a_screen_and_not_the_array() { + let mut s = ArraySource::from_bytes(doc(20_000), At::Key("steps")); + assert_eq!(s.len(), 0, "opening indexes nothing"); + s.set_width(120); + let _ = s.lines(0..40); + let known = s.len(); + assert!(known >= 40, "only {known} rows for a screen of 40"); + assert!(known < 20_001, "the whole array was indexed for one screen"); + assert!(matches!(s.end(), End::Scanning(_)), "the end is not known yet"); + let text = s.position_text(0).expect("a position"); + assert!(text.starts_with("record 1/\u{2265}"), "{text}"); + // And the idle tick converges, without any keystroke having waited on it. + while s.extend() {} + assert_eq!(s.len(), 20_001); + assert_eq!(s.end(), End::At(20_000)); + assert_eq!(s.position_text(0).as_deref(), Some("record 1/20001")); +} + +/// A record too big to display says how big it is instead of being loaded — +/// the same limit, and the same sentence, the document reader uses. +#[test] +fn a_record_past_the_parse_cap_reports_its_size() { + let big = "x".repeat(PARSE_CAP as usize + 16); + let body = format!("{{\"steps\":[{{\"a\":\"{big}\"}},{{\"b\":2}}]}}").into_bytes(); + let mut s = open(body, At::Key("steps")); + let rows = rows(&mut s); + assert!(rows[1].contains("record 2:") && rows[1].contains("display limit"), "{}", rows[1]); + assert!(rows[2].contains("b: 2"), "{}", rows[2]); +} + +/// An idle tick spends its whole budget, not the first chunk that happens to +/// yield a member. The store used to ask the structural index for `known + 1`, +/// which returned after ~4KB and threw the rest of the slice away — the index +/// crawled a chunk per tick and a multi-megabyte trajectory took a minute and a +/// half to settle. One tick must cover far more than one chunk. +#[test] +fn one_idle_tick_indexes_far_more_than_one_chunk() { + let mut s = ArraySource::from_bytes(doc(20_000), At::Key("steps")); + s.set_width(120); + // The first tick may be spent locating the array; from the tick after it, + // records arrive in thousands rather than in tens. + let mut ticks = 0; + while Records::known(&s) == 0 && Source::extend(&mut s) { + ticks += 1; + assert!(ticks < 100, "the array was not found in {ticks} ticks"); + } + let before = Records::known(&s); + Source::extend(&mut s); + let found = Records::known(&s) - before; + assert!(found > 1000, "one idle tick found only {found} records"); +} + +/// `--toc` on a document larger than one frame budget prints the records, not +/// nothing. A store whose records live inside a document serves none of them +/// until the whole top level has been walked, so a single budgeted slice left +/// `--toc --lens atif` printing zero lines and exiting 0 — which a script reads +/// as "this file has no records". +#[test] +fn the_toc_of_a_document_bigger_than_one_budget_is_not_empty() { + // Comfortably past the 4 MB a frame may spend on the index. + let body = doc(40_000); + assert!(body.len() > 4 * 1024 * 1024, "the fixture is {} bytes", body.len()); + let mut s = ArraySource::from_bytes(body, At::Key("steps")); + s.set_lens(crate::lens::find("atif").expect("the lens exists")); + s.set_width(120); + let toc = s.summaries(1000); + assert_eq!(toc.len(), 1000, "the toc truncated to nothing"); + assert!(toc[0].starts_with("1\tsession"), "{}", toc[0]); +} + +/// Against a real trajectory, when one is pointed at: set +/// `TREAD_ATIF_TRAJECTORY`. Fixtures only prove the shapes someone thought of, +/// and a real ATIF file is private, so this asserts *structure* — every record +/// reachable, every row painted, the lens recognising what it claims to — and +/// prints counts only. Skipped when unset, so CI (which has no trajectory) +/// stays green. +#[test] +fn a_real_atif_trajectory_reads() { + let Ok(path) = std::env::var("TREAD_ATIF_TRAJECTORY") else { + return; + }; + let at = match crate::lens::records_at("atif") { + Some(crate::lens::RecordsAt::Member(key)) => At::Key(key), + other => panic!("the atif lens no longer names a member: {other:?}"), + }; + let mut s = ArraySource::open(std::path::Path::new(&path), at).expect("open"); + s.set_lens(crate::lens::find("atif").expect("the lens exists")); + s.set_width(140); + // Read it all the way, the way `G` does: a slice at a time, never a whole + // file in one call. + let mut ticks = 0; + while s.extend() { + ticks += 1; + assert!(ticks < 100_000, "the idle tick never settles"); + } + let records = Records::known(&s); + assert!(records > 1, "a trajectory with no steps"); + // Every record is reachable and every row paints, with nothing lost: the + // rows a lens folds away are the rows its groups open. + let rows = s.len(); + // More rows than records now: a message carries what it said under it, and + // fewer than one row per record would mean a record had been hidden. + assert!(rows > 0, "{rows} rows for {records} records"); + let read = (0..records).filter(|&r| s.lens_row(r, false).is_some()).count(); + assert_eq!(read, records, "the lens left records unread"); + s.fold_all(true); + let shut = s.len(); + s.fold_all(false); + let open = s.len(); + assert!(open >= shut, "opening the runs lost rows"); + for row in 0..open.min(4000) { + assert!(s.row_line(row).is_some(), "row {row} of {open} did not paint"); + } + println!("{records} records, {read} read by the lens, {shut} rows shut, {open} rows open"); + + every_record_opens_one_rung(&mut s, records); + for width in [80usize, 140] { + every_call_opens(&mut s, records, width); + } +} + +/// Descend every record one rung and count what the open level showed. +/// +/// Counting rather than quoting: the file is private and nothing from it is +/// printed. What this holds to is that a descent shows *something* — a rung +/// that repaints the same screen is not a rung — and that every row of the +/// level paints. +fn every_record_opens_one_rung(s: &mut ArraySource, records: usize) { + s.fold_all(true); + let clipped = s.len(); + let (mut with_parts, mut part_rows, mut calls_opened) = (0usize, 0usize, 0usize); + for record in 0..records { + let row = s.row_of_record(record); + if !s.record_visible(record) { + continue; + } + let before = s.len(); + if s.fold_here(row).is_none() { + continue; + } + let grew = s.len() - before; + // The rows the open level added, over and above the message going whole. + let listed = s.with_parts(record, |laid| laid.rows.len()); + if listed > 0 { + with_parts += 1; + part_rows += listed; + // And at least one of those part rows is a call that opens. + if (0..listed).any(|i| s.with_parts(record, |l| l.call_at(i)).is_some()) { + calls_opened += 1; + } + let n = s.len(); + for r in 0..n { + assert!(s.row_line(r).is_some(), "row {r} of {n} at the open level"); + } + } + assert!(grew > 0 || listed == 0, "a descent that showed nothing"); + back_to_the_clip(s, row, clipped); + } + println!( + "{with_parts} records have an open level, {part_rows} part rows in all, \ + {calls_opened} of them carrying a call that opens" + ); +} + +/// One rung further: every call the file holds, opened, at one width. +/// +/// A real trajectory's arguments are the shapes no fixture thought of — a +/// `patchText`, a `todos` list, an `arguments` the wire format wrote as a +/// string — and what this holds to is that each of them lands in **one** +/// column. A call row scrolls sideways, exactly as a summary row and a tree row +/// do; the rows *under* it are a wrap and must fit. An argument's name is +/// written into the pad its value's wrap left for it, and one column out put +/// every argument's first row off the side of the view while its own +/// continuation rows sat at the right indent. +fn every_call_opens(s: &mut ArraySource, records: usize, width: usize) { + s.set_width(width); + s.fold_all(true); + let shut = s.len(); + let (mut opened, mut widest) = (0usize, 0usize); + for record in 0..records { + let row = s.row_of_record(record); + if !s.record_visible(record) || s.fold_here(row).is_none() { + continue; + } + for call_row in call_rows(s, record) { + if s.fold_here(call_row).is_none() { + continue; + } + opened += 1; + let n = s.len(); + for r in 0..n { + let line = s.row_line(r).unwrap_or_else(|| panic!("row {r} of {n} at width {width}")); + if under_a_call(s, r) { + widest = widest.max(crate::render::str_width(&line.text())); + } + } + s.fold_here(call_row); + } + back_to_the_clip(s, row, shut); + } + println!("width {width}: {opened} calls opened, widest row under one {widest} of {width} columns"); + assert!(opened > 0, "no call opened at width {width}"); + assert!(widest <= width, "an opened call overflowed the view at width {width}: {widest}"); +} + +/// The call rows of `record`, found the way the pager finds them: a row whose +/// spot is one of that record's parts, and whose part is one that opens. +fn call_rows(s: &ArraySource, record: usize) -> Vec { + (s.row_of_record(record)..s.len()) + .take_while(|&r| s.record_at(r) == record) + .filter(|&r| match s.spot(r) { + Spot::Part { record: p, line } => { + p == record && s.with_parts(p, |l| l.call_at(line)).is_some() + } + _ => false, + }) + .collect() +} + +/// Is `row` a row *under* a call — an argument, the `output` label, a line of +/// what came back — rather than the call's own row? +fn under_a_call(s: &ArraySource, row: usize) -> bool { + match s.spot(row) { + Spot::Part { record, line } => s.with_parts(record, |l| l.call_at(line)).is_none(), + _ => false, + } +} + +/// Round the ladder — at most a full turn — back to the clip, so the next +/// record is measured from the same place. +fn back_to_the_clip(s: &mut ArraySource, row: usize, clipped: usize) { + for _ in 0..3 { + if s.len() == clipped { + return; + } + s.fold_here(row); + } +} + +// -- through the lens ----------------------------------------------------------- + +/// End to end: the same document read through `--lens atif` is a conversation +/// with the mechanics folded into runs, and every record is still reachable. +#[test] +fn the_lens_folds_the_mechanics_of_a_document() { + let body = br#"{"schema_version":"ATIF-v1.7","session_id":"sxs_1","steps":[ + {"step_id":1,"source":"user","message":"do it"}, + {"step_id":2,"source":"agent","message":"", + "tool_calls":[{"function_name":"bash","arguments":{"command":"ls"}}]}, + {"step_id":3,"source":"agent","message":"", + "tool_calls":[{"function_name":"bash","arguments":{"command":"make"}}]}, + {"step_id":4,"source":"agent","message":"done"}]}"# + .to_vec(); + let mut s = ArraySource::from_bytes(body, At::Key("steps")); + s.set_lens(crate::lens::find("atif").expect("the lens exists")); + s.set_width(120); + let painted = rows(&mut s); + // session, the prompt, the folded run of two steps, the answer. Each + // message here is one line, so it is entirely on its own summary row and + // nothing is painted under it. The session row is a headline over the + // envelope's keys rather than something anyone said, so it has no body + // either. + assert_eq!(painted.len(), 4, "{painted:#?}"); + assert!(painted[0].contains("session") && painted[0].contains("ATIF-v1.7"), "{}", painted[0]); + assert!(painted[1].contains("user") && painted[1].contains("do it"), "{}", painted[1]); + assert!(painted[2].contains("\u{27e8}2 steps \u{b7} 2 tool calls\u{27e9}"), "{}", painted[2]); + assert!(painted[3].contains("done"), "{}", painted[3]); + // Once each: the row is the message's first line, not an excerpt with the + // message repeated under it. + assert_eq!(painted.iter().filter(|r| r.contains("do it")).count(), 1, "{painted:#?}"); + assert_eq!(painted.iter().filter(|r| r.contains("done")).count(), 1, "{painted:#?}"); + // The status bar names the lens, and the record numbering says plainly + // that the session row is record 1 and `steps[0]` is record 2. + let text = s.position_text(1).expect("a position"); + assert!(text.starts_with("atif \u{b7} record 2/5"), "{text}"); + // A search hit inside the folded run opens it rather than being lost. + s.set_query("make"); + let hit = s.preview_match(crate::source::Anchor(0), crate::source::Dir::Forward); + assert!(hit.is_some(), "the folded run hides a record from search"); + assert!(rows(&mut s).iter().any(|r| r.contains("make")), "the run did not open"); +} + +/// `--toc --lens atif`: the trajectory as a list, one line per record, with the +/// session row first and the records the lens did not read keeping a generic +/// summary. +#[test] +fn the_toc_is_the_trajectory_as_a_list() { + let body = br#"{"schema_version":"ATIF-v1.7","steps":[ + {"step_id":1,"source":"user","message":"go","timestamp":"2026-04-23T10:55:31.1+00:00"}, + {"nothing":"this dialect knows"}]}"# + .to_vec(); + let mut s = ArraySource::from_bytes(body, At::Key("steps")); + s.set_lens(crate::lens::find("atif").expect("the lens exists")); + s.set_width(120); + let toc = s.summaries(100); + assert_eq!(toc.len(), 3); + assert_eq!(toc[0], "1\tsession\t\tATIF-v1.7"); + assert_eq!(toc[1], "2\tuser\t10:55\tgo"); + // Not this dialect's record: the generic summary, never a wrong headline. + assert!(toc[2].starts_with("3\t") && toc[2].contains("1 key"), "{}", toc[2]); +} + diff --git a/src/source/jsonl/block_tests.rs b/src/source/jsonl/block_tests.rs new file mode 100644 index 0000000..ff6d339 --- /dev/null +++ b/src/source/jsonl/block_tests.rs @@ -0,0 +1,162 @@ +//! Blocks, under a lens: what one is, where it ends, and what `Tab` / `S-Tab` +//! jump between (SPEC.md §Lenses). `j` and `k` are rows here as everywhere. +//! +//! Split from `lensrow_tests.rs`, whose fixture these share, to keep both files +//! under the size limit. +#![deny(unsafe_code)] + +use super::lensrow_tests::{lensed, rows, RUN}; +use crate::source::Source; + +/// A lens is what makes a document read in blocks, and the blocks are exactly +/// the landmarks `Tab` already stepped between — one definition, not two. +#[test] +fn a_lens_makes_a_document_read_in_blocks() { + let mut s = lensed(RUN); + let _ = rows(&mut s); + assert!(s.blocks()); + // The user prompt fits its summary row; the answer has one body row under + // it; the folded run and the last two messages are one row each. + assert_eq!(s.block_at(0), Some(0..1)); + assert_eq!(s.block_at(1), Some(1..3)); + assert_eq!(s.block_at(2), Some(1..3), "a body row is in its message's block"); + assert_eq!(s.block_at(3), Some(3..4)); + assert_eq!(s.block_at(5), Some(5..6)); + // Every block starts where the landmark before it ends. + for row in 0..s.len() { + let block = s.block_at(row).expect("a block"); + assert_eq!(s.next_landmark(block.start, true), Some(block.end).filter(|e| *e < s.len())); + } +} + +/// A block's rows include the tree of a record opened inside it, or framing +/// would scroll to the top of a record whose tree is off the bottom. +#[test] +fn an_open_record_grows_the_block_it_is_in() { + let mut s = lensed(RUN); + let _ = rows(&mut s); + let before = s.block_at(1).expect("a block"); + let entry = s.section_at(1).expect("an outline entry"); + assert!(s.set_fold(entry, false), "the record opened"); + let _ = rows(&mut s); + let after = s.block_at(1).expect("a block"); + assert_eq!(after.start, before.start); + assert!(after.end > before.end, "{before:?} -> {after:?}"); + assert_eq!(s.block_at(after.end - 1), Some(after.clone()), "the last row is still in it"); +} + +/// `Tab` is the block jump, and a block is every kind of thing the document +/// has: the messages, the folded run of mechanics, and the record no dialect +/// recognised. Nothing is skipped, which is why block and not message. +#[test] +fn tab_steps_between_blocks_and_skips_nothing() { + let mut s = lensed(RUN); + let _ = rows(&mut s); + assert_eq!(s.next_landmark(0, true), Some(1)); + assert_eq!(s.next_landmark(1, true), Some(3), "the run is a block too"); + assert_eq!(s.next_landmark(2, true), Some(3), "and from inside the body too"); + assert_eq!(s.next_landmark(3, true), Some(4)); + assert_eq!(s.next_landmark(4, true), Some(5), "the unrecognised record"); + assert_eq!(s.next_landmark(5, true), None); + // `S-Tab` is the mirror: back to this block's own row first, then out. + assert_eq!(s.next_landmark(2, false), Some(1), "back to this message first"); + assert_eq!(s.next_landmark(4, false), Some(3), "then the run"); + assert_eq!(s.next_landmark(0, false), None); +} + +/// The status bar counts blocks alongside records, in one vocabulary. The +/// totals are exact here because the whole file has been read and classified. +#[test] +fn the_status_bar_counts_blocks_too() { + let mut s = lensed(RUN); + let _ = rows(&mut s); + let text = s.position_text(3).expect("position"); + assert!(text.contains("record 3/8"), "{text}"); + assert!(text.contains("block 3/5"), "{text}"); + let last = s.position_text(5).expect("position"); + assert!(last.contains("block 5/5"), "{last}"); +} + +/// A boundary **descends into an open run**: the run's own row, then one block +/// per step in it (SPEC.md §Lenses). A shut run is one block — that is what +/// makes it a summary — so opening one is what changes the sequence. +#[test] +fn a_boundary_descends_into_an_open_run() { + let mut s = lensed(RUN); + let _ = rows(&mut s); + // Shut: the run is one block, and the jump steps over it. + assert_eq!(s.block_at(3), Some(3..4)); + assert_eq!(s.next_landmark(3, true), Some(4)); + let entry = s.section_at(3).expect("the run is an outline entry"); + assert!(s.set_fold(entry, false), "the run opens"); + let _ = rows(&mut s); + // Open: rows 4..8 are its four steps, each one a block of its own. + assert_eq!(s.block_at(3), Some(3..4), "the run's own row"); + assert_eq!(s.next_landmark(3, true), Some(4), "then the first step"); + for row in 4..8 { + assert_eq!(s.block_at(row), Some(row..row + 1), "step {row}"); + assert_eq!(s.next_landmark(row, true), Some(row + 1)); + assert_eq!(s.next_landmark(row, false), Some(row - 1), "and `S-Tab` mirrors it"); + } + assert_eq!(s.next_landmark(8, false), Some(7), "back into the run from below"); + // And every block still starts where the one before it ends. + for row in 0..s.len() { + let block = s.block_at(row).expect("a block"); + assert_eq!(s.next_landmark(block.start, true), Some(block.end).filter(|e| *e < s.len())); + } +} + +/// A member and the tree it has open are one block, exactly as a message and +/// its body are: `Tab` clears the member's rows in one press. +#[test] +fn a_member_with_an_open_tree_is_one_block() { + let mut s = lensed(RUN); + let _ = rows(&mut s); + let run = s.section_at(3).expect("the run"); + assert!(s.set_fold(run, false)); + let _ = rows(&mut s); + let member = s.section_at(4).expect("the first step of the run"); + assert!(s.set_fold(member, false), "its record opens"); + let _ = rows(&mut s); + let block = s.block_at(4).expect("the member's block"); + assert_eq!(block.start, 4); + assert!(block.end > 5, "its tree is inside it: {block:?}"); + assert_eq!(s.block_at(block.end - 1), Some(block.clone()), "the last tree row too"); + assert_eq!(s.next_landmark(4, true), Some(block.end), "one step clears it"); + assert_eq!(s.next_landmark(block.end - 1, false), Some(4), "and one comes back"); +} + +/// The status bar counts what `Tab` jumps by, so opening a run grows the total +/// rather than leaving the counter saying something the jump disagrees with. +#[test] +fn the_block_count_grows_when_a_run_opens() { + let mut s = lensed(RUN); + let _ = rows(&mut s); + assert!(s.position_text(3).expect("position").contains("block 3/5"), "shut"); + let entry = s.section_at(3).expect("the run"); + assert!(s.set_fold(entry, false)); + let _ = rows(&mut s); + let at_run = s.position_text(3).expect("position"); + assert!(at_run.contains("block 3/9"), "{at_run}"); + let first_step = s.position_text(4).expect("position"); + assert!(first_step.contains("block 4/9"), "{first_step}"); + let after = s.position_text(9).expect("position"); + assert!(after.contains("block 9/9"), "{after}"); +} + +/// `Tab` **does** descend into an open run, because a block boundary does and +/// `Tab` is the block jump: opening a run is the reader asking for what is in +/// it, and a jump that stepped over it would put those steps out of reach of +/// everything but `j`. +#[test] +fn tab_descends_into_an_open_run() { + let mut s = lensed(RUN); + let _ = rows(&mut s); + assert_eq!(s.next_landmark(3, true), Some(4), "shut: over the run"); + let entry = s.section_at(3).expect("the run"); + assert!(s.set_fold(entry, false), "the run opens"); + let _ = rows(&mut s); + assert_eq!(s.next_landmark(3, true), Some(4), "open: into its first step"); + assert_eq!(s.next_landmark(7, true), Some(8), "and out again at its end"); + assert_eq!(s.next_landmark(8, false), Some(7), "the mirror steps back in"); +} diff --git a/src/source/jsonl/lensrow.rs b/src/source/jsonl/lensrow.rs deleted file mode 100644 index 9b6a913..0000000 --- a/src/source/jsonl/lensrow.rs +++ /dev/null @@ -1,237 +0,0 @@ -//! The lens behind the record source: the state, and the rows it produces. -//! -//! One per record the lens read, one per run it folded. -//! -//! Everything visual about a lens is here, so a second dialect adds a module -//! under `src/lens/` and changes nothing about how a row looks. The grammar: -//! -//! ```text -//! ▾ user 21:29 I want a reader for the terminal… -//! ▸ ⟨6 steps · 4 tool calls⟩ 21:31 -//! assistant 21:31 Bash(cargo test) -//! ``` -//! -//! Three columns — who, when, what — because that is the order a reader -//! scanning a log asks in. The clock is `HH:MM` of the timestamp the file -//! records (UTC in every agent log seen so far): the date is the same for -//! nearly every row of a session, and the seconds are noise. -//! -//! A row is never wrapped, the same rule the tree rows follow: it scrolls -//! sideways, so one record is always one row and the fold arithmetic holds. -#![deny(unsafe_code)] - -use super::*; -use crate::lens::{Class, Summary}; -use crate::render::{str_width, visible}; -use crate::theme; - -/// Columns the speaker's name gets before the clock. -const ACTOR: usize = 10; - -/// Columns the clock gets. `HH:MM`. -const TIME: usize = 5; - -impl JsonlSource { - // -- the lens --------------------------------------------------------------- - - /// Read this file through `lens` (SPEC.md §Lenses). - /// - /// A transform over the records, not a different source: the file, the - /// index, the parser and every row below a summary are unchanged. What the - /// lens decides is what each record's *one* row says and which runs of - /// records share one. - pub fn set_lens(&mut self, lens: Box) { - self.plan = Some(Plan::new(lens)); - } - - /// The lens in force, for the status bar. - pub fn lens_name(&self) -> Option<&'static str> { - self.plan.as_ref().map(|p| p.lens_name()) - } - - /// Classify records up to and including `upto`, parsing each one once. - /// - /// The prefix the lens has read decides the grouping, so this runs *ahead* - /// of the painted window: rows above the viewport never move because their - /// records were classified before they were painted. - pub(super) fn classify_to(&mut self, upto: usize) { - if self.plan.is_none() { - return; - } - let target = upto.saturating_add(1).min(self.known()); - // Both are moved out for the loop: reading a record borrows `self` - // immutably through the `RefCell`s, and the classification writes to - // these two. Put back before returning, always. - let mut plan = match self.plan.take() { - Some(p) => p, - None => return, - }; - let mut map = std::mem::take(&mut self.map); - while plan.classified() < target { - let record = plan.classified(); - self.with_record(record, |rec| plan.classify(record, rec.value(), &mut map)); - } - plan.sync(); - self.plan = Some(plan); - self.map = map; - } - - /// Where a screen row falls. The one translation between rows and records, - /// with or without a lens. - pub(crate) fn spot(&self, row: usize) -> Spot { - match &self.plan { - Some(p) => p.at(row, self.known(), &self.map), - None => { - let (record, sub) = self.map.at(row); - Spot::Record { record, sub } - } - } - } - - /// The row a record's summary sits on — the group's row when a lens has - /// folded it away. - pub(crate) fn row_of_record(&self, record: usize) -> usize { - match &self.plan { - Some(p) => p.row_of_record(record, &self.map), - None => self.map.row_of(record), - } - } - - /// Is `record` currently a row of its own? - pub(super) fn record_visible(&self, record: usize) -> bool { - match &self.plan { - None => true, - Some(p) => match p.item_of_record(record) { - None => true, - Some(i) => p.item(i).map(|it| !it.is_group() || it.open).unwrap_or(true), - }, - } - } - - /// A record's row as the lens reads it, or `None` when the lens did not - /// recognise the record — which is not a failure: the caller then renders - /// the generic tree row, and nothing is lost (SPEC.md §Lenses). - pub(super) fn lens_row(&self, record: usize, inset: bool) -> Option { - let sum = self.plan.as_ref()?.summary(record)?; - let foldable = self.tree_len(record) > 0; - let spans = lens_spans(sum, foldable, inset); - Some(summary_line(spans, record + 1)) - } - - /// A folded run of mechanics: `⟨6 steps · 4 tool calls⟩`, with the clock of - /// the first step so the run still sits on the timeline. - /// - /// It always carries a fold marker, because a group only exists when there - /// is something inside it to open. - pub(super) fn group_row(&self, item: usize) -> Line { - let (first, count, calls) = self.group_counts(item); - let time = self - .plan - .as_ref() - .and_then(|p| p.summary(first)) - .and_then(|s| s.time.clone()) - .unwrap_or_default(); - let rest = vec![ - Span::new(pad("", ACTOR), theme::text()), - Span::new(pad(&time, TIME), theme::lens_time()), - Span::new(" ", theme::text()), - Span::new(group_text(count, calls), theme::lens_group()), - ]; - summary_line(marker(rest), first + 1) - } - - /// `(first record, records, tool calls)` of a group. - fn group_counts(&self, item: usize) -> (usize, usize, usize) { - let Some(plan) = self.plan.as_ref() else { - return (0, 0, 0); - }; - let Some(it) = plan.item(item) else { - return (0, 0, 0); - }; - let calls = (it.first..it.first + it.count) - .filter_map(|r| plan.summary(r)) - .map(|s| s.calls) - .sum(); - (it.first, it.count, calls) - } -} - -/// One summary row: never wrapped, and a landmark for `Tab`. -fn summary_line(spans: Vec, source_line: usize) -> Line { - Line { - spans, - block: 0, - source_line, - heading: None, - scroll: true, - kind: LineKind::Heading, - } -} - -/// `who · when · what`, with the gutter a foldable row needs. -fn lens_spans(sum: &Summary, foldable: bool, inset: bool) -> Vec { - let rest = vec![ - Span::new(pad(&sum.actor, ACTOR), sum.who.style()), - Span::new(pad(sum.time.as_deref().unwrap_or(""), TIME), theme::lens_time()), - Span::new(" ", theme::text()), - Span::new(visible(&sum.what), body_style(sum.class)), - ]; - let spans = match foldable { - true => marker(rest), - false => leaf(rest), - }; - inset_after_gutter(spans, inset) -} - -/// Indent a row that belongs to an open run — *after* its fold marker, never -/// before it. -/// -/// The gutter has to stay the first span on the row: the painter rewrites the -/// open marker to the closed one there when a row is folded shut, and a -/// leading indent span would leave every member of an open run showing the -/// wrong glyph. -pub(super) fn inset_after_gutter(mut spans: Vec, inset: bool) -> Vec { - if inset && !spans.is_empty() { - spans.insert(1, Span::plain(" ")); - } - spans -} - -/// Speech is the document; mechanics recede. -fn body_style(class: Class) -> crate::term::Style { - match class { - Class::Message => theme::text(), - Class::Step => theme::muted(), - } -} - -/// `⟨6 steps · 4 tool calls⟩`, and the singulars, and no call clause when the -/// run made no calls (a run of thoughts and bookkeeping). -fn group_text(steps: usize, calls: usize) -> String { - let steps = match steps { - 1 => "1 step".to_string(), - n => format!("{n} steps"), - }; - let calls = match calls { - 0 => String::new(), - 1 => " \u{b7} 1 tool call".to_string(), - n => format!(" \u{b7} {n} tool calls"), - }; - format!("\u{27e8}{steps}{calls}\u{27e9}") -} - -/// Pad to `cols` display columns — display columns, not bytes, because an -/// actor may be `↳ assistant`. -fn pad(text: &str, cols: usize) -> String { - let mut s = visible(text); - let w = str_width(&s); - if w < cols { - s.push_str(&" ".repeat(cols - w)); - } - s.push(' '); - s -} - -#[cfg(test)] -#[path = "lensrow_tests.rs"] -mod tests; diff --git a/src/source/jsonl/lensrow_tests.rs b/src/source/jsonl/lensrow_tests.rs index 89dc5f5..5327507 100644 --- a/src/source/jsonl/lensrow_tests.rs +++ b/src/source/jsonl/lensrow_tests.rs @@ -4,14 +4,15 @@ use super::*; use crate::source::Source; +use crate::source::{Anchor, Dir}; /// A small trajectory in the shape a Claude Code session file has: a prompt, /// an answer, a run of mechanics, another answer, and one record from a /// dialect nobody has taught this lens. -const RUN: &str = concat!( +pub(super) const RUN: &str = concat!( r#"{"type":"user","timestamp":"2026-08-05T14:01:00.000Z","message":{"role":"user","content":"add a lens"}}"#, "\n", - r#"{"type":"assistant","timestamp":"2026-08-05T14:02:00.000Z","message":{"role":"assistant","content":[{"type":"text","text":"On it."}]}}"#, + r#"{"type":"assistant","timestamp":"2026-08-05T14:02:00.000Z","message":{"role":"assistant","content":[{"type":"text","text":"On it.\nStarting now."}]}}"#, "\n", r#"{"type":"assistant","timestamp":"2026-08-05T14:02:10.000Z","message":{"role":"assistant","content":[{"type":"tool_use","id":"t1","name":"Bash","input":{"command":"git status"}}]}}"#, "\n", @@ -27,7 +28,7 @@ const RUN: &str = concat!( "\n", ); -fn lensed(text: &str) -> JsonlSource { +pub(super) fn lensed(text: &str) -> JsonlSource { let mut s = JsonlSource::from_bytes(text.as_bytes().to_vec()); s.set_lens(crate::lens::find("agent").expect("the agent lens")); s.set_width(200); @@ -35,7 +36,7 @@ fn lensed(text: &str) -> JsonlSource { s } -fn rows(s: &mut JsonlSource) -> Vec { +pub(super) fn rows(s: &mut JsonlSource) -> Vec { let n = s.len(); s.lines(0..n) .iter() @@ -49,22 +50,46 @@ fn rows(s: &mut JsonlSource) -> Vec { fn messages_stay_and_mechanics_fold_into_one_row() { let mut s = lensed(RUN); let got = rows(&mut s); - assert_eq!(got.len(), 5, "{got:#?}"); + // Three messages — each *starting* on its own summary row, with whatever it + // said past the first line under it — one folded run, and the record the + // lens does not know. + assert_eq!(got.len(), 6, "{got:#?}"); assert!(got[0].starts_with("\u{25be} user 14:01"), "{:?}", got[0]); + // A one-line message is entirely on its row: nothing under it repeats it. assert!(got[0].ends_with("add a lens"), "{:?}", got[0]); assert!(got[1].contains("assistant 14:02 On it."), "{:?}", got[1]); + assert_eq!(got[2].trim(), "Starting now.", "the second line, once: {:?}", got[2]); // Four mechanical records — two calls, two results — as one row. assert!( - got[2].contains("\u{27e8}4 steps \u{b7} 2 tool calls\u{27e9}"), + got[3].contains("\u{27e8}4 steps \u{b7} 2 tool calls\u{27e9}"), "{:?}", - got[2] + got[3] ); - assert!(got[2].contains("14:02"), "the run is still on the clock: {:?}", got[2]); - assert!(got[3].contains("assistant 14:03 Done."), "{:?}", got[3]); + assert!(got[3].contains("14:02"), "the run is still on the clock: {:?}", got[3]); + assert!(got[4].contains("assistant 14:03 Done."), "{:?}", got[4]); // The record the lens does not know is *not* hidden and not summarised: it - // is the generic collapsed-record row, exactly as with no lens. - assert!(got[4].contains("{\u{2026}2 keys}"), "{:?}", got[4]); - assert!(got[4].contains("telemetry"), "{:?}", got[4]); + // is the generic collapsed-record row, exactly as with no lens — and it has + // no body, because nothing read it as a message. + assert!(got[5].contains("{\u{2026}2 keys}"), "{:?}", got[5]); + assert!(got[5].contains("telemetry"), "{:?}", got[5]); +} + +/// The defect: the summary row painted the excerpt and the body then started +/// again from the same words. The row *is* the first line now, so the opening +/// of a message is on the screen exactly once. +#[test] +fn a_message_s_first_line_is_the_row_and_is_not_repeated_under_it() { + let mut s = lensed(RUN); + let got = rows(&mut s); + let head = got[1].split("14:02").nth(1).expect("the what column").trim().to_string(); + assert_eq!(head, "On it."); + let under: Vec<&String> = got[2..3].iter().collect(); + assert!(!under.iter().any(|r| r.trim() == head), "{head:?} repeated: {under:#?}"); + assert_eq!( + got.iter().filter(|r| r.contains("On it.")).count(), + 1, + "the opening words appear once: {got:#?}" + ); } /// Without `--lens`, nothing changes (SPEC.md §Lenses). @@ -84,20 +109,20 @@ fn no_lens_is_the_generic_tree() { fn a_group_opens_into_its_records() { let mut s = lensed(RUN); let _ = rows(&mut s); - let group = 2; + let group = 3; let entry = s.section_at(group).expect("the group is an outline entry"); assert!(s.set_fold(entry, false), "the group opens"); let got = rows(&mut s); - assert_eq!(got.len(), 9, "four records appear under it: {got:#?}"); - assert!(got[3].contains("Bash(git status)"), "{:?}", got[3]); - assert!(got[4].contains("Bash \u{2192} 5 bytes"), "{:?}", got[4]); - assert!(got[5].contains("Read(src/lens/mod.rs)"), "{:?}", got[5]); - assert!(got[6].contains("Read \u{2192} 2 lines"), "{:?}", got[6]); + assert_eq!(got.len(), 10, "four records appear under it: {got:#?}"); + assert!(got[4].contains("Bash(git status)"), "{:?}", got[4]); + assert!(got[5].contains("Bash \u{2192} 5 bytes"), "{:?}", got[5]); + assert!(got[6].contains("Read(src/lens/mod.rs)"), "{:?}", got[6]); + assert!(got[7].contains("Read \u{2192} 2 lines"), "{:?}", got[7]); // Members are indented under the run they belong to — after the fold // marker, which has to stay the first thing on the row (the painter // rewrites it there when the row is shut). - assert!(got[3].starts_with("\u{25be} "), "{:?}", got[3]); - let spans = s.lines(3..4).pop().expect("row").spans; + assert!(got[4].starts_with("\u{25be} "), "{:?}", got[4]); + let spans = s.lines(4..5).pop().expect("row").spans; assert!(spans[0].text.starts_with('\u{25be}'), "{:?}", spans[0].text); // And it closes again. (The outline is a window over the last frame, so @@ -105,7 +130,7 @@ fn a_group_opens_into_its_records() { let _ = rows(&mut s); let entry = s.section_at(group).expect("still there"); assert!(s.set_fold(entry, true)); - assert_eq!(rows(&mut s).len(), 5); + assert_eq!(rows(&mut s).len(), 6); } /// A closed run says how much it hides, so the gutter can show it and the @@ -114,7 +139,7 @@ fn a_group_opens_into_its_records() { fn a_closed_run_reports_what_it_hides() { let mut s = lensed(RUN); let _ = rows(&mut s); - assert_eq!(s.hidden_at(2), Some(4)); + assert_eq!(s.hidden_at(3), Some(4)); // A message row is foldable too — it opens into the raw record — so what // it reports is its own tree, not a run. assert_eq!(s.hidden_at(0), Some(s.line(0).map(|_| 7).unwrap_or(0))); @@ -126,15 +151,15 @@ fn a_closed_run_reports_what_it_hides() { fn a_record_inside_a_run_still_opens_into_its_tree() { let mut s = lensed(RUN); let _ = rows(&mut s); - let entry = s.section_at(2).expect("group"); + let entry = s.section_at(3).expect("group"); s.set_fold(entry, false); let before = rows(&mut s); - // Row 3 is the first record of the run. - let entry = s.section_at(3).expect("a record inside the run"); + // Row 4 is the first record of the run. + let entry = s.section_at(4).expect("a record inside the run"); assert!(s.set_fold(entry, false), "the record opens"); let after = rows(&mut s); assert!(after.len() > before.len(), "its tree was spliced in"); - assert!(after[4].contains("\"type\""), "{:?}", after[4]); + assert!(after[5].contains("\"type\""), "{:?}", after[5]); // Everything after the expansion is still where it belongs. assert!(after[after.len() - 1].contains("telemetry"), "{:?}", after); assert!(after.iter().any(|r| r.contains("Done.")), "{after:#?}"); @@ -146,15 +171,15 @@ fn a_record_inside_a_run_still_opens_into_its_tree() { fn closing_a_run_closes_what_was_open_inside_it() { let mut s = lensed(RUN); let _ = rows(&mut s); - let entry = s.section_at(2).expect("group"); + let entry = s.section_at(3).expect("group"); s.set_fold(entry, false); let _ = rows(&mut s); - let inner = s.section_at(3).expect("record"); + let inner = s.section_at(4).expect("record"); s.set_fold(inner, false); let _ = rows(&mut s); - let entry = s.section_at(2).expect("group"); + let entry = s.section_at(3).expect("group"); s.set_fold(entry, true); - assert_eq!(rows(&mut s).len(), 5, "back to the folded conversation"); + assert_eq!(rows(&mut s).len(), 6, "back to the folded conversation"); } // -- the rest of the seam ---------------------------------------------------------- @@ -163,7 +188,7 @@ fn closing_a_run_closes_what_was_open_inside_it() { fn the_status_bar_names_the_lens_and_the_record() { let mut s = lensed(RUN); let _ = rows(&mut s); - let text = s.position_text(2).expect("position"); + let text = s.position_text(3).expect("position"); assert!(text.contains("agent"), "{text}"); assert!(text.contains("record 3/8"), "{text}"); } @@ -177,7 +202,7 @@ fn a_match_inside_a_folded_run_is_revealed() { s.set_query("git status"); let hit = s.cycle_match(Anchor(0), Dir::Forward).expect("found"); let got = rows(&mut s); - assert!(got.len() > 5, "the run was opened: {got:#?}"); + assert!(got.len() > 6, "the run was opened: {got:#?}"); assert!(got[hit.anchor.0].contains("Bash(git status)"), "{:?}", got); } @@ -186,7 +211,7 @@ fn a_match_inside_a_folded_run_is_revealed() { fn yanking_a_run_copies_every_record_in_it() { let mut s = lensed(RUN); let _ = rows(&mut s); - let y = s.yank_section(2).expect("a run yanks"); + let y = s.yank_section(3).expect("a run yanks"); assert_eq!(y.what, "4 records"); assert_eq!(y.text.lines().count(), 4, "{}", y.text); for line in y.text.lines() { @@ -199,7 +224,7 @@ fn yanking_a_run_copies_every_record_in_it() { fn fold_state_round_trips() { let mut s = lensed(RUN); let _ = rows(&mut s); - let entry = s.section_at(2).expect("group"); + let entry = s.section_at(3).expect("group"); s.set_fold(entry, false); let open = rows(&mut s); let folds = s.folds(); @@ -223,20 +248,25 @@ fn fold_all_both_ways_keeps_every_record() { assert!(opened.iter().any(|r| r.contains(needle)), "{needle} missing"); } s.fold_all(true); - assert_eq!(rows(&mut s).len(), 5); + assert_eq!(rows(&mut s).len(), 6, "every body back to its clip too"); } -/// `Tab` moves between the things that stand on their own — messages and -/// runs — rather than through the records a run has folded away. +/// A landmark is the thing that stands on its own — a message, or a run — +/// rather than a record the run has folded away. Renamed for what it always +/// tested: every run here is **shut**, and a shut run is one landmark. Opening +/// one adds its steps to the sequence, which +/// `block_tests::a_boundary_descends_into_an_open_run` is the test for. #[test] -fn landmarks_are_the_items() { +fn landmarks_are_the_items_while_every_run_is_shut() { let mut s = lensed(RUN); let _ = rows(&mut s); assert_eq!(s.next_landmark(0, true), Some(1)); - assert_eq!(s.next_landmark(1, true), Some(2)); + // From inside a message's body, the next item is the next item. assert_eq!(s.next_landmark(2, true), Some(3)); - assert_eq!(s.next_landmark(4, true), None); - assert_eq!(s.next_landmark(3, false), Some(2)); + assert_eq!(s.next_landmark(1, true), Some(3)); + assert_eq!(s.next_landmark(3, true), Some(4)); + assert_eq!(s.next_landmark(5, true), None); + assert_eq!(s.next_landmark(4, false), Some(3)); assert_eq!(s.next_landmark(0, false), None); } @@ -300,3 +330,145 @@ fn the_table_of_contents_is_sanitised() { assert!(toc[0].contains("be\u{b7}ll"), "{:?}", toc[0]); } + +// -- the two states of a message, and the key that reaches them ------------------ + +/// A message the clip already shows whole, that made no calls, has no **open** +/// level: there is nothing between its headline and its JSON, so `Enter` has no +/// rung to descend and nothing moves. The key is still **claimed** — a record's +/// row answers for `Enter` even when the answer is "nowhere to go", because the +/// caller's fall-through is the outline and a record's outline entry is the raw +/// tree `r` owns. `r` is what shows that record, and it is one press either way. +#[test] +fn a_message_with_nothing_under_it_has_no_rung_and_r_shows_it() { + let mut s = lensed(RUN); + let _ = rows(&mut s); + let clipped = s.len(); + // Row 1 is `assistant 14:02 On it.`, row 2 the rest of what it said. + assert_eq!(s.fold_here(1), Some(false), "claimed, and no rung to descend"); + assert_eq!(s.fold_here(1), Some(false), "and the second press says the same"); + assert_eq!(s.len(), clipped, "and nothing was repainted"); + s.toggle_tree(1).expect("`r` reaches the record"); + let got = rows(&mut s); + assert!(got.iter().any(|r| r.contains("timestamp")), "the tree: {got:#?}"); + assert!(s.len() > clipped); + s.toggle_tree(1).expect("and shuts it again"); + assert_eq!(s.len(), clipped); +} + +/// The two levels, on a message long enough to have both: clipped, the whole of +/// what was said, and back to the clip — with the record's own JSON at neither +/// of them, because that is `r`. +#[test] +fn a_clipped_message_opens_and_comes_back_without_showing_its_json() { + let text = format!( + "{{\"type\":\"assistant\",\"message\":{{\"role\":\"assistant\",\"content\":[{{\"type\":\"text\",\"text\":\"{}\"}}]}}}}\n", + (1..=20).map(|n| format!("line {n}")).collect::>().join("\\n") + ); + let mut s = lensed(&text); + let clipped = s.len(); + assert_eq!(s.fold_here(0), Some(false), "the open level: the whole message"); + let open = s.len(); + assert!(open > clipped, "the whole message is on screen"); + let got = rows(&mut s); + assert!(got.iter().any(|r| r.trim() == "line 20"), "all of it"); + assert!(!got.iter().any(|r| r.contains("\"type\"")), "and no JSON: {got:#?}"); + + assert!(s.fold_here(0).is_some(), "and back to the clip in one press"); + assert_eq!(s.len(), clipped); + let back = rows(&mut s); + assert!(!back.iter().any(|r| r.trim() == "line 20"), "clipped again: {back:#?}"); + assert!(!back.iter().any(|r| r.contains("\"type\"")), "still no JSON: {back:#?}"); + + // The record itself is `r`, and it leaves the level where it was. + s.toggle_tree(0).expect("the record"); + assert!(rows(&mut s).iter().any(|r| r.contains("\"type\""))); + s.toggle_tree(0).expect("shut"); + assert_eq!(s.len(), clipped); +} + +/// The summary row and the body rows are one wrap split in two, so they must be +/// wraps of the *same* text. A terminal wide enough for a row to hold more than +/// `lens::BODY_KEEP` bytes wrapped the head into the row and the whole record +/// into the rows under it, and every word between the two first rows was +/// painted nowhere — a lens hiding something (SPEC.md §Lenses), silently, +/// because the row arithmetic stayed self-consistent throughout. +#[test] +fn a_message_wider_than_the_kept_head_is_whole_on_a_wide_terminal() { + let words: Vec = (0..500).map(|n| format!("w{n:03}")).collect(); + let text = format!( + "{{\"type\":\"assistant\",\"message\":{{\"role\":\"assistant\",\"content\":[{{\"type\":\"text\",\"text\":\"{}\"}}]}}}}\n", + words.join(" ") + ); + for width in [92usize, 1100, 2000] { + let mut s = lensed(&text); + s.set_width(width); + assert_eq!(s.fold_here(0), Some(false), "{width}: the message opens"); + let got = rows(&mut s).join(" "); + for w in &words { + assert_eq!( + got.split_whitespace().filter(|t| t == w).count(), + 1, + "{width}: {w} is on the screen exactly once" + ); + } + } +} + +// -- what a mark means ----------------------------------------------------------- + +/// With no lens nothing wraps: a tree row is one row at every width, so a mark +/// is the row and the cursor keeps its place inside an open record. Paying the +/// lens's price here would snap it back to the record's summary row for +/// nothing — and `Pager::relayout` runs on a height-only resize too. +#[test] +fn without_a_lens_a_mark_keeps_the_place_inside_a_record() { + let mut plain = JsonlSource::from_bytes(RUN.as_bytes().to_vec()); + plain.set_width(200); + while plain.extend() {} + let n = plain.len(); + let _ = plain.lines(0..n); + let entry = plain.section_at(0).expect("record 0 is an outline entry"); + assert!(plain.set_fold(entry, false), "open record 0"); + let m = plain.mark(3).expect("a mark on a tree row"); + plain.set_width(80); + assert_eq!(plain.locate(m), Some(3), "the same row, because it is the same content"); +} + +/// Under a lens it is the record, because a resize re-wraps every body. +#[test] +fn under_a_lens_a_mark_is_the_record() { + let mut s = lensed(RUN); + let _ = rows(&mut s); + let m = s.mark(2).expect("a mark on a body row"); + s.set_width(60); + assert_eq!(s.locate(m), Some(s.row_of_record(1)), "record 2's own row"); +} + + +/// Search highlights the *painted* columns. The summary row's text is the +/// message's first wrapped line now, not the flattened excerpt, so `row_text` +/// and `matches_on` have to be looking at the same string — a highlight offset +/// by the difference would underline the wrong words. +#[test] +fn search_highlights_the_summary_row_where_the_words_actually_are() { + let mut s = lensed(RUN); + let _ = rows(&mut s); + s.set_query("On it."); + at(&mut s, 1, "On it."); + // And the second line of the message is highlighted on its own row. + s.set_query("Starting now."); + at(&mut s, 2, "Starting now."); +} + +/// The one hit on `row` covers exactly where `needle` is painted. A `MatchSpan` +/// is in **display columns**, so the check is against the width of the row text +/// before the needle — which is the arithmetic the painter does. +fn at(s: &mut JsonlSource, row: usize, needle: &str) { + let spans = s.matches_on(row); + assert_eq!(spans.len(), 1, "{row}: {spans:?}"); + let text = s.line(row).expect("the row").text(); + let byte = text.find(needle).unwrap_or_else(|| panic!("{needle:?} is not on {text:?}")); + assert_eq!(spans[0].start, crate::render::str_width(&text[..byte]), "{text:?}"); + assert_eq!(spans[0].end - spans[0].start, crate::render::str_width(needle), "{text:?}"); +} diff --git a/src/source/jsonl/mod.rs b/src/source/jsonl/mod.rs index a12bdca..7c42b28 100644 --- a/src/source/jsonl/mod.rs +++ b/src/source/jsonl/mod.rs @@ -1,217 +1,69 @@ -//! `.jsonl` / `.ndjson` behind the [`Source`] seam (SPEC.md §JSON, "`.jsonl` / +//! `.jsonl` / `.ndjson`: a record per *line* (SPEC.md §JSON, "`.jsonl` / //! `.ndjson`"). //! -//! # A record per line, and none of them read until they are shown +//! # All that is left here is where a record starts and stops +//! +//! Rows, folding, search, yanking, the outline and the lens machinery are +//! [`crate::source::record`], shared with every other record format. This file +//! is the [`Store`] behind them, and it says one thing: a record is a line. //! //! * **The index is the CSV one.** A line-oriented file is a CSV without //! quoting, so this reuses [`RowStore`] whole — the lazy byte-offset index, //! the block-delta offset encoding, the sliding read window, the progress //! report — through [`crate::csv::index::RowStore::lines`], a scanner with //! quoting turned off. Nothing about a multi-GB file is read on the open -//! path: a `stat` and a 3-byte BOM peek, exactly as a CSV. -//! * **A record is parsed when it is shown, and only then.** The user's own -//! trajectory has a single 41KB line among 2285; laying out a screen by -//! parsing every line would be as bad as reading the whole file. Rendering a -//! row parses that row's record, and a small [`Cache`] keeps the ones the -//! viewport keeps asking about. -//! * **The default view is a list of records**, one row each -//! ([`tree::record_spans`]). `Enter` / `za` expands one into the tree -//! [`tree`] renders, spliced in under its row by [`RowMap`]. +//! path: a `stat` and a 3-byte BOM peek, exactly as a CSV. There is exactly +//! one line indexer in the crate and this is a caller of it, not a copy. //! * **A line that is not JSON is an error row** carrying the reason and the //! line number, and the file keeps rendering: half a log is still worth //! reading. -//! -//! # What is not folded is not parsed -//! -//! Fold state here is the *open* records, not the closed ones: everything -//! starts closed, so the closed set of a million-record file would be a million -//! ids. That is the same "default plus exceptions" shape the document source -//! uses, in the same vocabulary ([`jsonrow::ALL_OPEN`]) — a fold id is a -//! member-index path from the root, and a record file's root is the implicit -//! list of records, so record 4 is `/4`. One scheme, two sources. #![deny(unsafe_code)] -pub mod lensrow; -pub mod plan; -pub mod rowmap; -mod rows; -pub mod tree; -mod view; - #[cfg(test)] #[path = "tests.rs"] mod tests; +#[cfg(test)] +#[path = "lensrow_tests.rs"] +mod lensrow_tests; + +#[cfg(test)] +#[path = "block_tests.rs"] +mod block_tests; + use std::cell::RefCell; use std::io; -use std::ops::Range; use std::path::Path; -use super::search::{self, Dir}; -use super::{Anchor, End, Entry, FoldState, Hit, LinkSite, Mark, MatchSpan, Source}; use crate::csv::index::RowStore; use crate::csv::read::{self, Reader}; -use crate::json::{self, Value}; -use crate::lens::Lens; -use crate::render::{Line, LineKind, Span}; -use crate::source::jsonrow; -use crate::select::Yank; -use plan::{Plan, Spot}; -use rowmap::RowMap; - -/// Records the index is pushed past the painted window on every frame. -const LOOKAHEAD: usize = 1024; - -/// Records indexed when the layout width is first set, so the first screen is -/// there without waiting for an idle tick. -const FIRST_RECORDS: usize = 256; - -/// Bytes of scanning one [`Source::lines`] call may spend on that lookahead. -const FRAME_BYTES: u64 = 4 * 1024 * 1024; - -/// Bytes of scanning one idle tick may spend. -const IDLE_BYTES: u64 = 8 * 1024 * 1024; +use crate::json; +use crate::source::record::{Record, RecordSource, Store}; -/// Records one search sweep reads before giving up, as in the CSV source: a -/// sweep must fit inside a keystroke, so search covers the neighbourhood of the -/// cursor rather than a whole multi-GB file. -const SEARCH_RECORDS: usize = 20_000; - -/// Records the lens classifies past the painted window, so grouping is decided -/// before the rows it changes are asked for. Smaller than [`LOOKAHEAD`]: -/// classifying a record parses it, and the index is far cheaper than that. -const CLASS_AHEAD: usize = 256; - -/// Records read through the lens before the first frame: a screenful, not the -/// [`FIRST_RECORDS`] the index reaches, because classifying costs a parse -/// apiece and the frame extends it as far as it actually needs. -const FIRST_CLASS: usize = 64; - -/// Records classified per idle tick, so `G` reaches a lens-read end without a -/// keystroke ever waiting on the whole file. -const CLASS_SLICE: usize = 2048; - -/// Parsed records kept in hand. Small on purpose: a record can be megabytes, -/// and the viewport only ever asks about a screenful plus whatever the cursor -/// just left. -const CACHE: usize = 64; - -/// Records `zR` will open at once. -/// -/// Opening *every* record of a million-record log means parsing the whole file, -/// which is the one thing this source exists not to do. `zR` therefore opens as -/// far as the reader is looking and says so, rather than freezing (see -/// [`Source::fold_all`]). -const EXPAND_CAP: usize = 50_000; - -/// A record, as far as this reader got with it. -pub enum Record { - /// Valid JSON. - Value(Value), - /// Not JSON, and why. Rendered as an error row rather than stopping the - /// file (SPEC.md §JSON). - Bad(String), -} - -impl Record { - fn value(&self) -> Option<&Value> { - match self { - Record::Value(v) => Some(v), - Record::Bad(_) => None, - } - } -} +/// A `.jsonl` file behind the record seam. +pub type JsonlSource = RecordSource; -/// The parsed-record cache: most-recently-used first, [`CACHE`] deep. -#[derive(Default)] -struct Cache { - items: Vec<(usize, Record)>, -} - -impl Cache { - fn position(&self, record: usize) -> Option { - self.items.iter().position(|(r, _)| *r == record) - } - - /// Insert at the front, dropping the oldest if the cache is full. - fn push(&mut self, record: usize, rec: Record) { - if self.items.len() >= CACHE { - self.items.pop(); - } - self.items.insert(0, (record, rec)); - } -} - -pub struct JsonlSource { - /// Interior mutability because reading a line is a *file* read and half the - /// trait is `&self`. Every borrow is taken and dropped inside one helper. +/// The lazy line index, as a record store. +pub struct Lines { + /// Interior mutability because reading a line is a *file* read and every + /// [`Store`] method is `&self`. Every borrow is taken and dropped inside + /// one method; none nest. store: RefCell, - cache: RefCell, - /// The rows of the one expanded record the viewport is working through. - /// Capacity one: rebuilding it costs a walk of that record, and a frame - /// asks for it once per painted row. - laid: RefCell)>>, - /// Which records are open, and what that does to the row numbering. - map: RowMap, - /// The `--lens` reading of this file, when one was asked for. `None` is - /// the generic record tree, unchanged in every respect (SPEC.md §Lenses: - /// "without it, records render as the generic tree"). - plan: Option, - /// `zR` was pressed (or a dump asked for everything): records are opened as - /// the viewport reaches them, up to [`EXPAND_CAP`]. - expand_all: bool, - /// Records `expand_all` has already considered. - filled: usize, - view: usize, - /// Outline entries for the records last painted — see [`Source::outline`]. - outline: Vec, - /// Rows last painted. - window: Range, - query: String, - needle: String, - sensitive: bool, - /// Row of the current match, when the last sweep found one. - found: Option, - /// Always empty: a record file has no links. Held so the trait can hand - /// out a slice. - none_links: Vec, } -impl JsonlSource { - /// Open `path`. Stats it and reads three bytes; no line is indexed and no - /// record is parsed until one is asked for. - pub fn open(path: &Path) -> io::Result { - Ok(JsonlSource::new(RowStore::lines(Reader::open(path)?))) +impl Store for Lines { + fn known(&self) -> usize { + self.store.borrow().known() } - /// A source over bytes that arrived on a pipe. - pub fn from_bytes(data: Vec) -> JsonlSource { - JsonlSource::new(RowStore::lines(Reader::memory(data))) + fn complete(&self) -> bool { + self.store.borrow().complete() } - fn new(store: RowStore) -> JsonlSource { - JsonlSource { - store: RefCell::new(store), - cache: RefCell::new(Cache::default()), - laid: RefCell::new(None), - map: RowMap::default(), - plan: None, - expand_all: false, - filled: 0, - view: 80, - outline: Vec::new(), - window: 0..0, - query: String::new(), - needle: String::new(), - sensitive: false, - found: None, - none_links: Vec::new(), - } + fn progress(&self) -> u8 { + self.store.borrow().progress().percent() } - // -- the store ------------------------------------------------------------ - - /// Grow the line index toward `records`, spending at most `budget` bytes. fn index_to(&self, records: usize, budget: u64) { let mut guard = self.store.borrow_mut(); let s = &mut *guard; @@ -225,30 +77,26 @@ impl JsonlSource { } } - /// Records indexed so far. - pub(crate) fn known(&self) -> usize { - self.store.borrow().known() - } - - fn complete(&self) -> bool { - self.store.borrow().complete() + fn extend(&self, budget: u64) -> bool { + let mut guard = self.store.borrow_mut(); + let s = &mut *guard; + match s.index.complete() { + true => false, + false => { + let before = s.index.known(); + s.index.ensure_bytes(budget, &mut s.reader); + !s.index.complete() || s.index.known() > before + } + } } - /// The raw bytes of record `n`, terminator stripped. + /// The raw bytes of line `n`, terminator stripped. fn raw(&self, record: usize) -> Vec { let mut store = self.store.borrow_mut(); store.row(record).map(|s| s.data).unwrap_or_default() } - /// The raw text of record `n`, for the search sweep: no value is built and - /// no row is laid out, which is what keeps a sweep inside a keystroke. - fn raw_text(&self, record: usize) -> String { - String::from_utf8_lossy(&self.raw(record)).into_owned() - } - - // -- records --------------------------------------------------------------- - - /// Read and parse record `n`. The only place a line becomes a value. + /// Read and parse line `n`. The only place a line becomes a value. fn load(&self, record: usize) -> Record { let span = { let mut store = self.store.borrow_mut(); @@ -270,146 +118,26 @@ impl JsonlSource { } } - /// Run `f` against record `n`, parsing it if it is not already in hand. - /// - /// A closure rather than a returned reference because the record lives in a - /// [`RefCell`] and can be tens of megabytes: handing out a clone to read one - /// field would undo the point of the cache. - fn with_record(&self, record: usize, f: impl FnOnce(&Record) -> T) -> T { - if self.cache.borrow().position(record).is_none() { - let loaded = self.load(record); - self.cache.borrow_mut().push(record, loaded); - } - let cache = self.cache.borrow(); - match cache.position(record).and_then(|i| cache.items.get(i)) { - Some((_, rec)) => f(rec), - // Unreachable: it was just inserted. Answering rather than - // panicking is the rule this whole seam is held to. - None => f(&Record::Bad("unavailable".to_string())), - } - } - - /// Rows record `n`'s tree occupies. `0` for a scalar or an error row, which - /// is what makes them leaves with nothing to open. - fn tree_len(&self, record: usize) -> usize { - self.with_record(record, |r| r.value().map(tree::row_count).unwrap_or(0)) - } - - /// Run `f` against the laid-out rows of record `n`'s tree, laying them out - /// if the cache is holding a different record. - fn with_tree(&self, record: usize, f: impl FnOnce(&[Line]) -> T) -> T { - let hit = matches!(&*self.laid.borrow(), Some((r, _)) if *r == record); - if !hit { - let rows = self.with_record(record, |r| match r.value() { - Some(v) => tree::rows(v, record + 1), - None => Vec::new(), - }); - *self.laid.borrow_mut() = Some((record, rows)); - } - match &*self.laid.borrow() { - Some((_, rows)) => f(rows), - None => f(&[]), - } - } - - // -- search ------------------------------------------------------------------- - - /// Look for the query from row `from`, wrapping once, over record *source - /// text* — far cheaper than laying rows out, and it finds a value whose - /// row the viewport has cut off the right-hand side. - /// Returns the *record* it found and whether the sweep wrapped; turning - /// that into a row is [`JsonlSource::hit`]'s job, because it may have to - /// open a fold first. - fn sweep(&self, from: usize, dir: Dir, inclusive: bool) -> Option<(usize, bool)> { - let n = self.known(); - if self.query.is_empty() || n == 0 { - return None; - } - let step: isize = match dir { - Dir::Forward => 1, - Dir::Backward => -1, - }; - let start = self.record_at(from.min(self.len_rows().saturating_sub(1))); - let mut record = start as isize + isize::from(!inclusive) * step; - let mut wrapped = false; - for _ in 0..SEARCH_RECORDS.min(n) + 1 { - if record < 0 { - record = n as isize - 1; - wrapped = true; - } else if record >= n as isize { - record = 0; - wrapped = true; - } - if self.hits(record as usize) { - return Some((record as usize, wrapped)); - } - record += step; - } - None - } - - fn hits(&self, record: usize) -> bool { - let text = self.raw_text(record); - match self.sensitive { - true => text.contains(&self.needle), - false => text.to_lowercase().contains(&self.needle), - } + fn unit(&self) -> &'static str { + "line" } +} - /// Turn a swept *record* into a row, opening whatever folds it away: a - /// match the reader cannot see is not a match (SPEC.md §Lenses — a lens - /// may fold, but it may never lose a record). - fn hit(&mut self, found: Option<(usize, bool)>) -> Option { - let (record, wrapped) = found?; - self.reveal_record(record); - let row = self.row_of_record(record); - self.found = Some(row); - Some(Hit { anchor: Anchor(row), wrapped }) - } - - // -- yank ---------------------------------------------------------------------- - - fn yank(text: String, what: String) -> Option { - match text.is_empty() { - true => None, - false => Some(Yank { text, what }), - } +impl JsonlSource { + /// Open `path`. Stats it and reads three bytes; no line is indexed and no + /// record is parsed until one is asked for. + pub fn open(path: &Path) -> io::Result { + Ok(RecordSource::new(Lines::new(RowStore::lines(Reader::open(path)?)))) } - /// One row as source-faithful text: a string's own characters, anything - /// else as compact JSON. A string is the value a reader wants to paste - /// somewhere; re-quoting it would be exporting rather than copying, which - /// is the same rule the CSV form applies to a field. - fn row_json(&self, row: usize) -> Option { - self.at_row(row, |_, v| match v { - Value::Str(s) => s.clone(), - other => other.to_json(), - }) + /// A source over bytes that arrived on a pipe. + pub fn from_bytes(data: Vec) -> JsonlSource { + RecordSource::new(Lines::new(RowStore::lines(Reader::memory(data)))) } } -/// The fold id of record `r`, in the shared vocabulary -/// ([`jsonrow::ALL_OPEN`]): a record file's root is the implicit list of -/// records, so record 4 is `/4`. -fn fold_id(record: usize) -> String { - jsonrow::child_id("", record) -} - -/// A row that can be opened: the fold marker the painter rewrites to `\u{25b8}` -/// when the record is shut, then the summary. -fn marker(mut rest: Vec) -> Vec { - // Always the *open* glyph: the painter rewrites it to `\u{25b8}` on any row - // `hidden_at` claims, so emitting the closed one here would double-negate. - let glyph = crate::theme::MARKER_OPEN; - let mut spans = vec![Span::new(format!("{glyph} "), crate::theme::json_marker())]; - spans.append(&mut rest); - spans -} - -/// A row with nothing under it: the gutter stays, empty, so the values on it -/// line up with the ones on rows that do open. -fn leaf(mut rest: Vec) -> Vec { - let mut spans = vec![Span::plain(" ")]; - spans.append(&mut rest); - spans +impl Lines { + fn new(store: RowStore) -> Lines { + Lines { store: RefCell::new(store) } + } } diff --git a/src/source/jsonl/tests.rs b/src/source/jsonl/tests.rs index 4096d79..cb66be8 100644 --- a/src/source/jsonl/tests.rs +++ b/src/source/jsonl/tests.rs @@ -6,6 +6,7 @@ use std::path::PathBuf; use super::*; use crate::source::Source; +use crate::source::{Anchor, Dir, End, Mark}; // -- fixtures ---------------------------------------------------------------- @@ -160,6 +161,22 @@ fn enter_expands_a_record_into_its_tree() { assert_eq!(s.len(), 3); } +/// With **no lens** there is no ladder, so a record row has no fold of its own +/// and `Enter` must reach the outline — where a record's entry is exactly the +/// tree above. Under a lens the same row claims the key instead (the record's +/// two levels, and `r` owning the tree); this is the other side of that fork, +/// and the reason it is asked as "is there a plan" rather than assumed. +#[test] +fn without_a_lens_enter_falls_through_to_the_outline() { + let mut s = src(THREE); + let _ = s.lines(0..3); + assert_eq!(s.fold_here(1), None, "no rung of its own without a lens"); + assert_eq!(s.len(), 3, "and nothing opened behind the answer"); + let entry = s.section_at(1).expect("the outline still has the record"); + assert!(s.set_fold(entry, false), "which is what opens its tree"); + assert!(s.len() > 3); +} + #[test] fn fold_state_is_the_open_records_and_survives_a_round_trip() { let mut s = src(THREE); @@ -459,3 +476,16 @@ fn lines_returns_exactly_the_window_it_was_asked_for() { } } } + +/// The one place a `blocks()` default could silently catch a format: this is +/// the same `impl` that answers `true` under a lens. Without one a record file +/// is the generic tree — one record per row, one node per row — with nothing +/// for a landing to frame, and `Tab` is the next record. +#[test] +fn a_record_file_with_no_lens_does_not_read_in_blocks() { + let mut s = src(THREE); + let _ = s.lines(0..3); + assert!(!s.blocks()); + assert_eq!(s.block_at(0), None); + assert_eq!(s.next_landmark(0, true), Some(1), "still the next record"); +} diff --git a/src/source/markdown_tests.rs b/src/source/markdown_tests.rs index dfaa776..75b472a 100644 --- a/src/source/markdown_tests.rs +++ b/src/source/markdown_tests.rs @@ -170,3 +170,14 @@ fn an_empty_document_answers_everything_without_panicking() { assert_eq!(s.yank_block(0), None); assert_eq!(s.yank_rows(0..3), None); } + +/// `j` stays one row here as everywhere (SPEC.md §"Moving through a document"). +/// What `blocks()` answers is narrower: whether `Tab` should *frame* its +/// landing. A heading starts and ends on its own row, so there is nothing to +/// frame, and prose keeps the `false` default. +#[test] +fn prose_does_not_read_in_blocks() { + let s = MarkdownSource::new(md::parse(DOC)); + assert!(!s.blocks()); + assert_eq!(s.block_at(0), None); +} diff --git a/src/source/mod.rs b/src/source/mod.rs index 82eba8a..87bc1a8 100644 --- a/src/source/mod.rs +++ b/src/source/mod.rs @@ -5,6 +5,8 @@ //! format they are showing — the same discipline that lets the whole crate //! above `sys` stay platform-agnostic. Adding a format is one module here plus //! one arm in the detector; formats are compiled in, never loaded at runtime. +//! Not every module here is a format: [`record`] is the seam *between* the +//! record formats — it has no detector arm and never opens a file. //! //! # Coordinate systems //! @@ -44,11 +46,14 @@ pub mod detect; pub mod fold; pub mod dir; pub mod json; +pub mod jsonarray; pub mod jsonl; pub mod jsonrow; pub mod markdown; +pub mod record; pub mod search; pub mod text; +mod types; use std::ops::Range; @@ -56,104 +61,7 @@ use crate::render::Line; use crate::select::Yank; use search::Dir; -/// A place in the document that survives folding but not re-layout. -/// -/// Opaque to everything above the seam: the pager only ever compares anchors -/// (they order the same way the document reads) and hands them back to the -/// source it got them from. -#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] -pub struct Anchor(pub usize); - -/// A place in the *content* that survives re-layout, so a resize can put the -/// cursor back where it was. -#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] -pub struct Mark(pub usize); - -/// Opaque, format-defined fold state: the ids of the closed sections. -/// -/// The pager stores it in a history [`Snapshot`](crate::nav::history::Snapshot) -/// and hands it back verbatim; it never inspects an id. Ids must be stable -/// across re-layout, which is what lets folds survive a resize. -pub type FoldState = Vec; - -/// One entry of the document outline (`o`, and the collapse tree). -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct Entry { - /// Nesting depth, 1 = outermost. Drives indentation and the fold ranges. - pub level: u8, - /// Stable id for this section: the key fold state is stored under, and the - /// target of an anchor link (`#some-heading`). - pub id: String, - /// Text shown in the outline overlay. - pub text: String, - /// Where the section starts. - pub anchor: Anchor, - /// True when this section is currently folded shut. - pub folded: bool, -} - -/// One link occurrence in the document, in reading order. -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct LinkSite { - /// The row the link sits on, as an anchor (it may be folded away). - pub anchor: Anchor, - /// Display column the link starts at, within its row. - pub col: usize, - pub url: String, -} - -/// One search match on a row, in display columns of that row. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub struct MatchSpan { - pub start: usize, - pub end: usize, - /// True for the match the cursor is currently sitting on. - pub current: bool, -} - -/// Where `G` lands, and whether the format still has work to do to know. -/// -/// A format that discovers its document lazily — a CSV's row index — genuinely -/// does not know where the end is until it has scanned there, and the *worst* -/// answer is the confident one: jumping to the end of whatever happens to be -/// indexed puts the cursor in the middle of the file and says nothing about it. -/// [`End::Scanning`] is that honest "not yet", carrying the percentage the -/// status bar shows while the pager drives the scan a slice at a time. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum End { - /// The last row `G` should put the cursor on. - At(usize), - /// The end is not known yet; `0..=100` of the way there. - Scanning(u8), -} - -/// Where a search landed, and whether it wrapped around the document. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub struct Hit { - pub anchor: Anchor, - pub wrapped: bool, -} - -/// One row expanded into labelled fields. -/// -/// A grid shows as many columns as fit and no more, which is exactly wrong for -/// the row you actually care about: a wide CSV hides most of it off-screen, and -/// a ragged row can carry fields the header never named. This is that row read -/// the other way round — one field per line, label beside value, nothing -/// hidden. A future tree format would return a node's children the same way. -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct Detail { - /// What the overlay is titled, e.g. `Row 41`. - pub title: String, - /// `(label, value)`, in the format's own order. A field the format has no - /// name for still appears — labelled positionally rather than dropped. - /// - /// Values are **raw**: exactly the bytes the document holds, control - /// characters and all. Painting them is what makes them safe - /// ([`crate::render::visible`]), so that copying one yields the real value - /// rather than the dotted display form. - pub fields: Vec<(String, String)>, -} +pub use types::{Anchor, Detail, End, Entry, FoldState, Hit, LinkSite, Mark, MatchSpan}; /// A document behind the format seam. /// @@ -223,6 +131,41 @@ pub trait Source { false } + /// True when this document reads in **blocks** — a message with what was + /// said under it, or a folded run of mechanics — so that landing on one + /// with `Tab` should show the whole block rather than the row it starts on + /// (SPEC.md §Lenses). + /// + /// A block is whatever [`Source::next_landmark`] already steps between — + /// one definition of a boundary, not two. It says nothing about `j`/`k`, + /// which move one row in every format: a closed block is one row, so a + /// block-sized cursor step differs from a row step only where it would skip + /// rows that are on the screen. + /// + /// The default is `false` and every format but one keeps it: prose, a CSV + /// row, a source line, a tree node and a record with no lens each start and + /// end on their own row, so framing a landing has nothing to add. + fn blocks(&self) -> bool { + false + } + + /// The rows the block containing `row` occupies, for framing it when `Tab` + /// lands on it: a block that fits is scrolled fully on screen, one taller + /// than the viewport puts its first row at the top. + /// + /// `None` — the default — is a format with no blocks, and also a row a + /// blocked format cannot place yet (past a lazily classified prefix); the + /// pager then frames the landing row alone. + /// + /// A real accessor rather than two [`Source::next_landmark`] calls, because + /// that pair cannot answer at either end: forward from the last block it is + /// `None` though the block does have an extent, and backward from inside + /// one it gives that block's own header row, not the previous boundary. + fn block_at(&self, row: usize) -> Option> { + let _ = row; + None + } + /// `h` / `l`: the horizontal offset one step in `dir` (`-1` left, `+1` /// right) from `hoff`, when the format scrolls by something other than a /// fixed number of columns. `None` — the default — leaves the pager's own @@ -327,8 +270,13 @@ pub trait Source { /// summary in the gutter. fn hidden_at(&self, row: usize) -> Option; - /// `Tab` / `S-Tab`: the next or previous structural landmark strictly after - /// (before) `row`, in rows. `None` when there is none that way. + /// The next or previous structural landmark strictly after (before) `row`, + /// in rows. `None` when there is none that way. + /// + /// The one definition of a boundary in the crate, and the only one: `Tab` / + /// `S-Tab` step between these — headings in prose, declarations in code, + /// blocks under a lens. `j` / `k` never do; they move one row, in every + /// format (SPEC.md §"Moving through a document"). fn next_landmark(&self, row: usize, forward: bool) -> Option; /// Jump to a section by its [`Entry::id`] — an anchor link (`#slug`). @@ -407,6 +355,18 @@ pub trait Source { None } + /// `r`: open the raw record under `row` — or shut it again — whatever + /// else that row is doing, returning what to say about it. + /// + /// `None` — the default — means this format has no such thing, or there is + /// nothing to open here, and the pager says so rather than appearing to do + /// nothing. It exists because a lens row's `Enter` belongs to what was + /// *said*, which leaves the record itself with no key: a lens may never + /// hide anything (SPEC.md §Lenses), so the raw record needs one of its own. + fn toggle_tree(&mut self, _row: usize) -> Option { + None + } + /// What a folded row says instead of `(N lines)`. /// /// A line count is the right summary for a function body. It is the wrong diff --git a/src/source/record/body.rs b/src/source/record/body.rs new file mode 100644 index 0000000..330a857 --- /dev/null +++ b/src/source/record/body.rs @@ -0,0 +1,374 @@ +//! The message under a summary row: how tall it is, and what it paints. +//! +//! A lens row is a headline — `who · when · what`, one line, never wrapped — +//! and for a record that said something **the headline is the message's own +//! first line**. What follows it goes *under* it, wrapped to the view width and +//! indented to the `what` column, so a conversation reads as a conversation +//! rather than as a list of first lines. +//! +//! ```text +//! ▾ assistant 10:55 Reading the failing test first. The suite names a +//! fixture that no longer exists, so that is where +//! this starts. +//! ⋯ +37 lines +//! ``` +//! +//! One wrap, split in two. [`first_line`] is row 1 — the summary row's `what`, +//! never wrapped because it is already one line's worth — and [`rows`] paints +//! rows 2..N. The message's opening words therefore appear once, which they did +//! not when the row painted [`crate::lens::Summary::what`] and the body then +//! started again from the top. `what` itself is unchanged: `--toc` still prints +//! the one-line excerpt, which is the right answer for a list. +//! +//! # The two states, and why a clip may never hide +//! +//! A body is **clipped** to [`CLIP`] rows until the reader opens it +//! (`Enter` / `za`), and the last row then says what is not on screen — in the +//! message's own lines when it has more of them, in bytes when the remainder is +//! the tail of one long line. Opening it paints the whole message, and the raw +//! record is still one `r` away either way. A lens never hides anything +//! (SPEC.md §Lenses), so a clip that did not say what it left out would be the +//! one thing this seam is not allowed to do. +//! +//! # Its own wrapper +//! +//! `render::wrap` wraps *atoms* — styled runs, links, clusters that may not be +//! split — because it lays out inline markdown. A message here is plain text +//! that happens to contain newlines, and the only decisions are the word break +//! and the hard split of a word longer than the column. That is small enough to +//! state directly, and doing so keeps the markdown layout engine free to make +//! markdown's decisions. +//! +//! Nothing here reads a file, parses a record or knows a dialect: it is text, +//! a width and a flag in, rows out. +#![deny(unsafe_code)] + +use crate::lens::Body; +use crate::render::{str_width, take_width, visible, Line, LineKind, Span}; +use crate::term::Style; +use crate::theme; + +/// Rows a clipped message occupies in total — the summary row it starts on, +/// and the body rows under it — before it says how much more there is. +pub const CLIP: usize = 6; + +/// Columns before the message text: the gutter, the actor column, the clock, +/// and the two spaces before `what` — so a body sits exactly under the words +/// it belongs to. +pub const INDENT: usize = 21; + +/// How one body is laid out. +/// +/// The wrap, the clip and the note are the same for every stretch of text this +/// seam shows — a message, a step's reasoning, a tool call's argument, its +/// output — and this is what those four disagree about. Passing it as one value +/// rather than four arguments is not tidiness: it is what makes "the height and +/// the rows come from the same walk" checkable, because both take exactly this. +#[derive(Clone, Copy)] +pub struct Shape { + /// The view width. Columns are what is left of it after [`Shape::indent`]. + pub width: usize, + /// Columns before the text. + pub indent: usize, + /// Is the first wrapped line painted by a **summary row above** rather than + /// here? True for a message, whose row *is* its first line; false for + /// everything else, which starts on a row of its own. + pub split: bool, + pub style: Style, +} + +impl Shape { + /// What was said, under the row that says who said it: one wrap split + /// between the summary row and the rows below. + pub fn message(width: usize) -> Shape { + Shape { width, indent: INDENT, split: true, style: theme::lens_body() } + } + + /// Text that starts on its own row — a step's reasoning, a call's output. + /// Muted, because the row above it is the thing and this is the detail. + pub fn under(width: usize, indent: usize) -> Shape { + Shape { width, indent, split: false, style: theme::muted() } + } +} + +/// Columns of message text at a given view width. Never zero: a terminal +/// narrower than the indent still gets a body rather than an empty column. +fn columns(shape: Shape) -> usize { + shape.width.saturating_sub(shape.indent).max(8) +} + +/// Rows the body occupies **under** the summary row, at `width`, clipped or +/// whole. Zero for a message the summary row already shows all of. +/// +/// The one number the row arithmetic runs on ([`super::plan::Plan`]), which is +/// why it is derived from the same walk that paints — the same `lay`, the same +/// `skip(1)`: a height that disagreed with the painted rows by one would move +/// every row below it. +pub fn height(body: &Body, text: &str, shape: Shape, full: bool) -> usize { + let laid = lay(body, text, shape, full); + under(&laid, shape) + usize::from(laid.note.is_some()) +} + +/// The body's rows, ready to paint: the wrap from its **second** line on when a +/// summary row above carries the first, and every row of it when nothing does. +pub fn rows(body: &Body, text: &str, shape: Shape, full: bool, source_line: usize) -> Vec { + let laid = lay(body, text, shape, full); + let pad = " ".repeat(shape.indent); + let mut out: Vec = laid + .rows + .iter() + .skip(usize::from(shape.split)) + .map(|r| row(&pad, r, shape.style, source_line)) + .collect(); + if let Some(note) = laid.note { + out.push(row(&pad, ¬e, theme::lens_more(), source_line)); + } + out +} + +/// The message's first wrapped line: what the **summary row** paints in its +/// `what` column, at the width the body is laid out for. +/// +/// `None` only where there is nothing to wrap at all — an empty message, or one +/// written entirely in blank lines. The row is never wrapped itself: this is +/// already one row's worth of text, and it scrolls sideways like every other +/// summary row. +/// +/// `text` must be the text [`rows`] is given for the same record, or the two +/// wraps are of different strings and the split between them loses whatever +/// falls between their first rows. +pub fn first_line(body: &Body, text: &str, width: usize) -> Option { + lay(body, text, Shape::message(width), false).rows.into_iter().next() +} + +/// Rows of the wrap that fall under the summary row rather than on it. +fn under(laid: &Laid, shape: Shape) -> usize { + laid.rows.len().saturating_sub(usize::from(shape.split)) +} + +/// Does the clip leave anything out at this width? +/// +/// What `Enter` / `za` opens — and, when this is false, what that key must +/// *not* consume: a message that is entirely on screen has one state rather +/// than two, and the row's fold marker is the record's own tree. +pub fn clips(body: &Body, text: &str, shape: Shape) -> bool { + lay(body, text, shape, false).note.is_some() +} + +/// One painted row of a body. Not a heading and not foldable: the summary row +/// above it owns the fold, and `Tab` walks items rather than lines. +fn row(pad: &str, text: &str, style: crate::term::Style, source_line: usize) -> Line { + // A blank line in a message is a blank row, not twenty-one spaces: this + // goes down a pipe as often as it goes on a screen. + let spans = match text.is_empty() { + true => Vec::new(), + false => vec![Span::plain(pad), Span::new(text.to_string(), style)], + }; + Line { + spans, + block: 0, + source_line, + heading: None, + scroll: true, + kind: LineKind::Paragraph, + } +} + +/// A laid-out body: the rows it shows, and what it admits to leaving out. +struct Laid { + rows: Vec, + note: Option, +} + +/// How far the walk got through the text it was given, in the text's *own* +/// bytes — never in painted ones. +/// +/// Painted bytes cannot answer for a remainder: `visible` substitutes a +/// two-byte `·` for a one-byte control character, so a row of tabs paints wider +/// than it reads and a remainder computed from it goes negative — which, being +/// a `saturating_sub`, came out as "nothing left to show" over a message that +/// had been cut. Characters survive that substitution one for one, and this +/// counts them off against the source. +struct Walk { + /// Whole source lines the walk got through — painted, or blank and walked + /// over. Never a line a head cut off in the middle. + lines: usize, + /// Source bytes those rows account for. Exact while every line was painted + /// whole — the walk jumps to the next line's offset — and a floor inside a + /// line the clip cut, because the wrap drops the space it broke on. A floor + /// is the safe direction: it can only make the remainder look bigger. + bytes: usize, +} + +/// Wrap `text` — as much of the message as the caller has — to the width, stop +/// where the state says to stop, and state the remainder. +fn lay(body: &Body, text: &str, shape: Shape, full: bool) -> Laid { + let cols = columns(shape); + let limit = match full { + true => usize::MAX, + false => CLIP, + }; + // A clip never walks more of the message than it could possibly show: at + // four bytes to a column this is past any width's worth of [`CLIP`] rows, + // and it is what keeps painting a clipped 40 MB message a screenful of + // work. The height comes off this same walk, so the two cannot disagree. + let text = match full { + true => text, + false => head_of(text, 4 * (limit + 1).saturating_mul(cols)), + }; + // The caller may only have the head of the message. + let short = text.len() < body.bytes; + let (rows, walk) = walk_text(text, cols, limit, short); + Laid { note: note(body, &walk, text.len(), short), rows } +} + +/// Wrap every line of `text` to `cols`, stopping at `limit` rows, and count off +/// against the source what those rows accounted for. +/// +/// The one walk: [`rows`] paints what this returns and [`height`] counts it, so +/// anything done to the row list here is done to the height by construction. +fn walk_text(text: &str, cols: usize, limit: usize, short: bool) -> (Vec, Walk) { + let mut rows: Vec = Vec::new(); + let mut walk = Walk { lines: 0, bytes: 0 }; + let mut at = 0usize; + for line in text.split('\n') { + if rows.len() >= limit { + break; + } + // Where the line after this one starts — the newline included, so a + // message that ends in one is fully accounted for and says nothing + // more. `split` yields a last empty segment for it, and counting that + // segment's absence as missing content was a `⋯` row over a message + // every byte of which was on screen. + let next = (at + line.len() + 1).min(text.len()); + // The head cut its last line where the head ended, not where the line + // does, so that segment is not a line anyone has seen in full however + // many rows it took — and it is the *only* line the -1 ever applied to. + // Subtracting one whenever the caller held a head made the six-row clip + // of a long message claim one more hidden line than it had. + let cut = short && at + line.len() >= text.len(); + // A message that opens with blank lines has no headline in them: the + // summary row would paint an empty `what` column and the clip would + // spend its rows on nothing. They are walked over, not painted — they + // carry no content to hide, and the walk still counts them off. + if rows.is_empty() && line.trim().is_empty() { + walk.lines += usize::from(!cut); + walk.bytes = next; + at = next; + continue; + } + let wrapped = fold_line(&visible(line), cols); + let room = limit - rows.len(); + let whole_line = wrapped.len() <= room; + let mut chars = 0usize; + for r in wrapped.into_iter().take(room) { + chars += r.chars().count(); + rows.push(r); + } + if !whole_line { + walk.bytes = at + prefix_bytes(line, chars); + break; + } + walk.lines += usize::from(!cut); + walk.bytes = next; + at = next; + } + (rows, walk) +} + +/// What the body is not showing: its own lines when whole ones are missing, and +/// bytes when what is left is the tail of a line already begun. `None` when +/// every byte of the message is on screen. +fn note(body: &Body, walk: &Walk, seen: usize, short: bool) -> Option { + // Whether anything is missing is structural: the walk stopped short of the + // text it was given, or that text was only the head. A byte count decides + // *what* the row says, never whether there is one. + if walk.bytes >= seen && !short { + return None; + } + // Lines the walk saw whole — never the one a head cut short, which the walk + // itself declines to count ([`walk_text`]). + let lines = body.lines.saturating_sub(walk.lines); + // A message written as one long line has no lines to count off, and + // "+1 line" would say nothing about the ten kilobytes left of it. + if lines == 0 || body.lines == 1 { + // Source bytes against source bytes, and never silent: a clip that hid + // something and said nothing is the one thing this seam may not do + // (SPEC.md §Lenses). + let left = body.bytes.saturating_sub(walk.bytes).max(1); + return Some(format!("\u{22ef} +{} more", crate::source::jsonrow::size(left as u64))); + } + Some(match lines { + 1 => "\u{22ef} +1 line".to_string(), + n => format!("\u{22ef} +{n} lines"), + }) +} + +/// Bytes the first `chars` characters of `line` occupy. +fn prefix_bytes(line: &str, chars: usize) -> usize { + line.char_indices().nth(chars).map(|(i, _)| i).unwrap_or(line.len()) +} + +/// The first `bytes` of `text`, cut on a character boundary — a message may be +/// any UTF-8 at all, and cutting mid-character would panic. +fn head_of(text: &str, bytes: usize) -> &str { + if text.len() <= bytes { + return text; + } + let mut cut = bytes; + while cut > 0 && !text.is_char_boundary(cut) { + cut -= 1; + } + &text[..cut] +} + +/// One source line, wrapped to `cols` columns on word boundaries, splitting a +/// word that cannot fit on a line of its own. An empty line stays a row: a +/// blank line in a message is a paragraph break, and dropping it would re-flow +/// what someone wrote. +fn fold_line(line: &str, cols: usize) -> Vec { + let mut out: Vec = Vec::new(); + let mut cur = String::new(); + let mut used = 0usize; + for word in line.split(' ') { + let w = str_width(word); + if used > 0 && used + 1 + w > cols { + out.push(std::mem::take(&mut cur)); + used = 0; + } + if used > 0 { + cur.push(' '); + used += 1; + } + if w <= cols { + cur.push_str(word); + used += w; + continue; + } + // A word wider than the column: split it, because the alternative is a + // row that runs off the screen and takes the rest of the message with + // it. Every piece is whole characters — `take_width` never cuts one. + let mut rest = word; + while !rest.is_empty() { + let (piece, wide) = take_width(rest, cols - used); + if piece.is_empty() { + out.push(std::mem::take(&mut cur)); + used = 0; + continue; + } + cur.push_str(piece); + used += wide; + rest = &rest[piece.len()..]; + if !rest.is_empty() { + out.push(std::mem::take(&mut cur)); + used = 0; + } + } + } + out.push(cur); + out +} + +#[cfg(test)] +#[path = "body_tests.rs"] +mod tests; diff --git a/src/source/record/body_tests.rs b/src/source/record/body_tests.rs new file mode 100644 index 0000000..ba5ba95 --- /dev/null +++ b/src/source/record/body_tests.rs @@ -0,0 +1,340 @@ +//! The message under a row: its wrap, its clip, and what the clip admits to. +//! +//! Fixtures are written here by hand. The one rule every case below is really +//! testing is the non-negotiable one: what is not on screen is *stated*, and +//! the whole message is one keypress away. +//! +//! The wrap is *split*: row 1 is the summary row's `what` ([`first_line`]) and +//! rows 2..N are the body ([`rows`]). Every count below is therefore a count of +//! what is under the summary row, and the head of the message is asserted +//! against `first_line` rather than against the first body row. +use super::*; +use crate::lens::{Body, Step}; + +/// A body whose text is short enough to be kept whole. +fn body(text: &str) -> Body { + Body::new(text, vec![Step::Key("message")]) +} + +fn texts(lines: &[Line]) -> Vec { + lines.iter().map(|l| l.text().trim_end().to_string()).collect() +} + +/// What the summary row paints, and what the body paints under it. +fn split(b: &Body, width: usize, full: bool) -> (String, Vec) { + let head = first_line(b, b.text_in(None), width).unwrap_or_default(); + (head, texts(&rows(b, b.text_in(None), Shape::message(width), full, 1))) +} + +#[test] +fn one_short_line_is_the_summary_row_and_nothing_under_it() { + let b = body("hello there"); + let (head, laid) = split(&b, 80, false); + assert_eq!(head, "hello there"); + assert!(laid.is_empty(), "{laid:#?}"); + assert_eq!(height(&b, &b.head, Shape::message(80), false), 0); +} + +#[test] +fn an_empty_message_has_no_rows_under_its_summary() { + let b = body(""); + assert_eq!(height(&b, &b.head, Shape::message(80), false), 0); + // Nothing to wrap at all, so nothing on the summary row either — the actor + // and the clock still name the record. + assert_eq!(first_line(&b, &b.head, 80).unwrap_or_default(), ""); +} + +/// A message that opens with blank lines still says something on its summary +/// row: the headline is its first line with *content*, not the newline someone +/// happened to start with, and the clip is not spent on empty rows. +#[test] +fn leading_blank_lines_do_not_swallow_the_headline() { + let b = body("\n\n\nAFTER_BLANKS the real first line.\nsecond line here."); + for width in [20usize, 40, 60, 92, 400] { + let (head, laid) = split(&b, width, false); + // At eight columns the word itself is split; what matters at every + // width is that the headline starts where the message says something. + assert!(head.starts_with("AFTER_BL"), "{width}: {head:?}"); + assert!(!laid.iter().any(|r| r.trim().is_empty()), "{width}: {laid:#?}"); + assert_eq!(height(&b, &b.head, Shape::message(width), false), laid.len(), "{width}"); + // And past the width where this message needs the clip, nothing is + // claimed to be missing: the blank lines cost the reader no rows. + if width >= 40 { + assert!(!laid.iter().any(|r| r.trim().starts_with('\u{22ef}')), "{width}: {laid:#?}"); + } + } +} + +/// `⋯ +N lines` counts what is genuinely not shown. A message longer than the +/// head that the clip stops six rows into has *seen* those six lines; counting +/// the last of them as hidden — which the head's own cut line does deserve — +/// overstated every long message by one. +#[test] +fn the_clip_counts_only_the_lines_it_did_not_finish() { + let text = (0..20) + .map(|n| format!("line {n:02} aaaa bbbb cccc dddd eeee ffff gggg hhhh iiii jj")) + .collect::>() + .join("\n"); + let b = body(&text); + assert!(!b.whole(), "the fixture is longer than the head"); + let (head, laid) = split(&b, 92, false); + assert!(head.starts_with("line 00"), "{head:?}"); + // Lines 0..=5 are painted whole — one on the summary row, five under it — + // so fourteen of the twenty are not shown. + assert_eq!(laid[CLIP - 2].trim(), "line 05 aaaa bbbb cccc dddd eeee ffff gggg hhhh iiii jj"); + assert_eq!(laid[CLIP - 1].trim(), "\u{22ef} +14 lines"); +} + +#[test] +fn text_wraps_to_the_width_less_the_indent() { + let b = body("alpha beta gamma delta epsilon"); + // 40 columns leaves 19 for the message: "alpha beta gamma" on the summary + // row, then the rest under it. + let (head, laid) = split(&b, 40, false); + assert_eq!(head, "alpha beta gamma"); + assert_eq!(laid.len(), 1); + assert_eq!(laid[0].trim(), "delta epsilon"); + assert_eq!(height(&b, &b.head, Shape::message(40), false), laid.len()); +} + +/// The defect this split exists to fix: the opening words were on the summary +/// row *and* on the first row under it. +#[test] +fn the_first_line_is_never_painted_twice() { + let text = "I'll build from the sources only, diagnose any failures from local files, \ + then verify the result.\nsecond line\nthird line"; + let b = body(text); + for width in [40usize, 92, 200] { + let (head, laid) = split(&b, width, true); + assert!(!head.is_empty(), "{width}"); + assert!(!laid.iter().any(|r| r.trim() == head), "{width}: {head:?} repeated in {laid:#?}"); + let painted = std::iter::once(head.clone()).chain(laid.iter().map(|r| r.trim().to_string())); + let once: Vec = painted.filter(|r| *r == head).collect(); + assert_eq!(once.len(), 1, "{width}: {head:?}"); + } +} + +#[test] +fn a_narrower_width_makes_a_taller_body() { + let b = body("alpha beta gamma delta epsilon zeta eta theta"); + let wide = height(&b, &b.head, Shape::message(120), true); + let narrow = height(&b, &b.head, Shape::message(40), true); + assert!(narrow > wide, "{narrow} should exceed {wide}"); +} + +#[test] +fn a_word_longer_than_the_column_is_split_rather_than_lost() { + let b = body(&"x".repeat(40)); + let (head, laid) = split(&b, INDENT + 10, true); + assert_eq!(laid.len(), 3); + let all = format!("{head}{}", laid.concat()); + assert_eq!(all.replace(' ', ""), "x".repeat(40)); +} + +#[test] +fn blank_lines_survive_because_someone_wrote_them() { + let b = body("one\n\ntwo"); + // "one" is the summary row; the blank line and "two" are under it. + let (head, laid) = split(&b, 80, true); + assert_eq!(head, "one"); + assert_eq!(laid, vec![String::new(), format!("{}two", " ".repeat(INDENT))]); + assert_eq!(height(&b, &b.head, Shape::message(80), true), 2); +} + +/// Six rows for the message: its summary row, and five under it. +#[test] +fn a_clip_stops_at_six_rows_and_states_the_rest() { + let text = (1..=20).map(|n| format!("line {n}")).collect::>().join("\n"); + let b = body(&text); + assert_eq!(height(&b, &b.head, Shape::message(80), false), CLIP); + let (head, laid) = split(&b, 80, false); + assert_eq!(head, "line 1"); + assert_eq!(laid.len(), CLIP); + assert_eq!(laid[CLIP - 2].trim(), "line 6"); + // Six lines are on screen, so fourteen are not — counted from the summary + // row down, which is where the message now starts. + assert_eq!(laid[CLIP - 1].trim(), "\u{22ef} +14 lines"); +} + +#[test] +fn opening_the_body_shows_every_line_and_says_nothing_more() { + let text = (1..=20).map(|n| format!("line {n}")).collect::>().join("\n"); + let b = body(&text); + let (head, laid) = split(&b, 80, true); + assert_eq!(head, "line 1"); + assert_eq!(laid.len(), 19); + assert!(laid.last().unwrap().contains("line 20")); + assert_eq!(height(&b, &b.head, Shape::message(80), true), 19); +} + +#[test] +fn a_single_long_line_clipped_reports_what_is_left_in_bytes() { + let b = body(&"word ".repeat(2000)); + let laid = texts(&rows(&b, &b.head, Shape::message(80), false, 1)); + assert_eq!(laid.len(), CLIP); + let note = laid[CLIP - 1].trim().to_string(); + assert!(note.starts_with("\u{22ef} +"), "{note}"); + assert!(note.ends_with("more"), "{note}"); +} + +/// The clip may never be silent, and the arithmetic that says how much is left +/// runs on the message's own bytes. Painted bytes are not the message's: +/// `visible` substitutes a two-byte `·` for a one-byte control character, so a +/// line of tabs paints wider than it reads. +#[test] +fn a_line_of_control_characters_still_says_what_the_clip_cut() { + let mut text = "x".to_string(); + text.push_str(&"\t".repeat(1200)); + let b = body(&text); + let laid = texts(&rows(&b, b.text_in(None), Shape::message(300), false, 1)); + let note = laid.last().unwrap().trim().to_string(); + assert!(note.starts_with("\u{22ef} +"), "the clip cut, so it must say so: {laid:#?}"); + assert!(note.ends_with("more"), "{note}"); +} + +/// The same arithmetic on a line that *fits* the head: `word\t` repeated is +/// mostly control characters, and the tail after them is what must be counted. +#[test] +fn a_clipped_line_counts_its_tail_in_the_message_s_own_bytes() { + let mut text = "word\t".repeat(94); + text.push_str(&"z".repeat(79)); + let b = body(&text); + assert!(b.whole(), "the fixture fits the head"); + let laid = texts(&rows(&b, &b.head, Shape::message(100), false, 1)); + let note = laid.last().unwrap().trim().to_string(); + assert!(note.starts_with("\u{22ef} +"), "{laid:#?}"); + let left: usize = note + .trim_start_matches("\u{22ef} +") + .trim_end_matches(" bytes more") + .parse() + .expect("a byte count"); + // Six rows of message text were painted, and every one of those characters + // is one byte of the message: what is left is the rest of it, counted where + // the message lives rather than after `visible` widened it. + assert_eq!(left, text.len() - CLIP * (100 - INDENT), "{note}"); +} + +/// A message that ends in a newline is entirely on screen when its lines are. +/// `str::lines` drops that last empty segment and the wrap does not, and the +/// difference used to paint `⋯ +6 bytes more` over a message with nothing left. +#[test] +fn a_trailing_newline_is_not_a_line_that_was_left_out() { + let b = body("aaa\nbbb\nccc\nddd\neee\nfff\n"); + for width in [40usize, 100, 200] { + let laid = texts(&rows(&b, &b.head, Shape::message(width), false, 1)); + assert_eq!(laid.len(), 5, "{width}: {laid:#?}"); + assert!( + !laid.iter().any(|r| r.trim().starts_with('\u{22ef}')), + "{width}: nothing was left out: {laid:#?}" + ); + } +} + +/// And a message that wraps to exactly the clip says nothing more either. +#[test] +fn a_message_that_fills_the_clip_exactly_says_nothing_more() { + // Three lines, each wrapping to two rows at 40 columns (19 for the text). + let line = "alpha beta gamma delta epsilon"; + let b = body(&[line, line, line].join("\n")); + let laid = texts(&rows(&b, &b.head, Shape::message(40), false, 1)); + assert_eq!(laid.len(), CLIP - 1, "{laid:#?}"); + assert!(!laid.iter().any(|r| r.trim().starts_with('\u{22ef}')), "{laid:#?}"); +} + +/// The predicate `Enter` is answered with: two states, or one. +#[test] +fn a_body_that_fits_does_not_claim_to_clip() { + let short = body("hello there"); + assert!(!clips(&short, &short.head, Shape::message(80))); + let long = body(&(1..=20).map(|n| format!("line {n}")).collect::>().join("\n")); + assert!(clips(&long, &long.head, Shape::message(80))); + // And width decides it: the same message clips when it is narrow enough. + let wrapped = body("alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu"); + assert!(!clips(&wrapped, &wrapped.head, Shape::message(200))); + assert!(clips(&wrapped, &wrapped.head, Shape::message(30))); +} + +#[test] +fn a_message_longer_than_the_head_says_so_even_when_it_is_open() { + // Longer than `lens::BODY_KEEP`, so the summary only kept its head — and + // the caller (a test with no record in hand) can only paint that much. + let text = (1..=600).map(|n| format!("line {n}")).collect::>().join("\n"); + let b = body(&text); + assert!(!b.whole()); + let laid = texts(&rows(&b, b.text_in(None), Shape::message(80), true, 1)); + let note = laid.last().unwrap().trim().to_string(); + assert!(note.starts_with("\u{22ef} +"), "{note}"); + assert!(note.ends_with("lines"), "{note}"); + assert_eq!(height(&b, b.text_in(None), Shape::message(80), true), laid.len()); +} + +#[test] +fn a_control_character_never_reaches_the_screen() { + let b = body("before\u{7}after"); + let head = first_line(&b, &b.head, 80).unwrap_or_default(); + assert!(head.contains("before\u{b7}after"), "{head:?}"); +} + +#[test] +fn a_terminal_narrower_than_the_indent_still_paints_the_message() { + let b = body("alpha beta gamma"); + let (head, laid) = split(&b, 4, true); + assert!(!laid.is_empty()); + assert!(format!("{head}{}", laid.concat()).contains("alpha")); +} + +#[test] +fn the_height_is_always_the_number_of_rows_painted() { + let cases = ["", "one line", "a\nb\nc", &"word ".repeat(400), &"z".repeat(9000)]; + for text in cases { + let b = body(text); + for width in [20usize, 40, 80, 200] { + for full in [false, true] { + let n = height(&b, b.text_in(None), Shape::message(width), full); + let painted = rows(&b, b.text_in(None), Shape::message(width), full, 1).len(); + assert_eq!(n, painted, "{width} full={full} {}", text.len()); + } + } + } +} + +/// The height and the painted rows are one walk, and a disagreement of one row +/// moves everything below it. Messages of 1, 2, 7 and 300 lines, at the widths +/// a reader actually has, clipped and whole. +#[test] +fn the_height_and_the_rows_agree_at_every_width_and_length() { + for lines in [1usize, 2, 7, 300] { + let text = (1..=lines) + .map(|n| format!("line {n} of the message, long enough to wrap at forty columns")) + .collect::>() + .join("\n"); + let b = body(&text); + for width in [40usize, 92, 200] { + for full in [false, true] { + let n = height(&b, b.text_in(None), Shape::message(width), full); + let painted = rows(&b, b.text_in(None), Shape::message(width), full, 1); + assert_eq!(n, painted.len(), "{lines} lines, {width} cols, full={full}"); + // And the summary row is the wrap's first row, never repeated + // below: one message, painted once. + let head = first_line(&b, b.text_in(None), width).unwrap_or_default(); + assert!(head.starts_with("line 1 "), "{width}: {head:?}"); + assert!( + !texts(&painted).iter().any(|r| r.trim() == head), + "{lines} lines, {width} cols, full={full}: {head:?} painted twice" + ); + } + } + } +} + +/// A one-line message is entirely on its summary row at every width wide enough +/// to hold it, and nothing under it claims otherwise. +#[test] +fn a_message_that_fits_the_summary_row_has_no_body_at_all() { + let b = body("short enough"); + for width in [40usize, 92, 200] { + assert_eq!(height(&b, &b.head, Shape::message(width), false), 0, "{width}"); + assert!(rows(&b, &b.head, Shape::message(width), false, 1).is_empty(), "{width}"); + assert!(!clips(&b, &b.head, Shape::message(width)), "{width}"); + } +} diff --git a/src/source/record/ladder.rs b/src/source/record/ladder.rs new file mode 100644 index 0000000..973aff9 --- /dev/null +++ b/src/source/record/ladder.rs @@ -0,0 +1,160 @@ +//! The **ladder**: how far into a record `Enter` goes, and what that costs. +//! +//! SPEC.md §Lenses gives a record two levels, and this is the one place the key +//! that walks them lives: +//! +//! ```text +//! clipped <-> open +//! ``` +//! +//! The raw JSON tree is **not** a rung: it is `r`, which is orthogonal to the +//! level and lives in `rows.rs`. This module never opens a tree and never shuts +//! one, which is the whole of why `Enter` can no longer undo `r`'s work. +//! +//! A third `impl RecordSource`, beside `rows.rs` and `view.rs`, split out for +//! the same reason they were: three sides of one type, each under the size +//! limit. What is here rather than in `plan_rows.rs` is everything that needs a +//! **record** — the level a record is at is the plan's, but whether descending +//! to it would show anything, and how tall it is when it does, are questions +//! only whoever can read the file may answer. +#![deny(unsafe_code)] + +use super::*; + +impl RecordSource { + /// Run `f` against the laid-out parts of `record` — the open level's rows, + /// and which call each of them belongs to. + /// + /// The same one-record cache `body_laid` is, keyed by width, and for the + /// same reason: a frame asks per painted row, and asking a dialect what a + /// record contains once per row of it would be quadratic. + pub(crate) fn with_parts(&self, record: usize, f: impl FnOnce(&parts::Laid) -> T) -> T { + let width = self.view; + let hit = matches!(&*self.parts_laid.borrow(), Some((r, w, _)) if *r == record && *w == width); + if !hit { + let laid = match self.plan.as_ref() { + Some(plan) => self.with_value(record, |v| lensrow::part_rows(plan, record, v, width)), + None => parts::Laid::empty(), + }; + *self.parts_laid.borrow_mut() = Some((record, width, laid)); + } + match &*self.parts_laid.borrow() { + Some((_, _, laid)) => f(laid), + None => f(&parts::Laid::empty()), + } + } + + /// `Enter` / `za`: **the record's two levels** (SPEC.md §Lenses). + /// + /// ```text + /// clipped <-> open + /// ``` + /// + /// A record with nothing under its headline and no calls has no *open* + /// level, so it has no rung at all and says `None` — at **either** level, + /// asked before the level is consulted, because the two show the same rows. + /// `None` here does not mean the key is free: [`Source::fold_here`] claims + /// it for the record's row anyway, since the fall-through above is the + /// outline and a record's outline entry is the raw tree that belongs to + /// `r`. Its bytes are `r`, and only `r`. + /// + /// **With the record's tree open this leaves the tree alone** and toggles + /// the record's own rows underneath it. `r` owns the tree, and a key that + /// silently shut what another key opened is exactly what the third rung was + /// removed for; the way out of a tree is the `r` that opened it. + /// + /// On a **call row** the key belongs to that call rather than to the record: + /// it shows the arguments it was made with and the output it returned, and + /// shuts them again. That is the one place a row inside a record has a fold + /// of its own, and it is why this is not simply a level flag. + /// + /// `None` on a group's row: `Enter` there opens the run, through the + /// outline, exactly as it did. + pub(crate) fn descend(&mut self, row: usize) -> Option { + let record = match self.spot(row) { + // A call row's key is the call's. Anything else on the open + // level — a named stretch of text — is a row of the record, and + // the key means what it means on a body row. + Spot::Part { record, line } => match self.with_parts(record, |l| l.call_at(line)) { + Some(part) => return self.toggle_part(record, part), + None => record, + }, + Spot::Body { record, .. } => record, + Spot::Record { record, sub: 0 } => record, + _ => return None, + }; + // No rung at all — the clip already shows the whole of this record's + // text and it made no calls. Asked *before* the level is consulted, + // not only on the way in: `zR` puts every record at the open level, + // and a rungless one that answered "was open, now clipped" would spend + // a press repainting rows that are the same at both levels. + // `opens_further` is a question about the record and the width, never + // about which level it is at, so it is the same answer either way. + if !self.opens_further(record) { + return None; + } + let plan = self.plan.as_ref()?; + // Two levels, and nothing here touches `self.map`: a tree the reader + // opened with `r` stays open across both of them. + if !plan.full_at(record) { + return self.set_level(record, true); + } + // Already open: back to the clip. + self.set_level(record, false) + } + + /// Is there an **open** rung on this record — anything the clip is not + /// already showing? A whole message with no tool calls has none, and the + /// key would otherwise repaint the same rows and call it a descent. + fn opens_further(&self, record: usize) -> bool { + let Some(plan) = self.plan.as_ref() else { + return false; + }; + let clips = match plan.body_of(record) { + None => false, + Some((body, _)) => { + let shape = lensrow::shape_of(plan, record, plan.width()); + body::clips(body, body.text_in(None), shape) + } + }; + clips || self.with_value(record, |v| v.is_some_and(|v| !plan.detail(v).is_empty())) + } + + /// Put `record` at the open level or back to its clip, re-measuring it. + /// `None` when it was already there, so the caller can fall through. + fn set_level(&mut self, record: usize, full: bool) -> Option { + let plan = self.plan.as_mut()?; + let was = plan.full_at(record); + if !plan.set_full(record, full) { + return None; + } + self.remeasure_record(record); + Some(was) + } + + /// `Enter` on a call row: its arguments and its output, or neither. + fn toggle_part(&mut self, record: usize, part: usize) -> Option { + let plan = self.plan.as_mut()?; + let was = plan.part_open(record, part); + // One call opens at a time, so opening this one may have shut a call on + // another record — whose rows are gone and whose height has to be + // restated before anything asks for a row. + let other = plan.open_part_record().filter(|r| *r != record); + if !plan.set_part_open(record, part, !was) { + return None; + } + if let Some(other) = other { + self.remeasure_record(other); + } + self.remeasure_record(record); + Some(was) + } + + /// Re-lay one record — what a rung of the ladder costs. + fn remeasure_record(&mut self, record: usize) { + let item = self.plan.as_ref().and_then(|p| p.item_of_record(record)); + if let Some(item) = item { + self.remeasure_item(item); + } + } +} diff --git a/src/source/record/lensrow.rs b/src/source/record/lensrow.rs new file mode 100644 index 0000000..b913487 --- /dev/null +++ b/src/source/record/lensrow.rs @@ -0,0 +1,459 @@ +//! The lens over a record document: the rows it produces, and the arithmetic +//! that places them. +//! +//! One per record the lens read, one per run it folded. +//! +//! Everything visual about a lens is here, so a second dialect adds a module +//! under `src/lens/` and changes nothing about how a row looks. The grammar: +//! +//! ```text +//! ▾ user 21:29 I want a reader for the terminal that understands +//! markdown. +//! ▸ ⟨6 steps · 4 tool calls⟩ 21:31 +//! assistant 21:31 Bash(cargo test) +//! ``` +//! +//! The message starts in the `what` column and continues under it; a step — +//! `Bash(cargo test)` — has no message and is its own one line. +//! +//! Three columns — who, when, what — because that is the order a reader +//! scanning a log asks in. The clock is `HH:MM` of the timestamp the file +//! records (UTC in every agent log seen so far): the date is the same for +//! nearly every row of a session, and the seconds are noise. +//! +//! A *summary* row is never wrapped, the same rule the tree rows follow: it +//! scrolls sideways, so the headline of a record is always one row. For a +//! record that said something that headline **is the message's first line** +//! ([`headline`]), and the rest of what it said goes under it, wrapped, as the +//! body rows [`super::body`] lays out — one wrap, split between the row and the +//! rows below it, so nothing is painted twice. That is why an item's own rows +//! depend on the width and why a resize re-measures every one of them +//! ([`remeasure`]). +//! +//! # Free functions, not methods +//! +//! Nothing here is a method on a source, because the plan, the row map and the +//! records are three separate borrows: reading a record is `&self` on the +//! format, and classifying writes to the other two. Taking them as arguments is +//! what lets one loop hold all three, and it is why [`Records`] is a trait the +//! format implements rather than a struct that owns the lens state. +#![deny(unsafe_code)] + +use super::plan::{Plan, Spot, Under}; +use super::rowmap::RowMap; +use super::{body, leaf, marker, parts, Records}; +use crate::lens::{Class, Summary}; +use crate::json::Value; +use crate::render::{str_width, visible, Line, LineKind, Span}; +use crate::theme; + +/// Columns the speaker's name gets before the clock. +const ACTOR: usize = 10; + +/// Columns the clock gets. `HH:MM`. +const TIME: usize = 5; + +/// Classify records up to and including `upto`, parsing each one once. +/// +/// The prefix the lens has read decides the grouping, so this runs *ahead* +/// of the painted window: rows above the viewport never move because their +/// records were classified before they were painted. +pub(crate) fn classify_to(src: &R, plan: &mut Plan, map: &mut RowMap, upto: usize) { + let target = upto.saturating_add(1).min(src.known()); + while plan.classified() < target { + let record = plan.classified(); + src.with_value(record, |v| plan.classify(record, v, map)); + // The item the record landed in — a new one, or the run it extended. + measure(src, plan, plan.items().len().saturating_sub(1)); + } + flush(src, plan); + plan.sync(); +} + +/// Measure the members of every run that has just been opened. +/// +/// Opening a run is `Plan`'s decision and measuring its members is not: a +/// member's rows are a wrap at a width, over text this module has to read the +/// record for. The plan queues, this drains, and nothing asks a row question in +/// between — [`Plan::take_pending`]. +pub(crate) fn flush(src: &R, plan: &mut Plan) { + for item in plan.take_pending() { + measure(src, plan, item); + } +} + +/// How tall item `i`'s message is at the plan's width. +/// +/// The record is read only for a body that is **open and longer than +/// [`crate::lens::BODY_KEEP`]**, which is the one case the head cannot answer. +/// Every other item — a step, a group, a clipped body, a short message — is +/// measured from the summary alone, so a resize costs no parse. +pub(crate) fn measure(src: &R, plan: &mut Plan, item: usize) { + let Some(it) = plan.item(item).cloned() else { + return; + }; + // A shut run hides its members, and a hidden record owns no rows: measuring + // one would put rows into a prefix sum for a row that is never painted. + if it.is_group() { + if it.open { + for record in it.first..it.first + it.count { + measure_record(src, plan, record, true); + } + } + return; + } + measure_record(src, plan, it.first, false); +} + +/// How tall `record`'s own rows are: what it said, then its parts. +/// +/// The record is read for a body that is **open and longer than +/// [`crate::lens::BODY_KEEP`]**, and for a record at the **open level**, which +/// is the one that has parts. A clipped record — the default, and every record +/// a resize re-lays — is measured from its summary alone, so neither the first +/// frame nor a resize costs a parse. +fn measure_record(src: &R, plan: &mut Plan, record: usize, member: bool) { + let width = plan.width(); + let body = match plan.body_of(record) { + None => 0, + Some((b, full)) => { + let shape = shape_of(plan, record, width); + match full && !b.whole() { + true => src.with_value(record, |v| body::height(b, b.text_in(v), shape, full)), + false => body::height(b, b.text_in(None), shape, full), + } + } + }; + // Parts are the open level and nothing else: a clipped record does not + // reach for its dialect, so the default state of a document is free. + let parts = match plan.full_at(record) { + false => 0, + true => src.with_value(record, |v| part_rows(plan, record, v, width).rows.len()), + }; + plan.set_under(record, Under { body, parts }, member); +} + +/// The parts of `record`, laid out — the one place a dialect is asked what a +/// record holds, and the one place that answer becomes rows. +pub(crate) fn part_rows(plan: &Plan, record: usize, value: Option<&Value>, width: usize) -> parts::Laid { + let Some(value) = value else { + return parts::Laid::empty(); + }; + if !plan.full_at(record) { + return parts::Laid::empty(); + } + let list = plan.detail(value); + let open = |i: usize| plan.part_open(record, i); + parts::lay(&list, &open, Some(value), width, body::INDENT + inset_of(plan, record), record + 1) +} + +/// Columns a record's under-rows are pushed right by: two for a member of an +/// open run, whose summary row is itself inset ([`inset_after_gutter`]), and +/// none for anything else. +/// +/// Without it a step's reasoning and its calls sat two columns left of the +/// step's own words, which is the one thing SPEC.md §Lenses says a body must +/// not do — it is "indented to the same column". +fn inset_of(plan: &Plan, record: usize) -> usize { + match in_open_group(Some(plan), record) { + true => 2, + false => 0, + } +} + +/// How a record's own text is laid out. A **message** is one wrap split between +/// its summary row and the rows under it; a **step** keeps its row for what it +/// did and puts its reasoning wholly underneath, muted. [`Class`] is the whole +/// of that decision, and this is where it is spent. +pub(crate) fn shape_of(plan: &Plan, record: usize, width: usize) -> body::Shape { + let mut shape = match plan.summary(record).map(|s| s.class) { + Some(Class::Message) => body::Shape::message(width), + _ => body::Shape::under(width, body::INDENT), + }; + // A member of an open run is inset, and what it said goes under what it + // said. Only a *step* is ever a member, so this never moves the half of a + // message's wrap that [`headline`] paints. + shape.indent += inset_of(plan, record); + shape +} + +/// Re-lay every body, which is what a width change means here. Classification +/// is not repeated — it runs once per record, in file order — only the wrap. +pub(crate) fn remeasure(src: &R, plan: &mut Plan) { + for i in 0..plan.items().len() { + measure(src, plan, i); + } + flush(src, plan); + plan.sync(); +} + +/// The rows of the message under `record`, at `width`. +pub(crate) fn body_rows(src: &R, plan: Option<&Plan>, record: usize, width: usize) -> Vec { + let Some(plan) = plan else { + return Vec::new(); + }; + let Some((body, full)) = plan.body_of(record) else { + return Vec::new(); + }; + let line = record + 1; + let shape = shape_of(plan, record, width); + match full && !body.whole() { + true => src.with_value(record, |v| body::rows(body, body.text_in(v), shape, full, line)), + false => body::rows(body, body.text_in(None), shape, full, line), + } +} + +/// The whole message under `record`, as text — what `y` copies off a body row. +pub(crate) fn body_text(src: &R, plan: Option<&Plan>, record: usize) -> Option { + let plan = plan?; + let (body, _) = plan.body_of(record)?; + match body.whole() { + true => Some(body.head.clone()), + false => Some(src.with_value(record, |v| body.text_in(v).to_string())), + } +} + +/// Where a screen row falls. The one translation between rows and records, +/// with or without a lens. +pub(crate) fn spot(plan: Option<&Plan>, map: &RowMap, known: usize, row: usize) -> Spot { + match plan { + Some(p) => p.at(row, known, map), + None => { + let (record, sub) = map.at(row); + Spot::Record { record, sub } + } + } +} + +/// The row a record's summary sits on — the group's row when a lens has +/// folded it away. +pub(crate) fn row_of_record(plan: Option<&Plan>, map: &RowMap, record: usize) -> usize { + match plan { + Some(p) => p.row_of_record(record, map), + None => map.row_of(record), + } +} + +/// The record a row belongs to. A group's row stands for its first record, +/// which is what keeps `record N/M`, search and yanking agreeing with what the +/// cursor is on. +pub(crate) fn record_at(plan: Option<&Plan>, map: &RowMap, known: usize, row: usize) -> usize { + match spot(plan, map, known, row) { + Spot::Record { record, .. } => record, + // A body row belongs to the record that said it, so `record N/M`, + // search and `Y` all still name what the cursor is reading. So does a + // part row: a call is something the record did. + Spot::Body { record, .. } | Spot::Part { record, .. } => record, + Spot::Group { item } => item_first(plan, item), + } +} + +/// The first record of a plan item. +pub(crate) fn item_first(plan: Option<&Plan>, item: usize) -> usize { + plan.and_then(|p| p.item(item)).map(|it| it.first).unwrap_or(0) +} + +/// Is item `i` an open group? What the outline reports as unfolded. +pub(crate) fn item_open(plan: Option<&Plan>, item: usize) -> bool { + plan.and_then(|p| p.item(item)).map(|it| it.open).unwrap_or(false) +} + +/// Is this record a member of a group that is currently open? Its row is then +/// indented under the group's, so a run reads as one thing. +pub(crate) fn in_open_group(plan: Option<&Plan>, record: usize) -> bool { + let Some(plan) = plan else { + return false; + }; + plan.item_of_record(record) + .and_then(|i| plan.item(i)) + .map(|it| it.is_group() && it.open) + .unwrap_or(false) +} + +/// Is `record` currently a row of its own? +pub(crate) fn record_visible(plan: Option<&Plan>, record: usize) -> bool { + match plan { + None => true, + Some(p) => match p.item_of_record(record) { + None => true, + Some(i) => p.item(i).map(|it| !it.is_group() || it.open).unwrap_or(true), + }, + } +} + +/// A record's row as the lens reads it, or `None` when the lens did not +/// recognise the record — which is not a failure: the caller then renders +/// the generic tree row, and nothing is lost (SPEC.md §Lenses). +pub(crate) fn lens_row( + src: &R, + plan: Option<&Plan>, + record: usize, + inset: bool, +) -> Option { + let plan = plan?; + let sum = plan.summary(record)?; + let what = headline(src, plan, record, sum); + let spans = lens_spans(sum, &what, src.foldable(record), inset); + Some(summary_line(spans, record + 1)) +} + +/// What the `what` column paints. +/// +/// For a record that said something it is the message's **first wrapped line** +/// at the layout width, and [`body::rows`] paints the rest — so the opening +/// words are on the screen once rather than twice. For a step, which has no +/// message under it, it is the summary's own one-line description, unchanged. +/// +/// The width is the plan's, not the caller's: it is the width every body was +/// measured at ([`measure`]), and a headline wrapped to a different one would +/// be a first line the rows below it do not continue. +/// +/// The **text** is chosen the same way, and for the same reason: whatever +/// [`body_rows`] wraps, this wraps, because the two are one wrap split in two. +/// Wrapping the head here while the rows wrapped the whole record put the +/// characters between the two first rows on no row at all, on any terminal +/// wide enough for a row to hold more than [`crate::lens::BODY_KEEP`] bytes. +fn headline(src: &R, plan: &Plan, record: usize, sum: &Summary) -> String { + let Some(b) = sum.body.as_ref() else { + return visible(&sum.what); + }; + // A step's row is what it *did* — its reasoning goes wholly underneath, + // so the row keeps `what` rather than becoming the first line of a thought. + if sum.class != Class::Message { + return visible(&sum.what); + } + let width = plan.width(); + let full = plan.full_at(record); + match full && !b.whole() { + true => src.with_value(record, |v| body::first_line(b, b.text_in(v), width)), + false => body::first_line(b, b.text_in(None), width), + } + .unwrap_or_default() +} + +/// One `--toc` line for a record the lens read: the same three columns the +/// painted row has, tab-separated for a shell rather than padded for a screen. +/// +/// Through the sanitiser, exactly as the painted row is: a record may hold any +/// byte, and `--toc` writes straight to a terminal (SPEC.md §JSON, the shared +/// sanitiser). `None` is a record the lens did not read — the caller then +/// prints the generic one, and nothing is hidden. +pub(crate) fn toc_line(plan: Option<&Plan>, record: usize) -> Option { + let sum = plan?.summary(record)?; + let time = sum.time.clone().unwrap_or_default(); + let actor = visible(&sum.actor); + let what = visible(&sum.what); + Some(format!("{}\t{actor}\t{time}\t{what}", record + 1)) +} + +/// A folded run of mechanics: `⟨6 steps · 4 tool calls⟩`, with the clock of +/// the first step so the run still sits on the timeline. +/// +/// It always carries a fold marker, because a group only exists when there +/// is something inside it to open. +pub(crate) fn group_row(plan: Option<&Plan>, item: usize) -> Line { + let (first, count, calls) = group_counts(plan, item); + let time = plan + .and_then(|p| p.summary(first)) + .and_then(|s| s.time.clone()) + .unwrap_or_default(); + let rest = vec![ + Span::new(pad("", ACTOR), theme::text()), + Span::new(pad(&time, TIME), theme::lens_time()), + Span::new(" ", theme::text()), + Span::new(group_text(count, calls), theme::lens_group()), + ]; + summary_line(marker(rest), first + 1) +} + +/// `(first record, records, tool calls)` of a group. +fn group_counts(plan: Option<&Plan>, item: usize) -> (usize, usize, usize) { + let Some(plan) = plan else { + return (0, 0, 0); + }; + let Some(it) = plan.item(item) else { + return (0, 0, 0); + }; + let calls = (it.first..it.first + it.count) + .filter_map(|r| plan.summary(r)) + .map(|s| s.calls) + .sum(); + (it.first, it.count, calls) +} + +/// One summary row: never wrapped, and a landmark for `Tab`. +fn summary_line(spans: Vec, source_line: usize) -> Line { + Line { + spans, + block: 0, + source_line, + heading: None, + scroll: true, + kind: LineKind::Heading, + } +} + +/// `who · when · what`, with the gutter a foldable row needs. `what` is +/// [`headline`]'s, already through the sanitiser. +fn lens_spans(sum: &Summary, what: &str, foldable: bool, inset: bool) -> Vec { + let rest = vec![ + Span::new(pad(&sum.actor, ACTOR), sum.who.style()), + Span::new(pad(sum.time.as_deref().unwrap_or(""), TIME), theme::lens_time()), + Span::new(" ", theme::text()), + Span::new(what.to_string(), body_style(sum.class)), + ]; + let spans = match foldable { + true => marker(rest), + false => leaf(rest), + }; + inset_after_gutter(spans, inset) +} + +/// Indent a row that belongs to an open run — *after* its fold marker, never +/// before it. +/// +/// The gutter has to stay the first span on the row: the painter rewrites the +/// open marker to the closed one there when a row is folded shut, and a +/// leading indent span would leave every member of an open run showing the +/// wrong glyph. +pub(crate) fn inset_after_gutter(mut spans: Vec, inset: bool) -> Vec { + if inset && !spans.is_empty() { + spans.insert(1, Span::plain(" ")); + } + spans +} + +/// Speech is the document; mechanics recede. +fn body_style(class: Class) -> crate::term::Style { + match class { + Class::Message => theme::text(), + Class::Step => theme::muted(), + } +} + +/// `⟨6 steps · 4 tool calls⟩`, and the singulars, and no call clause when the +/// run made no calls (a run of thoughts and bookkeeping). +fn group_text(steps: usize, calls: usize) -> String { + let steps = match steps { + 1 => "1 step".to_string(), + n => format!("{n} steps"), + }; + let calls = match calls { + 0 => String::new(), + 1 => " \u{b7} 1 tool call".to_string(), + n => format!(" \u{b7} {n} tool calls"), + }; + format!("\u{27e8}{steps}{calls}\u{27e9}") +} + +/// Pad to `cols` display columns — display columns, not bytes, because an +/// actor may be `↳ assistant`. +fn pad(text: &str, cols: usize) -> String { + let mut s = visible(text); + let w = str_width(&s); + if w < cols { + s.push_str(&" ".repeat(cols - w)); + } + s.push(' '); + s +} diff --git a/src/source/record/lensstate.rs b/src/source/record/lensstate.rs new file mode 100644 index 0000000..35f2b2e --- /dev/null +++ b/src/source/record/lensstate.rs @@ -0,0 +1,190 @@ +//! The lens state on the source, and the thin forwarders the rest of it calls. +//! +//! A fourth `impl RecordSource`, split out for the size limit and for one more +//! reason: nothing about lenses is *decided* here. `plan`, `lensrow`, `parts` +//! and `ops` own the plan, the row arithmetic, the levels, group folding and +//! every painted row; this holds the state, because that is where the fold state +//! already lives, and forwards. Every method below is one line long on purpose — +//! the one exception, [`RecordSource::settle`], says in its own doc comment why +//! it cannot be. +#![deny(unsafe_code)] + +use super::*; + +impl RecordSource { + /// Read this file through `lens` (SPEC.md §Lenses). + /// + /// A transform over the records, not a different source: the file, the + /// index, the parser and every row below a summary are unchanged. What the + /// lens decides is what each record's *one* row says and which runs of + /// records share one. + pub fn set_lens(&mut self, lens: Box) { + self.plan = Some(Plan::new(lens)); + } + + /// The lens in force, for the status bar. + pub fn lens_name(&self) -> Option<&'static str> { + self.plan.as_ref().map(|p| p.lens_name()) + } + + /// Classify records up to and including `upto`. + pub(crate) fn classify_to(&mut self, upto: usize) { + // Both are moved out for the call: reading a record borrows `self` + // immutably through the `RefCell`s, and the classification writes to + // these two. Put back before returning, always. + let Some(mut plan) = self.plan.take() else { + return; + }; + let mut map = std::mem::take(&mut self.map); + lensrow::classify_to(&*self, &mut plan, &mut map, upto); + self.plan = Some(plan); + self.map = map; + } + + /// Where a screen row falls. + pub(crate) fn spot(&self, row: usize) -> Spot { + lensrow::spot(self.plan.as_ref(), &self.map, self.known(), row) + } + + /// The row a record's summary sits on. + pub(crate) fn row_of_record(&self, record: usize) -> usize { + lensrow::row_of_record(self.plan.as_ref(), &self.map, record) + } + + /// Is `record` currently a row of its own? + pub(crate) fn record_visible(&self, record: usize) -> bool { + lensrow::record_visible(self.plan.as_ref(), record) + } + + /// A record's row as the lens reads it, or `None` when the lens did not + /// recognise it — the caller then renders the generic tree row. + pub(crate) fn lens_row(&self, record: usize, inset: bool) -> Option { + lensrow::lens_row(self, self.plan.as_ref(), record, inset) + } + + /// A folded run of mechanics: `⟨6 steps · 4 tool calls⟩`. + pub(crate) fn group_row(&self, item: usize) -> Line { + lensrow::group_row(self.plan.as_ref(), item) + } + + /// The record a row belongs to. + pub(crate) fn record_at(&self, row: usize) -> usize { + lensrow::record_at(self.plan.as_ref(), &self.map, self.known(), row) + } + + /// The first record of a plan item. + pub(crate) fn item_first(&self, item: usize) -> usize { + lensrow::item_first(self.plan.as_ref(), item) + } + + /// Is this record inside a group that is open? Its row is inset if so. + pub(crate) fn in_open_group(&self, record: usize) -> bool { + lensrow::in_open_group(self.plan.as_ref(), record) + } + + // -- folding --------------------------------------------------------------- + // + // Group folding is the seam's, whole: these forward and decide nothing. The + // only fold left is opening a *record* into its tree, which costs a parse. + + /// Measure whatever a fold has just made visible. + /// + /// Opening a run is the plan's decision; how tall its members are is not — + /// a member's reasoning is a wrap at a width over text only the format can + /// read. Every path that opens a group ends here, because a row question + /// asked in between would be asked of a prefix sum that has not been told + /// about the rows it is now supposed to include. + fn settle(&mut self) { + let Some(mut plan) = self.plan.take() else { + return; + }; + lensrow::flush(&*self, &mut plan); + plan.sync(); + self.plan = Some(plan); + self.drop_laid(); + } + + /// `zR` under a lens opens the groups the viewport has reached. + pub(crate) fn open_groups(&mut self, upto_row: usize) { + ops::open_upto(self.plan.as_mut(), &mut self.map, upto_row); + self.settle(); + } + + /// `zM`. + pub(crate) fn close_groups(&mut self) { + ops::close_all(self.plan.as_mut(), &mut self.map); + } + + /// Open the group holding `record`, so a search hit is never left folded. + pub(crate) fn reveal_record(&mut self, record: usize) { + ops::reveal(self.plan.as_mut(), &mut self.map, record); + self.settle(); + } + + /// The fold id of whatever sits on `at` — a group's or a record's. + pub(crate) fn fold_id_at(&self, at: Spot) -> String { + ops::id_at(self.plan.as_ref(), at) + } + + /// Apply a fold id that may name a group; `None` leaves it to the record + /// half of [`Source::set_fold`]. + pub(crate) fn set_group_by_id(&mut self, id: &str, open: bool) -> Option { + let changed = ops::set_by_id(self.plan.as_mut(), &mut self.map, id, open); + self.settle(); + changed + } + + /// The open groups, as fold ids. + pub(crate) fn group_folds(&self) -> Vec { + ops::open_ids(self.plan.as_ref()) + } + + /// Shut every group, then reopen the ones the fold state names. + pub(crate) fn restore_groups(&mut self, folds: &[String]) { + ops::restore(self.plan.as_mut(), &mut self.map, folds); + self.settle(); + } + + /// `Tab` / `S-Tab` under a lens: the next block boundary, not the next + /// record — and inside an open run, its steps are blocks. + pub(crate) fn next_block_row(&self, row: usize, forward: bool) -> Option { + ops::next_block(self.plan.as_ref(), &self.map, row, forward) + } + + /// The rows of the block a row falls in. + pub(crate) fn block_rows(&self, row: usize) -> Option> { + ops::block_at(self.plan.as_ref(), &self.map, row) + } + + /// `(index, count)` of the block a row is on, for the status bar. + pub(crate) fn block_of_row(&self, row: usize) -> Option<(usize, usize)> { + ops::block_of_row(self.plan.as_ref(), &self.map, row) + } + + /// The status bar's block clause — ` · block 2/23` — or nothing at all. + /// + /// Nothing in three cases, each of them honest rather than tidy: with no + /// lens there are no blocks, and past the classified prefix the block a row + /// will end up in is not decided yet, so a number there would be one the + /// next keystroke changes. The total carries the same `≥` the record count + /// does, and for the same reason twice over: the lens has only read a + /// prefix, and grouping makes that total *shrink* as it catches up. It + /// *grows* on the keystroke that opens a run, whose steps are blocks while + /// it is open; both numbers are read off the one table `Tab` jumps by. + pub(crate) fn block_text(&self, row: usize) -> String { + let Some((i, n)) = self.block_of_row(row) else { + return String::new(); + }; + let classified = self.plan.as_ref().map(|p| p.classified()).unwrap_or(0); + let total = match self.complete() && classified >= self.known() { + true => format!("{n}"), + false => format!("\u{2265}{n}"), + }; + format!(" \u{b7} block {}/{total}", i.saturating_add(1)) + } + + /// `Y` on a group's row: every record the run holds. + pub(crate) fn yank_group(&self, item: usize) -> Option { + ops::yank_group(self, self.plan.as_ref(), item) + } +} diff --git a/src/source/record/mod.rs b/src/source/record/mod.rs new file mode 100644 index 0000000..1331db1 --- /dev/null +++ b/src/source/record/mod.rs @@ -0,0 +1,118 @@ +//! Records in general, above any one record *format* (SPEC.md §Lenses). +//! +//! # What lives here +//! +//! **The** record source, and everything a lens does to it: [`RecordSource`] +//! ([`source`], with [`view`][`source`]'s `Source` impl, its rows, and the +//! ladder `Enter` walks beside it), [`tree`] (one record laid out as tree rows), +//! [`plan`] (which records share a row, and the row arithmetic — with the level +//! each record is at and the heights that follow from it in its `plan_rows` +//! child), [`rowmap`] (which rows an open record owns), [`lensrow`] (the +//! who/when/what rows a lens paints, and the row-to-record translation under +//! them), [`body`] (one wrap and one clip for every stretch of text this seam +//! shows, and what the clip admits to), [`parts`] (the open level: a record's +//! tool calls as rows, each opening into its arguments and its output), [`ops`] +//! (what `zR`, `zM`, `Tab`, +//! `Y` and a fold id off the outline do to a plan), and the shared gutter and +//! fold vocabulary below — [`marker`], [`leaf`], and `/4` for a record beside +//! [`plan::group_id`]'s `g4` for a group, the two spellings that must not +//! collide. +//! +//! Nothing here opens a file, names a format, or mentions a line. What a record +//! *format* supplies is [`Store`]: how many records the index has found, how to +//! push it along inside a byte budget, and record `i` as bytes or as a value. +//! `src/source/jsonl/` is a record per line over the CSV lazy line index and +//! `src/source/jsonarray/` is an array inside a JSON document over the +//! structural one; both get rows, grouping, folding, search, yanking and the +//! outline for nothing, and neither holds a row number. A change all lenses +//! need belongs here, not in a dialect under `src/lens/`. +//! +//! What the format still owns is what costs a *parse*: opening one record into +//! its own tree, and how many rows that tree has. That is the line — grouping +//! is decided from summaries the plan already holds, so it is free and lives +//! here; expansion reads the record, so it stays with whoever can read it. +//! +//! # Why the trait is generic rather than `dyn` +//! +//! [`Records::with_value`] hands the record to a closure instead of returning a +//! reference, which makes the trait non-object-safe — deliberately. A record +//! can be tens of megabytes and lives behind a `RefCell` in the format that +//! owns it, so handing out a clone to read one field would undo the point of +//! its cache, and a borrow cannot outlive the cell. The `&mut dyn FnMut` +//! alternative would also forbid *returning* a `T` from the closure, which +//! every caller of the jsonl equivalent relies on. +//! +//! The non-object-safety costs nothing above the seam: the functions in +//! [`lensrow`] are free and generic, they are monomorphised inside this module, +//! and the format itself keeps implementing [`crate::source::Source`] +//! concretely — the pager still holds a `Box` and never learns that +//! any of this exists. +#![deny(unsafe_code)] + +pub mod body; +pub mod lensrow; +pub mod ops; +pub mod parts; +pub mod plan; +pub mod rowmap; +mod source; +pub mod store; +pub mod tree; + +pub use source::RecordSource; +pub use store::{Record, Store}; + +use crate::json::Value; +use crate::render::Span; +use crate::source::jsonrow; + +/// A document that is a sequence of records, as the lens machinery needs to see +/// one: how many records are indexed, how to reach record `i`, and whether that +/// record has anything under it to open. +/// +/// Three methods, and each is here because something above it cannot be +/// answered without the format: the index is lazy so `known` grows, reading a +/// record is a file read the format owns, and "does this record open" is a +/// count of the rows the format's own tree renderer would produce. +pub trait Records { + /// Records indexed so far. Grows as the index is extended; never the whole + /// file unless the file has been read. + fn known(&self) -> usize; + + /// Run `f` against record `i`'s value, parsing it if the format does not + /// already have it in hand. `None` is a record that is not valid JSON — + /// which is not an error here: it keeps its own row and renders as + /// whatever the format renders a bad record as. + fn with_value(&self, i: usize, f: impl FnOnce(Option<&Value>) -> T) -> T; + + /// Has record `i` a tree under it? Decides whether a lens row gets a fold + /// marker. The format answers because the tree is the format's. + fn foldable(&self, i: usize) -> bool; +} + +/// The fold id of record `r`, in the shared vocabulary +/// ([`jsonrow::ALL_OPEN`]): a record document's root is the implicit list of +/// records, so record 4 is `/4`. The twin of [`plan::group_id`]'s `g4`, and +/// they live together so "these two cannot collide" is one fact in one place. +pub(crate) fn fold_id(record: usize) -> String { + jsonrow::child_id("", record) +} + +/// A row that can be opened: the fold marker the painter rewrites to `\u{25b8}` +/// when the record is shut, then the summary. +pub(crate) fn marker(mut rest: Vec) -> Vec { + // Always the *open* glyph: the painter rewrites it to `\u{25b8}` on any row + // `hidden_at` claims, so emitting the closed one here would double-negate. + let glyph = crate::theme::MARKER_OPEN; + let mut spans = vec![Span::new(format!("{glyph} "), crate::theme::json_marker())]; + spans.append(&mut rest); + spans +} + +/// A row with nothing under it: the gutter stays, empty, so the values on it +/// line up with the ones on rows that do open. +pub(crate) fn leaf(mut rest: Vec) -> Vec { + let mut spans = vec![Span::plain(" ")]; + spans.append(&mut rest); + spans +} diff --git a/src/source/record/ops.rs b/src/source/record/ops.rs new file mode 100644 index 0000000..2c4cf91 --- /dev/null +++ b/src/source/record/ops.rs @@ -0,0 +1,156 @@ +//! What the fold keys do to a plan: `zR`, `zM`, a fold id off the outline, +//! `Tab`, `Y` on a group's row, and revealing a search hit. +//! +//! # Why these are here and not on the format +//! +//! Every function below touches a [`Plan`] and a [`RowMap`] and nothing else — +//! no file, no parser, no `.jsonl`. They were methods on the record source once, +//! which meant a second record format would have implemented [`Records`], +//! watched `zR`, `zM`, `Tab` and fold-state restore do nothing, and copied them +//! across; two copies of the group arithmetic is exactly the drift the seam +//! exists to prevent. +//! +//! # The `Option<&mut Plan>` in every signature +//! +//! A document with no `--lens` has no plan, and "no lens" is the answer these +//! give: nothing to open, nothing to close, no group ids in the fold state. The +//! `None` arm lives here rather than at each call site so a format cannot get it +//! wrong in one place out of six. +//! +//! # The fold-id vocabulary +//! +//! [`id_at`] is the one place a group's `g4` and a record's `/4` meet +//! ([`super::fold_id`], [`plan::group_id`]), and [`set_by_id`] is the one place +//! an id off the outline is told apart: `None` is "this names no group", which +//! the format then reads as one of its own records — the only half of folding +//! that needs the format at all, because opening a record costs a parse. +#![deny(unsafe_code)] + +use crate::select::Yank; + +use super::plan::{self, Plan, Spot}; +use super::rowmap::RowMap; +use super::{fold_id, lensrow, Records}; + +/// `zR`: open the groups the viewport has reached, the same bounded way a +/// format opens records. Opening *every* group of a million-record log would +/// mean classifying all of it, which is the one thing a lens must never do. +pub(crate) fn open_upto(plan: Option<&mut Plan>, map: &mut RowMap, upto_row: usize) { + if let Some(plan) = plan { + plan.open_upto(upto_row, map); + } +} + +/// `zM`: shut every group, which is free — nothing is parsed to close one. +pub(crate) fn close_all(plan: Option<&mut Plan>, map: &mut RowMap) { + if let Some(plan) = plan { + plan.close_all(map); + plan.sync(); + } +} + +/// Open the group holding `record`, so a search hit is never left behind a +/// fold (SPEC.md §Lenses: a lens may fold, but it may never lose a record). +/// Does nothing when there is no lens, or the record is not in a group. +pub(crate) fn reveal(plan: Option<&mut Plan>, map: &mut RowMap, record: usize) { + let Some(plan) = plan else { + return; + }; + if let Some(item) = plan.item_of_record(record) { + plan.set_open(item, true, map); + plan.sync(); + } +} + +/// Open or close the group whose first record is `first`. +pub(crate) fn set_group(plan: Option<&mut Plan>, map: &mut RowMap, first: usize, open: bool) -> bool { + let Some(plan) = plan else { + return false; + }; + let changed = match plan.item_of_record(first) { + Some(item) => plan.set_open(item, open, map), + None => false, + }; + plan.sync(); + changed +} + +/// The fold id of whatever sits on `at`: a group's `g4`, or a record's `/4`. +pub(crate) fn id_at(plan: Option<&Plan>, at: Spot) -> String { + match at { + Spot::Group { item } => plan::group_id(lensrow::item_first(plan, item)), + // A body row answers with its record's id, so `za` on the message and + // `za` on the row above it are the same fold. A part row too: a call is + // a row inside the record, not a fold of its own. + Spot::Record { record, .. } | Spot::Body { record, .. } | Spot::Part { record, .. } => fold_id(record), + } +} + +/// Apply a fold id that may name a group, returning whether anything changed. +/// +/// `None` is an id that names no group — the format reads it as one of its own +/// records, which is the half of folding it has to own because opening a record +/// costs a parse. +pub(crate) fn set_by_id(plan: Option<&mut Plan>, map: &mut RowMap, id: &str, open: bool) -> Option { + let first = plan::group_first(id)?; + Some(set_group(plan, map, first, open)) +} + +/// The ids of the groups that are open: the exceptions a `FoldState` carries, +/// since a group starts shut. +pub(crate) fn open_ids(plan: Option<&Plan>) -> Vec { + let Some(plan) = plan else { + return Vec::new(); + }; + let open = plan.items().iter().filter(|it| it.is_group() && it.open); + open.map(|it| plan::group_id(it.first)).collect() +} + +/// Restore group folds from a `FoldState`: everything shut, then the ids that +/// disagree opened. Ids naming no group are left for the format. +pub(crate) fn restore(plan: Option<&mut Plan>, map: &mut RowMap, folds: &[String]) { + let Some(plan) = plan else { + return; + }; + close_all(Some(&mut *plan), map); + for id in folds { + if let Some(first) = plan::group_first(id) { + set_group(Some(&mut *plan), map, first, true); + } + } +} + +/// `Tab` / `S-Tab` under a lens: the next block boundary — a message, a shut +/// run, or, inside a run the reader has opened, one of the steps in it. +/// [`Plan::next_block`] is the crate's single definition of where a block +/// starts, and `Plan::block_at` is the same table read for an extent. +pub(crate) fn next_block(plan: Option<&Plan>, map: &RowMap, row: usize, forward: bool) -> Option { + plan?.next_block(row, map, forward) +} + +/// The rows of the block `row` falls in — what the pager frames when `j`/`k` +/// land on one. `None` with no lens (there are no blocks) and `None` past the +/// classified prefix. +pub(crate) fn block_at(plan: Option<&Plan>, map: &RowMap, row: usize) -> Option> { + plan?.block_at(row, map) +} + +/// `(index, count)` of the block `row` is on, for the status bar. +pub(crate) fn block_of_row(plan: Option<&Plan>, map: &RowMap, row: usize) -> Option<(usize, usize)> { + plan?.block_of_row(row, map) +} + +/// `Y` on a group's row: every record the run holds, one JSON document per +/// line — the folded rows as data, so what was copied is what was hidden. +pub(crate) fn yank_group(src: &R, plan: Option<&Plan>, item: usize) -> Option { + let it = plan?.item(item)?; + let (first, count) = (it.first, it.count); + let mut out = String::new(); + for record in first..first + count { + if let Some(text) = src.with_value(record, |v| v.map(|v| v.to_json())) { + out.push_str(&text); + out.push('\n'); + } + } + (!out.is_empty()).then(|| Yank { text: out, what: format!("{count} records") }) +} diff --git a/src/source/record/parts.rs b/src/source/record/parts.rs new file mode 100644 index 0000000..74a98ab --- /dev/null +++ b/src/source/record/parts.rs @@ -0,0 +1,309 @@ +//! The **open level**: a record read as what it is, one rung above its +//! headline and one below its JSON (SPEC.md §Lenses). +//! +//! ```text +//! ▾ assistant 10:55 Reading the failing test first, since the suite names a +//! fixture that no longer exists. +//! thinking +//! The fixture was renamed two commits ago. +//! ▾ bash cargo test -q parse → 32 lines +//! command cargo test -q parse +//! timeout 120 +//! output +//! test parse::empty_input … ok +//! ⋯ +26 lines +//! ▸ read src/parse.rs → 40 lines +//! ``` +//! +//! One row per call, and **that row opens too**: the arguments it was given, +//! one per line, and the output it returned, clipped exactly as a message is +//! clipped and saying exactly what it left out. +//! +//! # What this module decides, and what it does not +//! +//! It decides rows: the order, the indents, the glyph, the alignment, the +//! colours. It decides **nothing** about what a record contains — that is +//! [`crate::lens::Lens::detail`], a dialect's answer, and a `Part` reaching this +//! file has already been said. That split is the point of the seam: `atif` and +//! `agent` both come through here, and adding a third dialect adds no rows. +//! +//! The fold glyph is painted **in the text**, not in the gutter, and that is +//! deliberate: the gutter marker is rewritten by the painter on any row +//! `Source::hidden_at` claims, and a call row is not a record's fold — it is a +//! row inside one. A glyph of its own cannot be rewritten by the wrong hand. +//! +//! # Height and rows are one walk +//! +//! [`lay`] builds the rows, and the height is `rows.len()`. There is no second +//! function to disagree with it — the same discipline [`super::body`] follows, +//! and for the same reason: a height that is one out moves every row below it. +#![deny(unsafe_code)] + +use crate::json::Value; +use crate::lens::{Body, Part}; +use crate::render::{str_width, visible, Line, LineKind, Span}; +use crate::theme; + +use super::body::{self, Shape}; + +/// Columns an argument's name gets before its value, so a call's arguments read +/// as a column rather than as a paragraph. +const KEY: usize = 10; + +/// Columns a name written with [`column`] actually occupies: the field, and the +/// one space it always ends with. The value's wrap is laid out under exactly +/// this, so the name overwrites the pad rather than being pushed in front of it +/// — one column out and every argument's first row sat `KEY + 1` columns right +/// of its own continuation rows. +const KEY_COL: usize = KEY + 1; + +/// Columns the tool's own name gets before the argument on a call row. +const TOOL: usize = 9; + +/// Columns a call row's `→ what came back` is pushed out to, so the answers of +/// several calls line up under one another. A row wider than this simply +/// carries on: a summary row scrolls sideways, and so does this. +const ANSWER: usize = 56; + +/// The rows of a record's parts, and which part each row belongs to. +pub struct Laid { + pub rows: Vec, + /// `owner[i]` is the index into `parts` of the part row `i` came from — + /// what `Enter` on that row acts on. + pub owner: Vec, + /// Whether part `p` has anything under it to open. A call does; a named + /// stretch of text is already showing what it has. + pub opens: Vec, +} + +impl Laid { + /// Nothing at all — a record with no parts, or one that was not read. + pub fn empty() -> Laid { + Laid { rows: Vec::new(), owner: Vec::new(), opens: Vec::new() } + } + + /// The part row `line` belongs to, when that part is one `Enter` opens. + pub fn call_at(&self, line: usize) -> Option { + let part = self.owner.get(line).copied()?; + self.opens.get(part).copied().unwrap_or(false).then_some(part) + } +} + +/// Lay out every part of one record. +/// +/// `open` says which calls are showing their arguments and output; `value` is +/// the record being painted, which is how an output longer than the head it +/// was summarised from is read back whole (and `None` where the caller has no +/// record in hand, which falls back to the head and still says what it left +/// out). `base` is the column the whole level starts at — [`INDENT`], plus the +/// two columns a member of an open run is inset by, so a step's calls sit under +/// the step's own words rather than two columns to the left of them. +pub fn lay( + parts: &[Part], + open: &dyn Fn(usize) -> bool, + value: Option<&Value>, + width: usize, + base: usize, + source_line: usize, +) -> Laid { + let mut out = Laid::empty(); + for (i, part) in parts.iter().enumerate() { + let before = out.rows.len(); + let opens = part.opens(); + match part { + Part::Text { label, body } => text_part(&mut out.rows, label, body, value, width, base, source_line), + Part::Call { tool, arg, args, result } => { + let shown = opens && open(i); + out.rows.push(call_row(tool, arg, result.as_ref(), opens.then_some(shown), base, source_line)); + if shown { + call_detail(&mut out.rows, args, result.as_ref(), value, width, base, source_line); + } + } + } + out.owner.resize(out.rows.len(), i); + out.opens.push(opens); + debug_assert!(out.rows.len() > before, "a part is at least one row"); + } + out +} + +/// A named stretch of text: its name, then the text under it, **whole**. +/// +/// Whole rather than clipped because parts exist only at the open level, and +/// the open level is "the whole of that text" (SPEC.md §Lenses). A thought +/// beside a message is a `Part::Text`, and clipping it here left it cut at six +/// rows at every rung with no key that would ever expand it. +fn text_part( + rows: &mut Vec, + label: &str, + body: &Body, + value: Option<&Value>, + width: usize, + base: usize, + source_line: usize, +) { + rows.push(line(vec![pad_span(base), Span::new(visible(label), theme::lens_group())], source_line)); + let shape = Shape::under(width, base + 2); + rows.extend(body::rows(body, body.text_in(value), shape, true, source_line)); +} + +/// `▾ bash cargo test -q parse → 32 lines`. +/// +/// `state` is `None` for a call with nothing under it — no arguments and no +/// result — which then carries no glyph. A fold marker on a row that opens to +/// the same screen is the one thing `opens_further` refuses one rung up, and +/// the same rule holds here. +fn call_row( + tool: &str, + arg: &str, + result: Option<&Body>, + state: Option, + base: usize, + source_line: usize, +) -> Line { + let glyph = match state { + Some(true) => theme::MARKER_OPEN, + Some(false) => theme::MARKER_CLOSED, + None => ' ', + }; + let mut spans = vec![ + pad_span(base), + Span::new(format!("{glyph} "), theme::json_marker()), + Span::new(column(tool, TOOL), theme::lens_tool()), + ]; + let arg = visible(arg); + let mut used = base + 2 + str_width(&spans[2].text); + if !arg.is_empty() { + spans.push(Span::new(arg.clone(), theme::text())); + used += str_width(&arg); + } + if let Some(body) = result { + let gap = ANSWER.saturating_sub(used).max(1); + spans.push(Span::plain(" ".repeat(gap))); + spans.push(Span::new(format!("\u{2192} {}", size_of(body)), theme::lens_more())); + } + line(spans, source_line) +} + +/// What one call was given and what it gave back. +/// +/// The output gets a **named row of its own**, exactly as a [`Part::Text`] +/// does. Without it, output whose lines happen to read `key value` sat at the +/// argument-name column with nothing between it and the arguments, and a reader +/// could not tell which of the two they were looking at. +fn call_detail( + rows: &mut Vec, + args: &[(String, Body)], + result: Option<&Body>, + value: Option<&Value>, + width: usize, + base: usize, + source_line: usize, +) { + for (key, body) in args { + arg_rows(rows, key, body, value, width, base, source_line); + } + if let Some(body) = result { + rows.push(line( + vec![pad_span(base + 2), Span::new("output".to_string(), theme::lens_group())], + source_line, + )); + let shape = Shape::under(width, base + 4); + rows.extend(body::rows(body, body.text_in(value), shape, false, source_line)); + } +} + +/// One argument: its name in a column, its value beside it, and the value's own +/// continuation lines aligned under it. +/// +/// The value is clipped like everything else here — an argument may be a +/// sixty-line patch — and the clip says what it left out in the argument's own +/// lines or bytes. +fn arg_rows( + rows: &mut Vec, + key: &str, + body: &Body, + value: Option<&Value>, + width: usize, + base: usize, + source_line: usize, +) { + let shape = Shape::under(width, base + 4 + KEY_COL); + let mut laid = body::rows(body, body.text_in(value), shape, false, source_line); + let label = format!("{}{}", " ".repeat(base + 4), column(key, KEY)); + match laid.first_mut() { + // The value's first row starts in the column the wrap left for the + // name; every row after it is already aligned under that. + Some(first) => name_into(first, &label, shape.indent), + // An argument whose value is empty is still an argument. + None => rows.push(line(vec![Span::new(label.clone(), theme::lens_group())], source_line)), + } + rows.extend(laid); +} + +/// Put the argument's name into the space its value's indent left for it, +/// **overwriting** that pad rather than being pushed in front of it. +/// +/// Measured in display columns against the indent the value was wrapped to, not +/// in bytes against the pad's own length: a key with a multi-byte character in +/// it is shorter in columns than in bytes, and comparing the two put the value's +/// first row at an indent none of its continuation rows shared. +fn name_into(row: &mut Line, label: &str, cols: usize) { + let w = str_width(label); + let text = match w < cols { + true => format!("{label}{}", " ".repeat(cols - w)), + false => label.to_string(), + }; + match row.spans.first_mut() { + Some(pad) if pad.text.trim().is_empty() && str_width(&pad.text) == cols => { + *pad = Span::new(text, theme::lens_group()); + } + _ => row.spans.insert(0, Span::new(text, theme::lens_group())), + } +} + +/// How much a call returned: its own lines, or its size when it is one line. +/// The same vocabulary the row above it uses, so the two cannot disagree. +fn size_of(body: &Body) -> String { + match (body.bytes, body.lines) { + // `Body::lines` is `.max(1)`, so emptiness is a byte count and not a + // line count — reading it off `lines` said `0 bytes` where the summary + // row one rung up said `empty`. + (0, _) => "empty".to_string(), + (bytes, 0 | 1) => crate::source::jsonrow::size(bytes as u64), + (_, n) => format!("{n} lines"), + } +} + +/// `text`, padded to `cols` display columns and one space, and cut when it is +/// wider — a tool name is not allowed to push the whole column out of line. +fn column(text: &str, cols: usize) -> String { + let mut s = crate::lens::excerpt(text, cols); + let w = str_width(&s); + if w < cols { + s.push_str(&" ".repeat(cols - w)); + } + s.push(' '); + s +} + +fn pad_span(cols: usize) -> Span { + Span::plain(" ".repeat(cols)) +} + +/// One part row. Not a heading and not a landmark: the summary row above owns +/// the fold that `Tab` and the outline see, and this is a row inside it. +fn line(spans: Vec, source_line: usize) -> Line { + Line { + spans, + block: 0, + source_line, + heading: None, + scroll: true, + kind: LineKind::Paragraph, + } +} + +#[cfg(test)] +#[path = "parts_tests.rs"] +mod tests; diff --git a/src/source/record/parts_tests.rs b/src/source/record/parts_tests.rs new file mode 100644 index 0000000..3f9ac7a --- /dev/null +++ b/src/source/record/parts_tests.rs @@ -0,0 +1,271 @@ +//! The open level's rows, on their own: no file, no dialect, no pager — a +//! hand-built `Vec` in, rows out. +//! +//! What these hold to is the rule the whole seam is held to: a part may be +//! clipped and may never be silent about it. Every assertion below either +//! counts rows or reads the text of one; none of them reads a frame. +#![deny(unsafe_code)] + +use super::*; +use crate::lens::{Body, Part}; + +fn body(text: &str) -> Body { + Body::new(text, Vec::new()) +} + +fn call(tool: &str, arg: &str, args: &[(&str, &str)], result: Option<&str>) -> Part { + Part::Call { + tool: tool.to_string(), + arg: arg.to_string(), + args: args.iter().map(|(k, v)| (k.to_string(), body(v))).collect(), + result: result.map(body), + } +} + +/// Everything on one row, ANSI-free. +fn text_of(rows: &[Line]) -> Vec { + rows.iter().map(|r| r.text().trim_end().to_string()).collect() +} + +/// The **display column** `what` starts at on `row` — never the byte offset: a +/// cut key ends in a one-column `…` that is three bytes wide, and comparing +/// bytes to columns made a correct layout look two columns out. +fn column_of(row: &str, what: &str) -> Option { + Some(str_width(&row[..row.find(what)?])) +} + +fn shut(_: usize) -> bool { + false +} + +fn open_all(_: usize) -> bool { + true +} + +#[test] +fn a_shut_call_is_one_row_that_says_what_it_did_and_what_came_back() { + let parts = vec![call("bash", "cargo test -q parse", &[("command", "cargo test -q parse")], Some("a\nb\nc"))]; + let laid = lay(&parts, &shut, None, 92, body::INDENT, 7); + assert_eq!(laid.rows.len(), 1, "{:?}", text_of(&laid.rows)); + let row = text_of(&laid.rows).remove(0); + assert!(row.contains("bash"), "{row:?}"); + assert!(row.contains("cargo test -q parse"), "{row:?}"); + assert!(row.contains("\u{2192} 3 lines"), "{row:?}"); + assert!(row.contains(theme::MARKER_CLOSED), "a shut call carries its own glyph: {row:?}"); +} + +/// The user's words: "can we make this show the output if Enter on that line". +#[test] +fn an_open_call_shows_every_argument_and_the_output() { + let parts = vec![call( + "bash", + "make", + &[("command", "make -j8"), ("timeout", "120")], + Some("one\ntwo\nthree"), + )]; + let laid = lay(&parts, &open_all, None, 92, body::INDENT, 7); + let rows = text_of(&laid.rows); + assert!(rows[0].contains(theme::MARKER_OPEN), "{rows:#?}"); + assert!(rows.iter().any(|r| r.contains("command") && r.contains("make -j8")), "{rows:#?}"); + assert!(rows.iter().any(|r| r.contains("timeout") && r.contains("120")), "{rows:#?}"); + for line in ["one", "two", "three"] { + assert!(rows.iter().any(|r| r.trim() == line), "the output is under it: {rows:#?}"); + } + // Every row of it belongs to the one call, so `Enter` anywhere in it shuts + // the same thing it opened. + assert!(laid.owner.iter().all(|o| *o == 0), "{:?}", laid.owner); + assert_eq!(laid.call_at(rows.len() - 1), Some(0)); +} + +#[test] +fn the_height_is_the_rows_and_the_two_come_from_one_walk() { + let parts = vec![ + Part::Text { label: "thinking", body: body("a\nb") }, + call("read", "src/x.rs", &[("filePath", "src/x.rs")], Some("x")), + ]; + for open in [false, true] { + let f = |_: usize| open; + let laid = lay(&parts, &f, None, 92, body::INDENT, 1); + assert_eq!(laid.rows.len(), laid.owner.len(), "one owner per row"); + assert_eq!(laid.opens.len(), parts.len(), "one openable flag per part"); + } +} + +/// A named stretch of text is not a call: it has no glyph and `Enter` on it is +/// not its own fold. +#[test] +fn a_text_part_says_its_name_and_then_says_it() { + let parts = vec![Part::Text { label: "thinking", body: body("The fixture was renamed.") }]; + let laid = lay(&parts, &open_all, None, 92, body::INDENT, 1); + let rows = text_of(&laid.rows); + assert_eq!(rows[0].trim(), "thinking"); + assert_eq!(rows[1].trim(), "The fixture was renamed."); + assert_eq!(laid.call_at(0), None, "there is nothing under it to open"); + assert_eq!(laid.opens, vec![false]); +} + +/// The open level is "the whole of that text" (SPEC.md §Lenses). A thought +/// beside a message is a `Part::Text`, and clipping it here left it cut at six +/// rows at every rung of the ladder with no key that would ever expand it. +#[test] +fn a_text_part_is_whole_because_parts_are_the_open_level() { + let thought = (1..=30).map(|n| format!("thought {n}")).collect::>().join("\n"); + let parts = vec![Part::Text { label: "thinking", body: body(&thought) }]; + let laid = lay(&parts, &shut, None, 100, body::INDENT, 1); + let rows = text_of(&laid.rows); + assert_eq!(rows.len(), 31, "the name and every line of it: {}", rows.len()); + assert_eq!(rows[30].trim(), "thought 30"); + assert!(!rows.iter().any(|r| r.contains('\u{22ef}')), "nothing was left out: {rows:#?}"); +} + +/// The non-negotiable, at this level too (SPEC.md §Lenses): a clip states what +/// it left out, in the text's own lines. +#[test] +fn a_long_output_is_clipped_and_says_how_much_it_left_out() { + let out = (1..=40).map(|n| format!("line {n}")).collect::>().join("\n"); + let parts = vec![call("bash", "make", &[], Some(&out))]; + let laid = lay(&parts, &open_all, None, 92, body::INDENT, 1); + let rows = text_of(&laid.rows); + assert!(rows.len() < 12, "the output is clipped: {}", rows.len()); + let note = rows.last().expect("a last row"); + assert!(note.trim().starts_with('\u{22ef}'), "{rows:#?}"); + assert!(note.contains("+34 lines"), "{note:?} out of {rows:#?}"); +} + +/// And so is an argument: a sixty-line patch is not sixty rows, and the row +/// under it says so. +#[test] +fn a_multi_line_argument_is_clipped_and_says_so() { + let patch = (1..=60).map(|n| format!("+ added line {n}")).collect::>().join("\n"); + let parts = vec![call("apply_patch", "", &[("patchText", &patch)], None)]; + let laid = lay(&parts, &open_all, None, 92, body::INDENT, 1); + let rows = text_of(&laid.rows); + assert!(rows[1].contains("patchText"), "the name is beside its value: {rows:#?}"); + assert!(rows[1].contains("+ added line 1"), "{rows:#?}"); + assert!( + rows.iter().any(|r| r.contains("\u{22ef} +") && r.contains("lines")), + "{rows:#?}" + ); + assert!(rows.len() <= 8, "clipped, not sixty rows: {}", rows.len()); +} + +/// An argument whose value is one long line has no lines to count off, so the +/// remainder is stated in bytes — the same answer a one-line message gets. +#[test] +fn a_single_long_argument_states_its_remainder_in_bytes() { + let long = "x".repeat(4000); + let parts = vec![call("bash", "", &[("command", &long)], None)]; + let laid = lay(&parts, &open_all, None, 40, body::INDENT, 1); + let rows = text_of(&laid.rows); + let note = rows.last().expect("a last row"); + assert!(note.contains('\u{22ef}'), "{rows:#?}"); + assert!(!note.contains("lines"), "one line has no lines left: {note:?}"); +} + +/// A call with no arguments and no result still has a row — and carries **no** +/// glyph, because there is nothing under it. A marker on a row that opens to +/// the same screen is the rung `opens_further` refuses one level up. +#[test] +fn a_call_with_nothing_to_show_is_one_row_and_has_no_fold() { + let parts = vec![call("todowrite", "", &[], None)]; + for open in [false, true] { + let f = |_: usize| open; + let laid = lay(&parts, &f, None, 92, body::INDENT, 1); + assert_eq!(laid.rows.len(), 1, "open {open}: {:?}", text_of(&laid.rows)); + let row = text_of(&laid.rows).remove(0); + assert!(!row.contains(theme::MARKER_CLOSED), "no fold to advertise: {row:?}"); + assert!(!row.contains(theme::MARKER_OPEN), "no fold to advertise: {row:?}"); + assert_eq!(laid.opens, vec![false], "and `Enter` is the record's"); + assert_eq!(laid.call_at(0), None); + } +} + +/// A call that returned an empty string still opens: "it returned nothing" is +/// something to say, and the `output` row says it. +#[test] +fn a_call_that_returned_nothing_still_opens_and_says_so() { + let parts = vec![call("ls", "", &[], Some(""))]; + assert!(parts[0].opens(), "an answer is something to show"); + let laid = lay(&parts, &open_all, None, 92, body::INDENT, 1); + let rows = text_of(&laid.rows); + assert_eq!(rows.len(), 2, "{rows:#?}"); + assert!(rows[0].contains("empty"), "the row states the true size: {rows:#?}"); + assert_eq!(rows[1].trim(), "output"); +} + +/// Every argument's value starts in the column its own continuation rows use. +/// The name column is written *into* the pad the wrap left for it, and a +/// one-column disagreement pushed the first row `KEY + 1` columns right of the +/// rest of the value and off the side of the view. +#[test] +fn an_argument_value_is_laid_at_one_column_only() { + let long = "one two three four five six seven eight nine ten eleven twelve thirteen"; + let parts = vec![call("bash", "echo", &[("command", long)], None)]; + let laid = lay(&parts, &open_all, None, 92, body::INDENT, 1); + let rows = text_of(&laid.rows); + assert!(rows.len() >= 3, "the value wraps: {rows:#?}"); + let value_at = column_of(&rows[1], "one two").expect("the value is on the name's row"); + let cont_at = str_width(&rows[2]) - str_width(rows[2].trim_start()); + assert_eq!(value_at, cont_at, "one alignment, not two: {rows:#?}"); + assert_eq!(value_at, body::INDENT + 4 + KEY_COL, "{rows:#?}"); + for r in &rows { + assert!(str_width(r) <= 92, "no row overflows the view: {r:?}"); + } +} + +/// A key wider than the column is cut rather than pushing the value out of +/// line — the alignment is measured in display columns, not bytes. +#[test] +fn a_wide_or_multibyte_key_keeps_the_value_column() { + for key in ["a_very_long_argument_name", "\u{5f15}\u{6570}", "x"] { + let parts = vec![call("t", "", &[(key, "value")], None)]; + let laid = lay(&parts, &open_all, None, 92, body::INDENT, 1); + let rows = text_of(&laid.rows); + let at = column_of(&rows[1], "value").expect("the value is there"); + assert_eq!(at, body::INDENT + 4 + KEY_COL, "key {key:?}: {rows:#?}"); + } +} + +/// The output gets a row of its own naming it, so output whose lines read +/// `key value` cannot be mistaken for two more arguments. +#[test] +fn the_output_is_named_so_it_cannot_read_as_an_argument() { + let parts = vec![call("bash", "ls", &[("command", "ls")], Some("timeout 120\nverbose true"))]; + let laid = lay(&parts, &open_all, None, 92, body::INDENT, 1); + let rows = text_of(&laid.rows); + let at = rows.iter().position(|r| r.trim() == "output").expect("an output row: {rows:#?}"); + assert!(rows[at - 1].contains("command"), "it comes after the arguments: {rows:#?}"); + assert!(rows[at + 1].contains("timeout"), "and before the output: {rows:#?}"); +} + +/// A member of an open run is inset, and everything under its row goes with it. +#[test] +fn a_base_column_moves_the_whole_level() { + let parts = vec![ + Part::Text { label: "thinking", body: body("a thought") }, + call("bash", "make", &[("command", "make -j8")], Some("ok")), + ]; + let plain = text_of(&lay(&parts, &open_all, None, 92, body::INDENT, 1).rows); + let inset = text_of(&lay(&parts, &open_all, None, 92, body::INDENT + 2, 1).rows); + assert_eq!(plain.len(), inset.len(), "an inset moves rows sideways, not down"); + for (a, b) in plain.iter().zip(&inset) { + let (a_at, b_at) = (a.len() - a.trim_start().len(), b.len() - b.trim_start().len()); + assert_eq!(a_at + 2, b_at, "{a:?} vs {b:?}"); + } +} + +/// Rows are laid at any width, including one narrower than the indent — the +/// column floor in `body::columns` is what keeps that from being zero columns +/// and an endless wrap. +#[test] +fn every_width_lays_out() { + let parts = vec![ + Part::Text { label: "thinking", body: body("a thought worth having twice over") }, + call("bash", "make", &[("command", "make -j8 all")], Some("done\nok")), + ]; + for width in [1usize, 10, 21, 40, 92, 200] { + let laid = lay(&parts, &open_all, None, width, body::INDENT, 1); + assert!(!laid.rows.is_empty(), "width {width}"); + assert_eq!(laid.rows.len(), laid.owner.len(), "width {width}"); + } +} diff --git a/src/source/jsonl/plan.rs b/src/source/record/plan.rs similarity index 52% rename from src/source/jsonl/plan.rs rename to src/source/record/plan.rs index b3f6695..9b094e2 100644 --- a/src/source/jsonl/plan.rs +++ b/src/source/record/plan.rs @@ -1,16 +1,16 @@ -//! What a lens does to the *shape* of a record file (SPEC.md §Lenses). +//! What a lens does to the *shape* of a record document (SPEC.md §Lenses). //! //! # Items, not records //! -//! Without a lens a record file is one row per record. With one, consecutive +//! Without a lens a record document is one row per record. With one, consecutive //! mechanical records — a tool call, its result, a thought — collapse into a //! single row that opens: //! //! ```text -//! user 21:29 I want a reader for the terminal… -//! assistant 21:31 Goal: build a Rust TUI markdown pager… +//! user 21:29 I want a reader for the terminal. +//! assistant 21:31 Goal: build a Rust TUI markdown pager. //! ▸ ⟨6 steps · 4 tool calls⟩ 21:31 -//! assistant 21:36 The skeleton builds; here is what it does… +//! assistant 21:36 The skeleton builds; here is what it does. //! ``` //! //! So the document is a list of **items**, each covering a run of records: a @@ -18,6 +18,16 @@ //! else — search, folding, yanking, the status bar — keeps speaking records, //! and this module is the only place that translates. //! +//! What everything above calls a **block** — the unit `Tab`/`S-Tab` jump by, +//! and what the status bar counts (SPEC.md §Lenses) — is an item *while that +//! item is shut*, and not the same count once one is open: a boundary +//! descends into a run the reader has opened, so an open group is one item +//! and `1 + count` +//! blocks. `item` survives here because it is this file's own arithmetic; +//! `block` is the only word the reader, the keymap and the trait ever see. +//! Count blocks with [`Plan::blocks_of_item`] or the `bstarts` prefix sum +//! beside it, never `items().len()`; `plan_block.rs` defines every boundary. +//! //! # Laziness survives it //! //! Classifying a record means parsing it, so the plan is built the same way the @@ -32,16 +42,18 @@ //! An item owns rows; a record inside it can also be opened into its generic //! tree, and those rows belong to [`RowMap`]. Rather than duplicate the //! prefix-sum trick, this holds the *own* rows of the items (which do not -//! depend on any tree) and adds [`RowMap::extra_before`] on top. That is only +//! depend on any tree, though they do depend on the **width** — a message's +//! body is wrapped, so [`Plan::set_width`] re-lays every one of them) and adds +//! [`RowMap::extra_before`] on top. That is only //! correct while a *hidden* record has no tree rows, which is why closing a //! group closes its members ([`Plan::set_open`]) and why a step run growing //! past one record closes the record it swallows — but only while that run is //! shut. A member of an open group is visible and keeps its own tree. //! -//! Nothing here parses, reads a file, or recurses. +//! Nothing here parses, reads a file, names a record format, or recurses. #![deny(unsafe_code)] -use crate::lens::{Class, Lens, Summary}; +use crate::lens::{Lens, Summary}; use super::rowmap::RowMap; @@ -70,7 +82,22 @@ impl Item { #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum Spot { /// A record's summary row (`sub == 0`), or row `sub - 1` of its tree. + /// + /// The body rows between the two are [`Spot::Body`] and never `sub`: `sub` + /// has meant "row `sub - 1` of the tree" everywhere since before there were + /// bodies, and overloading it would have made every caller's `sub > 0` + /// silently wrong. Record { record: usize, sub: usize }, + /// Row `line` of the message under a record's summary row. + Body { record: usize, line: usize }, + /// Row `line` of the record's **parts** — the level between what was said + /// and the raw tree: its tool calls, listed as tool calls, each of which + /// opens into its arguments and its output. + /// + /// A fourth variant rather than more `sub`, for the reason [`Spot::Record`] + /// gives: `sub` has meant "row `sub - 1` of the tree" since before there + /// were bodies, and every caller's `sub > 0` would have gone quietly wrong. + Part { record: usize, line: usize }, /// A group's own row. Group { item: usize }, } @@ -81,10 +108,66 @@ pub struct Plan { /// did not recognise: it keeps its own row and renders as the generic tree. seen: Vec>, items: Vec, + /// Rows record `r` shows **under its own summary row**, at [`Plan::width`]: + /// its body, then its parts. One entry per classified record rather than + /// per item, because a member of an open run has them too — a step's + /// reasoning is text and is shown wherever the step is. + /// + /// Held rather than recomputed because `own` is asked per painted row and a + /// wrap is O(text). + under: Vec, + /// Which records are at the **open** level, as exceptions to + /// [`Plan::all_full`] — the "default plus what disagrees" shape the fold + /// state uses, which is what lets one record clip again after `zR`. + exc: Vec, + /// The call rows the reader has opened: `(record, part)`. Exceptions again, + /// so this is as long as what was opened and not as long as the file. + open_parts: Vec<(usize, usize)>, + /// The under-rows of the **members of open runs**, as a second prefix sum + /// beside the tree one. + /// + /// A group's own rows are `1 + count` and [`Plan::inside`] places its + /// members one row apart; a member that shows its reasoning underneath + /// breaks both. Rather than a third arithmetic, a member's extra rows get + /// exactly the treatment a member's *tree* rows already get — attributed at + /// that record's index, added on top by a prefix sum, and closed when the + /// record is hidden. `own`, `inside`, `row_of_record` and `blocks_of_item` + /// therefore keep the shape they had. + extra: RowMap, + /// Groups opened since the last measurement, for the caller to re-measure: + /// a member's rows need a width and a record, and this module has neither. + pending: Vec, /// Own rows of every item before this one — no tree rows in it. starts: Vec, + /// Blocks before item `i`. An item is one block while it is shut and + /// `1 + count` while it is an open group, because a block boundary descends + /// into a run the reader has opened ([`Plan::blocks_of_item`]). Kept as a + /// prefix sum next to `starts`, and rebuilt by the same `sync`, so the block + /// index the status bar prints and the boundary `Tab` jumps to are read off + /// one table rather than counted twice. + bstarts: Vec, /// First index of `starts` that may be wrong. dirty: usize, + /// Columns the bodies were laid out for. + width: usize, + /// Every record is at the open level: what a dump is, and what `zR` leaves + /// behind. + all_full: bool, +} + +/// The rows one record shows under its own summary row. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct Under { + /// Rows of the record's own text: what was said, or what it was thinking. + pub body: usize, + /// Rows of its parts: the tool calls, and whichever one is opened. + pub parts: usize, +} + +impl Under { + fn rows(self) -> usize { + self.body + self.parts + } } impl Plan { @@ -93,8 +176,16 @@ impl Plan { lens, seen: Vec::new(), items: Vec::new(), + under: Vec::new(), + exc: Vec::new(), + open_parts: Vec::new(), + extra: RowMap::default(), + pending: Vec::new(), starts: Vec::new(), + bstarts: Vec::new(), dirty: 0, + width: 80, + all_full: false, } } @@ -118,8 +209,10 @@ impl Plan { return; } let sum = value.and_then(|v| self.lens.read(v)); - let step = matches!(sum, Some(Summary { class: Class::Step, .. })); + let step = matches!(sum, Some(Summary { class: crate::lens::Class::Step, .. })); self.seen.push(sum); + self.under.push(Under::default()); + self.exc.push(false); let extends = matches!(self.items.last(), Some(last) if last.step && step); if !extends { self.items.push(Item { first: record, count: 1, step, open: false }); @@ -138,11 +231,15 @@ impl Plan { // away an expansion the reader can see (and can never get back, since // classification only ever runs once per record). if !open { - map.close(record); + self.hide(record, map); if grouped { - map.close(first); + self.hide(first, map); } + return; } + // A record joining a run the reader has open is visible at once, so its + // own rows have to be measured before anything asks for a row. + self.pending.push(at); } pub fn items(&self) -> &[Item] { @@ -158,59 +255,33 @@ impl Plan { self.seen.get(record).and_then(|s| s.as_ref()) } - /// Open or close a group. Closing closes the trees of the records it hides. - pub fn set_open(&mut self, item: usize, open: bool, map: &mut RowMap) -> bool { - let Some(it) = self.items.get_mut(item) else { - return false; - }; - if !it.is_group() || it.open == open { - return false; - } - it.open = open; - let (first, count) = (it.first, it.count); - if !open { - for r in first..first + count { - map.close(r); - } - } - self.mark(item); - true - } - - /// Close every group. - pub fn close_all(&mut self, map: &mut RowMap) { - for i in 0..self.items.len() { - self.set_open(i, false, map); - } - } - - /// Open every group up to and including the one on `row`, so `zR` opens - /// what the reader can see rather than parsing the whole file. - pub fn open_upto(&mut self, upto_row: usize, map: &mut RowMap) { - for i in 0..self.items.len() { - // Each opening moves everything after it, so the totals are - // brought up to date before the next item is placed. - self.sync(); - if self.row_of_item(i, map) > upto_row { - return; - } - self.set_open(i, true, map); - } - self.sync(); - } - // -- row arithmetic --------------------------------------------------------- - /// Own rows of item `i`: its summary row, plus one per member record when - /// it is an open group. + /// Own rows of item `i`: its summary row, whatever that record shows under + /// it, and one per member record when it is an open group. + /// + /// A **group** owns one row per member and nothing more, whatever those + /// members show underneath: a member's own rows are the [`Plan::extra`] + /// prefix sum's, exactly as its tree rows are the [`RowMap`]'s. That is what + /// keeps [`Plan::inside`] and [`Plan::blocks_of_item`] the shape they were. fn own(&self, i: usize) -> usize { match self.items.get(i) { - Some(it) if it.is_group() && it.open => 1 + it.count, - Some(_) => 1, + Some(it) if it.is_group() => match it.open { + true => 1 + it.count, + false => 1, + }, + Some(it) => 1 + self.under_rows(it.first), None => 0, } } + /// Rows spliced in before `record` by everything that opens: the trees the + /// reader expanded, and the under-rows of the members of open runs. The one + /// place the two prefix sums are added, so no caller can add only one. + fn before(&self, record: usize, map: &RowMap) -> usize { + map.extra_before(record) + self.extra.extra_before(record) + } + /// Recompute [`Plan::starts`] from the first dirty item, exactly as /// [`RowMap`] restates its running totals: a change late in a long log /// touches only what follows it. @@ -231,13 +302,20 @@ impl Plan { return; } self.starts.truncate(self.dirty); + self.bstarts.truncate(self.dirty); let mut run = match self.dirty { 0 => 0, n => self.starts[n - 1] + self.own(n - 1), }; + let mut blocks = match self.dirty { + 0 => 0, + n => self.bstarts[n - 1] + self.blocks_of_item(n - 1), + }; for i in self.dirty..self.items.len() { self.starts.push(run); + self.bstarts.push(blocks); run += self.own(i); + blocks += self.blocks_of_item(i); } self.dirty = self.items.len(); } @@ -249,7 +327,7 @@ impl Plan { 0 => 0, n => starts[n - 1] + self.own(n - 1), }; - own + map.extra_before(self.seen.len()) + own + self.before(self.seen.len(), map) } /// Rows in the document: the classified prefix, plus one per record the @@ -262,7 +340,7 @@ impl Plan { pub fn row_of_item(&self, i: usize, map: &RowMap) -> usize { let starts = self.settle(); match (starts.get(i), self.items.get(i)) { - (Some(start), Some(it)) => start + map.extra_before(it.first), + (Some(start), Some(it)) => start + self.before(it.first, map), _ => 0, } } @@ -282,21 +360,13 @@ impl Plan { (true, false) => base, (true, true) => { let inside = record - it.first; - let trees = map.extra_before(record) - map.extra_before(it.first); - base + 1 + inside + trees + let under = self.before(record, map) - self.before(it.first, map); + base + 1 + inside + under } (false, _) => base, } } - /// The item a record belongs to. - pub fn item_of_record(&self, record: usize) -> Option { - let i = self.items.partition_point(|it| it.first <= record); - let i = i.checked_sub(1)?; - let it = &self.items[i]; - (record < it.first + it.count).then_some(i) - } - /// Where a screen row falls. pub fn at(&self, row: usize, known: usize, map: &RowMap) -> Spot { let prefix = self.prefix_rows(map); @@ -310,7 +380,7 @@ impl Plan { let off = row - base; let it = &self.items[i]; if !it.is_group() { - return Spot::Record { record: it.first, sub: off }; + return self.place(it.first, off); } if !it.open || off == 0 { return Spot::Group { item: i }; @@ -318,46 +388,41 @@ impl Plan { self.inside(it, off - 1, map) } - /// A row inside an open group: which member, and how far into its tree. + /// Where a record's own `sub`-th row falls: its summary row, then what was + /// said, then its parts, then its tree. + /// + /// **The** order, and the only statement of it: a message and a step inside + /// an open run are placed by this same function, so the two cannot drift. + fn place(&self, record: usize, sub: usize) -> Spot { + let u = self.under_of(record); + match sub { + 0 => Spot::Record { record, sub: 0 }, + n if n <= u.body => Spot::Body { record, line: n - 1 }, + n if n <= u.rows() => Spot::Part { record, line: n - u.body - 1 }, + n => Spot::Record { record, sub: n - u.rows() }, + } + } + + /// A row inside an open group: which member, and how far into that member. fn inside(&self, it: &Item, off: usize, map: &RowMap) -> Spot { - let before = map.extra_before(it.first); + let before = self.before(it.first, map); // Rows consumed by members `first..k` is `(k - first) + trees`, which // only grows: one binary search rather than a walk, because a group can // be hundreds of records long and this runs per painted row. let (mut lo, mut hi) = (0usize, it.count); while lo < hi { let mid = (lo + hi).div_ceil(2); - let used = mid + (map.extra_before(it.first + mid) - before); + let used = mid + (self.before(it.first + mid, map) - before); match used <= off { true => lo = mid, false => hi = mid - 1, } } let record = it.first + lo; - let used = lo + (map.extra_before(record) - before); - Spot::Record { record, sub: off - used } + let used = lo + (self.before(record, map) - before); + self.place(record, off - used) } - /// The last item whose row is at or before `row`. - fn item_at_row(&self, row: usize, map: &RowMap) -> usize { - let (mut lo, mut hi) = (0usize, self.items.len().saturating_sub(1)); - while lo < hi { - let mid = (lo + hi).div_ceil(2); - match self.row_of_item(mid, map) <= row { - true => lo = mid, - false => hi = mid - 1, - } - } - lo - } - - /// Rows a closed group hides: one per record it holds. - pub fn hidden(&self, item: usize) -> usize { - match self.items.get(item) { - Some(it) if it.is_group() && !it.open => it.count, - _ => 0, - } - } } /// The fold id of a group, in a vocabulary that cannot collide with a record's @@ -372,6 +437,28 @@ pub fn group_first(id: &str) -> Option { id.strip_prefix('g')?.parse().ok() } +/// Where a block starts and ends — a child module so it can still reach the +/// prefix sums above, which are this file's own. +#[path = "plan_block.rs"] +mod block; + +/// What a record shows under its own row — the level it is at, its measured +/// heights, and which of its calls is open. A child module for the same reason +/// as `plan_block.rs`: it is this file's own state, and splitting it out is what +/// keeps both under the size limit. +#[path = "plan_rows.rs"] +mod rows; + +/// The double both test files are built on. One copy, so the two halves of the +/// arithmetic cannot be checked against two different plans. +#[cfg(test)] +#[path = "plan_fixture.rs"] +mod fixture; + #[cfg(test)] #[path = "plan_tests.rs"] mod tests; + +#[cfg(test)] +#[path = "plan_level_tests.rs"] +mod level_tests; diff --git a/src/source/record/plan_block.rs b/src/source/record/plan_block.rs new file mode 100644 index 0000000..1b5d58c --- /dev/null +++ b/src/source/record/plan_block.rs @@ -0,0 +1,161 @@ +//! Where a **block** starts and ends. +//! +//! A block is the unit `Tab`/`S-Tab` jump by and the unit the status bar counts +//! (SPEC.md §Lenses). It is *usually* an [`Item`](super::Item) — a run of +//! records that share a row — but **a boundary descends into an open group**: +//! opening a run is the reader saying "show me what is in here", so the steps +//! inside it become the blocks. The sequence over an open run is the run's own +//! row, then one block per member — a member and whatever tree it has open being +//! one block, exactly as a message and its body are. A **shut** group stays one +//! block; that is what makes it a summary. +//! +//! Everything here is read off one table: [`Plan::block_index_at`] and +//! [`Plan::row_of_block`] are inverses over the prefix sum `bstarts`, and the +//! extent, the boundary and the counter are all derived from that pair. Two +//! notions of a boundary is the drift this module exists to prevent. +//! +//! This is a child module of `plan` rather than a sibling so it can use the +//! prefix sums (`starts`, `bstarts`, `own`, `prefix_rows`, `inside`) without +//! making them public. +#![deny(unsafe_code)] + +use std::ops::Range; + +use super::super::rowmap::RowMap; +use super::{Plan, Spot}; + +impl Plan { + /// Blocks item `i` contributes: one, plus one per member while it is an + /// **open** group. This is the whole of the descent rule, and the only + /// place it is stated. + pub(super) fn blocks_of_item(&self, i: usize) -> usize { + match self.items.get(i) { + Some(it) if it.is_group() && it.open => 1 + it.count, + Some(_) => 1, + None => 0, + } + } + + /// Blocks in the classified prefix. Grouping makes this *shrink* as + /// classification catches up, and opening a run makes it *grow*; both are + /// why the status bar prints it with `≥` until the file is read. + pub fn blocks(&self) -> usize { + match self.items.len() { + 0 => 0, + n => self.bstarts.get(n - 1).copied().unwrap_or(0) + self.blocks_of_item(n - 1), + } + } + + /// The row block `b` starts on. `None` for a block that does not exist. + /// + /// O(log n): one search of `bstarts`, then the same row arithmetic + /// [`Plan::row_of_item`] and [`Plan::row_of_record`] already do. + pub fn row_of_block(&self, b: usize, map: &RowMap) -> Option { + if b >= self.blocks() { + return None; + } + let i = self.bstarts.partition_point(|&s| s <= b).checked_sub(1)?; + let it = self.items.get(i)?; + match b - self.bstarts[i] { + 0 => Some(self.row_of_item(i, map)), + sub => Some(self.row_of_record(it.first + (sub - 1), map)), + } + } + + /// The index of the block `row` falls in. + /// + /// `None` past the classified prefix: grouping there is not decided yet, so + /// the honest answer is that this row is not in a block — the caller frames + /// the row alone rather than being told an extent classification is about + /// to change. + pub fn block_index_at(&self, row: usize, map: &RowMap) -> Option { + if self.items.is_empty() || row >= self.prefix_rows(map) { + return None; + } + let i = self.item_at_row(row, map); + let base = self.bstarts.get(i).copied()?; + let it = self.items.get(i)?; + let off = row - self.row_of_item(i, map); + if !it.is_group() || !it.open || off == 0 { + return Some(base); + } + // Whatever kind of row it is — the member's own, a row of what it said, + // one of its parts, one of its tree — it belongs to *that member's* + // block. Naming only `Spot::Record` here was right only while a member + // had nothing under it: once a step shows its reasoning, the rows in + // between were attributed to the run and the jump walked backwards out of + // them. + match self.inside(it, off - 1, map) { + Spot::Record { record, .. } + | Spot::Body { record, .. } + | Spot::Part { record, .. } => Some(base + 1 + (record - it.first)), + Spot::Group { .. } => Some(base), + } + } + + /// The rows of the block `row` falls in: from its first row to the first row + /// of the block after it, which is the end of the classified prefix for the + /// last one. A member's block is that member and its open tree, never the + /// whole run. + pub fn block_at(&self, row: usize, map: &RowMap) -> Option> { + let b = self.block_index_at(row, map)?; + let start = self.row_of_block(b, map)?; + let end = self.row_of_block(b + 1, map).unwrap_or_else(|| self.prefix_rows(map)); + Some(start..end.max(start + 1)) + } + + /// The next or previous block boundary, strictly after (before) `row` — + /// `Tab`/`S-Tab`. From inside a block the first press back + /// goes to that block's own first row, which for a member of an open run is + /// the member's row rather than the run's: `S-Tab` mirrors `Tab` step for + /// step, including stepping back out of the run to the block above it. + /// + /// `None` at either end and past the classified prefix, where the pager + /// says the jump has no answer and moves nothing. No row is unreachable + /// regardless: `j`/`k` move one row in every format and never come here. + pub fn next_block(&self, row: usize, map: &RowMap, forward: bool) -> Option { + let b = self.block_index_at(row, map)?; + let start = self.row_of_block(b, map)?; + match forward { + true => self.row_of_block(b + 1, map), + false if row > start => Some(start), + false => self.row_of_block(b.checked_sub(1)?, map), + } + } + + /// The index of the block `row` is on, and how many blocks are classified — + /// for the status bar. The same pair `Tab` jumps by, so the counter cannot + /// disagree with what the key does. + pub fn block_of_row(&self, row: usize, map: &RowMap) -> Option<(usize, usize)> { + Some((self.block_index_at(row, map)?, self.blocks())) + } + + /// The item a record belongs to. + pub fn item_of_record(&self, record: usize) -> Option { + let i = self.items.partition_point(|it| it.first <= record); + let i = i.checked_sub(1)?; + let it = &self.items[i]; + (record < it.first + it.count).then_some(i) + } + + /// The last item whose row is at or before `row`. + pub(super) fn item_at_row(&self, row: usize, map: &RowMap) -> usize { + let (mut lo, mut hi) = (0usize, self.items.len().saturating_sub(1)); + while lo < hi { + let mid = (lo + hi).div_ceil(2); + match self.row_of_item(mid, map) <= row { + true => lo = mid, + false => hi = mid - 1, + } + } + lo + } + + /// Rows a closed group hides: one per record it holds. + pub fn hidden(&self, item: usize) -> usize { + match self.items.get(item) { + Some(it) if it.is_group() && !it.open => it.count, + _ => 0, + } + } +} diff --git a/src/source/record/plan_fixture.rs b/src/source/record/plan_fixture.rs new file mode 100644 index 0000000..b1ed52b --- /dev/null +++ b/src/source/record/plan_fixture.rs @@ -0,0 +1,112 @@ +//! The double every `plan` test is built on: a lens with no dialect, a plan +//! over a string of record kinds, and the two helpers that give records the +//! heights a source's measurement would have produced. +//! +//! One copy, shared by `plan_tests.rs` and `plan_level_tests.rs` — two halves of +//! one file, split to stay under the size limit. A second copy would drift, and +//! the whole point of these tests is that two answers agree. +#![deny(unsafe_code)] + +use super::*; +use crate::json::Value; +use crate::lens::{Class, Summary, Who}; + +/// A lens with no dialect: the fixture says what each record is. +/// +/// `{"k":"m"}` a message, `{"k":"s"}` a step, anything else unrecognised. +pub(super) struct Fake; + +impl crate::lens::Lens for Fake { + fn name(&self) -> &'static str { + "fake" + } + fn about(&self) -> &'static str { + "test double" + } + fn read(&mut self, v: &Value) -> Option { + let class = match v.get("k").and_then(|k| k.as_str())? { + "m" => Class::Message, + "s" => Class::Step, + _ => return None, + }; + Some(Summary { + class, + who: Who::System, + actor: "x".to_string(), + time: None, + what: String::new(), + calls: 1, + // A body is the source's to measure; these tests give one to an + // item directly (`with_bodies`), which is the same number the + // measurement would have produced and keeps this file file-free. + body: None, + }) + } +} + +/// Build a plan over `kinds` — `m`, `s`, or `?` for a record the lens does not +/// know — with a fresh row map. +pub(super) fn plan_of(kinds: &str) -> (Plan, RowMap) { + let mut plan = Plan::new(Box::new(Fake)); + let mut map = RowMap::default(); + for (i, ch) in kinds.chars().enumerate() { + let json = format!(r#"{{"k":"{ch}"}}"#); + let value = crate::json::parse(json.as_bytes()).expect("fixture"); + plan.classify(i, Some(&value), &mut map); + } + plan.sync(); + (plan, map) +} + +/// Every row of the document, as `(record, sub)` or `group N`. +pub(super) fn walk(plan: &Plan, map: &RowMap, known: usize) -> Vec { + (0..plan.rows(known, map)) + .map(|row| match plan.at(row, known, map) { + Spot::Group { item } => format!("group {item}"), + Spot::Body { record, line } => format!("{record}b{line}"), + Spot::Part { record, line } => format!("{record}p{line}"), + Spot::Record { record, sub } => format!("{record}.{sub}"), + }) + .collect() +} + +/// Give every message record a body `rows` tall — the number the source's +/// measurement would have produced at some width, which is all the arithmetic +/// here ever sees. +pub(super) fn with_bodies(plan: &mut Plan, rows: usize) { + let items: Vec<(usize, usize, bool)> = plan + .items() + .iter() + .map(|it| (it.first, it.count, it.step)) + .collect(); + for (first, count, step) in items { + if !step { + plan.set_under(first, Under { body: rows, parts: 0 }, false); + } + let _ = count; + } + plan.sync(); +} + +/// Give every record — messages *and* the steps inside an open run — a body +/// and a parts block. A step's reasoning is text and is shown wherever the +/// step is, which is the arithmetic this exercises. +pub(super) fn with_under(plan: &mut Plan, under: Under) { + let items: Vec<(usize, usize, bool, bool)> = plan + .items() + .iter() + .map(|it| (it.first, it.count, it.is_group(), it.open)) + .collect(); + for (first, count, group, open) in items { + match group { + false => plan.set_under(first, under, false), + true if open => { + for r in first..first + count { + plan.set_under(r, under, true); + } + } + true => {} + } + } + plan.sync(); +} diff --git a/src/source/record/plan_level_tests.rs b/src/source/record/plan_level_tests.rs new file mode 100644 index 0000000..3be0f05 --- /dev/null +++ b/src/source/record/plan_level_tests.rs @@ -0,0 +1,232 @@ +//! The **levels** a record has, and the rows they add: the parts between a body +//! and the tree, and the members of an open run that now show rows of their own. +//! +//! The other half of `plan_tests.rs`, split to stay under the size limit and +//! sharing its double ([`super::fixture`]). What both halves hold to is the one +//! invariant: if `rows()` and `at()` disagree by one, every row after the +//! disagreement shows the wrong record. +#![deny(unsafe_code)] + +use super::fixture::*; +use super::*; + +// -- the third kind of row, and the members that now have rows of their own ----- + +/// The order inside one record: its summary row, what it said, its parts, then +/// its own tree. Getting this wrong reads a call row as a tree row. +#[test] +fn parts_sit_between_the_body_and_the_tree() { + let (mut plan, mut map) = plan_of("mm"); + map.open(0, 2); + plan.set_under(0, Under { body: 2, parts: 3 }, false); + plan.set_under(1, Under { body: 1, parts: 0 }, false); + plan.sync(); + assert_eq!( + walk(&plan, &map, 2), + vec!["0.0", "0b0", "0b1", "0p0", "0p1", "0p2", "0.1", "0.2", "1.0", "1b0"] + ); + assert_eq!(plan.row_of_record(1, &map), 8); +} + +/// Reasoning is text, so it is shown: a step inside a run the reader has opened +/// has rows of its own, and the run is exactly as tall as what it holds. +/// +/// This is the invariant that used to read "a step never has a body". It was +/// traded deliberately: `own` for an open group is still `1 + count`, and a +/// member's own rows are the second prefix sum's — the same treatment its tree +/// rows have always had. +#[test] +fn an_open_run_is_as_tall_as_the_steps_and_what_they_show() { + let (mut plan, mut map) = plan_of("mssm"); + plan.set_open(1, true, &mut map); + with_under(&mut plan, Under { body: 2, parts: 1 }); + assert_eq!( + walk(&plan, &map, 4), + vec![ + "0.0", "0b0", "0b1", "0p0", "group 1", "1.0", "1b0", "1b1", "1p0", "2.0", "2b0", + "2b1", "2p0", "3.0", "3b0", "3b1", "3p0" + ] + ); + // And every member's row still maps back to that member. + for record in 0..4 { + let row = plan.row_of_record(record, &map); + assert_eq!(plan.at(row, 4, &map), Spot::Record { record, sub: 0 }, "record {record}"); + } +} + +/// A member's tree opens *under* what it showed, not in place of it. +#[test] +fn a_members_own_rows_and_its_tree_compose() { + let (mut plan, mut map) = plan_of("ssm"); + plan.set_open(0, true, &mut map); + with_under(&mut plan, Under { body: 1, parts: 2 }); + assert!(map.open(1, 2)); + plan.sync(); + assert_eq!( + walk(&plan, &map, 3), + vec!["group 0", "0.0", "0b0", "0p0", "0p1", "1.0", "1b0", "1p0", "1p1", "1.1", "1.2", "2.0", "2b0", "2p0", "2p1"] + ); + assert_eq!(plan.row_of_record(1, &map), 5); + assert_eq!(plan.row_of_record(2, &map), 11); +} + +/// Closing a run takes its members' own rows with it, exactly as it takes their +/// trees — a hidden record owns no rows, which is what both prefix sums assume. +#[test] +fn closing_a_run_takes_the_rows_its_members_showed() { + let (mut plan, mut map) = plan_of("mssm"); + plan.set_open(1, true, &mut map); + with_under(&mut plan, Under { body: 3, parts: 2 }); + let open = plan.rows(4, &map); + plan.set_open(1, false, &mut map); + plan.sync(); + let shut = plan.rows(4, &map); + assert_eq!(shut, open - 2 * (1 + 3 + 2), "both members and everything they showed"); + assert_eq!(walk(&plan, &map, 4).len(), shut); +} + +/// The whole thing at once: every fold combination, every level combination, +/// at three widths' worth of heights — `at(row)` inverts `row_of_record` and +/// the two totals agree, which is the failure this file exists to prevent. +#[test] +fn rows_and_records_round_trip_at_every_level() { + let kinds = "mssm?msm"; + let known = kinds.chars().count() + 1; + // Heights a body and its parts take at 40, 92 and 200 columns, plus the two + // ends: nothing under the row at all, and a level with only parts. + for under in [ + Under { body: 0, parts: 0 }, + Under { body: 6, parts: 0 }, + Under { body: 0, parts: 4 }, + Under { body: 12, parts: 9 }, + Under { body: 1, parts: 1 }, + ] { + for mask in 0..8u32 { + let (mut plan, mut map) = plan_of(kinds); + for item in 0..plan.items().len() { + if mask & (1 << (item % 3)) != 0 { + plan.set_open(item, true, &mut map); + } + } + with_under(&mut plan, under); + map.open(0, 3); + plan.sync(); + let total = plan.rows(known, &map); + for record in 0..known { + let row = plan.row_of_record(record, &map); + assert!(row < total, "record {record} at {row} of {total} ({under:?})"); + match plan.at(row, known, &map) { + Spot::Record { record: got, sub } => { + assert_eq!((got, sub), (record, 0), "{under:?} mask {mask}"); + } + Spot::Group { item } => { + let it = plan.item(item).expect("item"); + assert!(record >= it.first && record < it.first + it.count); + } + other => panic!("record {record} landed on {other:?} ({under:?})"), + } + } + assert_eq!(walk(&plan, &map, known).len(), total, "{under:?} mask {mask}"); + } + } +} + +/// The level is an *exception* to the default, per record — which is what lets +/// `zR` mean "everything, now" and one record clip again afterwards. +#[test] +fn a_level_is_an_exception_to_the_default() { + let (mut plan, _) = plan_of("mm"); + assert!(!plan.full_at(0)); + assert!(plan.set_full(0, true)); + assert!(plan.full_at(0) && !plan.full_at(1)); + plan.set_all_full(true); + assert!(plan.full_at(0) && plan.full_at(1), "everything, now"); + assert!(plan.set_full(0, false), "and one of them can clip again"); + assert!(!plan.full_at(0) && plan.full_at(1)); + plan.set_all_full(false); + assert!(!plan.full_at(0) && !plan.full_at(1), "exceptions are dropped either way"); +} + +/// An opened call is an exception too, and leaving the open level drops it: the +/// rows are gone, and a call left open would reappear on the next descent. +#[test] +fn an_opened_call_belongs_to_the_level_it_was_opened_at() { + let (mut plan, _) = plan_of("mm"); + plan.set_full(0, true); + assert!(plan.set_part_open(0, 1, true)); + assert!(plan.part_open(0, 1) && !plan.part_open(0, 0) && !plan.part_open(1, 1)); + assert!(!plan.set_part_open(0, 1, true), "twice is not a change"); + plan.set_full(0, false); + assert!(!plan.part_open(0, 1), "the level took it with it"); + plan.set_full(0, true); + plan.set_part_open(0, 0, true); + plan.set_all_full(false); + assert!(!plan.part_open(0, 0), "and so does `zM`"); +} + + +/// **One call opens at a time** (SPEC.md §Lenses). Opening a second one shuts +/// the first, wherever it was: two open calls put two screens of arguments and +/// output on the screen together, which is what the level exists to avoid. +#[test] +fn opening_a_call_shuts_whichever_was_open() { + let (mut plan, _) = plan_of("mm"); + plan.set_full(0, true); + plan.set_full(1, true); + assert!(plan.set_part_open(0, 0, true)); + assert_eq!(plan.open_part_record(), Some(0)); + // A second call on the same record. + assert!(plan.set_part_open(0, 2, true)); + assert!(plan.part_open(0, 2) && !plan.part_open(0, 0), "not both at once"); + // And one on another record, which is how the caller knows to re-measure it. + assert_eq!(plan.open_part_record(), Some(0)); + assert!(plan.set_part_open(1, 1, true)); + assert!(plan.part_open(1, 1) && !plan.part_open(0, 2)); + assert_eq!(plan.open_part_record(), Some(1)); + assert!(plan.set_part_open(1, 1, false)); + assert_eq!(plan.open_part_record(), None); +} + +// -- a block inside an open run --------------------------------------------------- + +/// A member's block is that member and everything under its row — its body, its +/// parts, its tree — never the whole run (docs/lenses.md). +/// +/// Attributing those rows to the run's own block made the jump move the cursor +/// *backwards* out of them, `k` skip the member's row, and the status bar count +/// down inside an open run. Before members had rows of their own the case could +/// not arise, so the arm that got it wrong was unreachable. +#[test] +fn a_members_under_rows_belong_to_the_members_block() { + let (mut plan, mut map) = plan_of("mssm"); + plan.set_open(1, true, &mut map); + with_under(&mut plan, Under { body: 2, parts: 1 }); + map.open(2, 2); + plan.sync(); + // 0.0 0b0 0b1 0p0 | group | 1.0 1b0 1b1 1p0 | 2.0 2b0 2b1 2p0 2.1 2.2 | 3.0 … + let rows = walk(&plan, &map, 4); + let block_of: Vec = (0..rows.len()) + .map(|r| plan.block_index_at(r, &map).expect("in the prefix")) + .collect(); + assert_eq!( + block_of, + vec![0, 0, 0, 0, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4], + "{rows:?}" + ); + // The jump never goes backwards, and back from anywhere inside a member's + // block comes back to that member's own row. + for (r, row) in rows.iter().enumerate() { + if let Some(next) = plan.next_block(r, &map, true) { + assert!(next > r, "Tab from row {r} ({row}) went to {next}"); + } + if let Some(back) = plan.next_block(r, &map, false) { + assert!(back < r, "S-Tab from row {r} ({row}) went to {back}"); + } + } + // The member's own extent is the member, not the run. + let member = plan.row_of_record(2, &map); + assert_eq!(plan.block_at(member, &map), Some(member..member + 6)); + let back = plan.next_block(member + 3, &map, false); + assert_eq!(back, Some(member), "S-Tab comes back to the member's own row"); + assert_eq!(plan.blocks(), 5, "the run's own row, then one block per step"); +} diff --git a/src/source/record/plan_rows.rs b/src/source/record/plan_rows.rs new file mode 100644 index 0000000..4359637 --- /dev/null +++ b/src/source/record/plan_rows.rs @@ -0,0 +1,270 @@ +//! What a record shows **under its own summary row**, and how much of it. +//! +//! Three pieces of state, all keyed by *record* rather than by item, because a +//! member of an open run is as much a thing a reader opens as a message is: +//! +//! | | what it is | who moves rows | +//! | --- | --- | --- | +//! | the **level** | clipped, or open — an exception to [`Plan::all_full`] | `Under::body` and `Under::parts` | +//! | the **heights** | rows the body and the parts occupy at this width | `Plan::own`, `Plan::extra` | +//! | the **open call** | which part row is showing its arguments and output | `Under::parts` | +//! +//! # The ladder +//! +//! `Enter` / `za` toggles the record's two levels (SPEC.md §Lenses): +//! +//! ```text +//! clipped <-> open +//! ``` +//! +//! **clipped** is the headline and a few rows of what was said; **open** is the +//! whole of it and the record's tool calls listed as tool calls. Whether a +//! record has the second level at all depends on what is in it — one with +//! nothing under the headline and no calls has no rung, and leaves the key to +//! the outline. +//! +//! The record's own **tree** is deliberately **not** a level here. Tree rows are +//! [`RowMap`]'s, keyed by record, shared with the no-lens path and toggled by +//! `r` from either level; the ladder writes *through* to it rather than owning +//! it, which is what leaves `r` orthogonal and the prefix sums untouched. +//! +//! # What is stored, and what is not +//! +//! The heights are stored; the *parts* are not. [`crate::lens::Lens::detail`] +//! is asked for the record being measured or painted and its answer is dropped +//! — so a document's cost is one `Under` (two `usize`) per record, and never a +//! tool's output. That is the same bargain [`crate::lens::Body`] strikes one +//! level down, and for the same reason. +#![deny(unsafe_code)] + +use crate::json::Value; +use crate::lens::{Body, Part}; + +use super::super::rowmap::RowMap; +use super::{Plan, Under}; + +impl Plan { + /// A record that has just been folded away owns no rows: its tree is shut + /// and its under-rows go with it, which is the invariant the two prefix + /// sums above are only correct under. + pub(super) fn hide(&mut self, record: usize, map: &mut RowMap) { + map.close(record); + self.extra.close(record); + } + + /// Columns the bodies are laid out for. + pub fn width(&self) -> usize { + self.width + } + + /// A new layout width. True when it changed — the caller's cue to re-measure + /// every body before asking a row question, since a body is wrapped and a + /// resize therefore moves rows in a way no fold does. + pub fn set_width(&mut self, cols: usize) -> bool { + let cols = cols.max(1); + if self.width == cols { + return false; + } + self.width = cols; + self.mark(0); + true + } + + /// Open or close a group. Closing closes the trees of the records it hides. + pub fn set_open(&mut self, item: usize, open: bool, map: &mut RowMap) -> bool { + let Some(it) = self.items.get_mut(item) else { + return false; + }; + if !it.is_group() || it.open == open { + return false; + } + it.open = open; + let (first, count) = (it.first, it.count); + match open { + // Its members are visible now: what each shows under its own row is + // a wrap at a width, so the caller re-measures before asking for a + // row ([`Plan::take_pending`]). + true => self.pending.push(item), + false => { + for r in first..first + count { + self.hide(r, map); + } + } + } + self.mark(item); + true + } + + /// Close every group. + pub fn close_all(&mut self, map: &mut RowMap) { + for i in 0..self.items.len() { + self.set_open(i, false, map); + } + } + + /// Open every group up to and including the one on `row`, so `zR` opens + /// what the reader can see rather than parsing the whole file. + pub fn open_upto(&mut self, upto_row: usize, map: &mut RowMap) { + for i in 0..self.items.len() { + // Each opening moves everything after it, so the totals are + // brought up to date before the next item is placed. + self.sync(); + if self.row_of_item(i, map) > upto_row { + return; + } + self.set_open(i, true, map); + } + self.sync(); + } + + + // -- the level --------------------------------------------------------------- + + /// Is `record` at the **open** level — its text whole, its parts listed? + /// + /// An *exception* to [`Plan::all_full`] rather than a state of its own, so + /// that `zR` can mean "everything, now" and one record can still clip again + /// afterwards without the two disagreeing. + pub fn full_at(&self, record: usize) -> bool { + self.exc.get(record).copied().unwrap_or(false) != self.all_full + } + + /// Put `record` at the open level, or back to its clip. True when it moved. + pub fn set_full(&mut self, record: usize, full: bool) -> bool { + if record >= self.exc.len() || self.full_at(record) == full { + return false; + } + self.exc[record] = !self.exc[record]; + // A record leaving the open level takes its opened call with it: the + // rows are gone, and a call left open would reappear on the next + // descent without the reader having asked for it. + if !full { + self.open_parts.retain(|(r, _)| *r != record); + } + self.mark_record(record); + true + } + + /// Every record at the open level (a dump, and what `zR` leaves behind), or + /// every record back to its clip. Exceptions are dropped either way: this is + /// "everything, now". + /// + /// Opened calls are dropped with them rather than being inverted. `zR` is + /// not "expand every argument of every call in the file" — a run of steps + /// would become thousands of rows behind one keystroke — it is every record + /// showing what it holds, with each call one row that still opens. The + /// record's own tree, which `zR` also opens, is where every byte is. + pub fn set_all_full(&mut self, full: bool) { + self.all_full = full; + for e in &mut self.exc { + *e = false; + } + self.open_parts.clear(); + self.mark(0); + } + + // -- the parts ---------------------------------------------------------------- + + /// Is part `part` of `record` showing its arguments and its output? + pub fn part_open(&self, record: usize, part: usize) -> bool { + self.open_parts.contains(&(record, part)) + } + + /// The record whose call is open, if any. One at a time, so there is at + /// most one — and the caller needs it to re-measure the record that is + /// about to lose its rows. + pub fn open_part_record(&self) -> Option { + self.open_parts.first().map(|(r, _)| *r) + } + + /// Open or shut one call. True when it changed, which is the caller's cue + /// to re-measure that record. + /// + /// **One call opens at a time** (SPEC.md §Lenses): opening one shuts + /// whichever was open, wherever it was. Two open calls put two screens of + /// arguments and output on the screen at once, which is what the level + /// exists to avoid — and the records that lost theirs are marked, so the + /// rows go with it. + pub fn set_part_open(&mut self, record: usize, part: usize, open: bool) -> bool { + let at = self.open_parts.iter().position(|p| *p == (record, part)); + match (at, open) { + (Some(_), true) | (None, false) => false, + (Some(i), false) => { + self.open_parts.remove(i); + self.mark_record(record); + true + } + (None, true) => { + let shut = std::mem::take(&mut self.open_parts); + for (r, _) in shut { + self.mark_record(r); + } + self.open_parts.push((record, part)); + self.mark_record(record); + true + } + } + } + + /// The parts of `record`, as its dialect reads them. Computed here and now + /// and thrown away by the caller — never stored, never per document. + pub fn detail(&self, value: &Value) -> Vec { + self.lens.detail(value) + } + + // -- what a record shows underneath ------------------------------------------- + + /// The text under `record`'s row, and whether it is shown whole. `None` for + /// a record the lens did not read and for one with nothing to say. + pub fn body_of(&self, record: usize) -> Option<(&Body, bool)> { + let body = self.summary(record)?.body.as_ref()?; + Some((body, self.full_at(record))) + } + + /// Rows `record` shows under its own row, split into the two kinds. + pub fn under_of(&self, record: usize) -> Under { + self.under.get(record).copied().unwrap_or_default() + } + + /// Rows `record` shows under its own row, in total. + pub fn under_rows(&self, record: usize) -> usize { + self.under_of(record).rows() + } + + /// Tell the plan how tall `record`'s body and parts are at the current + /// width. The measurement is the caller's because text longer than + /// [`crate::lens::BODY_KEEP`] is only whole inside the record, and this + /// module never reads one. + /// + /// `member` is whether the record is inside a run the reader has opened, in + /// which case its rows go into the second prefix sum rather than into its + /// item's own — the one place the two paths differ. + pub fn set_under(&mut self, record: usize, under: Under, member: bool) { + if record >= self.under.len() { + return; + } + let changed = self.under[record] != under; + self.under[record] = under; + if member { + self.extra.close(record); + self.extra.open(record, under.rows()); + } + if changed || member { + self.mark_record(record); + } + } + + /// Groups opened since this was last asked, for the caller to measure the + /// members of. Draining rather than reading, because measuring is the + /// answer to it. + pub fn take_pending(&mut self) -> Vec { + std::mem::take(&mut self.pending) + } + + /// Everything after `record`'s item may have moved. + fn mark_record(&mut self, record: usize) { + if let Some(i) = self.item_of_record(record) { + self.mark(i); + } + } +} diff --git a/src/source/jsonl/plan_tests.rs b/src/source/record/plan_tests.rs similarity index 72% rename from src/source/jsonl/plan_tests.rs rename to src/source/record/plan_tests.rs index 531ea89..a31ff11 100644 --- a/src/source/jsonl/plan_tests.rs +++ b/src/source/record/plan_tests.rs @@ -6,62 +6,8 @@ //! the failure this file exists to make impossible. #![deny(unsafe_code)] +use super::fixture::*; use super::*; -use crate::json::Value; -use crate::lens::{Class, Summary, Who}; - -/// A lens with no dialect: the fixture says what each record is. -/// -/// `{"k":"m"}` a message, `{"k":"s"}` a step, anything else unrecognised. -struct Fake; - -impl crate::lens::Lens for Fake { - fn name(&self) -> &'static str { - "fake" - } - fn about(&self) -> &'static str { - "test double" - } - fn read(&mut self, v: &Value) -> Option { - let class = match v.get("k").and_then(|k| k.as_str())? { - "m" => Class::Message, - "s" => Class::Step, - _ => return None, - }; - Some(Summary { - class, - who: Who::System, - actor: "x".to_string(), - time: None, - what: String::new(), - calls: 1, - }) - } -} - -/// Build a plan over `kinds` — `m`, `s`, or `?` for a record the lens does not -/// know — with a fresh row map. -fn plan_of(kinds: &str) -> (Plan, RowMap) { - let mut plan = Plan::new(Box::new(Fake)); - let mut map = RowMap::default(); - for (i, ch) in kinds.chars().enumerate() { - let json = format!(r#"{{"k":"{ch}"}}"#); - let value = crate::json::parse(json.as_bytes()).expect("fixture"); - plan.classify(i, Some(&value), &mut map); - } - plan.sync(); - (plan, map) -} - -/// Every row of the document, as `(record, sub)` or `group N`. -fn walk(plan: &Plan, map: &RowMap, known: usize) -> Vec { - (0..plan.rows(known, map)) - .map(|row| match plan.at(row, known, map) { - Spot::Group { item } => format!("group {item}"), - Spot::Record { record, sub } => format!("{record}.{sub}"), - }) - .collect() -} #[test] fn a_run_of_steps_becomes_one_item() { @@ -256,6 +202,9 @@ fn rows_and_lookups_agree_under_every_fold_combination() { Spot::Record { record: got, sub } => { assert_eq!((got, sub), (record, 0), "mask {mask}"); } + Spot::Body { record: got, line } | Spot::Part { record: got, line } => { + panic!("record {record} landed on row {line} under {got}"); + } // A folded record's row is its group's row. Spot::Group { item } => { let it = plan.item(item).expect("item"); @@ -319,3 +268,97 @@ fn joining_a_closed_group_still_closes_the_records_tree() { plan.sync(); assert_eq!(plan.rows(3, &map), 1, "a shut group is one row"); } + + +/// The sharpest edge in the file: an item's own rows now depend on the width, +/// so every fold combination is re-checked with bodies of 0, 1, 6 and 200 rows. +/// Every record's row maps back to that record, every row maps to *something*, +/// and the two totals agree. +#[test] +fn rows_and_records_round_trip_with_a_body_at_every_height() { + let kinds = "msssm?mssm"; + // Two records the lens has not reached: the unclassified tail must keep + // being one row each with a body present. + let known = kinds.chars().count() + 2; + for height in [0usize, 1, 6, 200] { + for mask in 0..8u32 { + let (mut plan, mut map) = plan_of(kinds); + for item in 0..plan.items().len() { + if mask & (1 << (item % 3)) != 0 { + plan.set_open(item, true, &mut map); + } + } + // A record opened into its tree as well, under the body. + map.open(0, 3); + with_bodies(&mut plan, height); + let total = plan.rows(known, &map); + for record in 0..known { + let row = plan.row_of_record(record, &map); + assert!(row < total, "record {record} at {row} of {total}"); + match plan.at(row, known, &map) { + Spot::Record { record: got, sub } => { + assert_eq!((got, sub), (record, 0), "height {height} mask {mask}"); + } + Spot::Body { record: got, line } | Spot::Part { record: got, line } => { + panic!("record {record} landed on row {line} under {got}"); + } + Spot::Group { item } => { + let it = plan.item(item).expect("item"); + assert!(record >= it.first && record < it.first + it.count); + } + } + } + assert_eq!(walk(&plan, &map, known).len(), total, "height {height} mask {mask}"); + } + } +} + +/// The order inside one item: the summary row, then the message, then the +/// record's own tree. Getting this wrong reads a body row as a tree row. +#[test] +fn a_body_sits_between_the_summary_row_and_the_tree() { + let (mut plan, mut map) = plan_of("mm"); + map.open(0, 2); + with_bodies(&mut plan, 3); + assert_eq!( + walk(&plan, &map, 2), + vec!["0.0", "0b0", "0b1", "0b2", "0.1", "0.2", "1.0", "1b0", "1b1", "1b2"] + ); + assert_eq!(plan.row_of_record(1, &map), 6); +} + +/// A step never has a body, so a run is exactly as tall as the steps in it — +/// which is what leaves the open-group arithmetic alone. +#[test] +fn an_open_run_is_untouched_by_bodies() { + let (mut plan, mut map) = plan_of("mssssm"); + with_bodies(&mut plan, 4); + plan.set_open(1, true, &mut map); + plan.sync(); + assert_eq!( + walk(&plan, &map, 6), + vec![ + "0.0", "0b0", "0b1", "0b2", "0b3", "group 1", "1.0", "2.0", "3.0", "4.0", "5.0", + "5b0", "5b1", "5b2", "5b3" + ] + ); +} + +/// A width change is the one thing that invalidates a height. The plan says so +/// and rebuilds its totals from the new measurements — without reclassifying, +/// which would read every record a second time. +#[test] +fn a_width_change_re_lays_every_body() { + let (mut plan, map) = plan_of("mm"); + with_bodies(&mut plan, 2); + assert_eq!(plan.rows(2, &map), 6); + let classified = plan.classified(); + assert!(plan.set_width(40), "a new width changed something"); + assert!(!plan.set_width(40), "the same width changes nothing"); + // The heights the caller re-measures at the new width. + with_bodies(&mut plan, 5); + assert_eq!(plan.rows(2, &map), 12); + assert_eq!(plan.classified(), classified, "nothing was reclassified"); + assert_eq!(plan.width(), 40); +} + diff --git a/src/source/jsonl/rowmap.rs b/src/source/record/rowmap.rs similarity index 98% rename from src/source/jsonl/rowmap.rs rename to src/source/record/rowmap.rs index 0c185b8..947a6c2 100644 --- a/src/source/jsonl/rowmap.rs +++ b/src/source/record/rowmap.rs @@ -1,6 +1,6 @@ //! Which record a screen row belongs to, when some records are expanded. //! -//! A `.jsonl` document is one row per record until the reader opens one, at +//! A record document is one row per record until the reader opens one, at //! which point that record's tree rows are spliced in under it. The mapping has //! to work both ways and it has to be O(log n): `lines()` turns rows into //! records, and `end`, `goto_id` and the fold code turn records back into rows. diff --git a/src/source/jsonl/rowmap_tests.rs b/src/source/record/rowmap_tests.rs similarity index 100% rename from src/source/jsonl/rowmap_tests.rs rename to src/source/record/rowmap_tests.rs diff --git a/src/source/jsonl/rows.rs b/src/source/record/rows.rs similarity index 64% rename from src/source/jsonl/rows.rs rename to src/source/record/rows.rs index a4e6b53..fc61681 100644 --- a/src/source/jsonl/rows.rs +++ b/src/source/record/rows.rs @@ -1,17 +1,19 @@ //! Turning records into rows: the summary row, the expanded tree, the window //! and the outline over it. //! -//! A second `impl JsonlSource` rather than a second type, split out of `mod.rs` -//! so both stay under the size limit. Everything here is `&self` where it can -//! be: laying a row out is a read of the file behind a [`RefCell`], not a -//! change to the document. +//! A second `impl RecordSource` rather than a second type, split out of +//! `source.rs` so both stay under the size limit. Everything here is `&self` +//! where it can be: laying a row out is a read of the file behind the store's +//! own `RefCell`, not a change to the document. The exceptions are the three +//! things that *re-lay* rows — a resize, a body toggled open, `r` — and they +//! are here because they are the same arithmetic seen from the other side. #![deny(unsafe_code)] use std::ops::Range; use super::*; -impl JsonlSource { +impl RecordSource { // -- rows ------------------------------------------------------------------- /// A record's own row: the lens's reading of it when there is one, and the @@ -28,18 +30,6 @@ impl JsonlSource { } } - /// Is this record a member of a group that is currently open? Its row is - /// then indented under the group's, so a run reads as one thing. - pub(super) fn in_open_group(&self, record: usize) -> bool { - let Some(plan) = self.plan.as_ref() else { - return false; - }; - plan.item_of_record(record) - .and_then(|i| plan.item(i)) - .map(|it| it.is_group() && it.open) - .unwrap_or(false) - } - /// The summary row of record `n`: the fold marker, then what the record is. pub(super) fn record_row(&self, record: usize, inset: bool) -> Line { let spans = self.with_record(record, |r| match r { @@ -48,7 +38,7 @@ impl JsonlSource { _ => marker(tree::record_spans(v)), }, Record::Bad(why) => leaf(vec![Span::new( - format!("line {}: {why}", record + 1), + format!("{} {}: {why}", self.store.unit(), record + 1), tree::style::error(), )]), }); @@ -71,6 +61,8 @@ impl JsonlSource { } match self.spot(row) { Spot::Group { item } => Some(self.group_row(item)), + Spot::Body { record, line } => self.with_body(record, |rows| rows.get(line).cloned()), + Spot::Part { record, line } => self.with_parts(record, |laid| laid.rows.get(line).cloned()), Spot::Record { record, sub } => match sub { 0 => Some(self.summary_row(record)), n => self.with_tree(record, |rows| rows.get(n - 1).cloned()), @@ -132,21 +124,15 @@ impl JsonlSource { } } - /// The record a row belongs to. A group's row stands for its first record, - /// which is what keeps `record N/M`, search and yanking agreeing with what - /// the cursor is on. - pub(crate) fn record_at(&self, row: usize) -> usize { - match self.spot(row) { - Spot::Record { record, .. } => record, - Spot::Group { item } => self.item_first(item), - } - } - /// The value (and its path) on a tree row, or the whole record on a /// summary row. What `y` copies and what the status bar names. pub(super) fn at_row(&self, row: usize, f: impl FnOnce(&str, &Value) -> T) -> Option { let (record, sub) = match self.spot(row) { Spot::Record { record, sub } => (record, sub), + // A body row is the record talking; the value under the cursor + // there is the record, exactly as on its summary row. A part row is + // the record doing something, which is the same answer. + Spot::Body { record, .. } | Spot::Part { record, .. } => (record, 0), // A group's row stands for the run it holds; the value under the // cursor there is its first record. Spot::Group { item } => (self.item_first(item), 0), @@ -197,102 +183,105 @@ impl JsonlSource { } } - /// `zR` under a lens opens the groups the viewport has reached, the same - /// bounded way it opens records. - pub(super) fn open_groups(&mut self, upto_row: usize) { + // -- the message under a row ------------------------------------------------- + + /// Re-lay every body at the current width, then bring the totals up to + /// date. Classification is untouched: a record is read once, in file order, + /// and a resize only changes how what it said is wrapped. + /// + /// **Only when the width actually moved.** `Source::set_width` runs on + /// every resize the pager sees, including a height-only one where nothing + /// wraps differently, and re-laying is O(classified records) — O(the file), + /// with a record read apiece once `zR` has opened the bodies. Dragging a + /// terminal's bottom edge may not cost half a second per SIGWINCH. + pub(crate) fn remeasure(&mut self) { let Some(mut plan) = self.plan.take() else { return; }; - let mut map = std::mem::take(&mut self.map); - plan.open_upto(upto_row, &mut map); + match plan.set_width(self.view) { + true => lensrow::remeasure(&*self, &mut plan), + false => plan.sync(), + } self.plan = Some(plan); - self.map = map; + self.body_laid.borrow_mut().take(); } - /// Open the group holding `record`, so a search hit is never left behind a - /// fold. Does nothing when there is no lens, or the record is not in one. - pub(super) fn reveal_record(&mut self, record: usize) { + /// Re-lay one item — what a rung of the ladder costs. + pub(super) fn remeasure_item(&mut self, item: usize) { let Some(mut plan) = self.plan.take() else { return; }; - let mut map = std::mem::take(&mut self.map); - if let Some(item) = plan.item_of_record(record) { - plan.set_open(item, true, &mut map); - plan.sync(); - } + lensrow::measure(&*self, &mut plan, item); + plan.sync(); self.plan = Some(plan); - self.map = map; + self.drop_laid(); } - /// The first record of a plan item. - pub(crate) fn item_first(&self, item: usize) -> usize { - self.plan - .as_ref() - .and_then(|p| p.item(item)) - .map(|it| it.first) - .unwrap_or(0) + /// Forget the one-record row caches: what they hold was laid out for a + /// state that has just changed. + pub(crate) fn drop_laid(&self) { + self.body_laid.borrow_mut().take(); + self.parts_laid.borrow_mut().take(); } - /// Open or close the group whose first record is `first`. - pub(super) fn set_group(&mut self, first: usize, open: bool) -> bool { - let Some(mut plan) = self.plan.take() else { - return false; - }; - let mut map = std::mem::take(&mut self.map); - let changed = match plan.item_of_record(first) { - Some(item) => plan.set_open(item, open, &mut map), - None => false, - }; - plan.sync(); - self.plan = Some(plan); - self.map = map; - changed + /// Run `f` against the wrapped rows of the message under `record`. + pub(crate) fn with_body(&self, record: usize, f: impl FnOnce(&[Line]) -> T) -> T { + let width = self.view; + let hit = matches!(&*self.body_laid.borrow(), Some((r, w, _)) if *r == record && *w == width); + if !hit { + let rows = lensrow::body_rows(self, self.plan.as_ref(), record, width); + *self.body_laid.borrow_mut() = Some((record, width, rows)); + } + match &*self.body_laid.borrow() { + Some((_, _, rows)) => f(rows), + None => f(&[]), + } } - /// `zM`: shut every group, which is free — nothing is parsed to close one. - pub(super) fn close_groups(&mut self) { + /// Every message whole, or every message back to its clip: `zR` / `zM`, + /// and what a dump asks for. + pub(crate) fn set_all_bodies(&mut self, full: bool) { let Some(mut plan) = self.plan.take() else { return; }; - let mut map = std::mem::take(&mut self.map); - plan.close_all(&mut map); - plan.sync(); + plan.set_all_full(full); + lensrow::remeasure(&*self, &mut plan); self.plan = Some(plan); - self.map = map; + self.drop_laid(); } - /// `Tab` / `S-Tab` under a lens: the next thing that stands on its own — - /// a message, or a folded run — rather than the next record, most of which - /// a lens has deliberately folded away. - pub(super) fn next_item(&self, row: usize, forward: bool) -> Option { - let plan = self.plan.as_ref()?; - let record = self.record_at(row); - let cur = plan.item_of_record(record)?; - let base = plan.row_of_item(cur, &self.map); - match forward { - true if cur + 1 < plan.items().len() => Some(plan.row_of_item(cur + 1, &self.map)), - true => None, - // Inside an item, the first press goes back to its header row. - false if row > base => Some(base), - false if cur > 0 => Some(plan.row_of_item(cur - 1, &self.map)), - false => None, - } + /// The whole message under `record`, when there is one. + pub(crate) fn body_text(&self, record: usize) -> Option { + lensrow::body_text(self, self.plan.as_ref(), record) } - /// `Y` on a group's row: every record the run holds, one JSON document per - /// line — the folded rows as data, so what was copied is what was hidden. - pub(super) fn yank_group(&self, item: usize) -> Option { - let plan = self.plan.as_ref()?; - let it = plan.item(item)?; - let (first, count) = (it.first, it.count); - let mut out = String::new(); - for record in first..first + count { - if let Some(text) = self.with_record(record, |r| r.value().map(|v| v.to_json())) { - out.push_str(&text); - out.push('\n'); - } + /// `r`: the record's own tree, open or shut, whatever its level is doing + /// (SPEC.md §Lenses). On a call row it is the row's **record**: a call has + /// no JSON of its own separate from the record it was made in. + /// `None` when there is nothing there to open — a group's row (which + /// `Enter` opens), or a record with no tree under it — and the pager says + /// so rather than appearing to do nothing. + pub(crate) fn open_tree_at(&mut self, row: usize) -> Option { + if let Spot::Group { .. } = self.spot(row) { + return None; + } + let record = self.record_at(row); + if record >= self.known() || !self.record_visible(record) || self.tree_len(record) == 0 { + return None; } - JsonlSource::yank(out, format!("{count} records")) + let shut = self.map.is_open(record); + match shut { + true => self.map.close(record), + false => self.open_record(record), + }; + Some(format!( + "record {} {}", + record + 1, + match shut { + true => "shut", + false => "opened", + } + )) } // -- the outline --------------------------------------------------------------- @@ -328,7 +317,7 @@ impl JsonlSource { self.outline.push(Entry { level: 1, id: fold_id(record), - text: format!("line {} {text}", record + 1), + text: format!("{} {} {text}", self.store.unit(), record + 1), anchor: Anchor(self.row_of_record(record)), folded: !self.map.is_open(record), }); @@ -341,27 +330,21 @@ impl JsonlSource { fn rebuild_lens_outline(&mut self, rows: Range) { let mut entries: Vec = Vec::new(); for row in rows.start..rows.end.min(self.len_rows()) { - match self.spot(row) { + let at = self.spot(row); + match at { Spot::Group { item } => { - let open = self - .plan - .as_ref() - .and_then(|p| p.item(item)) - .map(|it| it.open) - .unwrap_or(false); - let first = self.record_at(row); entries.push(Entry { level: 1, - id: plan::group_id(first), + id: self.fold_id_at(at), text: self.group_row(item).text().trim_end().to_string(), anchor: Anchor(row), - folded: !open, + folded: !lensrow::item_open(self.plan.as_ref(), item), }); } Spot::Record { record, sub: 0 } if self.tree_len(record) > 0 => { entries.push(Entry { level: u8::from(self.in_open_group(record)) + 1, - id: fold_id(record), + id: self.fold_id_at(at), text: self.summary_row(record).text().trim_end().to_string(), anchor: Anchor(row), folded: !self.map.is_open(record), @@ -378,19 +361,27 @@ impl JsonlSource { /// Through the lens when there is one, so `--toc --lens agent` is the /// conversation as a list rather than a column of `{…12 keys}`. pub fn summaries(&mut self, n: usize) -> Vec { - self.index_to(n, FRAME_BYTES); + // A `--toc` is a batch, not a frame: it may spend slice after slice, + // because printing nothing is the one answer a script cannot tell from + // "this file has no records". One budgeted slice was enough for a + // record per line and not for records inside a *document*, where the + // store serves nothing at all until the top level has been walked — a + // 6MB trajectory printed zero lines and exited 0. Ended the way + // `dump::write_source` ends: only "no more work *and* no new records" + // stops it, since the slice that finishes the index is also the one + // that finds the last records. + while self.known() < n && !self.complete() { + let before = self.known(); + if !self.store.extend(FRAME_BYTES) && self.known() <= before { + break; + } + } self.classify_to(n); let take = n.min(self.known()); (0..take) .map(|r| { - if let Some(sum) = self.plan.as_ref().and_then(|p| p.summary(r)) { - let time = sum.time.clone().unwrap_or_default(); - // Through the sanitiser, exactly as the painted row is: a - // record may hold any byte, and `--toc` writes straight to - // a terminal (SPEC.md §JSON, the shared sanitiser). - let actor = crate::render::visible(&sum.actor); - let what = crate::render::visible(&sum.what); - return format!("{}\t{actor}\t{time}\t{what}", r + 1); + if let Some(line) = lensrow::toc_line(self.plan.as_ref(), r) { + return line; } let body = self.with_record(r, |rec| match rec { Record::Value(v) => tree::record_spans(v) diff --git a/src/source/record/source.rs b/src/source/record/source.rs new file mode 100644 index 0000000..7d5109f --- /dev/null +++ b/src/source/record/source.rs @@ -0,0 +1,346 @@ +//! A record document behind the [`Source`] seam: the state, and the reads. +//! +//! # One source, two formats +//! +//! This was `JsonlSource` until a second record format arrived. Nothing in it +//! was ever about a line: the rows, the folding, the search sweep, the outline +//! and the status bar all speak *records*, and the only line-shaped thing was +//! where the bytes came from. That is now [`Store`], so `.jsonl` and a JSON +//! document's array share one implementation rather than two that drift. +//! +//! # None of them read until they are shown +//! +//! * **The index is the store's**, and it is lazy in both: a `stat` and a +//! handful of bytes on the open path, pushed a budget at a time from the +//! frame and from the idle tick. +//! * **A record is parsed when it is shown, and only then.** A real trajectory +//! has a single 32KB record among fifty; laying out a screen by parsing every +//! record would be as bad as reading the whole file. Rendering a row parses +//! that row's record, and a small [`Cache`] keeps the ones the viewport keeps +//! asking about. +//! * **The default view is a list of records**, one row each +//! ([`tree::record_spans`]). `Enter` / `za` expands one into the tree +//! [`tree`] renders, spliced in under its row by [`RowMap`]. +//! * **A record that is not JSON is an error row** carrying the reason and the +//! record number, and the file keeps rendering: half a log is still worth +//! reading. +//! +//! # What is not folded is not parsed +//! +//! Fold state here is the *open* records, not the closed ones: everything +//! starts closed, so the closed set of a million-record file would be a million +//! ids. That is the same "default plus exceptions" shape the document source +//! uses, in the same vocabulary ([`jsonrow::ALL_OPEN`]) — a fold id is a +//! member-index path from the root, and a record document's root is the +//! implicit list of records, so record 4 is `/4`. One scheme, two sources. +#![deny(unsafe_code)] + +/// The [`Source`] implementation itself, and the rows under it. Submodules of +/// this file rather than siblings, so all three see one set of imports and the +/// state above stays private to them. +#[path = "view.rs"] +mod view; + +#[path = "rows.rs"] +mod rows; + +/// `Enter` / `za`, and the levels it walks. +#[path = "ladder.rs"] +mod ladder; + +/// The lens state, and the thin forwarders that reach it. +#[path = "lensstate.rs"] +mod lensstate; + +use std::cell::RefCell; +use std::ops::Range; + +use super::super::search::{self, Dir}; +use super::super::{Anchor, End, Entry, FoldState, Hit, LinkSite, Mark, MatchSpan, Source}; +use super::plan::{Plan, Spot}; +use super::rowmap::RowMap; +use super::store::{Cache, Record, Store}; +use super::tree; +use super::{body, fold_id, leaf, lensrow, marker, ops, parts, Records}; +use crate::json::Value; +use crate::lens::Lens; +use crate::render::{Line, LineKind, Span}; +use crate::select::Yank; +use crate::source::jsonrow; + +/// Records the index is pushed past the painted window on every frame. +pub(crate) const LOOKAHEAD: usize = 1024; + +/// Records indexed when the layout width is first set, so the first screen is +/// there without waiting for an idle tick. +const FIRST_RECORDS: usize = 256; + +/// Bytes of scanning one [`Source::lines`] call may spend on that lookahead. +pub(crate) const FRAME_BYTES: u64 = 4 * 1024 * 1024; + +/// Bytes of scanning one idle tick may spend. +const IDLE_BYTES: u64 = 8 * 1024 * 1024; + +/// Records one search sweep reads before giving up, as in the CSV source: a +/// sweep must fit inside a keystroke, so search covers the neighbourhood of the +/// cursor rather than a whole multi-GB file. +const SEARCH_RECORDS: usize = 20_000; + +/// Records the lens classifies past the painted window, so grouping is decided +/// before the rows it changes are asked for. Smaller than [`LOOKAHEAD`]: +/// classifying a record parses it, and the index is far cheaper than that. +const CLASS_AHEAD: usize = 256; + +/// Records read through the lens before the first frame: a screenful, not the +/// [`FIRST_RECORDS`] the index reaches, because classifying costs a parse +/// apiece and the frame extends it as far as it actually needs. +const FIRST_CLASS: usize = 64; + +/// Records classified per idle tick, so `G` reaches a lens-read end without a +/// keystroke ever waiting on the whole file. +const CLASS_SLICE: usize = 2048; + +/// Records `zR` will open at once. +/// +/// Opening *every* record of a million-record log means parsing the whole file, +/// which is the one thing this source exists not to do. `zR` therefore opens as +/// far as the reader is looking and says so, rather than freezing (see +/// [`Source::fold_all`]). +pub(crate) const EXPAND_CAP: usize = 50_000; + +pub struct RecordSource { + /// Where the records come from. The one part of this that a format owns. + pub(crate) store: S, + cache: RefCell, + /// The rows of the one expanded record the viewport is working through. + /// Capacity one: rebuilding it costs a walk of that record, and a frame + /// asks for it once per painted row. + laid: RefCell)>>, + /// The wrapped message rows of the one record the viewport is reading, + /// keyed by the width they were laid out for. Same discipline as `laid`, + /// and for the same reason: a frame asks per painted row, and wrapping a + /// 40 KB message once per row of it would be quadratic. + body_laid: RefCell)>>, + /// The part rows of the one record the reader has open at the level between + /// its headline and its JSON, keyed by width. Same discipline as + /// `body_laid`: one record, rebuilt when the reader moves on. + parts_laid: RefCell>, + /// Which records are open, and what that does to the row numbering. + pub(crate) map: RowMap, + /// The `--lens` reading of this file, when one was asked for. `None` is + /// the generic record tree, unchanged in every respect (SPEC.md §Lenses: + /// "without it, records render as the generic tree"). + pub(crate) plan: Option, + /// `zR` was pressed (or a dump asked for everything): records are opened as + /// the viewport reaches them, up to [`EXPAND_CAP`]. + pub(crate) expand_all: bool, + /// Records `expand_all` has already considered. + pub(crate) filled: usize, + pub(crate) view: usize, + /// Outline entries for the records last painted — see [`Source::outline`]. + pub(crate) outline: Vec, + /// Rows last painted. + pub(crate) window: Range, + query: String, + needle: String, + sensitive: bool, + /// Row of the current match, when the last sweep found one. + found: Option, + /// Always empty: a record file has no links. Held so the trait can hand + /// out a slice. + pub(crate) none_links: Vec, +} + +/// A record document is a sequence of records: whatever the store indexed, +/// parsed when it is shown and not before. +impl Records for RecordSource { + fn known(&self) -> usize { + self.store.known() + } + + /// Straight through the parsed-record cache, so the seam pays nothing extra + /// for asking: a record the frame is about to paint is already in hand. + fn with_value(&self, record: usize, f: impl FnOnce(Option<&Value>) -> T) -> T { + self.with_record(record, |rec| f(rec.value())) + } + + fn foldable(&self, record: usize) -> bool { + self.tree_len(record) > 0 + } +} + +impl RecordSource { + pub(crate) fn new(store: S) -> RecordSource { + RecordSource { + store, + cache: RefCell::new(Cache::default()), + laid: RefCell::new(None), + body_laid: RefCell::new(None), + parts_laid: RefCell::new(None), + map: RowMap::default(), + plan: None, + expand_all: false, + filled: 0, + view: 80, + outline: Vec::new(), + window: 0..0, + query: String::new(), + needle: String::new(), + sensitive: false, + found: None, + none_links: Vec::new(), + } + } + + // -- the store ------------------------------------------------------------ + + /// Grow the index toward `records`, spending at most `budget` bytes. + pub(crate) fn index_to(&self, records: usize, budget: u64) { + self.store.index_to(records, budget); + } + + /// Records indexed so far. + pub(crate) fn known(&self) -> usize { + self.store.known() + } + + pub(crate) fn complete(&self) -> bool { + self.store.complete() + } + + /// The raw bytes of record `n`. + pub(crate) fn raw(&self, record: usize) -> Vec { + self.store.raw(record) + } + + /// The raw text of record `n`, for the search sweep: no value is built and + /// no row is laid out, which is what keeps a sweep inside a keystroke. + fn raw_text(&self, record: usize) -> String { + String::from_utf8_lossy(&self.raw(record)).into_owned() + } + + // -- records --------------------------------------------------------------- + + /// Run `f` against record `n`, parsing it if it is not already in hand. + /// + /// A closure rather than a returned reference because the record lives in a + /// [`RefCell`] and can be tens of megabytes: handing out a clone to read one + /// field would undo the point of the cache. + pub(crate) fn with_record(&self, record: usize, f: impl FnOnce(&Record) -> T) -> T { + if self.cache.borrow().position(record).is_none() { + let loaded = self.store.load(record); + self.cache.borrow_mut().push(record, loaded); + } + let cache = self.cache.borrow(); + match cache.position(record).and_then(|i| cache.items.get(i)) { + Some((_, rec)) => f(rec), + // Unreachable: it was just inserted. Answering rather than + // panicking is the rule this whole seam is held to. + None => f(&Record::Bad("unavailable".to_string())), + } + } + + /// Rows record `n`'s tree occupies. `0` for a scalar or an error row, which + /// is what makes them leaves with nothing to open. + pub(crate) fn tree_len(&self, record: usize) -> usize { + self.with_record(record, |r| r.value().map(tree::row_count).unwrap_or(0)) + } + + /// Run `f` against the laid-out rows of record `n`'s tree, laying them out + /// if the cache is holding a different record. + pub(crate) fn with_tree(&self, record: usize, f: impl FnOnce(&[Line]) -> T) -> T { + let hit = matches!(&*self.laid.borrow(), Some((r, _)) if *r == record); + if !hit { + let rows = self.with_record(record, |r| match r.value() { + Some(v) => tree::rows(v, record + 1), + None => Vec::new(), + }); + *self.laid.borrow_mut() = Some((record, rows)); + } + match &*self.laid.borrow() { + Some((_, rows)) => f(rows), + None => f(&[]), + } + } + + // -- search ------------------------------------------------------------------- + + /// Look for the query from row `from`, wrapping once, over record *source + /// text* — far cheaper than laying rows out, and it finds a value whose + /// row the viewport has cut off the right-hand side. + /// Returns the *record* it found and whether the sweep wrapped; turning + /// that into a row is [`RecordSource::hit`]'s job, because it may have to + /// open a fold first. + fn sweep(&self, from: usize, dir: Dir, inclusive: bool) -> Option<(usize, bool)> { + let n = self.known(); + if self.query.is_empty() || n == 0 { + return None; + } + let step: isize = match dir { + Dir::Forward => 1, + Dir::Backward => -1, + }; + let start = self.record_at(from.min(self.len_rows().saturating_sub(1))); + let mut record = start as isize + isize::from(!inclusive) * step; + let mut wrapped = false; + for _ in 0..SEARCH_RECORDS.min(n) + 1 { + if record < 0 { + record = n as isize - 1; + wrapped = true; + } else if record >= n as isize { + record = 0; + wrapped = true; + } + if self.hits(record as usize) { + return Some((record as usize, wrapped)); + } + record += step; + } + None + } + + fn hits(&self, record: usize) -> bool { + let text = self.raw_text(record); + match self.sensitive { + true => text.contains(&self.needle), + false => text.to_lowercase().contains(&self.needle), + } + } + + /// Turn a swept *record* into a row, opening whatever folds it away: a + /// match the reader cannot see is not a match (SPEC.md §Lenses — a lens + /// may fold, but it may never lose a record). + fn hit(&mut self, found: Option<(usize, bool)>) -> Option { + let (record, wrapped) = found?; + self.reveal_record(record); + let row = self.row_of_record(record); + self.found = Some(row); + Some(Hit { anchor: Anchor(row), wrapped }) + } + + // -- yank ---------------------------------------------------------------------- + + fn yank(text: String, what: String) -> Option { + match text.is_empty() { + true => None, + false => Some(Yank { text, what }), + } + } + + /// One row as source-faithful text: a string's own characters, anything + /// else as compact JSON. A string is the value a reader wants to paste + /// somewhere; re-quoting it would be exporting rather than copying, which + /// is the same rule the CSV form applies to a field. + fn row_json(&self, row: usize) -> Option { + // On a body row the value under the cursor is what was said, whole — + // not the record wrapped around it. `Y` still copies the record. + if let Spot::Body { record, .. } = self.spot(row) { + return self.body_text(record); + } + self.at_row(row, |_, v| match v { + Value::Str(s) => s.clone(), + other => other.to_json(), + }) + } +} + diff --git a/src/source/record/store.rs b/src/source/record/store.rs new file mode 100644 index 0000000..c051232 --- /dev/null +++ b/src/source/record/store.rs @@ -0,0 +1,101 @@ +//! Where records come from: the one thing a record *format* still owns. +//! +//! Everything above this — the plan, the row arithmetic, folding, search, the +//! outline, every painted row — is [`super::RecordSource`], written once. What +//! a format has to answer is narrow and entirely about bytes: how many records +//! the index has found, how to push that index along inside a budget, and how +//! to hand over record `i` as bytes or as a value. +//! +//! Two formats implement it today and they could hardly be less alike: a +//! `.jsonl` file, where a record is a line of the CSV lazy line index +//! ([`crate::source::jsonl`]), and an array *inside* a JSON document, where a +//! record is a member of the lazy structural index +//! ([`crate::source::jsonarray`]). Neither knows the other exists, and neither +//! contains a single row number. +//! +//! # Everything here is `&self` +//! +//! Reading a record is a *file* read, and half of [`crate::source::Source`] is +//! `&self`; a store therefore keeps its reader behind a `RefCell` exactly as +//! the two sources above it used to. The rule that comes with it is the one +//! that was already in force: take the borrow and drop it inside one small +//! helper, never across a call back into the seam. +#![deny(unsafe_code)] + +use crate::json::Value; + +/// A sequence of records, as bytes. +pub trait Store { + /// Records the index has found so far. Grows as the index is pushed. + fn known(&self) -> usize; + + /// The index has reached the end of the records: [`Store::known`] is now + /// the real count. + fn complete(&self) -> bool; + + /// How far the index has got, 0..=100, for the status bar's honest + /// `\u{2265}N (indexing 40%)`. + fn progress(&self) -> u8; + + /// Push the index toward `records`, spending at most `budget` bytes. May + /// stop short: a caller asks again next frame. + fn index_to(&self, records: usize, budget: u64); + + /// Spend an idle tick on the index. True while there is more to do *or* + /// this slice found new records. + fn extend(&self, budget: u64) -> bool; + + /// Record `i` as the bytes the file holds — no value is built, which is + /// what keeps a search sweep inside a keystroke and what `c` copies. + fn raw(&self, record: usize) -> Vec; + + /// Read and parse record `i`. The only place a record becomes a value. + fn load(&self, record: usize) -> Record; + + /// What one record is called, for a message a person reads: a `.jsonl` + /// record is a `line`, a member of a document's array is a `record`. + fn unit(&self) -> &'static str; +} + +/// A record, as far as this reader got with it. +pub enum Record { + /// Valid JSON. + Value(Value), + /// Not JSON, or too big to parse, and why. Rendered as an error row rather + /// than stopping the file (SPEC.md §JSON). + Bad(String), +} + +impl Record { + pub(crate) fn value(&self) -> Option<&Value> { + match self { + Record::Value(v) => Some(v), + Record::Bad(_) => None, + } + } +} + +/// Parsed records kept in hand. Small on purpose: a record can be megabytes, +/// and the viewport only ever asks about a screenful plus whatever the cursor +/// just left. +const CACHE: usize = 64; + +/// The parsed-record cache: most-recently-used first, [`CACHE`] deep. +#[derive(Default)] +pub(crate) struct Cache { + pub(crate) items: Vec<(usize, Record)>, +} + +impl Cache { + pub(crate) fn position(&self, record: usize) -> Option { + self.items.iter().position(|(r, _)| *r == record) + } + + /// Insert at the front, dropping the oldest if the cache is full. + pub(crate) fn push(&mut self, record: usize, rec: Record) { + if self.items.len() >= CACHE { + self.items.pop(); + } + self.items.insert(0, (record, rec)); + } +} diff --git a/src/source/jsonl/tree.rs b/src/source/record/tree.rs similarity index 100% rename from src/source/jsonl/tree.rs rename to src/source/record/tree.rs diff --git a/src/source/jsonl/view.rs b/src/source/record/view.rs similarity index 66% rename from src/source/jsonl/view.rs rename to src/source/record/view.rs index 0b7b98a..e1208b0 100644 --- a/src/source/jsonl/view.rs +++ b/src/source/record/view.rs @@ -1,21 +1,25 @@ //! The [`Source`] implementation itself: what the pager sees. //! -//! Split from `mod.rs`, which holds the state and the file access, so both stay -//! under the size limit. The order of the sections below is the order of the -//! trait's own (`src/source/mod.rs`), and every method a record file has no -//! honest answer for says so rather than inventing one. +//! Split from `source.rs`, which holds the state and the reads, so both stay +//! under the size limit. Written once for every record format: what a `.jsonl` +//! and an array inside a document disagree about is [`Store`] and nothing here. +//! The order of the sections below is the order of the trait's own +//! (`src/source/mod.rs`), and every method a record file has no honest answer +//! for says so rather than inventing one. #![deny(unsafe_code)] use std::ops::Range; use super::*; -impl Source for JsonlSource { +impl Source for RecordSource { // -- layout --------------------------------------------------------------- - /// Width changes nothing about the row *count*: a tree row is never - /// wrapped, so one node stays one row at any width and every [`Mark`] - /// survives a resize. + /// A tree row is never wrapped, so one node stays one row at any width — + /// but the **message** under a lens row is wrapped, so a width change + /// re-lays every body and moves every row below the first of them. That is + /// why [`Mark`] here is a *record* rather than a row: the cursor comes back + /// to what it was reading, not to row 41 of a different layout. fn set_width(&mut self, cols: usize) { self.view = cols.max(1); // Index a first slice so `len()` is not zero before the first paint. @@ -25,6 +29,11 @@ impl Source for JsonlSource { // And read that slice through the lens, so the first frame is already // grouped rather than re-flowing under the reader. self.classify_to(FIRST_CLASS); + // Re-wrap what is already classified. No record is re-read and none is + // re-classified: the summaries hold what the measurement needs. + self.remeasure(); + // The last search hit was a row in the old layout. + self.found = None; } fn len(&self) -> usize { @@ -56,21 +65,41 @@ impl Source for JsonlSource { true } + /// A lens turns a record document into a list of blocks — a message with + /// what was said under it, or a folded run of mechanics — so `Tab` frames + /// the block it lands on rather than its first row. With no lens this is + /// the generic tree again: one record per row, one node per row, each + /// already its own unit. `plan.is_some()` is the one spelling of + /// "is a lens on" this file uses (see [`Source::mark`]). + fn blocks(&self) -> bool { + self.plan.is_some() + } + + fn block_at(&self, row: usize) -> Option> { + self.block_rows(row) + } + fn position_text(&self, row: usize) -> Option { let (record, sub) = match self.spot(row) { Spot::Record { record, sub } => (record, sub), + // A body row is that record's row, and so is a part row. + Spot::Body { record, .. } | Spot::Part { record, .. } => (record, 0), Spot::Group { item } => (self.item_first(item), 0), }; let known = self.known(); let total = match self.complete() { true => format!("{known}"), false => { - let pct = self.store.borrow().progress().percent(); + let pct = self.store.progress(); format!("\u{2265}{known} (indexing {pct}%)") } }; let head = match self.lens_name() { - Some(lens) => format!("{lens} \u{b7} record {}/{total}", record.saturating_add(1)), + Some(lens) => format!( + "{lens} \u{b7} record {}/{total}{}", + record.saturating_add(1), + self.block_text(row) + ), None => format!("record {}/{total}", record.saturating_add(1)), }; // The path only means something inside an open record; on the summary @@ -89,7 +118,7 @@ impl Source for JsonlSource { /// drive it (SPEC.md §CSV, the same contract). fn end(&self) -> End { if !self.complete() { - return End::Scanning(self.store.borrow().progress().percent()); + return End::Scanning(self.store.progress()); } // The index has read the file, but the lens decides how many rows // those records make: answering now would put `G` in the middle. @@ -108,18 +137,7 @@ impl Source for JsonlSource { /// asks again only while this says yes, and the slice that finishes the /// index is also the one that produces most of its rows. fn extend(&mut self) -> bool { - let more = { - let mut guard = self.store.borrow_mut(); - let s = &mut *guard; - match s.index.complete() { - true => false, - false => { - let before = s.index.known(); - s.index.ensure_bytes(IDLE_BYTES, &mut s.reader); - !s.index.complete() || s.index.known() > before - } - } - }; + let more = self.store.extend(IDLE_BYTES); // The lens gets its own bounded slice of the same idle tick: a record // is only grouped once it has been read, and `G` waits on that. let done = self.plan.as_ref().map(|p| p.classified()); @@ -153,29 +171,54 @@ impl Source for JsonlSource { (n > 0).then(|| anchor.0.min(n - 1)) } + /// Under a lens a mark is the **record**, not the row: a body is wrapped, + /// so a resize moves rows and a row number would mean somewhere else + /// afterwards (`src/source/mod.rs`: a mark is the same *content*). + /// + /// What that costs is the offset *inside* a record: the cursor on tree row + /// seven of an open record, or on line four of a message, comes back to + /// that record's summary row. A `Mark` is one number, and packing an offset + /// into it would be a lie about which record a bigger number is in. + /// + /// With no lens there is no body and nothing wraps — a tree row is one row + /// at every width — so the row *is* the same content, and paying that cost + /// would throw away the place inside an open record for nothing. fn mark(&self, row: usize) -> Option { - (row < self.len_rows()).then_some(Mark(row)) + if row >= self.len_rows() { + return None; + } + Some(match self.plan.is_some() { + true => Mark(self.record_at(row)), + false => Mark(row), + }) } fn locate(&self, mark: Mark) -> Option { - let n = self.len_rows(); - (n > 0).then(|| mark.0.min(n - 1)) + let rows = self.len_rows(); + if rows == 0 { + return None; + } + if self.plan.is_none() { + return Some(mark.0.min(rows - 1)); + } + let known = self.known(); + if known == 0 { + return None; + } + Some(self.row_of_record(mark.0.min(known - 1))) } // -- structure --------------------------------------------------------------- /// The records the last frame painted, not the whole file — see - /// [`JsonlSource::rebuild_outline`] for why a million-record outline is a + /// [`RecordSource::rebuild_outline`] for why a million-record outline is a /// million parses. fn outline(&self) -> &[Entry] { &self.outline } fn section_at(&self, row: usize) -> Option { - let id = match self.spot(row) { - Spot::Group { item } => plan::group_id(self.item_first(item)), - Spot::Record { record, .. } => fold_id(record), - }; + let id = self.fold_id_at(self.spot(row)); self.outline.iter().position(|e| e.id == id) } @@ -184,10 +227,10 @@ impl Source for JsonlSource { Some(e) => e.id.clone(), None => return false, }; - // A group id (`g12`) cannot collide with a record id (`/12`), so one - // outline holds both and this is the only place that tells them apart. - if let Some(first) = plan::group_first(&id) { - let changed = self.set_group(first, !closed); + // A group id (`g12`) cannot collide with a record id (`/12`); the seam + // tells them apart, and `None` is "not a group" — this file only has to + // answer for its own records, which is the half that costs a parse. + if let Some(changed) = self.set_group_by_id(&id, !closed) { if changed { if let Some(e) = self.outline.get_mut(entry) { e.folded = closed; @@ -219,6 +262,10 @@ impl Source for JsonlSource { self.expand_all = !closed; self.filled = 0; self.map.clear(); + // Messages go with it: `zR` is "show me everything here", and a dump + // (`--plain`, which is `fold_all(false)` and nothing else) must print + // the whole of what was said rather than a viewport's clip of it. + self.set_all_bodies(!closed); if closed { self.close_groups(); } @@ -228,6 +275,43 @@ impl Source for JsonlSource { } } + /// `Enter` / `za`: the record's two levels — clipped and open, and back + /// again (SPEC.md §Lenses). The row's own fold, which is not an outline + /// entry and could not be reached through one. The raw tree is `r`, not a + /// rung, so this never opens or shuts one. A group row falls through to the + /// outline, which is what opens a run. + /// + /// **A record's row claims the key even when it has no rung.** Falling + /// through would hand `Enter` to the outline, and a record's outline entry + /// *is* its raw tree — so on exactly the records with nothing between the + /// headline and the JSON (a message the clip already showed whole, which + /// made no calls) the key would open and shut what `r` owns, and the tree + /// `r` had opened would vanish under a press of `Enter`. Under a lens the + /// answer there is "nothing to descend into", and it is answered here. + /// With **no lens** there is no ladder and no lens rows: a record row is a + /// collapsed tree and the outline is what opens it, so the key falls + /// through exactly as it always did. + fn fold_here(&mut self, row: usize) -> Option { + if let Some(was) = self.descend(row) { + return Some(was); + } + // No lens, no ladder: the key belongs to the outline, as it always did. + self.plan.as_ref()?; + match self.spot(row) { + // A run is opened *through* the outline, which is the one thing + // here that is an outline entry in its own right. + Spot::Group { .. } => None, + // Claimed, and nothing moved: no level changed, so "was open" is + // false and the pager repaints the rows it already had. + _ => Some(false), + } + } + + /// `r`: the raw record under the cursor, whatever its body is doing. + fn toggle_tree(&mut self, row: usize) -> Option { + self.open_tree_at(row) + } + /// The shared fold-id vocabulary ([`jsonrow::ALL_OPEN`]): a default plus the /// ids that disagree with it, each id a member-index path from the root. /// A record file's root is the implicit list of records, so record 4 is @@ -242,10 +326,7 @@ impl Source for JsonlSource { if self.expand_all { out.push(jsonrow::ALL_OPEN.to_string()); } - if let Some(p) = self.plan.as_ref() { - let open = p.items().iter().filter(|it| it.is_group() && it.open); - out.extend(open.map(|it| plan::group_id(it.first))); - } + out.extend(self.group_folds()); out.extend(self.map.records().map(fold_id)); out } @@ -254,12 +335,7 @@ impl Source for JsonlSource { self.map.clear(); self.expand_all = folds.iter().any(|s| s == jsonrow::ALL_OPEN); self.filled = 0; - self.close_groups(); - for id in &folds { - if let Some(first) = plan::group_first(id) { - self.set_group(first, true); - } - } + self.restore_groups(&folds); for id in &folds { if let Some(record) = jsonrow::top_index(id) { self.open_record(record); @@ -272,6 +348,10 @@ impl Source for JsonlSource { fn hidden_at(&self, row: usize) -> Option { let (record, sub) = match self.spot(row) { + // A body row hides nothing: the clip says what it is not showing. + // Nor does a part row — a shut call carries its own glyph, and the + // gutter marker this answer drives belongs to the record. + Spot::Body { .. } | Spot::Part { .. } => return None, // A closed group hides one row per record it holds; their trees // are closed with it, so there is nothing else under it. Spot::Group { item } => { @@ -289,10 +369,11 @@ impl Source for JsonlSource { } } - /// `Tab` / `S-Tab`: the next record, which is the only landmark a log has. + /// Block boundaries — what `Tab` / `S-Tab` jump between; see + /// `plan_block.rs`. fn next_landmark(&self, row: usize, forward: bool) -> Option { if self.plan.is_some() { - return self.next_item(row, forward); + return self.next_block_row(row, forward); } let (record, sub) = self.map.at(row); match forward { @@ -383,13 +464,14 @@ impl Source for JsonlSource { 1 => "1 value".to_string(), n => format!("{n} values"), }; - JsonlSource::yank(out, what) + RecordSource::::yank(out, what) } /// `y`: the value under the cursor (SPEC.md §JSON). fn yank_point(&self, row: usize) -> Option { let (record, sub) = match self.spot(row) { Spot::Record { record, sub } => (record, sub), + Spot::Body { record, .. } | Spot::Part { record, .. } => (record, 0), Spot::Group { item } => (self.item_first(item), 0), }; let text = self.row_json(row)?; @@ -397,7 +479,7 @@ impl Source for JsonlSource { Some(p) if sub > 0 => format!("{p} \u{b7} record {}", record + 1), _ => format!("record {}", record + 1), }; - JsonlSource::yank(format!("{text}\n"), what) + RecordSource::::yank(format!("{text}\n"), what) } /// `Y`: the whole record as one line of valid JSON, wherever in it the @@ -414,17 +496,19 @@ impl Source for JsonlSource { return None; } let text = self.with_record(record, |r| r.value().map(|v| v.to_json()))?; - JsonlSource::yank(format!("{text}\n"), format!("record {}", record + 1)) + RecordSource::::yank(format!("{text}\n"), format!("record {}", record + 1)) } - /// `c`: the record's source line, verbatim — the bytes the file holds, - /// not this reader's re-serialisation of them. + /// `c`: the record's source text, verbatim — the bytes the file holds, not + /// this reader's re-serialisation of them. A `.jsonl` record is a line and + /// says so; a record inside a document is a member and says that. fn yank_block(&self, row: usize) -> Option { let record = self.record_at(row); if record >= self.known() { return None; } let raw = String::from_utf8_lossy(&self.raw(record)).into_owned(); - JsonlSource::yank(format!("{raw}\n"), format!("line {} verbatim", record + 1)) + let what = format!("{} {} verbatim", self.store.unit(), record + 1); + RecordSource::::yank(format!("{raw}\n"), what) } } diff --git a/src/source/text/tests.rs b/src/source/text/tests.rs index a927696..4674c3e 100644 --- a/src/source/text/tests.rs +++ b/src/source/text/tests.rs @@ -438,3 +438,12 @@ fn a_short_last_line_is_still_settled_and_whole() { assert_eq!(rows(&mut t), ["alpha", "omega"]); assert!(!t.store.borrow_mut().row(1).expect("row 1").truncated); } + +/// Verbatim lines, and no landmarks to move between: `j` stays one row. +#[test] +fn plain_text_does_not_read_in_blocks() { + let mut s = src("one\ntwo\nthree\n"); + finish(&mut s); + assert!(!s.blocks()); + assert_eq!(s.block_at(0), None); +} diff --git a/src/source/types.rs b/src/source/types.rs new file mode 100644 index 0000000..2e5c3e5 --- /dev/null +++ b/src/source/types.rs @@ -0,0 +1,105 @@ +//! The seam's vocabulary: the handles, the state and the small records every +//! [`Source`](super::Source) hands back. +//! +//! None of it is a format and none of it has behaviour — split out of `mod.rs` +//! so that file is the trait and its contract, and nothing else. +#![deny(unsafe_code)] + +/// A place in the document that survives folding but not re-layout. +/// +/// Opaque to everything above the seam: the pager only ever compares anchors +/// (they order the same way the document reads) and hands them back to the +/// source it got them from. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub struct Anchor(pub usize); + +/// A place in the *content* that survives re-layout, so a resize can put the +/// cursor back where it was. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub struct Mark(pub usize); + +/// Opaque, format-defined fold state: the ids of the closed sections. +/// +/// The pager stores it in a history [`Snapshot`](crate::nav::history::Snapshot) +/// and hands it back verbatim; it never inspects an id. Ids must be stable +/// across re-layout, which is what lets folds survive a resize. +pub type FoldState = Vec; + +/// One entry of the document outline (`o`, and the collapse tree). +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Entry { + /// Nesting depth, 1 = outermost. Drives indentation and the fold ranges. + pub level: u8, + /// Stable id for this section: the key fold state is stored under, and the + /// target of an anchor link (`#some-heading`). + pub id: String, + /// Text shown in the outline overlay. + pub text: String, + /// Where the section starts. + pub anchor: Anchor, + /// True when this section is currently folded shut. + pub folded: bool, +} + +/// One link occurrence in the document, in reading order. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LinkSite { + /// The row the link sits on, as an anchor (it may be folded away). + pub anchor: Anchor, + /// Display column the link starts at, within its row. + pub col: usize, + pub url: String, +} + +/// One search match on a row, in display columns of that row. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct MatchSpan { + pub start: usize, + pub end: usize, + /// True for the match the cursor is currently sitting on. + pub current: bool, +} + +/// Where `G` lands, and whether the format still has work to do to know. +/// +/// A format that discovers its document lazily — a CSV's row index — genuinely +/// does not know where the end is until it has scanned there, and the *worst* +/// answer is the confident one: jumping to the end of whatever happens to be +/// indexed puts the cursor in the middle of the file and says nothing about it. +/// [`End::Scanning`] is that honest "not yet", carrying the percentage the +/// status bar shows while the pager drives the scan a slice at a time. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum End { + /// The last row `G` should put the cursor on. + At(usize), + /// The end is not known yet; `0..=100` of the way there. + Scanning(u8), +} + +/// Where a search landed, and whether it wrapped around the document. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Hit { + pub anchor: Anchor, + pub wrapped: bool, +} + +/// One row expanded into labelled fields. +/// +/// A grid shows as many columns as fit and no more, which is exactly wrong for +/// the row you actually care about: a wide CSV hides most of it off-screen, and +/// a ragged row can carry fields the header never named. This is that row read +/// the other way round — one field per line, label beside value, nothing +/// hidden. A future tree format would return a node's children the same way. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Detail { + /// What the overlay is titled, e.g. `Row 41`. + pub title: String, + /// `(label, value)`, in the format's own order. A field the format has no + /// name for still appears — labelled positionally rather than dropped. + /// + /// Values are **raw**: exactly the bytes the document holds, control + /// characters and all. Painting them is what makes them safe + /// ([`crate::render::visible`]), so that copying one yields the real value + /// rather than the dotted display form. + pub fields: Vec<(String, String)>, +} diff --git a/src/theme.rs b/src/theme.rs index 70537f5..0acc31f 100644 --- a/src/theme.rs +++ b/src/theme.rs @@ -259,6 +259,16 @@ pub const fn lens_time() -> Style { pub const fn lens_group() -> Style { Style::new().fg(MUTED_FG).italic() } +/// The message itself, under its summary row. Plain: this is the document, and +/// the summary above it is the headline. +pub const fn lens_body() -> Style { + Style::new() +} +/// `⋯ +12 lines` — what a clipped body is not showing. A notice, like the +/// group count, and never mistakable for the message it stands at the end of. +pub const fn lens_more() -> Style { + Style::new().fg(MUTED_FG).italic() +} /// Stands in for a row's left border when the row carries more fields than the /// header named. It replaces the bar rather than sitting beside it: a grid that diff --git a/tests/keymap_docs.rs b/tests/keymap_docs.rs index d2a66b2..bc26f95 100644 --- a/tests/keymap_docs.rs +++ b/tests/keymap_docs.rs @@ -134,7 +134,10 @@ fn the_readme_table_invents_nothing() { #[test] fn the_cli_help_mentions_the_core_keys() { let cli = read("src/cli.rs"); - for needle in ["za", "zM", "zR", "Tab", "Backspace", "q"] { + // `r` is pinned by its sentence rather than by the letter: a bare "r" + // matches almost any word in the help text, so it would pass with the key + // deleted — which is exactly the drift this test exists to catch. + for needle in ["za", "zM", "zR", "Tab", "Backspace", "q", "r shows the raw record"] { assert!(cli.contains(needle), "--help never mentions {needle}"); } }