Release FileContext 1.0.19

CHANGELOG.md

@@ -2,6 +2,14 @@
 
 All notable changes to ManagedCode.FileContext are documented here.
 
+## 1.0.19
+
+- Apply file, result, page, concurrency, and timeout limits only when explicitly configured or requested. Default text and document reads return the remaining content.
+- Inspect CSV records without materializing non-header values. Preserve quoted multiline fields, blank lines, and trailing tab-delimited columns.
+- Preserve DOCX tabs, line breaks, and continuation offsets. Return XLSX base cell text without phonetic annotations.
+- Reduce allocations during file search and XLSX inspection. Reuse report text styles and index table highlights once per table.
+- Paginate long PDF headings and KPI rows. Check configured output budgets before uploading generated files.
+
 ## 1.0.17
 
 - Update deterministic agent integration dependencies and APIs to `ManagedCode.LlmTestServer` 1.0.0.

Directory.Build.props

@@ -12,7 +12,7 @@
     <AnalysisMode>Recommended</AnalysisMode>
     <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
     <NoWarn>$(NoWarn);CS1591;MAAI001</NoWarn>
-    <Version>1.0.18</Version>
+    <Version>1.0.19</Version>
     <PackageVersion>$(Version)</PackageVersion>
   </PropertyGroup>
 

README.md

@@ -197,9 +197,9 @@ an existing host directory. Disposal, cancellation and staging failures remove t
 
 `IFileContextPdf.ReadPdfTextAsync(path)` returns bounded text, `PageCount`, and one-based `PagesWithoutText`. It does not perform OCR. A scanned page can instead be rendered with `RenderPdfPageAsync(path, pageNumber)`, which returns PNG `DataContent`. Use `CountPdfPageImagesAsync` and `ExtractPdfImageAsync` when the original embedded pictures are needed rather than the complete page. The four read-only `file_context_pdf_*` tools expose the same operations from scoped storage.
 
-For an authenticated PDF already held as bytes, `FileContextPdfTextExtractor.Extract`, `FileContextPdfImages.RenderPagePng`, and `FileContextPdfImages.ExtractPageImagesPng` work without storing it. PDF source reads default to 100 MiB and accept `FileContextOptions` for a different limit; page rasterization also uses configured pixel and PNG limits. `FileContextImageContent` creates model-visible `DataContent` from PNG bytes or base64 and `UriContent` from an HTTPS URL. A URL reference is not fetched by FileContext, so the model provider must be able to access it. A host must pass image content to its model as image content. A generic OpenAI Chat function result serializes it as text, so hosts must explicitly bridge image tool results into a multimodal model message.
+For an authenticated PDF already held as bytes, `FileContextPdfTextExtractor.Extract`, `FileContextPdfImages.RenderPagePng`, and `FileContextPdfImages.ExtractPageImagesPng` work without storing it. Hosts can set PDF source, pixel and PNG limits through `FileContextOptions`. `FileContextImageContent` creates model-visible `DataContent` from PNG bytes or base64 and `UriContent` from an HTTPS URL. A URL reference is not fetched by FileContext, so the model provider must be able to access it. A host must pass image content to its model as image content. A generic OpenAI Chat function result serializes it as text, so hosts must explicitly bridge image tool results into a multimodal model message.
 
-`file_context_docx_text(path, startParagraph?, startCharacter?, paragraphCount?)` reads ordinary paragraph and table text from a scoped DOCX package. The result contains numbered paragraph segments and `nextParagraph`/`nextCharacter`; use that cursor to continue a long document. Reads are limited to 50 paragraphs and 20,000 characters per call, with a configurable 25 MiB source limit (`MaximumDocxReadBytes`). It does not OCR embedded images. DOCX and XLSX packages are excluded from generic text reads and grep.
+`file_context_docx_text(path, startParagraph?, startCharacter?, paragraphCount?)` reads ordinary paragraph and table text from a scoped DOCX package. The result contains numbered paragraph segments and `nextParagraph`/`nextCharacter`; use that cursor to continue a long document. The caller can request a paragraph count. Hosts can also configure `MaximumDocxTextCharacters` and `MaximumDocxReadBytes`. It does not OCR embedded images. DOCX and XLSX packages are excluded from generic text reads and grep.
 
 ## Explore Markdown as a graph
 
@@ -253,32 +253,20 @@ var tools = workspaces.GetRequiredKeyedService<FileContextProvider>("research");
 
 Paths are logical, relative, and `/`-separated. `RootPrefix` scopes storage access. Path validation rejects traversal and unsafe path forms before storage calls. File contents remain untrusted tool data.
 
+Size, count, PDF concurrency, and deadline limits apply when configured by the host or requested by the caller. Range, DOCX and PDF text operations return the complete remaining content when the caller omits the window size and the host has left the relevant options at their defaults.
+
 | Option | Default |
-| --- | ---: |
-| `MaximumPdfReadBytes` | 100 MiB |
-| `MaximumImageBytes` | 8 MiB |
-| `MaximumRenderedPagePixels` / `MaximumImagesPerPdfPage` | 4,000,000 / 20 |
-| `DefaultPdfPageScale` / `MinimumPdfPageScale` / `MaximumPdfPageScale` | 1.5 / 0.5 / 3 |
-| `PdfPngQuality` | 100 |
-| `MaximumFullReadBytes` | 1 MiB |
-| `MaximumRangeReadBytes` | 256 KiB |
-| `MaximumDirectoryEntries` | 1,000 direct children |
-| `MaximumOpenXmlPartCharacters` | 32 Mi characters per decompressed XML part |
-| `MaximumGeneratedFileBytes` | 64 MiB per generated file or complete report |
-| `MaximumReportInputBytes` / `MaximumReportPages` | 10,000,000 bytes / 60 pages |
-| `DefaultRangeLineCount` / `MaximumRangeLineCount` | 200 / 1,000 |
-| `MaximumSearchFiles` | 500 |
-| `MaximumSearchFileBytes` | 4 MiB |
-| `MaximumSearchResults` / `MaximumMatchesPerFile` | 100 / 20 |
-| `RegexTimeout` | 2 seconds per regex match |
-| `OperationTimeout` | `null` (reports default to 60 seconds) |
+| --- | --- |
+| Numeric `Maximum*` capacity options | No FileContext limit. `int.MaxValue` or `long.MaxValue` represents the unset capacity |
+| `MaximumOpenXmlPartCharacters` | `0`, which delegates to Open XML's unlimited setting |
+| `DefaultRangeLineCount` | Remaining lines within any configured maximum |
+| `DefaultPdfPageScale` / `PdfPngQuality` | 1.5 / 100 |
+| `MinimumPdfPageScale` / `MaximumPdfPageScale` | Positive scales representable by the raster engine |
+| `RegexTimeout` | `Regex.InfiniteMatchTimeout` |
+| `OperationTimeout` | `null` |
 | `MarkdownGlob` | `**/*.md` |
-| `MaximumMarkdownFiles` | 100 |
-| `MaximumMarkdownSourceBytes` | 1 MiB per file |
-| `MaximumGraphResults` | 20 |
-| `MaximumGraphExportCharacters` | 200,000 |
 
-These settings are available through `FileContextOptions`; their named defaults are exposed by `FileContextDefaults`. Credentials, container lifecycle, authorization, and session persistence remain host responsibilities. When saving conversations, preserve tool-call/result pairs and handle interrupted turns before replaying history.
+These settings are available through `FileContextOptions`. Credentials, container lifecycle, authorization, and session persistence remain host responsibilities. When saving conversations, preserve tool-call/result pairs and handle interrupted turns before replaying history.
 
 ### Configure limits and timeouts
 
@@ -305,7 +293,7 @@ services.AddManagedCodeFileContext(storage, options =>
 });
 ```
 
-`RegexTimeout` limits **one regex match on one line**, protecting the process from excessive backtracking in a model-supplied pattern. It is not a two-second deadline for the whole search. Choose a finite positive value supported by .NET Regex. Size/count limits must be positive, and the default line count cannot exceed the maximum.
+`RegexTimeout` limits **one regex match on one line**, protecting the process from excessive backtracking in a model-supplied pattern. Set a finite positive value supported by .NET Regex to enable it. Configured size/count limits must be positive, except that zero disables the Open XML part limit. An explicitly configured default line count must fit its maximum.
 
 In a host that uses Microsoft configuration binding, the same options can come from `appsettings.json`, environment variables, or another configuration source:
 
@@ -421,7 +409,7 @@ await using var bytes = await store.OpenReadAsync(report.Path);
 
 `FileContextProvider` exposes `file_context_create_text`, `file_context_create_csv`, `file_context_create_workbook`, and `file_context_create_pdf` when writes are enabled. `FileContextDocumentTools.Create(store, options)` returns those same native functions for host-managed tool registries, including sandboxed JavaScript bridges.
 
-CSV preserves literal values and escapes quotes, delimiters and newlines. XLSX supports multiple sheets and text/number/boolean/formula cells; Excel calculates explicit formulas when opened. PDF supports paginated paragraphs and a title with embedded Noto Sans for Latin/Cyrillic text. The Noto Sans font is distributed under the bundled SIL Open Font License. The default generated-file budget is 64 MiB, configurable through `MaximumGeneratedFileBytes`; cancellation and `OperationTimeout` apply to generation and persistence.
+CSV preserves literal values and escapes quotes, delimiters and newlines. XLSX supports multiple sheets and text/number/boolean/formula cells; Excel calculates explicit formulas when opened. PDF supports paginated paragraphs and a title with embedded Noto Sans for Latin/Cyrillic text. The Noto Sans font is distributed under the bundled SIL Open Font License. Hosts can set `MaximumGeneratedFileBytes` explicitly. Cancellation and any configured `OperationTimeout` apply to generation and persistence.
 
 Document generation uses DocumentFormat.OpenXml (MIT), PdfPig (Apache-2.0), and bundled Noto Sans (OFL-1.1). CSV writing uses the .NET runtime. These components do not require a paid commercial license. See [dependency licenses](docs/Development/dependency-licenses.md) for the audited package boundary and font notice.
 
@@ -450,6 +438,7 @@ The read-only agent tool is `file_context_tables_info`; it follows read approval
 XLSX returns each named table (excluding its header and totals), or a worksheet summary
 when no named tables exist. Blank and duplicate headers retain column positions.
 CSV detects comma, semicolon, tab or pipe, or accepts an explicit single-character delimiter.
+Inspection materializes the header and scans data records for width and count. Quoted line endings and blank lines remain part of the field.
 Quoted multiline fields form one record; physically empty lines are skipped.
 The optional one-based headerRow selects a CSV logical record or worksheet row;
 otherwise the first nonempty record/row is the header. Named Excel tables own their headers.

docs/Architecture.md

@@ -80,13 +80,13 @@ flowchart TD
 ## Operational limits
 
 PDF callers share `FileContextPdfProcessor`, acquiring a permit before reading or downloading the
-source and retaining it through parsing/rendering. `MaximumConcurrentPdfOperations` defaults to one.
+source and retaining it through parsing/rendering when the host sets `MaximumConcurrentPdfOperations`.
 `FileContextPdfRenderDocument` exposes page count and sequential page rendering from one parsed PDF;
 dispose it after the batch. The low-level synchronous image helpers remain caller-scheduled APIs.
 
 PDF text, page rendering and embedded-image APIs accept bounded seekable streams. Storage-backed PDF tools retain seekable local `FileStream`/`MemoryStream` inputs but asynchronously stage other streams, including seekable cloud streams, to an automatically deleted temporary file. PdfPig's synchronous byte reads and seeks then remain local, without blocking on repeated network ranges. `PdfSourceStagingMode.TemporaryFile` also stages local inputs. `PdfSourceBufferBytes` bounds every copy read and `PdfTemporaryDirectory` optionally selects an existing host directory. No whole-document managed array is created. Native raster decoding has a separate per-page source-image pixel budget; lowering output scale does not reduce source bitmap allocation.
 
-All potentially large operations are controlled by `IOptions<FileContextOptions>`: PDF source/page/image budgets, full-read bytes, range bytes, files scanned, bytes per searched file, matches per file, total search results, graph documents, graph source bytes, and exported graph characters. Non-seekable cloud streams are supported by sequential streaming.
+Hosts can configure bounds for large operations through `IOptions<FileContextOptions>`: PDF source/page/image budgets, full-read bytes, range bytes, files scanned, bytes per searched file, matches per file, total search results, graph documents, graph source bytes, and exported graph characters. Non-seekable cloud streams are supported by sequential streaming.
 
 ## Start here
 
@@ -112,7 +112,7 @@ flowchart LR
 
 ## Native workbook reads
 
-`FileContextDocumentService` opens XLSX packages read-only from the scoped storage adapter.
+`FileContextDocumentService` opens XLSX packages read-only from the scoped storage adapter. Local seekable streams can be read directly when the host has left the source budget unset. Other streams and reads with a configured source budget are staged before parsing.
 `FileContextWorkbookReader` maps worksheets and cell rectangles into package-owned result records.
 It preserves coordinates and stored types without evaluating formulas or accessing external links.
 The same service opens DOCX packages read-only and streams bounded paragraph text through

docs/Features/ReportRendering.md

@@ -17,13 +17,13 @@ Block content is JSON:
 
 - `heading`: `title`, optional `subtitle`.
 - `text`: `text` with bold spans (`**bold**`), dash bullets, and HTTPS Markdown links.
-- `kpi-row`: `items` with 2-6 entries containing `label`, `value`, optional `change`, and optional `indicator` (`up` or `down`).
+- `kpi-row`: `items` with entries containing `label`, `value`, optional `change`, and optional `indicator` (`up` or `down`).
 - `chart`: `title`, `chartType` (`line`, `bar`, `stacked-bar`, `pie`, `donut`), `labels`, `series` entries with `name` and numeric `values`, optional `xAxis` and `yAxis`.
 - `table`: `columns` with `key`, `label`, optional .NET numeric `format`; `rows` as objects; optional `totals` object and `highlightCells` entries with zero-based `row` and column `key`.
 - `image`: base64 PNG/JPEG bytes in `base64`, optional `caption`.
 - `callout`: `text`.
-- `page-break`: empty content.
+- `page-break`: omit `content` or pass an empty object.
 
-All requested outputs render in memory before FileContext stores any file. The combined size of the outputs must fit `MaximumGeneratedFileBytes`. `MaximumReportInputBytes` and `MaximumReportPages` default to 10 MB and 60 pages. `OperationTimeout` covers rendering and upload, with a 60-second report default. Image blocks are capped at 5 MB. PDF chart and table rows are kept intact across pages, table headers repeat after a page break, and each page has a title, NYWD mark or theme logo, UTC date, and page number. Failures name the block index and type where possible.
+All requested outputs render in memory before FileContext stores any file. When configured, `MaximumGeneratedFileBytes` applies to the combined size of the outputs. Hosts can also set `MaximumReportInputBytes`, `MaximumReportPages`, and an `OperationTimeout` for rendering and upload. PDF headings can continue onto additional pages, and KPI cards flow onto additional rows and pages as needed. PDF chart and table rows are kept intact across pages, table headers repeat after a page break, and each page has a title, NYWD mark or theme logo, UTC date, and page number. Failures name the block index and type where possible.
 
 Named template and theme persistence, network allowlists, operator authorization, and authenticated download links belong to the host. `AIAgents.App` implements that boundary in its `AgentFiles/Reports` module.

docs/Features/file-context.md

@@ -16,7 +16,7 @@ In scope: standard file access, bounded line navigation, metadata, native PDF/DO
 6. Full reads fail before allocation when metadata exceeds `MaximumFullReadBytes`. Decoded UTF-8 limits retain encoder state across buffer boundaries, including split surrogate pairs.
 7. Range reads are 1-based by line, stream sequentially, return continuation metadata, and stop at configured line/byte limits.
 8. Content search is case-insensitive regular expression search with a timeout, file/match/result limits, and optional standard glob filtering.
-9. Directory listing returns direct child directories before direct child files and keeps the storage prefix private. A listing with more than `MaximumDirectoryEntries` distinct children fails with a limit error. The default is 1,000.
+9. Directory listing returns direct child directories before direct child files and keeps the storage prefix private. When a host sets `MaximumDirectoryEntries`, a listing that exceeds it fails with a limit error.
 10. Markdown graph builds include only `.md` and `.markdown` files that match the configured glob and limits. The default glob selects `.md` files. Builds use `ManagedCode.MarkdownLd.Kb`.
 11. Graph search returns package-owned match records; graph export supports JSON-LD, Turtle, Mermaid, and DOT with an output limit.
 12. Each graph operation builds from the current selected storage contents, so graph results cannot become stale between calls.
@@ -98,7 +98,7 @@ Independent writes and range reads on eight different files are tested concurren
 
 ## Generated documents
 
-The shared store and `IFileContext.Documents` create unique text/CSV/XLSX/PDF outputs. Standard listing, metadata, range reads and host streaming access use their returned paths. The document tool factory and context provider share the same implementations; hosts may register a selected subset in a native or JavaScript tool catalog. Writes must be explicitly enabled on shared options and human approval is enabled by default.
+The shared store and `IFileContext.Documents` create unique text/CSV/XLSX/PDF outputs. When configured, CSV generation checks the output byte budget as it writes UTF-8, including the BOM and escaped fields. Standard listing, metadata, range reads and host streaming access use their returned paths. The document tool factory and context provider share the same implementations; hosts may register a selected subset in a native or JavaScript tool catalog. Writes must be explicitly enabled on shared options and human approval is enabled by default.
 
 ```mermaid
 sequenceDiagram
@@ -125,10 +125,11 @@ delete any staged file. Source ownership continues through document rendering an
 DOCX reading uses the native `file_context_docx_text` tool. It reads ordinary paragraph and table
 text from the scoped `.docx` package in bounded windows. Each result includes the next paragraph
 and character offset when more text remains, so an agent can continue without loading a long
-document into one model call. The tool does not execute macros, fetch external links, or perform
-OCR on embedded images. Generic text reads refuse DOCX, and text search skips it.
-XLSX and DOCX reads also enforce `MaximumOpenXmlPartCharacters` per decompressed XML part.
-The default is 32 Mi characters.
+document into one model call. Tabs and explicit line breaks are included in paragraph text and
+cursor offsets. Text nested in an inline text box belongs to the containing paragraph and keeps the
+same paragraph numbering across windows. The tool does not execute macros, fetch external links,
+or perform OCR on embedded images. Generic text reads refuse DOCX, and text search skips it.
+Hosts can set `MaximumOpenXmlPartCharacters` to bound individual decompressed XLSX and DOCX XML parts.
 
 `file_context_pdf_text` reports a bounded text-layer prefix, total page count, and one-based pages with almost no text. It performs no OCR. `file_context_pdf_page_image` renders a complete page as PNG. `file_context_pdf_images_info` counts embedded image objects, and `file_context_pdf_image` returns one object as PNG. The direct `IFileContextPdf` methods and public byte-oriented PDF APIs support the same operations. Storage-scoped PDF reads enforce a byte cap; page rendering enforces pixel and image-byte caps. Image tools return `DataContent`; host chat pipelines must forward it as image content rather than stringify a function result.