fix(rag): detect top-level JSON arrays in auto format detection

[Theater-ahyeon]

PR Checklist

• The commit message follows our guidelines (conventional commits)
• Related issue(s) linked
• Tests for the changes have been added
• Docs have been added / updated
• Changesets have been added

What is the current behavior?

detectFormat() accepted { or [ as the opening character but only ever accepted } as the closing one:

if ((text.startsWith("{") || text.startsWith("[")) && text.endsWith("}")) {

A top-level JSON array ends with ], so it never entered the JSON.parse branch and was reported as "text". detectStrategy() in document/transformers.ts maps that result to a chunker, so with strategy: "auto" any document whose content is a JSON array — API response dumps, exported record sets, .json fixtures — went to RecursiveChunker instead of JsonChunker, losing the structural path: value chunks and the fields / samplePaths metadata.

Object payloads were unaffected.

What is the new behavior?

The brackets are paired, so arrays are detected too:

if (
  (text.startsWith("{") && text.endsWith("}")) ||
  (text.startsWith("[") && text.endsWith("]"))
) {

The existing try { JSON.parse(text) } catch still guards malformed input, so [unclosed] keeps falling through to the html / latex / code / table / markdown checks and still returns "text".

fixes (issue)

fixes #1458

Notes for reviewers

• Scope is the array case only. I deliberately did not touch the table check below (/\n\|.+\|\n/ requires a newline before the first row, so a table at the very start of a document is also missed) — that is a separate bug and I would rather not bundle it.
• Also noted in [BUG] rag: detectFormat never recognizes top-level JSON arrays, so "auto" chunking skips the JSON chunker #1458 but not changed here: JsonChunker's format: "json" is overwritten by the format: "text" it inherits from the token chunker, because buildMetadata ends with Object.assign(meta, params.extra) and only sourceType is stripped from the inherited metadata.
• New file packages/rag/src/__tests__/format-detector.spec.ts with four cases: array detection, object detection, malformed fall-through, and an end-to-end check that chunkByStrategy("auto", '[{"name":"alice"},{"name":"bob"}]') now yields metadata.fields === 2 and a chunk containing 0.name: alice.

Verified locally (Windows host, no network or API keys):

• vitest run in packages/rag -> 47 passed across 19 files.
• With format-detector.ts reverted to main, the array detection test and the end-to-end test fail (expected 'text' to be 'json', expected undefined to be 2); with the change they pass.
• biome check clean on both changed files (the CRLF complaint when checking a raw Windows checkout disappears against the committed LF blob).

AI assistance: this fix and its tests were produced with an AI coding agent under my direction. The bug, the paired-bracket fix, the duplicate check and every test result quoted above were verified by running those commands against a checkout of main at 72a46c7.

---

Summary by cubic

Fixes detectFormat() so top-level JSON arrays are detected as JSON instead of "text". Previously the check accepted { or [ as an opening bracket but only } as a closing one, so arrays ending in ] fell through to RecursiveChunker under the auto strategy, losing JSON structural chunks and metadata. Brackets are now paired ({...} and [...]), and JSON.parse still guards malformed input.

fixes #1458

^{Written for commit 6e047d7. Summary will update on new commits.}

Summary by CodeRabbit

• Bug Fixes
  • Automatic format detection now recognizes top-level JSON arrays, allowing them to be chunked as JSON. Malformed bracketed input continues to be treated as text.

[changeset-bot]

🦋 Changeset detected

Latest commit: 6e047d7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@voltagent/rag	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 4b34a0c1-a3af-415d-b803-598eeffc6eb4

📥 Commits

Reviewing files that changed from the base of the PR and between 72a46c7 and 6e047d7.

📒 Files selected for processing: 3

• .changeset/curly-donkeys-detect.md
• packages/rag/src/__tests__/format-detector.spec.ts
• packages/rag/src/utils/format-detector.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Walkthrough

detectFormat now recognizes top-level JSON arrays as candidates for JSON parsing. Tests cover format detection and automatic chunking of an array. A patch changeset records the fix.

Changes: JSON Array Detection

Layer / File(s)	Summary
Detect and validate JSON arrays packages/rag/src/utils/format-detector.ts, packages/rag/src/__tests__/format-detector.spec.ts, .changeset/curly-donkeys-detect.md	The JSON candidate check pairs { with } and [ with ]. Tests cover detection of arrays and objects, malformed input, and automatic array chunking. The changeset records a patch release for @voltagent/rag.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: omeraplak

Merge Risk: ⚪ Minimal · up to 6e047

Well-formed top-level JSON arrays are now routed to the JSON chunker under the auto strategy. Object payloads and malformed input behave as before. No merge-blocking risk is evident.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 2 files. (1 skipped: 1 …	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title is concise, specific, and accurately states the main change: detecting top-level JSON arrays during automatic format detection.
Description check		The description follows the repository template and includes the checklist, current and new behavior, issue reference, test coverage, changeset status, and reviewer notes. Documentation is marked as n…
Linked Issues check		Issue #1458 requires well-formed top-level JSON arrays to return "json" and route through JsonChunker under strategy: "auto". format-detector.ts now pairs {} and [] before JSON.parse. Th…
Out of Scope Changes check		The changes are limited to the bracket-pairing fix, focused regression tests, and a package changeset. These changes support issue #1458. The separate inherited metadata problem and table detection pr…

Full details: Description check

Explanation

The description follows the repository template and includes the checklist, current and new behavior, issue reference, test coverage, changeset status, and reviewer notes. Documentation is marked as not updated, but this is non-critical for the fix.

Full details: Linked Issues check

Explanation

Issue #1458 requires well-formed top-level JSON arrays to return "json" and route through JsonChunker under strategy: "auto". format-detector.ts now pairs {} and [] before JSON.parse. The new tests cover array detection, object compatibility, malformed input, and auto chunking with fields and indexed content. The changeset documents the fix.

Full details: Out of Scope Changes check

Explanation

The changes are limited to the bracket-pairing fix, focused regression tests, and a package changeset. These changes support issue #1458. The separate inherited metadata problem and table detection problem remain unchanged, as required by the stated scope.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 2 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cubic-dev-ai]

No issues found across 3 files

_{View guided diff | Re-trigger cubic}