[JayOfTheKeyboard]

Problem

A section over 800 tokens is split by splitAtParagraphs, which cuts at every blank line:

const paragraphs = content.split(/\n\n+/);

Code blocks often have blank lines in them, so when the budget runs out partway through a block, the block is cut in two. One part ends with an opening fence and no close, so everything after it in that part reads as code. The next part starts in the middle of the code with no fence, so its code reads as prose and its closing fence opens a new block. has_code is wrong for both parts.

Fastify's Guides/Database.md is one example. Its Mongo snippet has a blank line between two options, and the cut lands there. The Database section ends with:

*Usage:*
```javascript
const fastify = require('fastify')()

and Database (part 2) starts with the rest of the code and no fence:

fastify.register(require('@fastify/mongodb'), {
  // force to close the mongodb connection when app stopped
  // the default value is false
  forceClose: true,

  url: 'mongodb://mongo/mydb'
})

I built packages with context add <dir> --path <docs> on main and with this change, and counted sections with an odd number of fence lines:

Docs	Sections (main → fix)	Sections with an unpaired fence (main → fix)	Sections with code (main → fix)
Python 3.14 HTML (526 pages)	7,934 → 7,915	216 → 16	2,447 → 2,489
Drizzle ORM	1,619 → 1,601	343 → 17	1,366 → 1,387
Playwright	2,465 → 2,476	110 → 0	1,161 → 1,187
Vitest	1,498 → 1,503	40 → 0	1,052 → 1,061
Pydantic	580 → 583	62 → 0	387 → 406
Fastify	289 → 292	56 → 2	168 → 176
pg-boss	177 → 177	8 → 0	113 → 113
Starlette	253 → 253	6 → 0	78 → 80
Requests (reStructuredText)	107 → 107	0 → 0	0 → 0

No line of text is lost. Every distinct line in each package on main is also in the package with the fix. Token totals move by about 0.1% or less, except Drizzle (+0.2%): whitespace, plus a few whole sections that are now identical to another section and are removed by the existing deduplication (Playwright's trace-viewer.md shows the same C# block for two test frameworks). Drizzle grows because the custom-types.mdx pages have a long TypeScript block with ``` examples in its doc comments. On main that block was cut into dozens of small pieces at those fences, and the existing deduplication dropped the pieces that repeated. It is now cut by line only where the hard limit forces it, so the repeated comment text stays.

What is left: code blocks over the 1,200-token hard limit, which splitOversizedContent still cuts by line, so each leaves two sections with an unpaired fence. That is all 16 in Python (8 blocks), both in Fastify (1 block), and 12 of the 17 in Drizzle (6 blocks). The other 5 in Drizzle are pages whose own fences do not pair up (cache.mdx has a stray ```, data-querying.mdx has one inside a JSX comment).

Fix

Markdown sections are now split at blank lines outside fenced code only (splitMarkdownParagraphs). The fence rules follow CommonMark:

• An opening fence is three or more backticks or tildes. A backtick fence's info string cannot contain a backtick, so a line that starts with ```npm i``` is prose.
• Only a bare run of the same character, at least as long, closes it. A ```js line inside a ```md example does not close it, and neither does ``` inside ````.
• Any indent opens a fence, and so does a list marker in front of it (1. ```bash), because fences in list items are indented. A closing fence has no list marker and sits at most 3 columns right of the opening fence (tabs stop every 4 columns). A fence shown indented inside a code example therefore does not close the block around it.
• A fence that never closes is read as text, so one stray fence does not hold the rest of the section together. CommonMark would run it to the end of the document, but here that would turn every later paragraph into one chunk. Fences that cannot close are found in one backwards pass, so this stays linear (timings below).

These rules live in one helper, markdown-fences.ts. The paragraph splitter, the ## heading splitter that splitForParsing uses on very large files, hasCodeBlock and the cut at code blocks in splitOversizedContent all use it, in place of the heading splitter's own scanner and a backtick-only regex. For the heading splitter this adds fences behind a list marker and the closing-indent limit, and a fence that never closes no longer hides every later heading in the file. hasCode is now true when a line opens a fence, so ~~~ blocks, fences in a blockquote and each part of a block cut at the hard limit count as code, and ```npm i``` installs it does not. AsciiDoc and reStructuredText turn tilde fences off, because ~~~ underlines headings there.

The section is split first, and MDX tags are removed from each paragraph afterwards. Removing tags first can shift an indented fence to column 0 (<Tabs> / <Tab> around an indented block), and then its closing fence no longer matches. Cleaning each paragraph is safe for generics in code (#173), because no paragraph holds part of a block. A line holding only MDX tags (<Tip>, </Note>) outside a code block is a break, like a blank line, so MDX prose without blank lines around its tags still splits. Tag bodies are not kept together: a blank line inside <Tip> splits its body like any other paragraph break.

The split scans the text itself. Without MDX support, remark reads a fence directly under a JSX line such as <Section> as part of an HTML block. I tried splitting at remark's code nodes, and Drizzle still had 266 sections with an unpaired fence, against 17 with this change.

AsciiDoc and reStructuredText keep the plain blank-line split. Their code syntax is different, and an RST subheading underlined with ~~~~~~~ would otherwise read as a fence.

Not changed

• Code blocks over HARD_LIMIT_TOKENS are still cut by line in splitOversizedContent, with an unpaired fence in each part. Closing and reopening the fence at each cut would fix that. It is a separate function, so I left it for a follow-up if you want one. (Before that cut, a paragraph over the limit is now split at the edges of the blocks the fence rules find, not at every ```.)
• Fences inside blockquotes (the splitters do not pair them, though hasCode counts them), CRLF files (the old split did not handle them either), and AsciiDoc ---- / RST :: blocks with blank lines in them.
• A short lead-in such as "Run:" just before a code block that is now kept whole can end up as a section of its own, or be dropped if it is under 5 tokens. That is the existing packing of small paragraphs, which this change reaches more often.
• MDX_TAG_OUTSIDE_CODE keeps its own fence rules. Sharing one fence scanner between tag removal and splitting would be a larger refactor.

Test

build.test.ts:

• A long section whose budget runs out inside a code block with blank lines: the block stays whole in one section, every section has 0 or 2 fences, and <Tip> tags around it are still removed.
• One block of three ~325-token pieces (over the 800 budget, under the hard limit) stays in one section for each of: a longer fence around a shorter one, a tilde fence around a backtick one, a fence around a ```js line, a tilde fence, a fence in a list item, a fence after 1. , a fence shown indented inside a block, the same tab-indented, a list item fence shown inside a block, and a fence after 1. closed by ` \t```` (a tab stop, not 4 extra columns).
• An indented block inside MDX <Tabs> / <Tab> stays whole.
• A fence that never closes, and a line that opens with inline code, do not stop the section from splitting.
• MDX prose with <Tip> / <Note> lines and no blank lines around them still splits within the budget, with the tags removed.
• A tag inside a paragraph (<Tip>Run it twice.</Tip>) is removed from a split section, and its text is kept.
• An RST section with ~ subheadings still splits at paragraphs, and its sections, short or split, have hasCode: false.
• hasCode is true for a tilde fence, a fence in a blockquote, and each part of a block cut at the hard limit, and false for a line that opens with ```npm i```. An AsciiDoc ``` fence still sets it.
• A paragraph over the hard limit is cut at the edges of a tilde block, and of a ```ts block with ```js examples in its doc comments, so each block stays whole.

package-builder.test.ts:

• A ## line inside a fence behind 1. is not a heading.
• A fence that never closes does not stop a later ## from splitting.

The existing test "keeps generics in a code block split across section parts" used a 12-line block that is now kept whole. It now uses 20 lines, which is over the hard limit, so the block is still split and the generics check still applies.

On main 19 of these tests fail, for example expected [] to have a length of 1 but got +0 (no section holds the whole block). With the fix they pass. The blockquote, AsciiDoc and RST hasCode tests and the tag-inside-a-paragraph test pass on main too; they guard against this change taking those away.

I broke each part of the fix in turn. 24 of 26 changes turned a test red:

Mutation	Test that fails
Split at every blank line, as before	14 tests
Any run of the same character closes	longer fence around a shorter one
A run of either character closes	tilde fence around a backtick one
A closing fence may have an info string	fence around a ```js line, and the heading splitter's info-string test
No list marker before an opening fence	fence after 1. , and ## in a fence behind 1.
Only 0 to 3 spaces of indent open a fence	fence in a list item, fence after 1.
No tilde fences	both tilde cases
No backtick info-string rule	line that opens with inline code
No closing-indent limit	fence shown indented, and tab-indented, inside a block
A tab counts as one column	tab-indented case
A tab counts as 4 columns, not to the next stop	fence closed at a tab stop after 1.
A list marker can close	list item fence shown inside a block
No rewind for a fence that never closes	fence that never closes
Tags removed before splitting	MDX tabs case
Lines of MDX tags alone are not breaks	lines that hold only MDX tags
Paragraphs not cleaned of MDX tags	tag inside a paragraph
RST split with the Markdown splitter	RST ~ subheadings
Tilde fences on for AsciiDoc and RST	RST ~ subheadings hasCode
Backtick fences off for AsciiDoc and RST too	AsciiDoc and RST code detection
Fences in a blockquote not counted	blockquote hasCode
hasCode ignores the backtick info-string rule	line that opens with ```npm i```
hasCode from closed blocks only	blockquote hasCode, parts of a block cut at the hard limit
Oversized paragraph not cut at block edges	tilde block, and block with fences in its comments
Heading splitter ignores fences	4 heading splitter tests

The two that stayed green are the backwards pass that marks fences which cannot close, and remembering a fence that failed to close (below). Both change speed, not output, so no test covers them. The <Tip> check used to catch a missing paragraph cleanup, but once a line holding only tags became a break, those <Tip> lines never reached the cleanup, so the tag-inside-a-paragraph test now covers it.

The backwards pass is there for speed. Timed with parseMarkdown on one section with an unclosed ```js on every fourth line:

Lines	main	this change	without the backwards pass
4,000	244 ms	276 ms	1,260 ms
16,000	643 ms	756 ms	12,981 ms

A fence that fails to close is also remembered, so a later fence with the same character, at least as long and no further right, is read as text without scanning again. With 6,000 info-string fences and one closer indented too far at the end, parseMarkdown takes 3.1 s, against 2.7 s on main (most of it is remark).

Validation

• vitest run in packages/context: 18 files, 393 tests pass.
• biome ci --error-on-warnings: clean.
• tsc --noEmit -p tsconfig.build.json: clean, in packages/context and packages/registry.
• Real docs: the table above, from context add builds of nine doc sets on main and on this branch.
• Changeset: .changeset/tidy-fences-hold.md (patch).

[changeset-bot]

🦋 Changeset detected

Latest commit: 196f826

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@neuledge/context	Patch
@neuledge/registry	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[human0-reviewer]

Changes requested — Keeps fenced code blocks together when oversized Markdown sections are split, and adds coverage for Markdown fence and MDX cases.

Changes behavior: no — This corrects the existing section-splitting bug rather than changing the package's intended promise. First review; no prior behavior finding to clear.

Must fix

• The description promises MDX tag bodies stay together, but they can still split (packages/context/src/build.ts:410)
• The fence rules now live in two separate scanners (packages/context/src/build.ts:332)

1 note, no action needed

• Suggestion: Tilde-fenced sections still get marked as having no code The new scanner now keeps ~~~ blocks together, but hasCodeBlock still only recognizes backtick fences, so a preserved tilde-fenced section is stored with hasCode: false. That value was already false before this change, so it does not block the fix, but aligning the metadata with the newly recognized fence style would make it accurate. (packages/context/src/build.ts:354)

---

This codebase is run by Human0.

[human0-reviewer]

Changes requested — Keeps recognized Markdown code fences together when long sections are split, preserving complete code blocks for document ingestion.

Changes behavior: no — This is a bug fix toward the existing behavior promised by Markdown parsing: code blocks should remain intact when long sections are divided. It changes outputs that were malformed, not the project's intended contract. This full look confirms the change is a fix toward the existing promise; no new commits have landed since the earlier review.

Must fix

• Use one shared Markdown fence rule (packages/context/src/build.ts:384)

1 note, no action needed

• Suggestion: Keep code metadata consistent with tilde fences The new splitter preserves tilde-fenced blocks, but hasCodeBlock still recognizes only backticks, so those sections are stored with hasCode: false. It would be better to derive the flag from the same fence rules. (packages/context/src/build.ts:377)

---

This codebase is run by Human0.

[JayOfTheKeyboard]

The two splitters now share one fence rule in packages/context/src/markdown-fences.ts, and hasCode and the code-block cut in splitOversizedContent use it too, so ~~~ blocks count as code (AsciiDoc and reStructuredText leave tildes out, because ~~~ underlines headings there). For the heading splitter this adds fences behind a list marker and the closing-indent limit, and a fence that never closes no longer hides every later heading in the file; both changes have tests. On the MDX point, the code does what was intended: a line of tags alone is a break, and tags are removed from each paragraph after the split, so a blank line inside <Tip> does split its body. I reworded that paragraph of the description so it says this plainly.

[human0-reviewer]

Approved — Keeps fenced code intact when long Markdown sections are split, and uses the shared fence logic for heading detection and code flags. The change moves the splitter toward its existing promise rather than changing the intended contract.

Changes behavior: no — This corrects section splitting and hasCode when fenced code is present; it makes the implementation meet the existing intent rather than adding or changing a promised capability. Full review of the current base-to-head range confirms the behavior is a bug fix toward the existing promise; the earlier changes-requested findings are fixed in this head.

1 note, no action needed

• Suggestion: The tag remover still has its own fence check The MDX tag remover still carries a separate fence check, so tag cleanup and section splitting can drift on fence variants. Sharing the parser there would keep those paths aligned; this is a follow-up, not a blocker. (packages/context/src/build.ts:252)

---

This codebase is run by Human0.