Skip to content

Commit a0181dd

Browse files
CahidArdaCahidArda
authored
DX-2659 (#696)
* feat: add initial llm.txt generator * feat: llms.txt parity * feat: align llms-full.txt structure with upstream Mintlify output Refactor the builder to emit llms-full.txt in upstream's per-page format: # <title> Source: <url> # no .md suffix <blank lines> # 3 for MDX pages, 1 for OpenAPI ops <body> <blank lines> # 2 before the next page OpenAPI operations now carry a synthesized metadata line (`<spec> <method> <api-path>`) followed by the description, matching how Mintlify renders spec-driven pages. `expandOpenApi` now returns the method and API path alongside the existing fields. Apply the high-frequency content transforms Mintlify does at build time so MDX bodies look closer to the published markdown: - bullet `- ` → `* ` - horizontal rule `---` → `***` - code fence ``` ```lang ``` → ``` ```lang theme={"system"} ``` - internal links `/foo` → `/docs/foo` - link-URL `\&` → `&` - trailing whitespace trimmed on every line Known divergences from upstream llms-full.txt (~53K of 87K lines): - JSX component children (`<Tabs>`, `<Tab>`, `<Accordion>`, `<Steps>`) are re-indented by 4 spaces per nesting level in upstream. We emit them verbatim from the source MDX, because correct re-indentation requires a real MDX/JSX parser (remark-mdx + remark-stringify with component-aware rules). - Markdown tables are column-padded in upstream so cell boundaries align vertically. We emit tables verbatim. - Some `<img>` tags get `src=` attributes injected from external data (e.g. shields.io badge URLs). We can't reproduce these without Mintlify's component resolution data. - Blank-line spacing around components occasionally differs because of the above re-indentation pass. llms.txt remains byte-identical to upstream. Closing the llms-full.txt gap fully would require implementing a Mintlify-compatible MDX serializer; tracked as future work. * fix: add readme file
1 parent e037cfd commit a0181dd

12 files changed

Lines changed: 90283 additions & 1 deletion

File tree

‎.github/workflows/llms-check.yml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
name: llms.txt check
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
permissions:
10+
contents: write
11+
12+
jobs:
13+
check:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v4
17+
with:
18+
# Push back to the PR branch needs the branch ref, not the merge commit.
19+
ref: ${{ github.head_ref || github.ref }}
20+
# Use a token that can push back to PR branches.
21+
token: ${{ secrets.GITHUB_TOKEN }}
22+
23+
- uses: actions/setup-node@v4
24+
with:
25+
node-version: "20"
26+
cache: "npm"
27+
cache-dependency-path: llms/package-lock.json
28+
29+
- name: Install
30+
working-directory: llms
31+
run: npm ci
32+
33+
- name: Check llms.txt is up to date
34+
id: check
35+
working-directory: llms
36+
run: npm run check
37+
continue-on-error: true
38+
39+
- name: Auto-commit regenerated llms.txt
40+
if: steps.check.outcome == 'failure' && github.event_name == 'pull_request'
41+
run: |
42+
git config user.name "github-actions[bot]"
43+
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
44+
git add llms.txt llms-full.txt
45+
git commit -m "chore(llms): regenerate llms.txt and llms-full.txt"
46+
git push
47+
48+
- name: Fail if llms.txt was stale
49+
if: steps.check.outcome == 'failure'
50+
run: |
51+
echo "::error::llms.txt / llms-full.txt were out of date. A regeneration commit was pushed to this PR — please review it and re-run CI."
52+
exit 1

‎.gitignore‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,9 @@
33
.gitignore
44
package.json
55
package-lock.json
6-
node_modules/*
6+
node_modules/*
7+
8+
# llms/ is a real npm workspace — it needs its package files tracked.
9+
!llms/package.json
10+
!llms/package-lock.json
11+
llms/node_modules/*

0 commit comments

Comments
 (0)