Skip to content

Improve doc_md rendering in Dag Documentation - #66809

Merged
pierrejeambrun merged 45 commits into
apache:mainfrom
minyeamer:ui/markdown-rendering
Jun 29, 2026
Merged

pierrejeambrun merged 45 commits into
apache:mainfrom
minyeamer:ui/markdown-rendering

Conversation

@minyeamer

@minyeamer minyeamer commented May 12, 2026 •

Copy link
Copy Markdown
Contributor

This PR improves how doc_md content renders in the Dag Documentation dialog in the new UI.

It adds proper rendering for inline code, fenced code blocks, KaTeX math, and Mermaid diagrams.

What changed

  • render inline code separately so it stays in normal sentence flow
  • render fenced code blocks in a dedicated full-width container with broader syntax highlighting, line numbers, and a copy action
  • add KaTeX support for math expressions and load the bundled stylesheet needed to display them
  • add Mermaid support for diagram blocks with strict security settings
  • add regression tests for inline code, fenced code blocks, math, and Mermaid rendering

Test plan

  • install dependencies with the committed lockfile to confirm the markdown-related packages and KaTeX assets install correctly:
    cd airflow-core/src/airflow/ui && corepack pnpm install --frozen-lockfile
  • run the focused React markdown tests to verify inline code, fenced code blocks, math, and Mermaid rendering:
    cd airflow-core/src/airflow/ui && corepack pnpm vitest run src/components/ReactMarkdown.test.tsx
  • run eslint on the touched UI files to catch typing, import, and lint regressions in the markdown rendering changes:
    cd airflow-core/src/airflow/ui && corepack pnpm exec eslint src/components/ReactMarkdown.tsx src/components/ReactMarkdownBlocks.tsx src/utils/renderMermaid.ts src/utils/syntaxHighlighter.ts src/components/ReactMarkdown.test.tsx src/main.tsx
  • build the UI bundle to confirm the new markdown packages and KaTeX styles work in a production build:
    cd airflow-core/src/airflow/ui && corepack pnpm build

Screenshots

Mode Before After
Light Before: light mode markdown dialog before the rendering improvements. After: light mode markdown dialog after the rendering improvements.
Dark Before: dark mode markdown dialog before the rendering improvements. After: dark mode markdown dialog after the rendering improvements.

Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: GitHub Copilot (GPT-5.4) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Copilot AI review requested due to automatic review settings May 12, 2026 16:42
@boring-cyborg boring-cyborg Bot added the area:UI Related to UI/UX. For Frontend Developers. label May 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enhances the new UI’s DAG Documentation markdown rendering so doc_md displays richer content (inline code, fenced code blocks with improved UX, KaTeX math, and Mermaid diagrams), and adds regression tests for the new behavior.

Changes:

  • Refactors markdown rendering to separate inline code from fenced code blocks and introduce dedicated block renderers (code + Mermaid).
  • Adds KaTeX + math plugin support and globally loads KaTeX styles.
  • Switches syntax highlighting setup and adds targeted Vitest coverage for markdown rendering cases.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
airflow-core/src/airflow/ui/src/utils/syntaxHighlighter.ts Switches exported highlighter + theme source (now hljs-based) and updates types.
airflow-core/src/airflow/ui/src/utils/renderMermaid.ts Adds Mermaid rendering helper with strict security configuration.
airflow-core/src/airflow/ui/src/main.tsx Imports KaTeX stylesheet globally.
airflow-core/src/airflow/ui/src/components/ReactMarkdownBlocks.tsx Adds block-level renderers for fenced code blocks and Mermaid diagrams (copy UX, loading state, SVG injection).
airflow-core/src/airflow/ui/src/components/ReactMarkdown.tsx Integrates remark-math/rehype-katex, new pre/code handling, Mermaid routing, and KaTeX display styling.
airflow-core/src/airflow/ui/src/components/ReactMarkdown.test.tsx Adds regression tests for inline code, fenced blocks, math, and Mermaid rendering.
airflow-core/src/airflow/ui/package.json Adds dependencies for KaTeX, Mermaid, and markdown math plugins.
airflow-core/src/airflow/ui/pnpm-lock.yaml Updates lockfile for new dependencies.
Files not reviewed (1)
  • airflow-core/src/airflow/ui/pnpm-lock.yaml: Language not supported

@minyeamer
minyeamer requested a review from vincbeck as a code owner May 13, 2026 04:39
@Srabasti

Copy link
Copy Markdown
Contributor

Static checks are failing. Please run prek locally in your branch to fix the errors and then commit to repo. Remember to rebase to ensure all latest commits from community are incorporated.

@choo121600

Copy link
Copy Markdown
Member

@minyeamer A few things need addressing before review — see our Pull Request quality criteria.

  • ❌ Pre-commit / static checks — Failing: CI image checks / Static checks. See docs.
  • ❌ Provider tests — Failing: provider distributions tests / Compat 3.0.6:P3.10:. See docs.

No rush.


Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

minyeamer and others added 6 commits May 14, 2026 01:03
…rams

- Fix inline code rendering to properly handle single backtick snippets
- Add syntax highlighting, line numbers, and copy buttons to code blocks
- Core support for LaTeX math expressions using KaTeX (inline and display)
- Integrate Mermaid.js to render diagrams directly from markdown blocks
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot follow-ups
- Lazy-load Mermaid so markdown rendering only pulls it in when Mermaid blocks are rendered
- Reuse existing i18n clipboard labels in markdown block actions instead of hard-coded JSX strings
- Replace the Mermaid loading text with a spinner-only state and remove the redundant jest-dom test import

Compat 3.0.6 fix
- Pin pyjwt>=2.11.0 in providers/fab so compat installs paired with Airflow 3.0.6 do not fall back to PyJWT 2.10.1
- Fix flask_jwt_extended imports that expect jwt.types.Options during FAB and Google provider test setup
- Keep the FAB wheel metadata aligned with the dependency floor already required by current airflow-core

Validation
- Pass targeted eslint, vitest, and UI build checks for the markdown rendering changes
- Reproduce the FAB compat install sequence locally and confirm the failing import path is restored
- add jest-dom type support for ReactMarkdown tests so UI static checks use the current matcher types
- keep the markdown UI files in the formatter output expected by the UI lint/type hook
- sync FAB provider docs with the existing pyjwt>=2.11.0 dependency used for compat coverage
- use origin/main as the correct baseline for branch-only pre-commit checks
@minyeamer
minyeamer force-pushed the ui/markdown-rendering branch from 4b91adc to 1114cde Compare May 13, 2026 16:03
@minyeamer

Copy link
Copy Markdown
Contributor Author

@Srabasti @choo121600

Thanks for pointing those out! I’ve resolved the failures locally and pushed the fixes:

Static Checks: Fixed the pre-commit and CI image check failures.
Provider Distributions: All failed provider tests (Compat 3.0.6) are now passing.

@choo121600 choo121600 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice feature :)
one question

@parkhojeong parkhojeong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this. This looks very useful, and I left a few comments.

minyeamer added 2 commits May 14, 2026 23:38
- revert the markdown syntax highlighter back to PrismLight with selective language registration
- simplify language resolution to only allow explicitly registered languages
- apply the Prism pre styling to the outer code block container
- remove inherited margin and border radius that caused background bleed in the documentation modal
- trim ReactMarkdown tests down to a minimal smoke-test set
- keep basic math integration coverage
- keep mermaid render failure fallback coverage
- drop broader layout and highlighting assertions that were not essential to preserve
@minyeamer

Copy link
Copy Markdown
Contributor Author

@bbovenzi Thanks for the follow-up review. I addressed both of the remaining points and pushed follow-up commits:

  • Removed the 3rd-party license additions in 44c6acf
  • Changed Mermaid from the context provider back to a small utility in 39dc0f1

@jscheffl
jscheffl requested a review from bbovenzi June 14, 2026 20:08
@jscheffl

Copy link
Copy Markdown
Contributor

That improved markdown also including mermaid is really cool! But after merge nobody will know (except when reading the code). So can you please (1) ass a small hint to docs what is supported (proposal: (a) airflow-core/docs/core-concepts/dags.rst and (b) airflow-core/docs/tutorial/fundamentals.rst) and add an extended example also to one of the example dags (e.g. airflow-core/src/airflow/example_dags/tutorial.py)

@minyeamer

minyeamer commented Jun 21, 2026 •

Copy link
Copy Markdown
Contributor Author

@jscheffl Thanks, updated this in the docs and examples in 0f5935d.

I added a short note to airflow-core/docs/core-concepts/dags.rst describing the Markdown features now rendered in Dag documentation, including fenced code blocks, Mermaid fences, and KaTeX display math.

For airflow-core/docs/tutorial/fundamentals.rst, I updated the existing section that already references airflow-core/src/airflow/example_dags/tutorial.py. Instead of adding a separate tutorial example, I extended the existing print_date task's doc_md so the rendered Task documentation now demonstrates inline code, a fenced code block, KaTeX, and Mermaid while still describing the tutorial task and its downstream sleep / templated tasks.

I also added new screenshots as task_doc_md.png and dag_doc_md.png for both light and dark themes. I kept the existing task_doc.png and dag_doc.png files untouched, since they may still be referenced elsewhere.

For the Dag-level doc_md, I added example_markdown_doc.py as a small example Dag whose doc_md demonstrates the same richer Markdown rendering features at the Dag documentation level; this file can be included in the example docs as well if needed.

task_doc_md

@pierrejeambrun pierrejeambrun left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, tested and working as expected

A few small things before we can merge:

  • CI need fixing.
  • One comment to add, check bellow
  • Title and description needs to be updated. That is not limited to the 'doc_md' only. That's also rendering for TI and DagRun note. (math + mermaid blocks). Basically apply to the Markdown renderer.

minyeamer and others added 4 commits June 29, 2026 01:43
….tsx

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Use ```math fenced blocks for KaTeX rendering in the markdown UI instead of
detecting $$ delimiters in raw markdown text.

This keeps fenced block handling in the dedicated block renderer, avoids
ambiguous $$ detection, preserves the existing Mermaid block flow, and updates
the tutorial and docs to describe the supported Markdown syntax accurately.
@minyeamer

Copy link
Copy Markdown
Contributor Author

@pierrejeambrun Thanks for the review — I pushed 359cbce and fcda8a7.

In 359cbce, I moved Mermaid branching and KaTeX math rendering so both are handled in ReactMarkdownBlocks, and also updated the related docs/tutorial text to match the fenced math behavior.

In fcda8a7, I updated the docs images to reflect that same rendering change.

@pierrejeambrun
pierrejeambrun merged commit 154bfd5 into apache:main Jun 29, 2026
94 checks passed
@pierrejeambrun pierrejeambrun changed the title UI: Improve doc_md rendering in Dag Documentation Improve doc_md rendering in Dag Documentation Jun 29, 2026
karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Jun 30, 2026
* UI: Improve markdown rendering with code highlighting, math, and diagrams

- Fix inline code rendering to properly handle single backtick snippets
- Add syntax highlighting, line numbers, and copy buttons to code blocks
- Core support for LaTeX math expressions using KaTeX (inline and display)
- Integrate Mermaid.js to render diagrams directly from markdown blocks

* UI: Add horizontal scroll for code blocks and fix container overflow

* Fix markdown code block language parsing for hyphenated identifiers

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* UI: Apply markdown review feedback and fix FAB compat CI

Copilot follow-ups
- Lazy-load Mermaid so markdown rendering only pulls it in when Mermaid blocks are rendered
- Reuse existing i18n clipboard labels in markdown block actions instead of hard-coded JSX strings
- Replace the Mermaid loading text with a spinner-only state and remove the redundant jest-dom test import

Compat 3.0.6 fix
- Pin pyjwt>=2.11.0 in providers/fab so compat installs paired with Airflow 3.0.6 do not fall back to PyJWT 2.10.1
- Fix flask_jwt_extended imports that expect jwt.types.Options during FAB and Google provider test setup
- Keep the FAB wheel metadata aligned with the dependency floor already required by current airflow-core

Validation
- Pass targeted eslint, vitest, and UI build checks for the markdown rendering changes
- Reproduce the FAB compat install sequence locally and confirm the failing import path is restored

* UI: Fix markdown test typing and sync FAB provider docs

- add jest-dom type support for ReactMarkdown tests so UI static checks use the current matcher types
- keep the markdown UI files in the formatter output expected by the UI lint/type hook
- sync FAB provider docs with the existing pyjwt>=2.11.0 dependency used for compat coverage
- use origin/main as the correct baseline for branch-only pre-commit checks

* Remove direct PyJWT dependency from FAB provider

* UI: Restore PrismLight and fix markdown code block framing

- revert the markdown syntax highlighter back to PrismLight with selective language registration
- simplify language resolution to only allow explicitly registered languages
- apply the Prism pre styling to the outer code block container
- remove inherited margin and border radius that caused background bleed in the documentation modal

* UI: Reduce markdown rendering tests to smoke coverage

- trim ReactMarkdown tests down to a minimal smoke-test set
- keep basic math integration coverage
- keep mermaid render failure fallback coverage
- drop broader layout and highlighting assertions that were not essential to preserve

* UI: Drop inline markdown math and lazy-load KaTeX

- disable single-dollar inline math parsing to avoid breaking plain dollar text
- keep display math support for `$$...$$` blocks
- remove the global KaTeX stylesheet import from the app entrypoint
- load KaTeX styles only when markdown content needs math rendering

* docs: Add third-party license notices for markdown deps

- update `airflow-core/LICENSE` for the newly bundled markdown-rendering UI dependencies
- add MIT license texts for `katex`, `mermaid`, `rehype-katex`, and `remark-math`
- follow the existing `airflow-core/3rd-party-licenses` convention used for bundled UI packages

* fix: restore Link component props and align with library contract

- restore Link components required props (href/children) to match main branch
- isolate optional handling to the react-markdown adapter boundary
- handle invalid links by rendering children directly without a Link wrapper

* Update airflow-core/src/airflow/ui/src/components/ReactMarkdownBlocks.tsx

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>

* Update airflow-core/src/airflow/ui/src/components/ReactMarkdown.tsx

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>

* UI: Simplify ReactMarkdown code rendering helpers

Rename the inline code renderer for clarity, simplify markdown text extraction,
and remove the one-off mermaid label variable.

* UI: Move markdown Mermaid rendering into provider

* fix: Rename KatexStyleLoader to PascalCase and update imports

* UI: Remove redundant syntax language resolution logic

- Delete unnecessary `resolveSyntaxLanguage` helper from `syntaxHighlighter.ts`
- Simplify code block rendering by passing raw language strings to Prism

* chore: remove pnpm.minimumReleaseAge

* remove extractText & extractTextContent in ReactMarkdown

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>

* UI: Simplify MermaidProvider by removing redundant hooks and state tracking

- Remove `useCallback`/`useMemo` — `babel-plugin-react-compiler` handles
  memoization automatically via `reactCompilerPreset()`
- Remove `useState`/`useRef`/`useEffect` theme-tracking chain and the
  separate `initializeMermaid` function; `mermaid.initialize()` is called
  directly in `renderDiagram` on every invocation instead

* fix typeof guards in `ReactMarkdown` to resolve no-base-to-string lint error

* restore pnpm minimumReleaseAge setting

* UI: Fix markdown override behavior and Mermaid initialization

* remove 3rd party licenses for markdown rendering

* UI: Move Mermaid markdown rendering from context provider to util

* Docs: Add documentation hints for Markdown rendering support in the tutorial Task docs example

* remove example_markdown_doc Dag

* Update airflow-core/src/airflow/ui/src/components/ReactMarkdownBlocks.tsx

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>

* UI: Render markdown math from fenced code blocks

Use ```math fenced blocks for KaTeX rendering in the markdown UI instead of
detecting $$ delimiters in raw markdown text.

This keeps fenced block handling in the dedicated block renderer, avoids
ambiguous $$ detection, preserves the existing Mermaid block flow, and updates
the tutorial and docs to describe the supported Markdown syntax accurately.

* docs: Update task_doc_md images

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
senthh pushed a commit to acceldata-io/airflow that referenced this pull request Sep 18, 2026
…6809)

* UI: Improve markdown rendering with code highlighting, math, and diagrams

- Fix inline code rendering to properly handle single backtick snippets
- Add syntax highlighting, line numbers, and copy buttons to code blocks
- Core support for LaTeX math expressions using KaTeX (inline and display)
- Integrate Mermaid.js to render diagrams directly from markdown blocks

* UI: Add horizontal scroll for code blocks and fix container overflow

* Fix markdown code block language parsing for hyphenated identifiers

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* UI: Apply markdown review feedback and fix FAB compat CI

Copilot follow-ups
- Lazy-load Mermaid so markdown rendering only pulls it in when Mermaid blocks are rendered
- Reuse existing i18n clipboard labels in markdown block actions instead of hard-coded JSX strings
- Replace the Mermaid loading text with a spinner-only state and remove the redundant jest-dom test import

Compat 3.0.6 fix
- Pin pyjwt>=2.11.0 in providers/fab so compat installs paired with Airflow 3.0.6 do not fall back to PyJWT 2.10.1
- Fix flask_jwt_extended imports that expect jwt.types.Options during FAB and Google provider test setup
- Keep the FAB wheel metadata aligned with the dependency floor already required by current airflow-core

Validation
- Pass targeted eslint, vitest, and UI build checks for the markdown rendering changes
- Reproduce the FAB compat install sequence locally and confirm the failing import path is restored

* UI: Fix markdown test typing and sync FAB provider docs

- add jest-dom type support for ReactMarkdown tests so UI static checks use the current matcher types
- keep the markdown UI files in the formatter output expected by the UI lint/type hook
- sync FAB provider docs with the existing pyjwt>=2.11.0 dependency used for compat coverage
- use origin/main as the correct baseline for branch-only pre-commit checks

* Remove direct PyJWT dependency from FAB provider

* UI: Restore PrismLight and fix markdown code block framing

- revert the markdown syntax highlighter back to PrismLight with selective language registration
- simplify language resolution to only allow explicitly registered languages
- apply the Prism pre styling to the outer code block container
- remove inherited margin and border radius that caused background bleed in the documentation modal

* UI: Reduce markdown rendering tests to smoke coverage

- trim ReactMarkdown tests down to a minimal smoke-test set
- keep basic math integration coverage
- keep mermaid render failure fallback coverage
- drop broader layout and highlighting assertions that were not essential to preserve

* UI: Drop inline markdown math and lazy-load KaTeX

- disable single-dollar inline math parsing to avoid breaking plain dollar text
- keep display math support for `$$...$$` blocks
- remove the global KaTeX stylesheet import from the app entrypoint
- load KaTeX styles only when markdown content needs math rendering

* docs: Add third-party license notices for markdown deps

- update `airflow-core/LICENSE` for the newly bundled markdown-rendering UI dependencies
- add MIT license texts for `katex`, `mermaid`, `rehype-katex`, and `remark-math`
- follow the existing `airflow-core/3rd-party-licenses` convention used for bundled UI packages

* fix: restore Link component props and align with library contract

- restore Link components required props (href/children) to match main branch
- isolate optional handling to the react-markdown adapter boundary
- handle invalid links by rendering children directly without a Link wrapper

* Update airflow-core/src/airflow/ui/src/components/ReactMarkdownBlocks.tsx

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>

* Update airflow-core/src/airflow/ui/src/components/ReactMarkdown.tsx

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>

* UI: Simplify ReactMarkdown code rendering helpers

Rename the inline code renderer for clarity, simplify markdown text extraction,
and remove the one-off mermaid label variable.

* UI: Move markdown Mermaid rendering into provider

* fix: Rename KatexStyleLoader to PascalCase and update imports

* UI: Remove redundant syntax language resolution logic

- Delete unnecessary `resolveSyntaxLanguage` helper from `syntaxHighlighter.ts`
- Simplify code block rendering by passing raw language strings to Prism

* chore: remove pnpm.minimumReleaseAge

* remove extractText & extractTextContent in ReactMarkdown

Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>

* UI: Simplify MermaidProvider by removing redundant hooks and state tracking

- Remove `useCallback`/`useMemo` — `babel-plugin-react-compiler` handles
  memoization automatically via `reactCompilerPreset()`
- Remove `useState`/`useRef`/`useEffect` theme-tracking chain and the
  separate `initializeMermaid` function; `mermaid.initialize()` is called
  directly in `renderDiagram` on every invocation instead

* fix typeof guards in `ReactMarkdown` to resolve no-base-to-string lint error

* restore pnpm minimumReleaseAge setting

* UI: Fix markdown override behavior and Mermaid initialization

* remove 3rd party licenses for markdown rendering

* UI: Move Mermaid markdown rendering from context provider to util

* Docs: Add documentation hints for Markdown rendering support in the tutorial Task docs example

* remove example_markdown_doc Dag

* Update airflow-core/src/airflow/ui/src/components/ReactMarkdownBlocks.tsx

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>

* UI: Render markdown math from fenced code blocks

Use ```math fenced blocks for KaTeX rendering in the markdown UI instead of
detecting $$ delimiters in raw markdown text.

This keeps fenced block handling in the dedicated block renderer, avoids
ambiguous $$ detection, preserves the existing Mermaid block flow, and updates
the tutorial and docs to describe the supported Markdown syntax accurately.

* docs: Update task_doc_md images

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:UI Related to UI/UX. For Frontend Developers. ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants