Skip to content

feat: improve ABNF grammar clarity and inline all secondary grammars#116

Open
char0n wants to merge 5 commits intomainfrom
feat/improve-abnf-grammar
Open

feat: improve ABNF grammar clarity and inline all secondary grammars#116
char0n wants to merge 5 commits intomainfrom
feat/improve-abnf-grammar

Conversation

@char0n
Copy link
Copy Markdown
Member

@char0n char0n commented Apr 2, 2026

Summary

  • Add $self expression support (new in Arazzo 1.1.0)
  • Add $inputs/$outputs JSON Pointer support (e.g., $inputs.customer#/firstName)
  • Inline all secondary grammars ($steps, $workflows, $sourceDescriptions, $components) into the primary grammar, eliminating two-pass parsing
  • Extract shared identifier and identifier-strict rules to reduce duplication
  • Adapt json-pointer to exclude { and } from unescaped, fixing the embedded expression extraction limitation for body expressions and all other expressions with JSON pointers
  • Require explicit component types (parameters/successActions/failureActions)
  • Update README with current grammar, examples, and removal of the "known limitation" note

This PR was created as a verification of the proposed ABNF grammar changes in OAI/Arazzo-Specification#454. During implementation, several issues with the spec PR were identified:

  1. unescaped in json-pointer still includes { and } — breaks embedded expression parsing; we fix this by excluding them
  2. Single identifier rule is too broad — the spec PR uses one rule for both IDs (no dots) and field names (with dots); we use identifier and identifier-strict to respect the spec's different constraints
  3. source-descriptions-reference is too restrictive — the spec PR constrains it to identifier, but operationIds in OpenAPI are unconstrained strings; we keep 1*CHAR

Resolves: OAI/Arazzo-Specification#424, OAI/Arazzo-Specification#425, OAI/Arazzo-Specification#426, OAI/Arazzo-Specification#428

Test plan

  • All 152 tests pass
  • New test fixtures for $self, $inputs/$outputs with JSON Pointer, dotted/hyphenated/underscored names
  • Extract tests updated: body expressions with JSON pointers now work in embedded context
  • Validation tests updated: {/} in JSON pointer paths now correctly rejected
  • Snapshot tests updated for new CST/AST node structure

🤖 Generated with Claude Code

Restructure the ABNF grammar to use explicit, typed reference rules in the
primary grammar instead of relying on secondary grammars with two-pass parsing.
This improves grammar clarity and aligns with the proposed spec changes in
OAI/Arazzo-Specification#454.

Key changes:
- Add $self expression support
- Add $inputs/$outputs JSON Pointer support (e.g., $inputs.customer#/firstName)
- Inline all secondary grammars into the primary grammar
- Extract shared identifier and identifier-strict rules
- Adapt json-pointer to exclude { and } from unescaped for unambiguous
  embedded expression parsing, fixing the body expression extract limitation
- Require explicit component types (parameters/successActions/failureActions)
- Update README with current grammar and examples

Resolves: OAI/Arazzo-Specification#424, OAI/Arazzo-Specification#425,
OAI/Arazzo-Specification#426, OAI/Arazzo-Specification#428

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@char0n char0n requested a review from Copilot April 2, 2026 13:25
@char0n char0n self-assigned this Apr 2, 2026
@char0n char0n added the enhancement New feature or request label Apr 2, 2026
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Arazzo runtime expression grammar and translators to support newer spec features and simplify parsing by inlining formerly secondary grammars.

Changes:

  • Add $self support and JSON Pointer support for $inputs/$outputs.
  • Inline $steps/$workflows/$sourceDescriptions/$components parsing into the primary grammar (eliminating two-pass parsing) and introduce shared identifier rules.
  • Adjust JSON Pointer unescaped handling to address embedded-expression extraction ambiguity; update tests, fixtures, and README accordingly.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
test/test.js Updates validation expectations for {/} in JSON Pointer paths.
test/parse/translators/snapshots/CSTTranslator.js.snap Updates CST snapshots to reflect new reference node structure.
test/parse/translators/snapshots/ASTTranslator.js.snap Updates AST snapshots for revised node shapes (e.g., steps/workflows/components).
test/parse/snapshots/cst-corpus.js.snap Refreshes CST corpus snapshots for new grammar structure and new expressions.
test/parse/snapshots/ast-corpus.js.snap Refreshes AST corpus snapshots (adds $self, pointers on inputs/outputs, renamed fields).
test/fixtures/expressions-valid.js Adds new valid expression fixtures ($self, pointers, dotted/hyphenated identifiers).
test/extract.js Updates extraction tests to confirm embedded body expressions with JSON pointers now extract correctly.
src/parse/translators/CSTTranslator.js Aligns CST callbacks with the new inlined grammar rules.
src/parse/translators/ASTTranslator/transformers.js Removes secondary parsing and maps new CST node types directly to updated AST shapes; adds $self.
src/grammar.js Regenerated grammar output reflecting inlined rules and updated JSON Pointer ranges.
src/grammar.bnf Updates the source ABNF (primary change set) including inlined references and identifier rules.
README.md Updates documented ABNF and examples; removes previous JSON-pointer extraction limitation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

char0n and others added 3 commits April 3, 2026 13:21
The unescaped range incorrectly started at %x7E instead of %x7F,
re-introducing ~ which RFC 6901 requires to only appear as part
of escape sequences (~0 / ~1).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add SelfExpression type to ASTNode union
- Add optional jsonPointer to InputsExpression and OutputsExpression
- Restore field: 'outputs' on StepsExpression
- Add fieldName to WorkflowsExpression, deprecate subField
- Add componentType/componentName to ComponentsExpression, deprecate field/subField
- Fix stale json-pointer-safe reference in test comment

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ambiguity in runtime expressions embedded in strings

2 participants