feat: improve ABNF grammar clarity and inline all secondary grammars#116
Open
feat: improve ABNF grammar clarity and inline all secondary grammars#116
Conversation
Restructure the ABNF grammar to use explicit, typed reference rules in the primary grammar instead of relying on secondary grammars with two-pass parsing. This improves grammar clarity and aligns with the proposed spec changes in OAI/Arazzo-Specification#454. Key changes: - Add $self expression support - Add $inputs/$outputs JSON Pointer support (e.g., $inputs.customer#/firstName) - Inline all secondary grammars into the primary grammar - Extract shared identifier and identifier-strict rules - Adapt json-pointer to exclude { and } from unescaped for unambiguous embedded expression parsing, fixing the body expression extract limitation - Require explicit component types (parameters/successActions/failureActions) - Update README with current grammar and examples Resolves: OAI/Arazzo-Specification#424, OAI/Arazzo-Specification#425, OAI/Arazzo-Specification#426, OAI/Arazzo-Specification#428 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR updates the Arazzo runtime expression grammar and translators to support newer spec features and simplify parsing by inlining formerly secondary grammars.
Changes:
- Add
$selfsupport and JSON Pointer support for$inputs/$outputs. - Inline
$steps/$workflows/$sourceDescriptions/$componentsparsing into the primary grammar (eliminating two-pass parsing) and introduce sharedidentifierrules. - Adjust JSON Pointer
unescapedhandling to address embedded-expression extraction ambiguity; update tests, fixtures, and README accordingly.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| test/test.js | Updates validation expectations for {/} in JSON Pointer paths. |
| test/parse/translators/snapshots/CSTTranslator.js.snap | Updates CST snapshots to reflect new reference node structure. |
| test/parse/translators/snapshots/ASTTranslator.js.snap | Updates AST snapshots for revised node shapes (e.g., steps/workflows/components). |
| test/parse/snapshots/cst-corpus.js.snap | Refreshes CST corpus snapshots for new grammar structure and new expressions. |
| test/parse/snapshots/ast-corpus.js.snap | Refreshes AST corpus snapshots (adds $self, pointers on inputs/outputs, renamed fields). |
| test/fixtures/expressions-valid.js | Adds new valid expression fixtures ($self, pointers, dotted/hyphenated identifiers). |
| test/extract.js | Updates extraction tests to confirm embedded body expressions with JSON pointers now extract correctly. |
| src/parse/translators/CSTTranslator.js | Aligns CST callbacks with the new inlined grammar rules. |
| src/parse/translators/ASTTranslator/transformers.js | Removes secondary parsing and maps new CST node types directly to updated AST shapes; adds $self. |
| src/grammar.js | Regenerated grammar output reflecting inlined rules and updated JSON Pointer ranges. |
| src/grammar.bnf | Updates the source ABNF (primary change set) including inlined references and identifier rules. |
| README.md | Updates documented ABNF and examples; removes previous JSON-pointer extraction limitation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The unescaped range incorrectly started at %x7E instead of %x7F, re-introducing ~ which RFC 6901 requires to only appear as part of escape sequences (~0 / ~1). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add SelfExpression type to ASTNode union - Add optional jsonPointer to InputsExpression and OutputsExpression - Restore field: 'outputs' on StepsExpression - Add fieldName to WorkflowsExpression, deprecate subField - Add componentType/componentName to ComponentsExpression, deprecate field/subField - Fix stale json-pointer-safe reference in test comment Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
$selfexpression support (new in Arazzo 1.1.0)$inputs/$outputsJSON Pointer support (e.g.,$inputs.customer#/firstName)$steps,$workflows,$sourceDescriptions,$components) into the primary grammar, eliminating two-pass parsingidentifierandidentifier-strictrules to reduce duplicationjson-pointerto exclude{and}fromunescaped, fixing the embedded expression extraction limitation for body expressions and all other expressions with JSON pointersparameters/successActions/failureActions)This PR was created as a verification of the proposed ABNF grammar changes in OAI/Arazzo-Specification#454. During implementation, several issues with the spec PR were identified:
unescapedin json-pointer still includes{and}— breaks embedded expression parsing; we fix this by excluding themidentifierrule is too broad — the spec PR uses one rule for both IDs (no dots) and field names (with dots); we useidentifierandidentifier-strictto respect the spec's different constraintssource-descriptions-referenceis too restrictive — the spec PR constrains it toidentifier, but operationIds in OpenAPI are unconstrained strings; we keep1*CHARResolves: OAI/Arazzo-Specification#424, OAI/Arazzo-Specification#425, OAI/Arazzo-Specification#426, OAI/Arazzo-Specification#428
Test plan
$self,$inputs/$outputswith JSON Pointer, dotted/hyphenated/underscored names{/}in JSON pointer paths now correctly rejected🤖 Generated with Claude Code