Add a code generator from the DSL to C++#58
Merged
ruoso merged 14 commits intoPacktPublishing:mainfrom Nov 27, 2025
Merged
Conversation
Comprehensive design document for the Network Protocol DSL to C++ code generation feature, including: - Architecture overview and sans-IO design - Type mapping rules - Parser and serializer generation - State machine coordination - Phase-by-phase implementation plan
OutputContext provides helper methods for C++ code generation: - Namespace opening/closing declarations - Header guard generation from namespace + filename - String literal escaping for C++ code output This is the foundation for the code generation module that will translate Network Protocol DSL into compile-time C++ code.
TypeMapping provides utilities to convert DSL types to C++ equivalents: - int<bits=N, unsigned=bool> -> uint8_t, int32_t, etc. - str<...> -> std::string - array<element_type=T> -> std::vector<T> - tuple<field1=T1, ...> -> generated struct Also includes helpers for converting message and state names to valid C++ identifiers.
ProtocolInfo extracts and organizes information from the semantic AST: - Collects all unique protocol states - Extracts message info with identifiers, states, and data types - Separates read transitions (parsing) from write transitions (serializing) - Preserves action sequences for code generation
Generates C++ struct definitions for each protocol message's data fields. Maps DSL types (int, str, array, tuple) to C++ types and handles nested struct generation for tuple types within arrays.
Generates the State enum with all protocol states, plus input/output variant types for each state representing possible message transitions. This enables type-safe state machine transitions in generated code.
Generates incremental parsers for each message type with: - Stage-based state machine for resumable parsing - Static octet matching and terminator detection - Buffer management for partial data - take_data() to extract parsed message structs
Generates incremental serializers for each message type with: - Stage-based state machine for chunked output - Static octet writing and field-to-string conversion - Loop handling for array fields - advance()/next_chunk() interface for IO integration
…oordinators Generates ClientStateMachine and ServerStateMachine classes that: - Coordinate parsing and serialization based on current state - Route incoming bytes to appropriate parser - Provide type-safe send methods per message type - Track state transitions with current_state() accessor - Support thread-safe operation with queues for IO integration
CppGenerator orchestrates all code generation modules to produce: - protocol.hpp (main include aggregating all headers) - data_types.hpp/.cpp (message data structs) - states.hpp/.cpp (State enum and transition variants) - parser.hpp/.cpp (sans-IO parsers) - serializer.hpp/.cpp (sans-IO serializers) - state_machine.hpp/.cpp (Client/Server state machines) - CMakeLists.txt (optional, for standalone library) Test 037 verifies the generated SMTP protocol code compiles and can be instantiated correctly.
Command-line tool to generate C++ code from .networkprotocoldsl files: protocol_generator <input> -n <namespace> -o <output_dir> [-l <library>] Options: -n/--namespace: Target C++ namespace (required) -o/--output: Output directory (default: current) -l/--library: Generate CMakeLists.txt with this library name -v/--verbose: Show detailed progress Example: protocol_generator smtp.networkprotocoldsl -n myapp::smtp -o generated/
- Fix parser completion bug: Changed while loop condition from 'while (!complete_ && !input.empty())' to 'while (!complete_)' so the default case that sets complete_=true is reached even when input is exhausted after the last parsing stage. - Fix state machine parser reset bug: Removed parser reset calls at the start of on_bytes_received() which were discarding buffered partial data. Now properly handles NeedMoreData status to preserve parser state across multiple calls for incremental parsing. - Add SMTP-specific parser tests to verify EHLO command parsing with proper field extraction and stage-based parsing. - Add SMTP-specific state machine tests to verify on_bytes_received behavior including parser dispatch and state transitions.
This commit introduces infrastructure for using generated protocol code with typed callbacks, eliminating the need for manual std::visit in handlers. New files: - src/networkprotocoldsl/codegen/generate_runner.hpp/cpp: Generates ServerRunner<Handler> and ClientRunner<Handler> template classes with C++20 concepts that enforce type-safe handler interfaces. - src/networkprotocoldsl_uv/generatedserverwrapper.hpp/cpp: Libuv adapter for integrating generated protocol runners with libuv event loop. - tests/data/codegen/test_uv_adapter.cpp: Integration test for the libuv adapter with a complete SMTP EHLO/QUIT exchange. Key design decisions: - Handlers implement on_<MessageType>() methods for each concrete message type, receiving parsed data directly (not variants) - Runner internally uses std::visit to dispatch to the correct handler - Concepts enforce at compile-time that handlers implement all required methods with correct signatures - on_Open() takes no arguments for servers that send initial greeting - C++20 required for concepts support (updated CMakeLists generator)
There was a problem hiding this comment.
Pull request overview
This PR introduces a comprehensive C++ code generation system that translates the Network Protocol DSL into efficient, type-safe C++ code. The implementation follows a sans-IO architecture pattern and includes libuv integration for async I/O.
Key Changes:
- Implements complete code generation pipeline from DSL AST to C++ (data types, parsers, serializers, state machines)
- Adds concept-based callback runner with compile-time type safety
- Provides libuv adapter for network I/O integration
- Includes comprehensive test suite with integration tests comparing generated vs interpreted behavior
Reviewed changes
Copilot reviewed 47 out of 47 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
src/networkprotocoldsl/codegen/*.cpp/.hpp |
Core code generation modules (typemapping, protocolinfo, outputcontext, generators for each component) |
src/networkprotocoldsl_uv/generatedserverwrapper.* |
Libuv adapter for generated protocol code with connection management |
tests/data/codegen/test_*.cpp |
Test cases for generated code (greeting, parsing, roundtrip, conversation, partial data, uv adapter) |
tests/0*-codegen-*.cpp |
Unit tests for each code generation module |
tests/CMakeLists.txt |
Added test definitions and SMTP_SOURCE_DIR macro |
examples/smtpserver_generated/* |
Example SMTP server using generated code with handler callbacks |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This example demonstrates using the NetworkProtocolDSL code generator to create a type-safe SMTP server implementation: - Uses generated C++ code from smtp.npdsl protocol definition - Handler is shared across all connections (not per-connection) - All handler methods are const to enforce no per-connection state - Per-connection data flows through method inputs/outputs only - Integrates with libuv via GeneratedServerWrapper
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This introduces a code generation step to translate the DSL into C++, the various commits go through the entire implementation step by step, the last commit introduces an example C++ implementation of a smtp sever, like it was for the interpreter, but in this case it uses C++.