Release 0.23.23

What's Changed

• Updated code and templates: bug fix and added file encodings in the desc.xml. by @kaby76 in #563

Release 0.23.21

This is a minor release. (Prior releases 0.23.19 and 0.23.20 contained errors.)

• Add "trquery assert" command for testing parse tree structure.
• Changed trgen parser driver templates.
  • Add .trq file testing for trquery asserts.
  • Fix templates of several bugs.
  • Remove use of '--' in dotnet commands.
  • Fix incorrect use of single quotes in .ps1's.
  • Remove triconv since it adds nothing.
  • Rewrite tests.txt construction using .Net calls.
  • Fix grep patterns with '.'.
• Clean up source code.

Release 0.23.18

This is a minor release.

• Removed very old trst.
• Restructured code usings/namespace scoping.
• Reformatted code.
• Updated Antlr4 grammar from grammars-v4, and updated usage in various tools, e.g., trcover.
• Update trperf help.
• Add trdistill.
• Cleaned up trgen code.
• Cleaned up trnullable.
• Fixed trparse switching on file extension of supported grammar types.

0.23.17

This release (along with changes from 0.23.16 and 0.23.15) correct and update the trgen code and templates. There were problems in generating code for some grammars, like grammars-v4/antlr/antlr4, which is the main issue being addressed with this release.

• All the templates were updated, and propagated to the grammars-v4 repo. Some of the updates had to deal with the new antlr-ng tool that will be required eventually by the Antlr4ng runtime. As of version 3.0.16, antlr4ng still has the patched Antlr4 tool .jar for Antlr4ng. But, we now use the new Antlr-ng tool for Antlr4ng runtime. (Eventually, there will be a way to specify which Antlr4 tool generator to use.) Other changes in templates had to do with a rename of an attribute.
• One of the big problems in grammars-v4 is "grammar forking." This is where someone takes a .g4 and replaces all the grammar "actions" in one programming language, like Java, with another, like C#. This is bad practice because we've found that people tend to work in one language and changes in one grammar are not propagated to another. Another type of "grammar forking" is copying one grammar, then changing it to parse a slightly different variant of the original language. This PR adds cross-grammar importing. <imports> in a desc.xml is the equivalent to the -lib command-line option to the Antlr4 tool. An example grammar of its use is the TrapC grammar. I decided to implement the desc.xml this way rather than have a general way (e.g., "Here's a list of parameters to pass to the Antlr4 Tool."), not only because the command-line options between Antlr-ng and the Official Antlr4 Tool are different, but because I need to analyze the grammars specified in the paths.
• The trgen tool now outputs a dependency graph for grammars, in order to understand what trgen is doing.
• Updated various Nuget package dependencies.

0.23.15

This release updates the dependency analysis of grammars used in trgen. Dependency analysis determines what are the "top-level" parser and lexer grammars, which the tool uses to generate a driver. The graph is displayed when you use the tool.

Example 1

abb has two .g4's, one for the parser and the other for the lexer. The graph for this is:

Dependency graph of grammar:
2 vertices, 1 edges
abbParser: abbParser->abbLexer
abbLexer:

In other words, there are two nodes in the graph and one edge between them. The top level grammars are abbParser and abbLexer.

Example 2

antlr/antlr4 has three .g4's. ANLTRv4Lexer.g4, ANTLRv4Parser.g4, and LexBasic.g4. The dependency graph is:

Dependency graph of grammar:
3 vertices, 2 edges
LexBasic:
ANTLRv4Parser: ANTLRv4Parser->ANTLRv4Lexer
ANTLRv4Lexer: ANTLRv4Lexer->LexBasic

Here we see ANTLRv4Lexer "imports" LexBasic and ANTLRv4Parser uses the token vocabulary from ANTLRv4Lexer. The top level grammars are ANTLRv4Parser and ANTLRv4Lexer.

Example 3

cobal85 has two .g4's: Cobol85.g4 and Cobol85Preprocessor.g4. Both have start rules, but the desc.xml says that the grammar to test is Cobol85. The dependency graph is:

Dependency graph of grammar:
4 vertices, 2 edges
Cobol85PreprocessorParser: Cobol85PreprocessorParser->Cobol85PreprocessorLexer
Cobol85PreprocessorLexer:
Cobol85Parser: Cobol85Parser->Cobol85Lexer
Cobol85Lexer:

The top level grammars are Cobol85Parser and CobolLexer.

0.23.14

This release is a minor change for the trgen templates. The templates and tool now have a way to switch to the new generator Antlr-ng. The templates currently use a pre-built version of the tool in the directory specified by $HOME. Use these steps to build.

cd $HOME
git clone https://github.com/mike-lischke/antlr-ng.git
cd antlr-ng
npm i
npm run build

There should not be any errors, but the tool is still under development.

0.23.13

Minor changes to 0.23.12 for Go target. #531

0.23.12

This point release is to change the code in trgen to allow different parser generators. We now have Antlr-ng, which is port of Antlr4 to TypeScript. I am planning to make Trash not so Antlr-specific.

Fixes #518, #524, #525, #527.

0.23.11

This is a point release for the Trash Toolkit. There are only a couple of minor changes to correct some bugs.

• Fix trgen to handle names like star.g4. The code did have a bug in considering those files as template files, e.g., st.build.sh, where the name should begin with st. to indicate that it is a template file. That error caused a cascade of problems in generating the driver.
• Fix trgen to accept template files in the "target-specific" code directories for a file. Prior to this change, target-specific code was just copied. This is necessary to have a st.CMakeLists.txt template file so it can be instatiated with OS specific C++ compiler options, for Windows, Linux, and Mac OS. antlr/grammars-v4#4362

0.23.9

This is a minor change.

• Fixed trgen problems with grammar files that end with st.g4, e.g., tst.g4. Trgen would not create a good driver for the grammar. I included a test to make sure this doesn't regress with a future change.
• Fixed the built-in grammar for Bison, so that instead of recursion, kleene closure is used. This cuts back on excessive stack size for parsing some Bison grammars. Here are some example rules that were changed.

token_decls_for_prec
    : token_decl_for_prec+
    | TAG token_decl_for_prec+
    | token_decls_for_prec TAG token_decl_for_prec+
    ;
symbol_decls
    : symbol+
    | TAG symbol+
    | symbol_decls TAG symbol+
    ;
bison_grammar
    : rules_or_grammar_declaration
    | bison_grammar rules_or_grammar_declaration
    ;
params
    : params actionBlock
    | actionBlock
    ;

vs.

token_decls_for_prec
    : (token_decl_for_prec+ | TAG token_decl_for_prec+) (TAG token_decl_for_prec+)*
    ;
symbol_decls
    : (symbol+ | TAG symbol+) (TAG symbol+)*
    ;
bison_grammar
    : rules_or_grammar_declaration+
    ;
params
    : actionBlock+
    ;

• trclonereplace was fixed up to perform proper rule cloning and replacing.
• trextract was added. _I haven't really refind what this command should do. But, essentially, I want to have it extract out actions and replace with macros. I can then add a tool to expand the macros. This code is intended to fix the "target agnostic problem."
• tragl now has "multi-tree" display. Prior to this change, each tree passed via a parsing result set would have to be viewed in order, and closed off each time. Instead, we can now view all the trees simultaneously, and compared. There are two modes for trparse: one to not group, and the other to group. This affects how tragl displays the trees, whether in one "doc" window, or multiple. The default is "ungrouped", i.e., multiple.

12/14-20:44:10 ~/issues/g4-current/c/Generated-CSharp
$ trparse --ambig ../examples/TypeCast.c | tragl
CSharp 0 ../examples/TypeCast.c success 0.071354
12/14-20:44:40 ~/issues/g4-current/c/Generated-CSharp

• trgen templates for performance. The templates for the drivers where changed a little to help in tracing performance issues. For example, instead of Total Time: 123.123 being displayed, a prefix can be passed in to display group Total Time: 123.123. This helps with another change for testing performance and extracting out information for statistical analysis. Some of the old double-minus -- syntax for dotnet calls were fixed as well, as this is not accepted by the dotnet tool anymore. The getAllPossibleParseTrees() function was changed in the CSharp templates to now return a string corresponding to the "decision/alt" for an ambiguous parse tree that was computed. This helps one to understand where grammar ambiguity resides. This could be considered a "major change", but since most don't use Trash, this isn't a problem.
• trglob now produces a list of files and directories with relative paths instead of absolute.