jison-lex

A lexical analyzer generator used by jison. It takes a lexical grammar definition (either in JSON or Bison's lexical grammar format) and outputs a JavaScript lexer.

install

The easiest way is to install the package from the GitHub repository:

npm install https://github.com/ull-esit-pl/jison-lex.git

Alternatively, be sure you have in your ~/.npmrc file the following line to be able to install the package from the GitHub registry inside the ULL-ESIT-PL organization:

@ull-esit-pl:registry=https://npm.pkg.github.com

Then you can install the package with npm:

npm install @ull-esit-pl/jison-lex

usage

Usage: jison-lex [file] [options]

file     file containing a lexical grammar

Options:
   -o FILE, --outfile FILE       Filename and base module name of the generated parser
   -t TYPE, --module-type TYPE   The type of module to generate (commonjs, js)
   --version                     print version and exit

Example

Given the following example.l file:

➜  test-jison-lex cat example.l

comment [/][*](.|[\r\n])*?[*][/]
%%
\s+|{comment}         /* skip whitespace */
[0-9]+                return 'NUMBER';
[-+*/]                return 'OPERATOR';
<<EOF>>               return 'EOF';
.                     return 'INVALID';

Which we compile with the following command:

test-jison-lex npx jison-lex -t commonjs example.l  -o example.js

And the following main.js file:

➜  test-jison-lex cat main.js 

const { lexer, lex } = require("./example.js");
const input = process.argv[2] || "2\n-/* a comment*/\n3";
lexer.setInput(input);

const results = [];

results.push({ type: lex(), lexeme: lexer.yytext, loc: lexer.yylloc });

results.push({ type: lex(), lexeme: lexer.yytext, loc: lexer.yylloc });
results.push({ type: lex(), lexeme: lexer.yytext, loc: lexer.yylloc });
results.push({ type: lex(), lexeme: lexer.yytext, loc: lexer.yylloc });

console.log(results);

The output of running main.js should be the following:

➜  test-jison-lex node main.js 

[
  {
    type: 'NUMBER',
    lexeme: '2',
    loc: { first_line: 1, last_line: 1, first_column: 0, last_column: 1 }
  },
  {
    type: 'OPERATOR',
    lexeme: '-',
    loc: { first_line: 2, last_line: 2, first_column: 0, last_column: 1 }
  },
  {
    type: 'NUMBER',
    lexeme: '3',
    loc: { first_line: 3, last_line: 3, first_column: 0, last_column: 1 }
  },
  {
    type: 'EOF',
    lexeme: '',
    loc: { first_line: 3, last_line: 3, first_column: 1, last_column: 1 }
  }
]

programatic usage

Here is an example using the node.js REPL:

➜  test-jison-lex node
> JisonLex = require('@ull-esit-pl/jison-lex');
[Function: RegExpLexer] { generate: [Function: generate] }
> grammar = {
|  rules: [
|     ["x", "return 'X';" ],
|     ["y", "return 'Y';" ],
|     ["$", "return 'EOF';" ]
|   ]
| };
> // or load from a file
> // var grammar = fs.readFileSync('mylexer.l', 'utf8');
> lexerSource = JisonLex.generate(grammar);
'/* generated by jison-lex 0.3.5 */\n' +
  'var lexer = (function(){\n' +
  'var lexer = ({\n' +
  '        if (this.conditionStack.length && this.conditionStack[this.co'... 1199 more characters
> lexer = new JisonLex(grammar);
{  // The lexer object!
  EOF: 1,
  parseError: [Function: parseError],
  setInput: [Function: setInput],
  input: [Function: input],
  unput: [Function: unput],
  more: [Function: more],
  reject: [Function: reject],
  less: [Function: less],
  pastInput: [Function: pastInput],
  upcomingInput: [Function: upcomingInput],
  showPosition: [Function: showPosition],
  test_match: [Function: test_match],
  next: [Function: next],
  lex: [Function: lex],
  begin: [Function: begin],
  popState: [Function: popState],
  _currentRules: [Function: _currentRules],
  topState: [Function: topState],
  pushState: [Function: pushState],
  stateStackSize: [Function: stateStackSize],
  options: {},
  performAction: [Function: anonymous],
  rules: [ /^(?:x)/, /^(?:y)/, /^(?:$)/ ],
  conditions: { INITIAL: { rules: [Array], inclusive: true } },
  yy: {},
  generate: [Function (anonymous)],
  generateModule: [Function (anonymous)],
  generateCommonJSModule: [Function (anonymous)],
  generateAMDModule: [Function (anonymous)]
}
> lexer.setInput('xyxxy');
> lexer.lex();
'X'
> lexer.lex();
'Y'
> target = lexer.generateCommonJSModule() // Creates a string containing the source code of a CommonJS module that
> target.length                           // exports the lexer and lex
11295
> target.substr(11100) // The last 195 characters of the generated module, which contains the export statements
'?:y)/,/^(?:$)/],\n' +
  'conditions: {"INITIAL":{"rules":[0,1,2],"inclusive":true}}\n' +
  '});\n' +
  'return lexer;\n' +
  '})();\n' +
  'exports.lexer = lexer;\n' +
  'exports.lex = function () { return lexer.lex.apply(lexer, arguments); };'

Use with Jison

Given the grammar:

➜  test-jison-lex cat grammar.jison 

%locations
%token NUMBER OPERATOR EOF

%left OPERATOR
%%

s: expr EOF { return $1; }
;

expr
  : expr OPERATOR expr
      {
        $$ = {
          type: "OPERATOR",
          lexeme: $2,
          left: $1,
          right: $3,
          loc: @2
        };
      }
  | NUMBER
      {
        $$ = {
          type: "number",
          value: Number($1),
          loc: @1
        };
      }
  ;

%%

const { lexer, lex } = require("./example.js");
parser.lexer = lexer;

We compile it with the following command:

➜  test-jison-lex npx jison grammar.jison -o parser.js

And then we can use the generated parser in the node.js REPL:

➜  test-jison-lex node                                
> p = require("./parser")
{
  parser: { yy: {} },
  Parser: [Function: Parser],
  parse: [Function (anonymous)],
  main: [Function: commonjsMain]
}
> p.parse("3+4")
{
  type: 'OPERATOR',
  lexeme: '+',
  left: {
    type: 'number',
    value: 3,
    loc: { first_line: 1, last_line: 1, first_column: 0, last_column: 1 }
  },
  right: {
    type: 'number',
    value: 4,
    loc: { first_line: 1, last_line: 1, first_column: 2, last_column: 3 }
  },
  loc: { first_line: 1, last_line: 1, first_column: 1, last_column: 2 }
}
> 

Be sure the example.js file is generated by @ull-esit-pl/jison-lex and not by jison-lex using the command npx jison-lex -t commonjs example.l -o example.js. Check that the node_modules/.bindirectory contains a symlink to the@ull-esit-pl/jison-lex` CLI:

➜  test-jison-lex ls -l node_modules/.bin 
total 0
lrwxr-xr-x@ 1 casianorodriguezleon  staff  29  1 mar.  22:08 escodegen -> ../escodegen/bin/escodegen.js
lrwxr-xr-x@ 1 casianorodriguezleon  staff  30  1 mar.  22:08 esgenerate -> ../escodegen/bin/esgenerate.js
lrwxr-xr-x@ 1 casianorodriguezleon  staff  25  1 mar.  22:08 esparse -> ../esprima/bin/esparse.js
lrwxr-xr-x@ 1 casianorodriguezleon  staff  28  1 mar.  22:08 esvalidate -> ../esprima/bin/esvalidate.js
lrwxr-xr-x@ 1 casianorodriguezleon  staff  19  1 mar.  22:08 jison -> ../jison/lib/cli.js
lrwxr-xr-x@ 1 casianorodriguezleon  staff  32  1 mar.  22:11 jison-lex -> ../@ull-esit-pl/jison-lex/cli.js
lrwxr-xr-x@ 1 casianorodriguezleon  staff  22  1 mar.  22:08 jsonlint -> ../jsonlint/lib/cli.js

Writing a lexer for Jison by hand

See examples/manual-lexer.js for an example of a lexer written by hand for use with Jison.

Author

Zach Carter hi@znc.dev

Contributor: Casiano Rodríguez León (2026).

• This version fixes a bug with the --t --module-type option and uses commander instead of nomnom for the command line interface.
• It also changes the way the function lex is exported

license

MIT